diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a1386d..0e2a3f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,36 @@ All notable changes to `xtafkit` will be documented in this file. +## [Unreleased] + +### ISO / disc-image support +- Added `xtafkit extract` for streaming Xbox / Xbox 360 XISO contents to a local directory, with `$SystemUpdate` skipped by default. Supports `--keep-systemupdate` to override and `--dry-run` to preview file list + byte totals without writing. +- Added `xtafkit god` for XISO → Games-on-Demand conversion. Default trim is `compact`; `preserve-layout` and `none` stay available for debugging and compatibility. Game-title slot in the CON header auto-fills from the bundled catalog; pass `--game-title TITLE` to override. +- TUI upload (`u`) now sniffs every local file and, on XISO detection, prompts **e(X)tract / (G)oD / (R)aw / Esc**. Default flips by cwd context: inside `/Content//` defaults to GoD (BC playback target); everywhere else defaults to Extract (alt-dashboard target). +- Extract destination folder name is resolved from the catalog when the title is known — `disc1.iso` with TitleID `4D5307E6` extracts as `Halo 3/` rather than `disc1/`. Falls back to the file stem on catalog miss. Names are sanitized for FATX (illegal chars replaced with `-`, runs of whitespace collapsed, truncated to 42 bytes). +- Introduced a shared `fatxlib::iso` namespace for image reading, manifest planning, compact repacking, and GoD conversion. +- Reworked compact GoD conversion to stream a virtual dense XDVDFS layout instead of staging a temporary ISO on disk — peak local disk usage during conversion is zero. +- Centralized ISO filtering and planning so extract, compact trim, and dry-run reporting share the same manifest. +- Removed the old public `fatxlib::xiso` and `fatxlib::iso2god` entry points in favor of `fatxlib::iso::{image,manifest,compact,god}`. +- Refactored GoD conversion to share its engine between host-filesystem and FATX-volume targets via an internal `GodSink` trait — one `run_conversion` loop, two sink implementations. + +### Performance +- Hot-path SHA-1 in GoD conversion routes through `openssl::sha::sha1` by default (ARMv8 SHA on Apple Silicon, SHA-NI on x86). Gated by the default-on `openssl-hash` cargo feature; disable to fall back to RustCrypto's `sha1` crate with zero system OpenSSL dependency. +- Fixed a double-I/O bug in `write_part`: the upstream implementation read each subpart, hashed it, then `seek_relative`d back and re-read it via `io::copy` to write the part file. Now writes from the buffer it already has, halving I/O on the hot path (~33 % wall-time reduction on large ISOs). +- 1 MiB `BufReader` on the source ISO during the metadata pre-pass cuts syscall tax on multi-GiB inputs. +- Streaming variant of GoD conversion to FATX (`convert_iso_to_fatx`) builds each part in a reused ~163 MiB buffer and streams straight into the volume — no local staging. + +### TUI / quality of life +- Mid-conversion `Esc` cancels GoD conversion cleanly (checked between parts and between MHT-chain steps); no partial silent failures. +- Per-part byte-level progress with MiB/s throughput, rate-limited to ~200 ms intervals. +- Upload prompt no longer prefills with the last-used path — always starts blank. +- TUI extract worker skips `$SystemUpdate` and surfaces the skip count + bytes in the completion message. + +### Library API additions +- `fatxlib::iso::image::XisoImage::title_info()` parses the embedded `Default.xex` / `default.xbe` and returns the title's execution info. Used by catalog name resolution and by the GoD conversion pipeline. +- `fatxlib::executable` (top-level module) holds `TitleInfo` / `TitleExecutionInfo` and the XEX/XBE parsers — shared between `iso::image` and `iso::god`. +- `fatxlib::volume::FatxVolume::create_file_from_reader` streams a file into FATX cluster-by-cluster from any `Read` source, capping working-set at one cluster regardless of total file size. + ## [1.1.0] - 2026-05-16 First release under the `xtafkit` name. Forked from diff --git a/CLAUDE.md b/CLAUDE.md index 37d5db0..69cb635 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ Rust toolkit for reading and writing FATX/XTAF file systems on Xbox/Xbox 360 for ## Architecture - **Cargo workspace** with two crates: - `fatxlib` — Library crate. FATX/XTAF volume implementation, types, partition detection, platform I/O. Also: bundled title catalog (Xbox 360 + Original Xbox), STFS header parser, profile (Account) blob decryption, slot-aware display formatting. - - `xtafkit` (root) — Single binary (`xtafkit`). Five subcommands via clap (`browse`, `ls`, `scan`, `mkimage`, `resolve`); no-args entry point launches the TUI via guided picker. Ratatui-based TUI is the primary UX. Test image generator (`mkimage`) is the only non-TUI write path that ships. + - `xtafkit` (root) — Single binary (`xtafkit`). Seven subcommands via clap (`browse`, `ls`, `scan`, `mkimage`, `resolve`, `extract`, `god`); no-args entry point launches the TUI via guided picker. Ratatui-based TUI is the primary UX. `extract` and `god` handle XISO work; `mkimage` is the only non-TUI write path that targets FATX/XTAF itself. ## Key Technical Details @@ -84,10 +84,15 @@ cargo run -p fatxlib --example check_profile -- /path/to/profile-file - Commit and push at each milestone (working feature, major fix, etc.) ### XISO / disc-image support -- `fatxlib::xiso` wraps `xdvdfs` (sync feature, no async runtime) and exposes `XisoImage::{open, walk_files, read_into, file_reader, read_at}` plus a `LAYOUTS` table for raw / XGD1 / XGD2 / XGD3 pre-partition offsets. +- `fatxlib::iso` owns ISO-domain work: + - `image` wraps `xdvdfs` (sync feature, no async runtime) and exposes `XisoImage::{open, walk_files, read_into, file_reader, read_at}` plus a `LAYOUTS` table for raw / XGD1 / XGD2 / XGD3 pre-partition offsets. + - `manifest` builds the shared ISO file manifest, skips `$SystemUpdate` by default, and feeds both extract and compact planning. + - `compact` builds a dense virtual XDVDFS image for hard-trim GoD conversion. + - `god` owns GoD packaging. +- `xtafkit extract` streams XISO files to disk; `xtafkit god` converts XISO to GoD and defaults to `compact` trim. - TUI upload (`u`) sniffs every local file with `XisoImage::open`. On a hit, the user is prompted **Extract contents (Y/n)** — default extracts via `IoCmd::ExtractXiso`, `n` falls back to raw `WriteFile`. Extraction streams each entry through `XisoFileReader` → `FatxVolume::create_file_from_reader`, which keeps the working set at one cluster regardless of image size. - Useful because Aurora / FreeStyle Dash / XBMC4XBOX scan the drive for loose `default.xex` / `default.xbe` and launch them directly; STFS-wrapped GoD packaging is **not** required for those dashboards. ## Future Work (Deferred) - Eager / deferred-sync auto-resolve for files inside STFS content-type folders (Marketplace/Arcade/etc.) — currently on-demand only -- `iso2god`-style ISO → Games-on-Demand conversion (cherry-picked from iso2god-rs, refactored for streaming) — needed only for Xbox 360 BC, which requires STFS GoD packages; alt-dashboard playback already works via the XISO extract flow above +- Further split `fatxlib::iso::god::convert` into a pure conversion core plus transport-specific sinks if the current host-FS / FATX split starts accumulating more policy diff --git a/Cargo.lock b/Cargo.lock index 53540e7..91594f2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -219,6 +219,12 @@ version = "1.25.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec" +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + [[package]] name = "castaway" version = "0.2.4" @@ -359,6 +365,27 @@ version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" +[[package]] +name = "const_format" +version = "0.2.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4481a617ad9a412be3b97c5d403fef8ed023103368908b9c50af598ff467cc1e" +dependencies = [ + "const_format_proc_macros", + "konst", +] + +[[package]] +name = "const_format_proc_macros" +version = "0.2.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d57c2eccfb16dbac1f4e61e206105db5820c9d26c3c472bc17c774259ef7744" +dependencies = [ + "proc-macro2", + "quote", + "unicode-xid", +] + [[package]] name = "convert_case" version = "0.10.0" @@ -643,10 +670,13 @@ name = "fatxlib" version = "1.1.0" dependencies = [ "bitflags 2.11.1", + "byteorder", "hmac", "libc", "log", "nix 0.31.3", + "num_enum", + "openssl", "phf 0.13.1", "phf_codegen 0.13.1", "proptest", @@ -706,6 +736,21 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" +[[package]] +name = "foreign-types" +version = "0.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1" +dependencies = [ + "foreign-types-shared", +] + +[[package]] +name = "foreign-types-shared" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b" + [[package]] name = "futures-core" version = "0.3.32" @@ -898,6 +943,15 @@ version = "1.70.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" +[[package]] +name = "itertools" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57" +dependencies = [ + "either", +] + [[package]] name = "itertools" version = "0.14.0" @@ -969,6 +1023,21 @@ dependencies = [ "cpufeatures 0.2.17", ] +[[package]] +name = "konst" +version = "0.2.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "128133ed7824fcd73d6e7b17957c5eb7bacb885649bd8c69708b2331a10bcefb" +dependencies = [ + "konst_macro_rules", +] + +[[package]] +name = "konst_macro_rules" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4933f3f57a8e9d9da04db23fb153356ecaf00cbd14aee46279c33dc80925c37" + [[package]] name = "lab" version = "0.11.0" @@ -1168,6 +1237,28 @@ dependencies = [ "autocfg", ] +[[package]] +name = "num_enum" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d0bca838442ec211fa11de3a8b0e0e8f3a4522575b5c4c06ed722e005036f26" +dependencies = [ + "num_enum_derive", + "rustversion", +] + +[[package]] +name = "num_enum_derive" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "680998035259dcfcafe653688bf2aa6d3e2dc05e98be6ab46afb089dc84f1df8" +dependencies = [ + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "num_threads" version = "0.1.7" @@ -1189,6 +1280,43 @@ version = "1.70.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" +[[package]] +name = "openssl" +version = "0.10.79" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf0b434746ee2832f4f0baf10137e1cabb18cbe6912c69e2e33263c45250f542" +dependencies = [ + "bitflags 2.11.1", + "cfg-if", + "foreign-types", + "libc", + "openssl-macros", + "openssl-sys", +] + +[[package]] +name = "openssl-macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "openssl-sys" +version = "0.9.115" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "158fe5b292746440aa6e7a7e690e55aeb72d41505e2804c23c6973ad0e9c9781" +dependencies = [ + "cc", + "libc", + "pkg-config", + "vcpkg", +] + [[package]] name = "ordered-float" version = "4.6.0" @@ -1361,6 +1489,21 @@ version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "pori" +version = "0.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4a63d338dec139f56dacc692ca63ad35a6be6a797442479b55acd611d79e906" +dependencies = [ + "nom", +] + [[package]] name = "portable-atomic" version = "1.13.1" @@ -1421,6 +1564,15 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "proc-macro-crate" +version = "3.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" +dependencies = [ + "toml_edit", +] + [[package]] name = "proc-macro2" version = "1.0.106" @@ -1570,7 +1722,7 @@ dependencies = [ "compact_str", "hashbrown 0.16.1", "indoc", - "itertools", + "itertools 0.14.0", "kasuari", "lru", "strum", @@ -1622,7 +1774,7 @@ dependencies = [ "hashbrown 0.16.1", "indoc", "instability", - "itertools", + "itertools 0.14.0", "line-clipping", "ratatui-core", "strum", @@ -2058,6 +2210,36 @@ version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" +[[package]] +name = "toml_datetime" +version = "1.1.1+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_edit" +version = "0.25.11+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b59c4d22ed448339746c59b905d24568fcbb3ab65a500494f7b8c3e97739f2b" +dependencies = [ + "indexmap", + "toml_datetime", + "toml_parser", + "winnow", +] + +[[package]] +name = "toml_parser" +version = "1.1.2+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2abe9b86193656635d2411dc43050282ca48aa31c2451210f4202550afb7526" +dependencies = [ + "winnow", +] + [[package]] name = "twox-hash" version = "2.1.2" @@ -2100,7 +2282,7 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "16b380a1238663e5f8a691f9039c73e1cdae598a30e9855f541d29b08b53e9a5" dependencies = [ - "itertools", + "itertools 0.14.0", "unicode-segmentation", "unicode-width", ] @@ -2135,6 +2317,12 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "vcpkg" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" + [[package]] name = "version_check" version = "0.9.5" @@ -2262,6 +2450,20 @@ dependencies = [ "semver", ] +[[package]] +name = "wax" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8d12a78aa0bab22d2f26ed1a96df7ab58e8a93506a3e20adb47c51a93b4e1357" +dependencies = [ + "const_format", + "itertools 0.11.0", + "nom", + "pori", + "regex", + "thiserror 1.0.69", +] + [[package]] name = "wezterm-bidi" version = "0.2.3" @@ -2424,6 +2626,15 @@ dependencies = [ "windows-link", ] +[[package]] +name = "winnow" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0592e1c9d151f854e6fd382574c3a0855250e1d9b2f99d9281c6e6391af352f1" +dependencies = [ + "memchr", +] + [[package]] name = "wit-bindgen" version = "0.51.0" @@ -2534,6 +2745,7 @@ dependencies = [ "serde", "serde-big-array", "sha3", + "wax", ] [[package]] diff --git a/NOTICE b/NOTICE index f84c12a..05539b2 100644 --- a/NOTICE +++ b/NOTICE @@ -16,3 +16,62 @@ Licensed under the Apache License, Version 2.0. xtafkit and the upstream fatx-rs are both licensed under the Apache License, Version 2.0. See LICENSE for the full license text. + +Third-party code vendored under Apache-2.0-compatible licenses: + +iso2god (in fatxlib/src/iso/god/) +--------------------------------- +Vendored from QAston/iso2god-rs (xdvdfx branch), itself a fork of +iliazeus/iso2god-rs. Both are MIT-licensed: + + Copyright (c) 2023 Ilia Pozdnyakov + + Permission is hereby granted, free of charge, to any person obtaining + a copy of this software and associated documentation files (the + "Software"), to deal in the Software without restriction, including + without limitation the rights to use, copy, modify, merge, publish, + distribute, sublicense, and/or sell copies of the Software, and to + permit persons to whom the Software is furnished to do so, subject to + the following conditions: + + The above copyright notice and this permission notice shall be + included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY + CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, + TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE + SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Upstream sources: + https://github.com/iliazeus/iso2god-rs (parent) + https://github.com/QAston/iso2god-rs/tree/xdvdfx (vendored branch) + +Local adaptations made when vendoring: + - anyhow::Error replaced with fatxlib's FatxError so callers see one + error type across the library. + - Intra-crate imports rewritten from `crate::god` / `crate::executable` + to `crate::iso::god` / `crate::executable`. + - The upstream `src/game_list/` (~5 KLOC compiled-in title catalog) + was NOT vendored; fatxlib's existing `titles` module covers the + same purpose with broader data. + +XellLaunch2_retail.xex (in fatxlib/tests/fixtures/tiny.xiso) +----------------------------------------------------------- +XellLaunch is a public homebrew launcher from the Free60.org project +that boots `xell-2f.bin` (the XeLL bootloader) from a CON / LIVE +container. We embed `XellLaunch2_retail.xex` inside the `tiny.xiso` +test fixture so the round-trip and walk tests can exercise our XEX +parser and full ISO → GoD pipeline against a real Xbox 360 executable +without shipping a copyrighted game. The XEX is consumed as a data +file for testing only; it is not linked into the xtafkit binary. + +Upstream: + https://free60project.github.io/wiki/XeLL.html + +XellLaunch carries the Title ID 0xFFFF011D (homebrew / dev range). +If you have license concerns about this fixture, delete +`fatxlib/tests/fixtures/tiny.xiso` and the dependent tests will +gracefully skip. diff --git a/README.md b/README.md index ad2e6e5..186d483 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE) -Mac-native TUI workbench for Xbox 360 (XTAF) and Original Xbox (FATX) drives. Plug a console drive into your Mac over USB, then browse, transfer files, resolve game titles, and decode profile gamertags from a polished terminal UI. +Mac-native TUI workbench for Xbox 360 (XTAF) and Original Xbox (FATX) drives. Plug a console drive into your Mac over USB, then browse, transfer files, resolve game titles, decode profile gamertags, and work with XISO disc images from a polished terminal UI plus a small CLI surface. ## Highlights @@ -10,6 +10,7 @@ Mac-native TUI workbench for Xbox 360 (XTAF) and Original Xbox (FATX) drives. Pl - **Title resolution** — ~5,500 Xbox 360 and Original Xbox games compiled in, with on-demand STFS header parsing for anything the catalog misses - **Profile gamertag extraction** — decrypts the embedded Account file (ARC4 + HMAC-SHA1) to label profile XUIDs - **Per-file resolution** for Arcade / XNA / Marketplace / Installer folders, with one-keystroke bulk scan +- **XISO extraction and GoD conversion** — stream XISO contents to a local directory or build Games-on-Demand packages for Xbox 360 backward-compatible titles - **Sort toggle** — by resolved name or by raw ID (display format flips to match) - **Persistent caches** under `~/.config/xtafkit/` — plain text, human-editable - macOS-native I/O (`F_NOCACHE`, `F_RDAHEAD`, device-optimal alignment) @@ -34,6 +35,8 @@ cargo build --release Produces a single binary: `target/release/xtafkit`. +The default build links against the system OpenSSL for hardware-accelerated SHA-1 during GoD conversion. On macOS install via Homebrew (`brew install openssl@3`); on Debian/Ubuntu install `libssl-dev`. To skip the OpenSSL dependency entirely and fall back to portable Rust SHA-1, build with `cargo build --release --no-default-features`. + ## Quick start ```bash @@ -59,9 +62,15 @@ xtafkit ls [PATH] [-l] list files (text in TTY, JSON when xtafkit scan [--deep] detect FATX/XTAF partitions xtafkit mkimage [--size 1G] [--populate] [--format fatx|xtaf] xtafkit resolve STFS-based title / file resolution +xtafkit extract [--keep-systemupdate] [--dry-run] +xtafkit god [--trim compact|preserve-layout|none] [--dry-run] [--game-title TITLE] ``` -Six subcommands total — file operations (download/upload/mkdir/rm/rename/copy/info/cleanup) live inside the TUI. +Seven subcommands total — file operations (download/upload/mkdir/rm/rename/copy/info/cleanup) live inside the TUI. + +## XISO Tools + +`xtafkit extract` streams every file from an XISO to a local directory and skips `$SystemUpdate` by default. `xtafkit god` converts an XISO into a Games-on-Demand package; the default trim mode is `compact`, which repacks XDVDFS densely before GoD packaging. Pass `--trim preserve-layout` to keep mastered holes, or `--trim none` to use the full data partition. ## TUI @@ -83,7 +92,7 @@ sudo xtafkit browse /dev/rdisk4 --partition "360 Data" | `R` | Resolve title or bulk-scan files (slot-aware) | | `s` | Toggle sort: by name ⇄ by ID (flips bracket order) | | `m` | Create directory | -| `d` / `u` | Download / upload | +| `d` / `u` | Download / upload (XISO uploads prompt for e**(X)**tract / **(G)**oD / **(R)**aw — see below) | | `D` / `r` | Delete / rename | | `i` | Volume info | | `c` | Clean up macOS metadata | @@ -91,6 +100,24 @@ sudo xtafkit browse /dev/rdisk4 --partition "360 Data" Entries that can be resolved show a `?` marker. Resolution results are cached under `~/.config/xtafkit/` and persist across runs. +### Uploading an XISO + +When the file you point at is an Xbox / Xbox 360 disc image (XDVDFS volume detected automatically), the upload prompt becomes: + +``` +Detected XISO 'Halo.iso'. e(X)tract / (G)oD / (R)aw / Esc: +``` + +| Choice | Result | +|---|---| +| **(X)tract** | Walks the XISO and writes each file into `//` on the drive. `$SystemUpdate` is skipped automatically. `` is the catalog-known game title when available, otherwise the local filename stem. Best for alt dashboards (Aurora / FreeStyle / XBMC4XBOX) that launch loose `default.xex` / `default.xbe` directly. | +| **(G)oD** | Streams a Games-on-Demand package into `//00007000/{,.data/}`. Uses the compact trim by default so the output is sized to actual content, not the original mastered layout. Required for stock Xbox 360 backward-compatibility playback. | +| **(R)aw** | Plain byte-for-byte copy of the source ISO file. | + +The default action (the capitalized letter) flips by context: inside `/Content//` the default is **G** (where the dashboard looks for BC packages); everywhere else the default is **X**. + +Press `Esc` to cancel mid-conversion at any time — the worker checks between parts and between hash-tree steps. + ## `xtafkit resolve` Auto-dispatches by what you point at: @@ -142,7 +169,7 @@ Tests use file-backed FATX/XTAF images generated by `xtafkit mkimage`. No hardwa ## Origin -`xtafkit` started as a fork of [joshuareisbord/fatx-rs](https://github.com/joshuareisbord/fatx-rs), which provided the FATX/XTAF filesystem core. The project has since diverged — title catalog with merged Xbox 360 + Original Xbox sources, on-demand STFS resolution, profile gamertag decryption, slot-aware folder display, TUI-first workflow, and a deliberate scope reduction (the NFS Finder-mount server was removed in favor of doing all file operations inside the TUI). Credit to the original author for the filesystem foundation. +`xtafkit` started as a fork of [joshuareisbord/fatx-rs](https://github.com/joshuareisbord/fatx-rs), which provided the FATX/XTAF filesystem core. The project has since diverged — title catalog with merged Xbox 360 + Original Xbox sources, on-demand STFS resolution, profile gamertag decryption, slot-aware folder display, TUI-first workflow, XISO extraction, and Games-on-Demand conversion. Credit to the original author for the filesystem foundation. ## License diff --git a/fatxlib/Cargo.toml b/fatxlib/Cargo.toml index dbefc24..245d22d 100644 --- a/fatxlib/Cargo.toml +++ b/fatxlib/Cargo.toml @@ -16,7 +16,17 @@ libc = "0.2" phf = "0.13" hmac = "0.13" sha1 = "0.11" -xdvdfs = { version = "0.8", default-features = false, features = ["std", "read", "sync"] } +byteorder = "1.5" +num_enum = "0.7" +openssl = { version = "0.10", optional = true } +xdvdfs = { version = "0.8", default-features = false, features = ["std", "read", "write", "sync"] } + +[features] +default = ["openssl-hash"] +# Route iso2god's hot-path SHA-1 through `openssl::sha::sha1` (ARMv8 SHA +# on Apple Silicon, SHA-NI on x86). Disable if the host's OpenSSL is +# hard to find; everything falls back to the RustCrypto `sha1` crate. +openssl-hash = ["dep:openssl"] [target.'cfg(target_os = "macos")'.dependencies] nix = { version = "0.31", features = ["fs", "ioctl"] } diff --git a/fatxlib/examples/iso2god.rs b/fatxlib/examples/iso2god.rs new file mode 100644 index 0000000..ee70460 --- /dev/null +++ b/fatxlib/examples/iso2god.rs @@ -0,0 +1,95 @@ +//! Minimal CLI wrapper around [`fatxlib::iso::god::convert_iso`]. Argument +//! shape: +//! +//! ```text +//! iso2god [--trim MODE] [--dry-run] [--game-title TITLE] +//! ``` +//! +//! `--trim compact` is the default. Pass `--trim preserve-layout` to retain +//! mastered holes, or `--trim none` to convert the full source partition. +//! +//! `-j N` isn't exposed; `convert_iso` is single-threaded. + +use std::env; +use std::path::PathBuf; +use std::process; +use std::time::Instant; + +use fatxlib::iso::god::{ConvertOptions, TrimMode, convert_iso}; + +fn usage_and_exit() -> ! { + eprintln!( + "usage: iso2god [--trim compact|preserve-layout|none] [--dry-run] [--game-title TITLE] " + ); + process::exit(2); +} + +fn main() { + let mut args = env::args().skip(1); + let mut trim = TrimMode::Compact; + let mut dry_run = false; + let mut game_title: Option = None; + let mut positional: Vec = Vec::new(); + + while let Some(arg) = args.next() { + match arg.as_str() { + "--trim" => { + trim = match args.next().as_deref() { + Some("preserve-layout") => TrimMode::PreserveLayout, + Some("none") => TrimMode::None, + Some("compact") => TrimMode::Compact, + _ => usage_and_exit(), + }; + } + "--dry-run" => dry_run = true, + "--game-title" => { + game_title = Some(args.next().unwrap_or_else(|| usage_and_exit())); + } + "-h" | "--help" => usage_and_exit(), + _ => positional.push(arg), + } + } + + if positional.len() != 2 { + usage_and_exit(); + } + let source = PathBuf::from(&positional[0]); + let dest = PathBuf::from(&positional[1]); + + let started = Instant::now(); + let mut last_stage = String::new(); + let mut progress_cb = |stage: &str, current: u64, total: u64| { + if stage != last_stage { + eprintln!("[{stage}] {current}/{total}"); + last_stage = stage.to_string(); + } else if total > 0 && (current == total || current.is_multiple_of(total.max(1) / 10 + 1)) { + eprintln!("[{stage}] {current}/{total}"); + } + }; + + let mut opts = ConvertOptions { + trim, + game_title: game_title.as_deref(), + dry_run, + progress: Some(&mut progress_cb), + should_abort: None, + }; + + match convert_iso(&source, &dest, &mut opts) { + Ok(report) => { + let elapsed = started.elapsed(); + eprintln!(); + eprintln!("Title ID: {:08X}", report.title_id); + eprintln!("Media ID: {:08X}", report.media_id); + eprintln!("Content: {:?}", report.content_type); + eprintln!("Block count: {}", report.block_count); + eprintln!("Part count: {}", report.part_count); + eprintln!("Data size: {} bytes", report.data_size); + eprintln!("Elapsed: {:?}", elapsed); + } + Err(e) => { + eprintln!("convert_iso failed: {e}"); + process::exit(1); + } + } +} diff --git a/fatxlib/examples/list_xiso.rs b/fatxlib/examples/list_xiso.rs index f882ea4..e647e28 100644 --- a/fatxlib/examples/list_xiso.rs +++ b/fatxlib/examples/list_xiso.rs @@ -15,7 +15,7 @@ use std::path::PathBuf; use std::process; use std::time::Instant; -use fatxlib::xiso::XisoImage; +use fatxlib::iso::image::XisoImage; fn human_size(n: u64) -> String { const UNITS: &[&str] = &["B", "KiB", "MiB", "GiB", "TiB"]; diff --git a/fatxlib/src/executable/mod.rs b/fatxlib/src/executable/mod.rs new file mode 100644 index 0000000..df4c29b --- /dev/null +++ b/fatxlib/src/executable/mod.rs @@ -0,0 +1,108 @@ +use crate::error::{FatxError, Result}; +use crate::iso::god::ContentType; +use byteorder::{BE, LE, ReadBytesExt}; +use std::io::{Read, Seek, SeekFrom}; +use xdvdfs::{blockdev::BlockDeviceRead, layout::VolumeDescriptor}; + +pub mod xbe; +pub mod xex; + +#[derive(Clone, Debug)] +pub struct TitleExecutionInfo { + pub media_id: u32, + pub version: u32, + pub base_version: u32, + pub title_id: u32, + pub platform: u8, + pub executable_type: u8, + pub disc_number: u8, + pub disc_count: u8, +} + +#[derive(Clone, Debug)] +pub struct TitleInfo { + pub content_type: ContentType, + pub execution_info: TitleExecutionInfo, +} + +impl TitleExecutionInfo { + pub fn from_xex(mut reader: R) -> Result { + Ok(TitleExecutionInfo { + media_id: reader.read_u32::().map_err(FatxError::Io)?, + version: reader.read_u32::().map_err(FatxError::Io)?, + base_version: reader.read_u32::().map_err(FatxError::Io)?, + title_id: reader.read_u32::().map_err(FatxError::Io)?, + platform: reader.read_u8().map_err(FatxError::Io)?, + executable_type: reader.read_u8().map_err(FatxError::Io)?, + disc_number: reader.read_u8().map_err(FatxError::Io)?, + disc_count: reader.read_u8().map_err(FatxError::Io)?, + }) + } + + pub fn from_xbe(mut reader: R) -> Result { + reader.seek(SeekFrom::Current(8)).map_err(FatxError::Io)?; + let title_id = reader.read_u32::().map_err(FatxError::Io)?; + + reader.seek(SeekFrom::Current(164)).map_err(FatxError::Io)?; + let version = reader.read_u32::().map_err(FatxError::Io)?; + + Ok(TitleExecutionInfo { + media_id: 0, + version, + base_version: 0, + title_id, + platform: 0, + executable_type: 0, + disc_number: 1, + disc_count: 1, + }) + } +} + +impl TitleInfo { + pub fn from_image + Seek, E: std::fmt::Debug>( + xiso: &mut R, + volume: VolumeDescriptor, + ) -> Result { + if let Ok(direntnode) = volume.root_table.walk_path(xiso, "Default.xex") { + let mut data = direntnode + .node + .dirent + .read_data_all(xiso) + .map_err(|e| FatxError::Other(format!("xdvdfs read Default.xex: {e:?}")))?; + let mut data_slice = std::io::Cursor::new(data.as_mut()); + + let default_xex_header = xex::XexHeader::read(&mut data_slice) + .map_err(|e| FatxError::Other(format!("error reading default.xex: {e}")))?; + let execution_info = default_xex_header.fields.execution_info.ok_or_else(|| { + FatxError::Other("no execution info in default.xex header".to_string()) + })?; + + Ok(TitleInfo { + content_type: ContentType::GamesOnDemand, + execution_info, + }) + } else if let Ok(direntnode) = volume.root_table.walk_path(xiso, "default.xbe") { + let mut data = direntnode + .node + .dirent + .read_data_all(xiso) + .map_err(|e| FatxError::Other(format!("xdvdfs read default.xbe: {e:?}")))?; + let mut data_slice = std::io::Cursor::new(data.as_mut()); + let default_xbe_header = xbe::XbeHeader::read(&mut data_slice) + .map_err(|e| FatxError::Other(format!("error reading default.xbe: {e}")))?; + let execution_info = default_xbe_header.fields.execution_info.ok_or_else(|| { + FatxError::Other("no execution info in default.xbe header".to_string()) + })?; + + Ok(TitleInfo { + content_type: ContentType::XboxOriginal, + execution_info, + }) + } else { + Err(FatxError::Other( + "no executable found in this image".to_string(), + )) + } + } +} diff --git a/fatxlib/src/executable/xbe.rs b/fatxlib/src/executable/xbe.rs new file mode 100644 index 0000000..54baada --- /dev/null +++ b/fatxlib/src/executable/xbe.rs @@ -0,0 +1,57 @@ +use crate::error::{FatxError, Result}; +use crate::executable::TitleExecutionInfo; +use byteorder::{LE, ReadBytesExt}; +use std::io::{Read, Seek, SeekFrom}; + +pub struct XbeHeader { + // We only need these fields to get the cert address + pub dw_base_addr: u32, + pub dw_certificate_addr: u32, + pub fields: XbeHeaderFields, +} + +#[derive(Clone, Default, Debug)] +pub struct XbeHeaderFields { + pub execution_info: Option, +} + +impl XbeHeader { + pub fn read(mut reader: R) -> Result { + Self::check_magic_bytes(&mut reader)?; + + // Offset 0x0104 + reader.seek(SeekFrom::Current(256)).map_err(FatxError::Io)?; + let dw_base_addr = reader.read_u32::().map_err(FatxError::Io)?; + + // Offset 0x0118 + reader.seek(SeekFrom::Current(16)).map_err(FatxError::Io)?; + let dw_certificate_addr = reader.read_u32::().map_err(FatxError::Io)?; + + let offset = reader.stream_position().map_err(FatxError::Io)? - 284; + let cert_address = dw_certificate_addr - dw_base_addr; + reader + .seek(SeekFrom::Start(offset + (cert_address as u64))) + .map_err(FatxError::Io)?; + + Ok(XbeHeader { + dw_base_addr, + dw_certificate_addr, + fields: XbeHeaderFields { + execution_info: Some(TitleExecutionInfo::from_xbe(reader)?), + }, + }) + } + + fn check_magic_bytes(mut reader: R) -> Result<()> { + let mut magic_bytes = [0u8; 4]; + reader.read_exact(&mut magic_bytes).map_err(FatxError::Io)?; + + if &magic_bytes != b"XBEH" { + return Err(FatxError::Other( + "missing 'XBEH' magic bytes in XBE header".to_string(), + )); + } + + Ok(()) + } +} diff --git a/fatxlib/src/executable/xex.rs b/fatxlib/src/executable/xex.rs new file mode 100644 index 0000000..21c01cd --- /dev/null +++ b/fatxlib/src/executable/xex.rs @@ -0,0 +1,142 @@ +use std::io::{Read, Seek, SeekFrom}; + +use byteorder::{BE, ReadBytesExt}; + +use bitflags::bitflags; +use num_enum::TryFromPrimitive; + +use crate::error::{FatxError, Result}; +use crate::executable::TitleExecutionInfo; + +#[derive(Clone, Debug)] +pub struct XexHeader { + pub module_flags: XexModuleFlags, + pub code_offset: u32, + pub certificate_offset: u32, + pub fields: XexHeaderFields, +} + +bitflags! { + // based on https://free60.org/System-Software/Formats/XEX/#xex-header + #[derive(Clone, Copy, PartialEq, Eq, Debug)] + pub struct XexModuleFlags: u32 { + const TITLE_MODULE = 0x01; + const EXPORTS_TO_TITLE = 0x02; + const SYSTEM_DEBUGGER = 0x04; + const DLL_MODULE = 0x08; + const MODULE_PATCH = 0x10; + const FULL_PATCH = 0x20; + const DELTA_PATCH = 0x40; + const USER_MODE = 0x80; + } +} + +#[derive(Clone, Default, Debug)] +pub struct XexHeaderFields { + pub execution_info: Option, + // other fields will be added if and when necessary +} + +// based on https://free60.org/System-Software/Formats/XEX/#header-ids +#[repr(u32)] +#[derive(Clone, Debug, PartialEq, Eq, TryFromPrimitive)] +#[allow(dead_code)] +enum XexHeaderFieldId { + ResourceInfo = 0x_00_00_02_ff, + BaseFileFormat = 0x_00_00_03_ff, + BaseReference = 0x_00_00_04_05, + DeltaPatchDescriptor = 0x_00_00_05_ff, + BoundingPath = 0x_00_00_80_ff, + DeviceId = 0x_00_00_81_05, + OriginalBaseAddress = 0x_00_01_00_01, + EntryPoint = 0x_00_01_01_00, + ImageBaseAddress = 0x_00_01_02_01, + ImportLibraries = 0x_00_01_03_ff, + ChecksumTimestamp = 0x_00_01_80_02, + EnabledForCallcap = 0x_00_01_81_02, + EnabledForFastcap = 0x_00_01_82_00, + OriginalPeName = 0x_00_01_83_ff, + StaticLibraries = 0x_00_02_00_ff, + TlsInfo = 0x_00_02_01_04, + DefaultStackSize = 0x_00_02_02_00, + DefaultFilesystemCacheSize = 0x_00_02_03_01, + DefaultHeapSize = 0x_00_02_04_01, + PageHeapSizeAndFlags = 0x_00_02_80_02, + SystemFlags = 0x_00_03_00_00, + ExecutionId = 0x_00_04_00_06, + ServiceIdList = 0x_00_04_01_ff, + TitleWorkspaceSize = 0x_00_04_02_01, + GameRatings = 0x_00_04_03_10, + LanKey = 0x_00_04_04_04, + Xbox360Logo = 0x_00_04_05_ff, + MultidiscMediaIds = 0x_00_04_06_ff, + AlternateTitleIds = 0x_00_04_07_ff, + AdditionalTitleMemory = 0x_00_04_08_01, + ExportsByName = 0x_00_e1_04_02, +} + +impl XexHeader { + pub fn read(mut reader: R) -> Result { + Self::check_magic_bytes(&mut reader)?; + Self::read_checked(reader) + } + + fn check_magic_bytes(mut reader: R) -> Result<()> { + let mut buf = [0_u8; 4]; + reader.read_exact(&mut buf).map_err(FatxError::Io)?; + + reader.seek(SeekFrom::Current(-4)).map_err(FatxError::Io)?; + + if buf != "XEX2".as_bytes() { + return Err(FatxError::Other( + "missing 'XEX2' magic bytes in XEX header".to_string(), + )); + } + + Ok(()) + } + + fn read_checked(mut reader: R) -> Result { + let header_offset = reader.stream_position().map_err(FatxError::Io)?; + + let _ = reader.read_u32::().map_err(FatxError::Io)?; + + let module_flags = reader.read_u32::().map_err(FatxError::Io)?; + let module_flags = XexModuleFlags::from_bits_truncate(module_flags); + + let code_offset = reader.read_u32::().map_err(FatxError::Io)?; + + let _ = reader.read_u32::().map_err(FatxError::Io)?; + + let certificate_offset = reader.read_u32::().map_err(FatxError::Io)?; + + let mut fields: XexHeaderFields = Default::default(); + let field_count = reader.read_u32::().map_err(FatxError::Io)?; + + for _ in 0..field_count { + let key = reader.read_u32::().map_err(FatxError::Io)?; + let value = reader.read_u32::().map_err(FatxError::Io)?; + + let key = XexHeaderFieldId::try_from(key).ok(); + type Key = XexHeaderFieldId; + + if let Some(Key::ExecutionId) = key { + let offset = reader.stream_position().map_err(FatxError::Io)?; + reader + .seek(SeekFrom::Start(header_offset + (value as u64))) + .map_err(FatxError::Io)?; + fields.execution_info = Some(TitleExecutionInfo::from_xex(&mut reader)?); + reader + .seek(SeekFrom::Start(offset)) + .map_err(FatxError::Io)?; + }; + } + + Ok(XexHeader { + module_flags, + code_offset, + certificate_offset, + fields, + }) + } +} diff --git a/fatxlib/src/iso/compact.rs b/fatxlib/src/iso/compact.rs new file mode 100644 index 0000000..ff7c50f --- /dev/null +++ b/fatxlib/src/iso/compact.rs @@ -0,0 +1,373 @@ +//! Compact virtual XDVDFS layout planning. +//! +//! Builds an in-memory plan for a dense XDVDFS image without materializing a +//! temporary `.iso` on disk. Metadata regions are synthesized in memory; file +//! regions are read lazily from the source image when the reader is consumed. + +use std::collections::{BTreeMap, HashMap, HashSet}; +use std::fs::File; +use std::io::{Read, Seek, SeekFrom}; +use std::path::Path; + +use crate::error::{FatxError, Result}; +use crate::executable::TitleExecutionInfo; + +use super::god::ContentType; +use super::image::XisoImage; +use super::manifest::{IsoFilterPolicy, build_manifest}; + +use xdvdfs::layout::{DirectoryEntryTable, SECTOR_SIZE, VolumeDescriptor}; +use xdvdfs::write::dirtab::DirectoryEntryTableWriter; +use xdvdfs::write::fs::{FileEntry, FileType, Filesystem, PathVec, XDVDFSFilesystem}; +use xdvdfs::write::sector::SectorAllocator; + +type SourceOffsetDevice = xdvdfs::blockdev::OffsetWrapper; +type SourceFilesystem = XDVDFSFilesystem; + +#[derive(Clone)] +struct CompactTreeEntry { + dir: PathVec, + listing: Vec, +} + +enum CompactRegionData { + Bytes(Box<[u8]>), + Source { source_offset: u64 }, +} + +struct CompactRegion { + start: u64, + len: u64, + data: CompactRegionData, +} + +struct CompactImagePlan { + data_size: u64, + regions: Vec, +} + +pub(crate) struct CompactSource { + exe_info: TitleExecutionInfo, + content_type: ContentType, + partition_offset: u64, + plan: CompactImagePlan, +} + +pub(crate) struct CompactImageReader<'a> { + source: File, + partition_offset: u64, + plan: &'a CompactImagePlan, + cursor: u64, +} + +impl CompactSource { + pub(crate) fn open_reader(&self, source_iso: &Path) -> Result> { + Ok(CompactImageReader { + source: File::open(source_iso).map_err(FatxError::Io)?, + partition_offset: self.partition_offset, + plan: &self.plan, + cursor: 0, + }) + } + + pub(crate) fn exe_info(&self) -> &TitleExecutionInfo { + &self.exe_info + } + + pub(crate) fn content_type(&self) -> ContentType { + self.content_type + } + + pub(crate) fn data_size(&self) -> u64 { + self.plan.data_size + } +} + +impl CompactImageReader<'_> { + fn read_at(&mut self, offset: u64, buf: &mut [u8]) -> std::io::Result<()> { + buf.fill(0); + if buf.is_empty() { + return Ok(()); + } + + let end = offset.saturating_add(buf.len() as u64); + let mut idx = self + .plan + .regions + .partition_point(|region| region.start.saturating_add(region.len) <= offset); + + while idx < self.plan.regions.len() { + let region = &self.plan.regions[idx]; + let region_end = region.start.saturating_add(region.len); + if region.start >= end { + break; + } + + let overlap_start = offset.max(region.start); + let overlap_end = end.min(region_end); + if overlap_start < overlap_end { + let dst_start = (overlap_start - offset) as usize; + let dst_end = (overlap_end - offset) as usize; + let dst = &mut buf[dst_start..dst_end]; + let src_offset = overlap_start - region.start; + match ®ion.data { + CompactRegionData::Bytes(bytes) => { + let src_start = src_offset as usize; + let src_end = src_start + dst.len(); + dst.copy_from_slice(&bytes[src_start..src_end]); + } + CompactRegionData::Source { source_offset } => { + self.source.seek(SeekFrom::Start( + self.partition_offset + source_offset + src_offset, + ))?; + self.source.read_exact(dst)?; + } + } + } + idx += 1; + } + + Ok(()) + } +} + +impl Read for CompactImageReader<'_> { + fn read(&mut self, buf: &mut [u8]) -> std::io::Result { + if self.cursor >= self.plan.data_size || buf.is_empty() { + return Ok(0); + } + let want = ((self.plan.data_size - self.cursor) as usize).min(buf.len()); + self.read_at(self.cursor, &mut buf[..want])?; + self.cursor += want as u64; + Ok(want) + } +} + +impl Seek for CompactImageReader<'_> { + fn seek(&mut self, pos: SeekFrom) -> std::io::Result { + let len = self.plan.data_size as i128; + let next = match pos { + SeekFrom::Start(pos) => pos as i128, + SeekFrom::Current(delta) => self.cursor as i128 + delta as i128, + SeekFrom::End(delta) => len + delta as i128, + }; + if next < 0 { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "negative seek in CompactImageReader", + )); + } + self.cursor = next as u64; + Ok(self.cursor) + } +} + +fn cancelled(op: &str) -> FatxError { + FatxError::Other(format!("{op}: cancelled")) +} + +fn xdvdfs_other(ctx: &str, err: E) -> FatxError { + FatxError::Other(format!("{ctx}: {err:?}")) +} + +fn collect_compact_tree( + fs: &mut SourceFilesystem, + should_abort: Option<&dyn Fn() -> bool>, + kept_paths: &HashSet, + kept_dirs: &HashSet, +) -> Result> { + let mut dirs = vec![PathVec::default()]; + let mut out = Vec::new(); + + while let Some(dir) = dirs.pop() { + if let Some(abort) = should_abort + && abort() + { + return Err(cancelled("compact_tree")); + } + + let mut listing = + >::read_dir(fs, &dir) + .map_err(|e| FatxError::Other(format!("xdvdfs compact read_dir: {e}")))?; + listing.retain(|entry| { + let path = PathVec::from_base(&dir, &entry.name).as_string(); + let path = normalize_path(&path); + match entry.file_type { + FileType::Directory => kept_dirs.contains(path), + FileType::File => kept_paths.contains(path), + } + }); + + for entry in &listing { + if matches!(entry.file_type, FileType::Directory) { + dirs.push(PathVec::from_base(&dir, &entry.name)); + } + } + + out.push(CompactTreeEntry { dir, listing }); + } + + Ok(out) +} + +fn build_compact_dirent_tables( + tree: &[CompactTreeEntry], +) -> Result> { + let mut dirent_tables: BTreeMap = BTreeMap::new(); + + for entry in tree.iter().rev() { + let mut dirtab = DirectoryEntryTableWriter::default(); + for child in &entry.listing { + match child.file_type { + FileType::Directory => { + let child_path = PathVec::from_base(&entry.dir, &child.name); + let dir_size = dirent_tables + .get(&child_path) + .ok_or_else(|| { + FatxError::Other(format!( + "xdvdfs compact: missing dirtab for {}", + child_path.as_string() + )) + })? + .dirtab_size(); + dirtab + .add_dir::(&child.name, dir_size) + .map_err(|e| xdvdfs_other("xdvdfs add_dir", e))?; + } + FileType::File => { + let size = child + .len + .try_into() + .map_err(|_| FatxError::Other(format!("file too large: {}", child.len)))?; + dirtab + .add_file::(&child.name, size) + .map_err(|e| xdvdfs_other("xdvdfs add_file", e))?; + } + } + } + dirtab + .compute_size::() + .map_err(|e| xdvdfs_other("xdvdfs compute_size", e))?; + dirent_tables.insert(entry.dir.clone(), dirtab); + } + + Ok(dirent_tables) +} + +pub(crate) fn build_compact_source( + source_iso: &Path, + should_abort: Option<&dyn Fn() -> bool>, +) -> Result { + if let Some(abort) = should_abort + && abort() + { + return Err(cancelled("compact_source")); + } + + let manifest = { + let file = File::open(source_iso).map_err(FatxError::Io)?; + let mut img = XisoImage::open(file)?; + build_manifest( + &mut img, + IsoFilterPolicy { + keep_systemupdate: false, + }, + )? + }; + let title_info = manifest + .title_info + .clone() + .ok_or_else(|| FatxError::Other("xdvdfs compact: no executable found".into()))?; + let exe_info = title_info.execution_info; + let content_type = title_info.content_type; + let partition_offset = manifest.partition_offset; + let file_offsets: HashMap = manifest.kept_offset_map(); + let kept_paths = manifest.kept_path_set(); + let kept_dirs = manifest.kept_dir_set(); + + let file = File::open(source_iso).map_err(FatxError::Io)?; + let xiso = xdvdfs::blockdev::OffsetWrapper::new(file) + .map_err(|e| xdvdfs_other("xdvdfs offset detect", e))?; + let mut fs = XDVDFSFilesystem::new(xiso) + .ok_or_else(|| FatxError::Other("xdvdfs compact: could not open source image".into()))?; + let tree = collect_compact_tree(&mut fs, should_abort, &kept_paths, &kept_dirs)?; + let dirent_tables = build_compact_dirent_tables(&tree)?; + + let mut dir_sectors = BTreeMap::new(); + let mut allocator = SectorAllocator::default(); + let (root_path, root_dirtab) = dirent_tables + .first_key_value() + .ok_or_else(|| FatxError::Other("xdvdfs compact: empty directory tree".into()))?; + let root_sector = allocator.allocate_contiguous(root_dirtab.dirtab_size() as u64); + let root_table = DirectoryEntryTable::new(root_dirtab.dirtab_size(), root_sector); + dir_sectors.insert(root_path.clone(), root_sector as u64); + + let volume_bytes = VolumeDescriptor::new(root_table) + .serialize::() + .map_err(|e| xdvdfs_other("xdvdfs serialize volume", e))?; + let mut regions = vec![CompactRegion { + start: 32 * SECTOR_SIZE as u64, + len: volume_bytes.len() as u64, + data: CompactRegionData::Bytes(Box::from(volume_bytes)), + }]; + + for (path, dirtab) in dirent_tables { + if let Some(abort) = should_abort + && abort() + { + return Err(cancelled("compact_source")); + } + + let sector = *dir_sectors + .get(&path) + .ok_or_else(|| FatxError::Other(format!("missing sector for {}", path.as_string())))?; + let repr = dirtab + .disk_repr::(&mut allocator) + .map_err(|e| xdvdfs_other("xdvdfs disk_repr", e))?; + regions.push(CompactRegion { + start: sector * SECTOR_SIZE as u64, + len: repr.entry_table.len() as u64, + data: CompactRegionData::Bytes(repr.entry_table), + }); + + for entry in repr.file_listing { + let child_path = PathVec::from_base(&path, &entry.name); + if entry.is_dir { + dir_sectors.insert(child_path, entry.sector); + continue; + } + + let logical_path = child_path.as_string(); + let logical_path = logical_path.trim_start_matches('/').to_string(); + let source_offset = *file_offsets.get(&logical_path).ok_or_else(|| { + FatxError::Other(format!( + "xdvdfs compact: missing source offset for {}", + logical_path + )) + })?; + regions.push(CompactRegion { + start: entry.sector * SECTOR_SIZE as u64, + len: entry.size, + data: CompactRegionData::Source { source_offset }, + }); + } + } + + regions.sort_by_key(|region| region.start); + let data_size = regions + .iter() + .map(|region| region.start + region.len) + .max() + .unwrap_or(0); + + Ok(CompactSource { + exe_info, + content_type, + partition_offset, + plan: CompactImagePlan { data_size, regions }, + }) +} + +fn normalize_path(path: &str) -> &str { + path.trim_start_matches('/') +} diff --git a/fatxlib/src/iso/god/con_header.rs b/fatxlib/src/iso/god/con_header.rs new file mode 100644 index 0000000..4c648dc --- /dev/null +++ b/fatxlib/src/iso/god/con_header.rs @@ -0,0 +1,122 @@ +use byteorder::{BE, ByteOrder, LE}; + +use crate::executable::TitleExecutionInfo; + +use super::sha1_digest; + +const EMPTY_LIVE: &[u8] = include_bytes!("empty_live.bin"); + +pub struct ConHeaderBuilder { + buffer: Vec, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ContentType { + GamesOnDemand = 0x7000, + XboxOriginal = 0x5000, +} + +impl Default for ConHeaderBuilder { + fn default() -> Self { + Self::new() + } +} + +impl ConHeaderBuilder { + pub fn new() -> Self { + ConHeaderBuilder { + buffer: Vec::from(EMPTY_LIVE), + } + } + + fn write_u8(&mut self, offset: usize, value: u8) { + self.buffer[offset] = value; + } + + fn write_u16_be(&mut self, offset: usize, value: u16) { + BE::write_u16(&mut self.buffer[offset..], value); + } + + fn write_u24_be(&mut self, offset: usize, value: u32) { + BE::write_u24(&mut self.buffer[offset..], value); + } + + fn write_u32_be(&mut self, offset: usize, value: u32) { + BE::write_u32(&mut self.buffer[offset..], value); + } + + fn write_u32_le(&mut self, offset: usize, value: u32) { + LE::write_u32(&mut self.buffer[offset..], value); + } + + fn write_bytes(&mut self, offset: usize, buf: &[u8]) { + self.buffer[offset..offset + buf.len()].copy_from_slice(buf); + } + + fn write_utf16_be(&mut self, offset: usize, s: &str) { + for (i, c) in s.encode_utf16().chain([0]).enumerate() { + self.write_u16_be(offset + i * 2, c); + } + } + + pub fn with_block_counts(mut self, blocks_allocated: u32, blocks_not_allocated: u16) -> Self { + self.write_u24_be(0x0392, blocks_allocated); + self.write_u16_be(0x0395, blocks_not_allocated); + self + } + + pub fn with_content_type(mut self, content_type: ContentType) -> Self { + self.write_u32_be(0x0344, content_type as u32); + self + } + + pub fn with_data_parts_info(mut self, part_count: u32, parts_total_size: u64) -> Self { + self.write_u32_le(0x03a0, part_count); // sic! + self.write_u32_be(0x03a4, (parts_total_size / 0x0100) as u32); + self + } + + pub fn with_execution_info(mut self, exe_info: &TitleExecutionInfo) -> Self { + // TODO: maybe just pick a suitable repr() for the struct, and write it whole? + self.write_u32_be(0x0354, exe_info.media_id); + self.write_u32_be(0x0360, exe_info.title_id); + self.write_u8(0x0364, exe_info.platform); + self.write_u8(0x0365, exe_info.executable_type); + self.write_u8(0x0366, exe_info.disc_number); + self.write_u8(0x0367, exe_info.disc_count); + self + } + + pub fn with_game_icon(mut self, png_bytes: Option<&[u8]>) -> Self { + let png_bytes = png_bytes.unwrap_or(&[]); + assert!(png_bytes.len() <= 0x0400); + + self.write_u32_be(0x1712, png_bytes.len() as u32); + self.write_u32_be(0x1716, png_bytes.len() as u32); + self.write_bytes(0x171a, png_bytes); + self.write_bytes(0x571a, png_bytes); + self + } + + pub fn with_game_title(mut self, game_title: &str) -> Self { + self.write_utf16_be(0x0411, game_title); + self.write_utf16_be(0x1691, game_title); + self + } + + pub fn with_mht_hash(mut self, mht_hash: &[u8; 20]) -> Self { + self.write_bytes(0x037d, mht_hash); + self + } + + pub fn finalize(mut self) -> Vec { + self.buffer[0x035b] = 0; + self.buffer[0x035f] = 0; + self.buffer[0x0391] = 0; + + let digest = sha1_digest(&self.buffer[0x0344..(0x0344 + 0xacbc)]); + self.write_bytes(0x032c, &digest); + + self.buffer + } +} diff --git a/fatxlib/src/iso/god/convert.rs b/fatxlib/src/iso/god/convert.rs new file mode 100644 index 0000000..638eff5 --- /dev/null +++ b/fatxlib/src/iso/god/convert.rs @@ -0,0 +1,96 @@ +//! Public entry point for ISO → Games-on-Demand conversion. +//! +//! The actual work is split across: +//! - `prepare` for source analysis and layout sizing +//! - `core` for the shared conversion loop +//! - `sink_host` / `sink_fatx` for transport-specific outputs +//! +//! See `NOTICE` for the upstream sources this code descends from. + +use std::path::Path; + +use crate::error::Result; +use crate::volume::FatxVolume; + +use super::ContentType; +use super::core::run_conversion; +use super::prepare::prepare_source; +use super::sink_fatx::FatxSink; +use super::sink_host::HostFsSink; + +/// Progress callback shape: `(stage, current, total)` where `stage` is one +/// of `"parts"`, `"mht"`, `"header"`. +pub type ProgressFn<'a> = &'a mut dyn FnMut(&str, u64, u64); + +/// How to size the output GoD relative to the source ISO. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum TrimMode { + /// Walk the existing directory tree, find the max `(offset + size)`, + /// and pack only that many bytes. Preserves any mastered holes inside + /// the XDVDFS layout while trimming trailing slack after the highest + /// file extent. + PreserveLayout, + /// Pack every byte from the start of the data partition to the end of + /// the source file. Larger output, but useful when the directory tree + /// is suspect. + None, + /// Rebuild the XDVDFS image densely as a virtual layout and stream + /// those bytes directly through the GoD pipeline. + #[default] + Compact, +} + +/// Knobs the caller can adjust per conversion. +#[derive(Default)] +pub struct ConvertOptions<'a> { + pub trim: TrimMode, + pub game_title: Option<&'a str>, + pub dry_run: bool, + pub progress: Option>, + pub should_abort: Option<&'a dyn Fn() -> bool>, +} + +/// Metadata extracted from the source ISO and the resulting layout sizing. +#[derive(Debug, Clone, Copy)] +pub struct ConvertReport { + pub title_id: u32, + pub media_id: u32, + pub content_type: ContentType, + pub part_count: u64, + pub block_count: u64, + pub data_size: u64, +} + +/// Convert an Xbox 360 / original-Xbox ISO into a Games-on-Demand package. +pub fn convert_iso<'a>( + source_iso: &Path, + dest_dir: &Path, + opts: &'a mut ConvertOptions<'a>, +) -> Result { + let source = prepare_source(source_iso, opts)?; + if opts.dry_run { + return Ok(source.report); + } + + let mut sink = HostFsSink::new(dest_dir); + run_conversion(&source, &mut sink, opts, "convert_iso") +} + +/// Convert an ISO directly into a Games-on-Demand package rooted at a FATX volume. +pub fn convert_iso_to_fatx<'a, T>( + source_iso: &Path, + vol: &mut FatxVolume, + dest_dir: &str, + opts: &'a mut ConvertOptions<'a>, +) -> Result +where + T: std::io::Read + std::io::Seek + std::io::Write, +{ + let source = prepare_source(source_iso, opts)?; + if opts.dry_run { + return Ok(source.report); + } + + let mut sink = FatxSink::new(vol, dest_dir); + run_conversion(&source, &mut sink, opts, "convert_iso_to_fatx") +} diff --git a/fatxlib/src/iso/god/core.rs b/fatxlib/src/iso/god/core.rs new file mode 100644 index 0000000..579bbbf --- /dev/null +++ b/fatxlib/src/iso/god/core.rs @@ -0,0 +1,132 @@ +use crate::error::{FatxError, Result}; + +use super::prepare::PreparedSource; +use super::{ + BLOCK_SIZE, BLOCKS_PER_PART, ConHeaderBuilder, ConvertOptions, ConvertReport, HashList, +}; + +pub(crate) trait GodSink { + fn begin(&mut self, source: &PreparedSource) -> Result<()>; + fn write_part<'a>( + &mut self, + source: &PreparedSource, + part_index: u64, + opts: &mut ConvertOptions<'a>, + ) -> Result<()>; + fn read_master_hash(&mut self, source: &PreparedSource, part_index: u64) -> Result; + fn write_master_hash( + &mut self, + source: &PreparedSource, + part_index: u64, + mht: &HashList, + ) -> Result<()>; + fn last_part_size(&self, source: &PreparedSource) -> Result; + fn write_con_header(&mut self, source: &PreparedSource, con_bytes: Vec) -> Result<()>; + fn flush_after_parts(&mut self) -> Result<()> { + Ok(()) + } + fn flush_after_mht(&mut self) -> Result<()> { + Ok(()) + } + fn flush_after_header(&mut self) -> Result<()> { + Ok(()) + } +} + +pub(crate) fn run_conversion<'a, S: GodSink>( + source: &PreparedSource, + sink: &mut S, + opts: &mut ConvertOptions<'a>, + cancel_ctx: &str, +) -> Result { + if source.report.part_count == 0 { + return Err(FatxError::Other(format!( + "{cancel_ctx}: source has no data to convert" + ))); + } + + sink.begin(source)?; + + if let Some(cb) = opts.progress.as_deref_mut() { + cb("parts", 0, source.report.part_count); + } + for part_index in 0..source.report.part_count { + if let Some(abort) = opts.should_abort + && abort() + { + return Err(FatxError::Other(format!("{cancel_ctx}: cancelled"))); + } + sink.write_part(source, part_index, opts)?; + if let Some(cb) = opts.progress.as_deref_mut() { + cb("parts", part_index + 1, source.report.part_count); + } + } + sink.flush_after_parts()?; + + if let Some(cb) = opts.progress.as_deref_mut() { + cb("mht", 0, source.report.part_count); + } + let mut mht = sink.read_master_hash(source, source.report.part_count - 1)?; + for prev_part_index in (0..source.report.part_count - 1).rev() { + if let Some(abort) = opts.should_abort + && abort() + { + return Err(FatxError::Other(format!("{cancel_ctx}: cancelled"))); + } + let mut prev_mht = sink.read_master_hash(source, prev_part_index)?; + prev_mht.add_hash(&mht.digest()); + sink.write_master_hash(source, prev_part_index, &prev_mht)?; + mht = prev_mht; + if let Some(cb) = opts.progress.as_deref_mut() { + cb( + "mht", + source.report.part_count - prev_part_index, + source.report.part_count, + ); + } + } + sink.flush_after_mht()?; + + if let Some(cb) = opts.progress.as_deref_mut() { + cb("header", 0, 1); + } + let last_part_size = sink.last_part_size(source)?; + let con_header = build_con_header(source, &mht.digest(), opts.game_title, last_part_size); + sink.write_con_header(source, con_header)?; + sink.flush_after_header()?; + if let Some(cb) = opts.progress.as_deref_mut() { + cb("header", 1, 1); + } + + Ok(source.report) +} + +pub(crate) fn build_con_header( + source: &PreparedSource, + mht_digest: &[u8; 20], + game_title: Option<&str>, + last_part_size: u64, +) -> Vec { + let mut con_header = ConHeaderBuilder::new() + .with_execution_info(&source.exe_info) + .with_block_counts(source.report.block_count as u32, 0) + .with_data_parts_info( + source.report.part_count as u32, + last_part_size + (source.report.part_count - 1) * BLOCK_SIZE * 0xa290, + ) + .with_content_type(source.content_type) + .with_mht_hash(mht_digest); + if let Some(title) = game_title { + con_header = con_header.with_game_title(title); + } + con_header.finalize() +} + +pub(crate) fn part_payload_bytes(data_size: u64, part_index: u64) -> u64 { + let part_start = part_index + .saturating_mul(BLOCKS_PER_PART) + .saturating_mul(BLOCK_SIZE); + data_size + .saturating_sub(part_start) + .min(BLOCKS_PER_PART * BLOCK_SIZE) +} diff --git a/fatxlib/src/iso/god/empty_live.bin b/fatxlib/src/iso/god/empty_live.bin new file mode 100644 index 0000000..0bc7abd Binary files /dev/null and b/fatxlib/src/iso/god/empty_live.bin differ diff --git a/fatxlib/src/iso/god/file_layout.rs b/fatxlib/src/iso/god/file_layout.rs new file mode 100644 index 0000000..2c5829d --- /dev/null +++ b/fatxlib/src/iso/god/file_layout.rs @@ -0,0 +1,62 @@ +use std::path::{Path, PathBuf}; + +use crate::executable::TitleExecutionInfo; + +use super::*; + +pub struct FileLayout<'a> { + base_path: &'a Path, + exe_info: &'a TitleExecutionInfo, + content_type: ContentType, +} + +impl<'a> FileLayout<'a> { + pub fn new( + base_path: &'a Path, + exe_info: &'a TitleExecutionInfo, + content_type: ContentType, + ) -> FileLayout<'a> { + FileLayout { + base_path, + exe_info, + content_type, + } + } + + fn title_id_string(&self) -> String { + format!("{:08X}", self.exe_info.title_id) + } + + fn content_type_string(&self) -> String { + format!("{:08X}", self.content_type as u32) + } + + fn media_id_string(&self) -> String { + match self.content_type { + ContentType::GamesOnDemand => { + format!("{:08X}", self.exe_info.media_id) + } + ContentType::XboxOriginal => { + format!("{:08X}", self.exe_info.title_id) + } + } + } + + pub fn data_dir_path(&self) -> PathBuf { + self.base_path + .join(self.title_id_string()) + .join(self.content_type_string()) + .join(self.media_id_string() + ".data") + } + + pub fn part_file_path(&'a self, part_index: u64) -> PathBuf { + self.data_dir_path().join(format!("Data{:04}", part_index)) + } + + pub fn con_header_file_path(&self) -> PathBuf { + self.base_path + .join(self.title_id_string()) + .join(self.content_type_string()) + .join(self.media_id_string()) + } +} diff --git a/fatxlib/src/iso/god/gdf_sector.rs b/fatxlib/src/iso/god/gdf_sector.rs new file mode 100644 index 0000000..7364190 --- /dev/null +++ b/fatxlib/src/iso/god/gdf_sector.rs @@ -0,0 +1,132 @@ +#[rustfmt::skip] +pub const GDF_SECTOR: [u8; 2055] = [ + 1, 0x43, 0x44, 0x30, 0x30, 0x31, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0x17, 0x4b, 0, 0, 0, 0, 0x4b, 0x17, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, + 0, 8, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x30, 0x30, 0x30, + 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0, 0x30, 0x30, + 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0, 0x30, + 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0, + 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, + 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0xff, 0x43, 0x44, 0x30, 0x30, 0x31, 1, +]; diff --git a/fatxlib/src/iso/god/hash_list.rs b/fatxlib/src/iso/god/hash_list.rs new file mode 100644 index 0000000..b94de04 --- /dev/null +++ b/fatxlib/src/iso/god/hash_list.rs @@ -0,0 +1,61 @@ +use std::io::{Read, Write}; + +use crate::error::{FatxError, Result}; + +use super::sha1_digest; + +#[derive(Clone)] +pub struct HashList { + buffer: [u8; 4096], + len: usize, +} + +impl Default for HashList { + fn default() -> Self { + Self::new() + } +} + +impl HashList { + pub fn bytes(&self) -> &[u8; 4096] { + &self.buffer + } + + pub fn new() -> HashList { + HashList { + buffer: [0u8; 4096], + len: 0, + } + } + + pub fn read(mut reader: R) -> Result { + let mut buffer = [0u8; 4096]; + reader.read_exact(&mut buffer).map_err(FatxError::Io)?; + + let len = buffer + .chunks(20) + .position(|c| *c == [0u8; 20]) + .map(|p| p * 20) + .unwrap_or(buffer.len()); + + Ok(HashList { buffer, len }) + } + + pub fn add_hash(&mut self, hash: &[u8; 20]) { + self.buffer[self.len..self.len + 20].copy_from_slice(hash); + self.len += 20; + } + + pub fn add_block_hash(&mut self, block: &[u8]) { + self.add_hash(&sha1_digest(block)) + } + + pub fn digest(&self) -> [u8; 20] { + sha1_digest(&self.buffer) + } + + pub fn write(&self, mut writer: W) -> Result<()> { + writer.write_all(&self.buffer).map_err(FatxError::Io)?; + Ok(()) + } +} diff --git a/fatxlib/src/iso/god/mod.rs b/fatxlib/src/iso/god/mod.rs new file mode 100644 index 0000000..debc4e3 --- /dev/null +++ b/fatxlib/src/iso/god/mod.rs @@ -0,0 +1,138 @@ +//! ISO → Games-on-Demand conversion pipeline. +//! +//! Vendored from [QAston/iso2god-rs `xdvdfx` branch](https://github.com/QAston/iso2god-rs/tree/xdvdfx) +//! (parent: [iliazeus/iso2god-rs](https://github.com/iliazeus/iso2god-rs); +//! both MIT-licensed). Local deviations from upstream: +//! +//! - `anyhow::Error` → [`crate::error::FatxError`] so errors flow through +//! the same channel as the rest of fatxlib. +//! - Upstream's `src/executable/` lives at [`crate::executable`] now and is +//! shared with the XDVDFS image reader. +//! - The original `src/game_list/` (4.9 KLOC of compiled-in title catalog) is +//! dropped; fatxlib already has a richer catalog via [`crate::titles`]. +//! - The upstream binary (`src/bin/iso2god.rs`) lives elsewhere — fatxlib only +//! provides the library surface; the CLI/TUI wraps it in `xtafkit`. +//! +//! See `NOTICE` at the repo root for the full attribution. + +use std::io::{Read, Seek, SeekFrom, Write}; + +use crate::error::{FatxError, Result}; + +pub const SOURCE_BUFFER_SIZE: usize = 1 << 20; + +mod convert; +pub use convert::{ConvertOptions, ConvertReport, TrimMode, convert_iso, convert_iso_to_fatx}; + +mod con_header; +mod core; +pub use con_header::*; + +mod file_layout; +pub use file_layout::*; + +mod gdf_sector; +pub use gdf_sector::*; + +mod hash_list; +pub use hash_list::*; + +mod prepare; +mod sink_fatx; +mod sink_host; + +/// Single hot-path SHA-1 entry point used by [`HashList`] and +/// [`ConHeaderBuilder`]. With the `openssl-hash` feature (default on) +/// this routes to `openssl::sha::sha1`, which uses ARMv8 SHA on Apple +/// Silicon and SHA-NI on x86. Without the feature it falls back to the +/// portable-Rust `sha1` crate. +#[inline] +pub(crate) fn sha1_digest(data: &[u8]) -> [u8; 20] { + #[cfg(feature = "openssl-hash")] + { + openssl::sha::sha1(data) + } + #[cfg(not(feature = "openssl-hash"))] + { + use sha1::{Digest, Sha1}; + Sha1::digest(data).into() + } +} + +pub const BLOCKS_PER_PART: u64 = 0xa1c4; +pub const BLOCKS_PER_SUBPART: u64 = 0xcc; +pub const BLOCK_SIZE: u64 = 0x1000; +pub const SUBPARTS_PER_PART: u32 = 0xcb; +pub const SUBPART_SIZE: u64 = BLOCK_SIZE * BLOCKS_PER_SUBPART; + +pub fn write_part( + mut data_volume: R, + part_index: u64, + remaining_bytes: u64, + mut part_file: W, +) -> Result<()> { + data_volume + .seek_relative((part_index * BLOCKS_PER_PART * BLOCK_SIZE) as i64) + .map_err(FatxError::Io)?; + + let mut master_hash_list = HashList::new(); + + let master_hash_list_position = part_file.stream_position().map_err(FatxError::Io)?; + master_hash_list.write(&mut part_file)?; + + // Pre-allocated subpart buffer — avoids `take + read_to_end`'s repeated + // grow/check ceremony and the Vec-append work that came with it. We read + // straight into a fixed-size buffer and slice off the actual length. + let mut subpart_buf = vec![0u8; SUBPART_SIZE as usize]; + let mut bytes_left = remaining_bytes; + + for _subpart_index in 0..SUBPARTS_PER_PART { + if bytes_left == 0 { + break; + } + // Fill subpart_buf one read at a time. The last subpart may be + // short — that's fine, we slice with `got` below. + let want = (subpart_buf.len() as u64).min(bytes_left) as usize; + let mut got = 0usize; + while got < want { + let n = data_volume + .read(&mut subpart_buf[got..want]) + .map_err(FatxError::Io)?; + if n == 0 { + break; + } + got += n; + } + if got == 0 { + break; + } + let subpart = &subpart_buf[..got]; + + let mut sub_hash_list = HashList::new(); + + for block in subpart.chunks(BLOCK_SIZE as usize) { + sub_hash_list.add_block_hash(block); + } + + sub_hash_list.write(&mut part_file)?; + master_hash_list.add_block_hash(sub_hash_list.bytes()); + + // Write the subpart we already buffered. An earlier shape + // seeked back and re-read via `io::copy` (a `reflink` hint for + // CoW filesystems), but APFS doesn't honor reflink on partial- + // file writes — the re-read just doubled I/O without benefit. + part_file.write_all(subpart).map_err(FatxError::Io)?; + bytes_left -= got as u64; + + if got < want { + break; + } + } + + part_file + .seek(SeekFrom::Start(master_hash_list_position)) + .map_err(FatxError::Io)?; + master_hash_list.write(&mut part_file)?; + + Ok(()) +} diff --git a/fatxlib/src/iso/god/prepare.rs b/fatxlib/src/iso/god/prepare.rs new file mode 100644 index 0000000..bb185fe --- /dev/null +++ b/fatxlib/src/iso/god/prepare.rs @@ -0,0 +1,156 @@ +use std::fs::File; +use std::io::{BufReader, Read, Seek, SeekFrom}; +use std::path::{Path, PathBuf}; + +use crate::error::{FatxError, Result}; +use crate::executable::TitleInfo; +use crate::iso::compact::build_compact_source; + +use super::{ + BLOCK_SIZE, BLOCKS_PER_PART, ContentType, ConvertOptions, ConvertReport, SOURCE_BUFFER_SIZE, + TrimMode, +}; + +pub(crate) trait ReadSeek: Read + Seek {} + +impl ReadSeek for T {} + +pub(crate) struct PreparedSource { + pub(crate) report: ConvertReport, + pub(crate) exe_info: crate::executable::TitleExecutionInfo, + pub(crate) content_type: ContentType, + reader: ReaderSource, +} + +enum ReaderSource { + Raw { + source_iso: PathBuf, + root_offset: u64, + }, + Compact { + source_iso: PathBuf, + compact: crate::iso::compact::CompactSource, + }, +} + +impl PreparedSource { + pub(crate) fn open_reader(&self) -> Result> { + self.reader.open_reader() + } +} + +impl ReaderSource { + fn open_reader(&self) -> Result> { + match self { + Self::Raw { + source_iso, + root_offset, + } => { + let mut iso = File::open(source_iso).map_err(FatxError::Io)?; + iso.seek(SeekFrom::Start(*root_offset)) + .map_err(FatxError::Io)?; + Ok(Box::new(iso)) + } + Self::Compact { + source_iso, + compact, + } => Ok(Box::new(compact.open_reader(source_iso)?)), + } + } +} + +pub(crate) fn prepare_source( + source_iso: &Path, + opts: &ConvertOptions<'_>, +) -> Result { + if matches!(opts.trim, TrimMode::Compact) { + let compact = build_compact_source(source_iso, opts.should_abort)?; + let report = build_report( + compact.exe_info().title_id, + compact.exe_info().media_id, + compact.content_type(), + compact.data_size(), + ); + return Ok(PreparedSource { + exe_info: compact.exe_info().clone(), + content_type: compact.content_type(), + report, + reader: ReaderSource::Compact { + source_iso: source_iso.to_path_buf(), + compact, + }, + }); + } + + let source_iso_file_meta = std::fs::metadata(source_iso).map_err(FatxError::Io)?; + let img = File::open(source_iso).map_err(FatxError::Io)?; + let xiso = BufReader::with_capacity(SOURCE_BUFFER_SIZE, img); + let mut xiso = xdvdfs::blockdev::OffsetWrapper::new(xiso) + .map_err(|e| FatxError::Other(format!("xdvdfs offset detect: {e:?}")))?; + let volume = xdvdfs::read::read_volume(&mut xiso) + .map_err(|e| FatxError::Other(format!("xdvdfs read_volume: {e:?}")))?; + let title_info = TitleInfo::from_image(&mut xiso, volume)?; + let exe_info = title_info.execution_info; + let content_type = title_info.content_type; + let root_offset = { + xiso.seek(SeekFrom::Start(0)).map_err(FatxError::Io)?; + xiso.get_mut().stream_position().map_err(FatxError::Io)? + }; + let data_size = match opts.trim { + TrimMode::PreserveLayout => volume + .root_table + .file_tree(&mut xiso) + .map_err(|e| FatxError::Other(format!("xdvdfs file_tree: {e:?}")))? + .iter() + .map(|dirent| { + if dirent.1.node.dirent.data.is_empty() { + return 0; + } + let offset = dirent + .1 + .node + .dirent + .data + .offset::(0) + .unwrap_or(0); + offset + dirent.1.node.dirent.data.size() as u64 + }) + .max() + .unwrap_or(0), + TrimMode::None => source_iso_file_meta.len() - root_offset, + TrimMode::Compact => unreachable!("compact handled before metadata pass"), + }; + let report = build_report( + exe_info.title_id, + exe_info.media_id, + content_type, + data_size, + ); + Ok(PreparedSource { + exe_info, + content_type, + report, + reader: ReaderSource::Raw { + source_iso: source_iso.to_path_buf(), + root_offset, + }, + }) +} + +fn build_report( + title_id: u32, + media_id: u32, + content_type: ContentType, + data_size: u64, +) -> ConvertReport { + let block_count = data_size.div_ceil(BLOCK_SIZE); + let part_count = block_count.div_ceil(BLOCKS_PER_PART); + ConvertReport { + title_id, + media_id, + content_type, + part_count, + block_count, + data_size, + } +} diff --git a/fatxlib/src/iso/god/sink_fatx.rs b/fatxlib/src/iso/god/sink_fatx.rs new file mode 100644 index 0000000..1314a66 --- /dev/null +++ b/fatxlib/src/iso/god/sink_fatx.rs @@ -0,0 +1,245 @@ +use std::io::{Cursor, Read, Seek, Write}; + +use crate::error::{FatxError, Result}; +use crate::volume::FatxVolume; + +use super::core::{GodSink, part_payload_bytes}; +use super::prepare::PreparedSource; +use super::{BLOCK_SIZE, ConvertOptions, HashList, SUBPART_SIZE, SUBPARTS_PER_PART}; + +pub(crate) struct FatxSink<'a, T: Read + Seek + Write> { + vol: &'a mut FatxVolume, + dest_dir: &'a str, + data_dir: Option, + con_header_path: Option, + part_buf: Vec, + master_lists: Vec, + last_part_size: u64, +} + +impl<'a, T: Read + Seek + Write> FatxSink<'a, T> { + pub(crate) fn new(vol: &'a mut FatxVolume, dest_dir: &'a str) -> Self { + Self { + vol, + dest_dir, + data_dir: None, + con_header_path: None, + part_buf: vec![0u8; MAX_PART_BYTES], + master_lists: Vec::new(), + last_part_size: 0, + } + } + + fn data_dir(&self) -> Result<&str> { + self.data_dir + .as_deref() + .ok_or_else(|| FatxError::Other("fatx sink not initialized".to_string())) + } + + fn con_header_path(&self) -> Result<&str> { + self.con_header_path + .as_deref() + .ok_or_else(|| FatxError::Other("fatx sink not initialized".to_string())) + } + + fn part_path(&self, part_index: u64) -> Result { + Ok(format!("{}/Data{:04}", self.data_dir()?, part_index)) + } +} + +impl GodSink for FatxSink<'_, T> { + fn begin(&mut self, source: &PreparedSource) -> Result<()> { + let title_id_str = format!("{:08X}", source.exe_info.title_id); + let content_type_str = format!("{:08X}", source.content_type as u32); + let media_id_str = match source.content_type { + super::ContentType::GamesOnDemand => format!("{:08X}", source.exe_info.media_id), + super::ContentType::XboxOriginal => format!("{:08X}", source.exe_info.title_id), + }; + let dest_root = self.dest_dir.trim_end_matches('/'); + let title_dir = format!("{}/{}", dest_root, title_id_str); + let content_dir = format!("{}/{}", title_dir, content_type_str); + let con_header_path = format!("{}/{}", content_dir, media_id_str); + let data_dir = format!("{}/{}.data", content_dir, media_id_str); + + ensure_fatx_dir(self.vol, &title_dir)?; + ensure_fatx_dir(self.vol, &content_dir)?; + ensure_fatx_dir(self.vol, &data_dir)?; + self.data_dir = Some(data_dir); + self.con_header_path = Some(con_header_path); + self.master_lists.clear(); + self.master_lists.reserve(source.report.part_count as usize); + self.last_part_size = 0; + Ok(()) + } + + fn write_part<'a>( + &mut self, + source: &PreparedSource, + part_index: u64, + opts: &mut ConvertOptions<'a>, + ) -> Result<()> { + let remaining_bytes = part_payload_bytes(source.report.data_size, part_index); + let mut iso = source.open_reader()?; + let (len, master) = + fill_part_buf(&mut iso, part_index, remaining_bytes, &mut self.part_buf)?; + let part_path = self.part_path(part_index)?; + let reader = Cursor::new(&self.part_buf[..len]); + + let mut outer = opts.progress.take(); + let part_idx_now = part_index; + let part_count_now = source.report.part_count; + { + let mut inner = |bytes: u64, total: u64| { + if let Some(cb) = outer.as_deref_mut() { + let stage = format!("part {}/{}", part_idx_now + 1, part_count_now); + cb(&stage, bytes, total); + } + }; + self.vol + .create_file_from_reader(&part_path, len as u64, reader, Some(&mut inner))?; + } + opts.progress = outer; + + self.master_lists.push(master); + self.last_part_size = len as u64; + Ok(()) + } + + fn read_master_hash(&mut self, _source: &PreparedSource, part_index: u64) -> Result { + self.master_lists + .get(part_index as usize) + .cloned() + .ok_or_else(|| FatxError::Other(format!("missing FATX part {}", part_index))) + } + + fn write_master_hash( + &mut self, + _source: &PreparedSource, + part_index: u64, + mht: &HashList, + ) -> Result<()> { + let slot = self + .master_lists + .get_mut(part_index as usize) + .ok_or_else(|| FatxError::Other(format!("missing FATX part {}", part_index)))?; + *slot = mht.clone(); + let part_path = self.part_path(part_index)?; + overwrite_part_master(self.vol, &part_path, mht.bytes()) + } + + fn last_part_size(&self, _source: &PreparedSource) -> Result { + Ok(self.last_part_size) + } + + fn write_con_header(&mut self, _source: &PreparedSource, con_bytes: Vec) -> Result<()> { + let con_len = con_bytes.len() as u64; + let path = self.con_header_path()?.to_string(); + self.vol + .create_file_from_reader(&path, con_len, Cursor::new(con_bytes), None) + } + + fn flush_after_parts(&mut self) -> Result<()> { + let _ = self.vol.flush(); + Ok(()) + } + + fn flush_after_mht(&mut self) -> Result<()> { + let _ = self.vol.flush(); + Ok(()) + } + + fn flush_after_header(&mut self) -> Result<()> { + let _ = self.vol.flush(); + Ok(()) + } +} + +const MAX_PART_BYTES: usize = 4096 + (SUBPARTS_PER_PART as usize) * (4096 + SUBPART_SIZE as usize); + +fn fill_part_buf( + data_volume: &mut R, + part_index: u64, + remaining_bytes: u64, + out: &mut [u8], +) -> Result<(usize, HashList)> { + data_volume + .seek_relative((part_index * super::BLOCKS_PER_PART * BLOCK_SIZE) as i64) + .map_err(FatxError::Io)?; + + let mut master = HashList::new(); + let mut cursor = 4096usize; + let mut subpart_buf = vec![0u8; SUBPART_SIZE as usize]; + let mut bytes_left = remaining_bytes; + + for _ in 0..SUBPARTS_PER_PART { + if bytes_left == 0 { + break; + } + let want = (subpart_buf.len() as u64).min(bytes_left) as usize; + let mut got = 0usize; + while got < want { + let n = data_volume + .read(&mut subpart_buf[got..want]) + .map_err(FatxError::Io)?; + if n == 0 { + break; + } + got += n; + } + if got == 0 { + break; + } + let subpart = &subpart_buf[..got]; + let mut sub_hash = HashList::new(); + for block in subpart.chunks(BLOCK_SIZE as usize) { + sub_hash.add_block_hash(block); + } + out[cursor..cursor + 4096].copy_from_slice(sub_hash.bytes()); + cursor += 4096; + out[cursor..cursor + got].copy_from_slice(subpart); + cursor += got; + bytes_left -= got as u64; + master.add_block_hash(sub_hash.bytes()); + if got < want { + break; + } + } + + out[0..4096].copy_from_slice(master.bytes()); + Ok((cursor, master)) +} + +fn overwrite_part_master( + vol: &mut FatxVolume, + path: &str, + new_master: &[u8; 4096], +) -> Result<()> +where + T: Read + Seek + Write, +{ + let entry = vol.resolve_path(path)?; + let first_cluster = entry.first_cluster; + let cluster_size = vol.superblock.cluster_size() as usize; + let mut cluster_buf = vec![0u8; cluster_size]; + vol.read_cluster(first_cluster, &mut cluster_buf)?; + cluster_buf[..new_master.len()].copy_from_slice(new_master); + vol.write_cluster(first_cluster, &cluster_buf)?; + Ok(()) +} + +fn ensure_fatx_dir(vol: &mut FatxVolume, path: &str) -> Result<()> +where + T: Read + Seek + Write, +{ + match vol.create_directory(path) { + Ok(()) => Ok(()), + Err(FatxError::FileExists(_)) => { + let existing = vol.resolve_path(path)?; + if !existing.is_directory() { + return Err(FatxError::NotADirectory(path.to_string())); + } + Ok(()) + } + Err(e) => Err(e), + } +} diff --git a/fatxlib/src/iso/god/sink_host.rs b/fatxlib/src/iso/god/sink_host.rs new file mode 100644 index 0000000..faf0e47 --- /dev/null +++ b/fatxlib/src/iso/god/sink_host.rs @@ -0,0 +1,113 @@ +use std::fs::{self, File}; +use std::io::{BufWriter, Write}; +use std::path::Path; + +use crate::error::{FatxError, Result}; + +use super::core::{GodSink, part_payload_bytes}; +use super::prepare::PreparedSource; +use super::{ConvertOptions, FileLayout, HashList, SOURCE_BUFFER_SIZE}; + +pub(crate) struct HostFsSink<'a> { + dest_dir: &'a Path, +} + +impl<'a> HostFsSink<'a> { + pub(crate) fn new(dest_dir: &'a Path) -> Self { + Self { dest_dir } + } + + fn data_dir_path(&self, source: &PreparedSource) -> std::path::PathBuf { + FileLayout::new(self.dest_dir, &source.exe_info, source.content_type).data_dir_path() + } + + fn part_file_path(&self, source: &PreparedSource, part_index: u64) -> std::path::PathBuf { + FileLayout::new(self.dest_dir, &source.exe_info, source.content_type) + .part_file_path(part_index) + } + + fn con_header_file_path(&self, source: &PreparedSource) -> std::path::PathBuf { + FileLayout::new(self.dest_dir, &source.exe_info, source.content_type).con_header_file_path() + } +} + +impl GodSink for HostFsSink<'_> { + fn begin(&mut self, source: &PreparedSource) -> Result<()> { + ensure_empty_dir(&self.data_dir_path(source)) + } + + fn write_part<'a>( + &mut self, + source: &PreparedSource, + part_index: u64, + _opts: &mut ConvertOptions<'a>, + ) -> Result<()> { + let part_path = self.part_file_path(source, part_index); + let part_file = File::options() + .write(true) + .create(true) + .truncate(true) + .open(&part_path) + .map_err(FatxError::Io)?; + let part_file = BufWriter::with_capacity(SOURCE_BUFFER_SIZE, part_file); + let remaining_bytes = part_payload_bytes(source.report.data_size, part_index); + let iso_data_volume = source.open_reader()?; + super::write_part(iso_data_volume, part_index, remaining_bytes, part_file) + } + + fn read_master_hash(&mut self, source: &PreparedSource, part_index: u64) -> Result { + let part_path = self.part_file_path(source, part_index); + read_part_mht(&part_path) + } + + fn write_master_hash( + &mut self, + source: &PreparedSource, + part_index: u64, + mht: &HashList, + ) -> Result<()> { + let part_path = self.part_file_path(source, part_index); + write_part_mht(&part_path, mht) + } + + fn last_part_size(&self, source: &PreparedSource) -> Result { + fs::metadata(self.part_file_path(source, source.report.part_count - 1)) + .map_err(FatxError::Io) + .map(|meta| meta.len()) + } + + fn write_con_header(&mut self, source: &PreparedSource, con_bytes: Vec) -> Result<()> { + let mut con_header_file = File::options() + .write(true) + .create(true) + .truncate(true) + .open(self.con_header_file_path(source)) + .map_err(FatxError::Io)?; + con_header_file.write_all(&con_bytes).map_err(FatxError::Io) + } +} + +fn ensure_empty_dir(path: &Path) -> Result<()> { + if fs::exists(path).map_err(FatxError::Io)? { + fs::remove_dir_all(path).map_err(FatxError::Io)?; + } + fs::create_dir_all(path).map_err(FatxError::Io)?; + Ok(()) +} + +fn read_part_mht(path: &Path) -> Result { + let mut part_file = File::options() + .read(true) + .open(path) + .map_err(FatxError::Io)?; + HashList::read(&mut part_file) +} + +fn write_part_mht(path: &Path, mht: &HashList) -> Result<()> { + let mut part_file = File::options() + .write(true) + .open(path) + .map_err(FatxError::Io)?; + mht.write(&mut part_file)?; + Ok(()) +} diff --git a/fatxlib/src/xiso/mod.rs b/fatxlib/src/iso/image.rs similarity index 87% rename from fatxlib/src/xiso/mod.rs rename to fatxlib/src/iso/image.rs index 1742220..ccea4b7 100644 --- a/fatxlib/src/xiso/mod.rs +++ b/fatxlib/src/iso/image.rs @@ -141,6 +141,28 @@ impl XisoImage { LAYOUTS.iter().find(|l| l.offset == self.partition_offset) } + /// Parse the embedded `Default.xex` (Xbox 360) or `default.xbe` + /// (Original Xbox) and return the title's execution info — TitleID, + /// MediaID, version, content type, etc. Returns `None` if the image + /// has neither executable. + /// + /// Useful for resolving a human-readable game title via + /// [`crate::titles::lookup`] before extracting, so on-drive folder + /// names track the game rather than the local filename. + pub fn title_info(&mut self) -> Result> { + let mut shifted = ShiftedSource { + inner: &mut self.source, + offset: self.partition_offset, + }; + match crate::executable::TitleInfo::from_image(&mut shifted, self.volume) { + Ok(info) => Ok(Some(info)), + Err(crate::error::FatxError::Other(msg)) if msg.contains("no executable found") => { + Ok(None) + } + Err(e) => Err(e), + } + } + /// Walk the entire directory tree, returning every file (not directories) /// as a flat list with image-relative paths and data-partition-relative /// byte offsets. @@ -254,6 +276,20 @@ impl BlockDeviceRead for ShiftedSo } } +// `xdvdfs::executable::TitleInfo::from_image` requires `R: BlockDeviceRead + Seek`. +// We pass the inner Seek through, shifting `Start` positions into the data +// partition; `Current` / `End` are forwarded unchanged. +impl Seek for ShiftedSource<'_, R> { + fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result { + let adjusted = match pos { + std::io::SeekFrom::Start(s) => std::io::SeekFrom::Start(s + self.offset), + other => other, + }; + let abs = self.inner.seek(adjusted)?; + Ok(abs.saturating_sub(self.offset)) + } +} + /// Recurse through the directory tree. `prefix` is the parent directory path /// (empty at the root); each entry is prefixed with it to form the full path. fn walk_recursive( diff --git a/fatxlib/src/iso/manifest.rs b/fatxlib/src/iso/manifest.rs new file mode 100644 index 0000000..b1d9062 --- /dev/null +++ b/fatxlib/src/iso/manifest.rs @@ -0,0 +1,176 @@ +//! Shared ISO manifest planning. + +use std::collections::{HashMap, HashSet}; +use std::io::{Read, Seek}; + +use crate::error::Result; +use crate::executable::TitleInfo; + +use super::image::{XisoFile, XisoImage}; +use super::policy::is_systemupdate_path; + +#[derive(Debug, Clone, Copy, Default)] +pub struct IsoFilterPolicy { + pub keep_systemupdate: bool, +} + +impl IsoFilterPolicy { + pub fn keeps(&self, path: &str) -> bool { + self.keep_systemupdate || !is_systemupdate_path(path) + } +} + +#[derive(Debug, Clone)] +pub struct ManifestEntry { + pub file: XisoFile, + pub skipped: bool, +} + +impl ManifestEntry { + pub fn path(&self) -> &str { + &self.file.path + } +} + +#[derive(Debug, Clone)] +pub struct IsoManifest { + pub layout: String, + pub partition_offset: u64, + pub title_info: Option, + pub entries: Vec, + pub kept_bytes: u64, + pub skipped_bytes: u64, +} + +impl IsoManifest { + pub fn kept_files(&self) -> usize { + self.entries.iter().filter(|entry| !entry.skipped).count() + } + + pub fn skipped_files(&self) -> usize { + self.entries.iter().filter(|entry| entry.skipped).count() + } + + pub fn kept(&self) -> impl Iterator { + self.entries + .iter() + .filter(|entry| !entry.skipped) + .map(|entry| &entry.file) + } + + pub fn skipped(&self) -> impl Iterator { + self.entries + .iter() + .filter(|entry| entry.skipped) + .map(|entry| &entry.file) + } + + pub fn kept_path_set(&self) -> HashSet { + self.kept() + .map(|entry| normalize_path(&entry.path).to_string()) + .collect() + } + + pub fn kept_dir_set(&self) -> HashSet { + let mut dirs = HashSet::from([String::new()]); + for entry in self.kept() { + let mut prefix = String::new(); + let path = normalize_path(&entry.path); + for component in path.split('/').take(path.matches('/').count()) { + if !prefix.is_empty() { + prefix.push('/'); + } + prefix.push_str(component); + dirs.insert(prefix.clone()); + } + } + dirs + } + + pub fn kept_offset_map(&self) -> HashMap { + self.kept() + .map(|entry| (normalize_path(&entry.path).to_string(), entry.offset)) + .collect() + } +} + +pub fn build_manifest( + img: &mut XisoImage, + policy: IsoFilterPolicy, +) -> Result { + let files = img.walk_files()?; + let entries: Vec = files + .into_iter() + .map(|file| ManifestEntry { + skipped: !policy.keeps(&file.path), + file, + }) + .collect(); + let kept_bytes = entries + .iter() + .filter(|entry| !entry.skipped) + .map(|entry| entry.file.size) + .sum(); + let skipped_bytes = entries + .iter() + .filter(|entry| entry.skipped) + .map(|entry| entry.file.size) + .sum(); + let layout = img + .layout() + .map(|layout| format!("{} (0x{:08X})", layout.name, layout.offset)) + .unwrap_or_else(|| format!("unknown @ 0x{:08X}", img.partition_offset())); + + Ok(IsoManifest { + layout, + partition_offset: img.partition_offset(), + title_info: img.title_info()?, + entries, + kept_bytes, + skipped_bytes, + }) +} + +fn normalize_path(path: &str) -> &str { + path.trim_start_matches('/') +} + +#[cfg(test)] +mod tests { + use super::{IsoManifest, ManifestEntry}; + use crate::iso::image::XisoFile; + + #[test] + fn kept_dir_set_contains_all_parent_directories() { + let manifest = IsoManifest { + layout: String::new(), + partition_offset: 0, + title_info: None, + kept_bytes: 12, + skipped_bytes: 0, + entries: vec![ + ManifestEntry { + file: XisoFile { + path: "default.xex".to_string(), + size: 4, + offset: 0, + }, + skipped: false, + }, + ManifestEntry { + file: XisoFile { + path: "Media/Videos/intro.bik".to_string(), + size: 8, + offset: 4, + }, + skipped: false, + }, + ], + }; + + let dirs = manifest.kept_dir_set(); + assert!(dirs.contains("")); + assert!(dirs.contains("Media")); + assert!(dirs.contains("Media/Videos")); + } +} diff --git a/fatxlib/src/iso/mod.rs b/fatxlib/src/iso/mod.rs new file mode 100644 index 0000000..3b6ed7b --- /dev/null +++ b/fatxlib/src/iso/mod.rs @@ -0,0 +1,10 @@ +//! ISO-domain functionality. +//! +//! This namespace groups operations on XDVDFS/XISO images independently of +//! FATX/XTAF transport concerns. + +pub mod compact; +pub mod god; +pub mod image; +pub mod manifest; +pub mod policy; diff --git a/fatxlib/src/iso/policy.rs b/fatxlib/src/iso/policy.rs new file mode 100644 index 0000000..ff75a4b --- /dev/null +++ b/fatxlib/src/iso/policy.rs @@ -0,0 +1,9 @@ +//! Shared policy decisions for ISO-derived operations. + +pub fn is_systemupdate_path(path: &str) -> bool { + path.trim_start_matches('/') + .split('/') + .next() + .unwrap_or("") + .eq_ignore_ascii_case("$SystemUpdate") +} diff --git a/fatxlib/src/lib.rs b/fatxlib/src/lib.rs index 2e92796..4cff031 100644 --- a/fatxlib/src/lib.rs +++ b/fatxlib/src/lib.rs @@ -28,13 +28,14 @@ pub mod content_types; pub mod display; pub mod error; +pub mod executable; +pub mod iso; pub mod partition; pub mod platform; pub mod stfs; pub mod titles; pub mod types; pub mod volume; -pub mod xiso; pub mod xuids; pub use error::{FatxError, Result}; diff --git a/fatxlib/tests/fixtures/tiny.xiso b/fatxlib/tests/fixtures/tiny.xiso index fb17b0a..3ea4ed5 100644 Binary files a/fatxlib/tests/fixtures/tiny.xiso and b/fatxlib/tests/fixtures/tiny.xiso differ diff --git a/fatxlib/tests/iso2god_roundtrip.rs b/fatxlib/tests/iso2god_roundtrip.rs new file mode 100644 index 0000000..3f742ec --- /dev/null +++ b/fatxlib/tests/iso2god_roundtrip.rs @@ -0,0 +1,384 @@ +//! Integration smoke test for [`fatxlib::iso::god::convert_iso`] and +//! [`fatxlib::iso::god::convert_iso_to_fatx`]. +//! +//! Runs end-to-end against the bundled `tiny.xiso` fixture — a synthetic +//! XISO packed via `xdvdfs pack` that contains a real `default.xex` +//! (XellLaunch2_retail, a public homebrew launcher from the Free60 +//! project). The XEX has valid `XEX2` magic + execution-info fields, so +//! `TitleInfo::from_image` parses it cleanly and the full pipeline runs. +//! +//! Focuses on "the pipeline runs to completion and the output is shaped +//! correctly", not byte-equality — the GoD format is deterministic, and +//! byte-equality is best validated against an external reference +//! conversion when one is available. + +mod common; + +use std::fs; +use std::fs::OpenOptions; +use std::io::Write; +use std::path::PathBuf; + +use fatxlib::iso::god::{ConvertOptions, TrimMode, convert_iso, convert_iso_to_fatx}; + +fn fixture_path() -> Option { + let p = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/tiny.xiso"); + if p.exists() { Some(p) } else { None } +} + +fn padded_fixture_path() -> Option<(tempfile::TempDir, PathBuf)> { + let source = fixture_path()?; + let tmp = tempfile::TempDir::new().expect("tempdir"); + let padded = tmp.path().join("tiny-padded.xiso"); + fs::copy(&source, &padded).expect("copy padded fixture"); + let mut file = OpenOptions::new() + .append(true) + .open(&padded) + .expect("open padded fixture"); + file.write_all(&vec![0xA5; 16 * 1024 * 1024]) + .expect("append padding"); + Some((tmp, padded)) +} + +fn expected_part_len(payload_bytes: u64) -> u64 { + let subpart_size = fatxlib::iso::god::SUBPART_SIZE; + let subparts = payload_bytes.div_ceil(subpart_size); + 4096 + (subparts * 4096) + payload_bytes +} + +#[test] +fn converts_fixture_into_valid_god_package() { + let Some(iso) = fixture_path() else { + eprintln!("skipping: fatxlib/tests/fixtures/tiny.xiso missing"); + return; + }; + + let tmp = tempfile::TempDir::new().expect("tempdir"); + let dest = tmp.path(); + + let mut opts = ConvertOptions { + trim: TrimMode::PreserveLayout, + game_title: Some("XellLaunch2 fixture"), + dry_run: false, + progress: None, + should_abort: None, + }; + + let report = convert_iso(&iso, dest, &mut opts).expect("convert_iso"); + + assert!(report.title_id != 0, "title id should be non-zero"); + assert!( + report.part_count >= 1, + "fixture must produce at least one Data part; got {:?}", + report + ); + assert!(report.block_count >= 1); + + // CON header lives at /// + let title_hex = format!("{:08X}", report.title_id); + let ctype_hex = format!("{:08X}", report.content_type as u32); + let media_hex = if matches!( + report.content_type, + fatxlib::iso::god::ContentType::XboxOriginal + ) { + title_hex.clone() + } else { + format!("{:08X}", report.media_id) + }; + + let con_header_path = dest.join(&title_hex).join(&ctype_hex).join(&media_hex); + let data_dir = dest + .join(&title_hex) + .join(&ctype_hex) + .join(format!("{}.data", media_hex)); + let first_part = data_dir.join("Data0000"); + + assert!( + con_header_path.exists(), + "CON header missing at {}", + con_header_path.display() + ); + assert!( + first_part.exists(), + "Data0000 missing at {}", + first_part.display() + ); + + let con_header_size = fs::metadata(&con_header_path).expect("stat header").len(); + assert_eq!( + con_header_size, 0xB000, + "CON header should be 45 056 bytes (empty_live template)" + ); + + let first_part_size = fs::metadata(&first_part).expect("stat data").len(); + assert!( + first_part_size > 0, + "Data0000 should be non-empty; got {} bytes", + first_part_size + ); + + // CON header should start with "LIVE" (`empty_live.bin` magic). + let head = fs::read(&con_header_path).expect("read header"); + assert_eq!( + &head[..4], + b"LIVE", + "CON header missing LIVE magic; got {:?}", + &head[..4] + ); +} + +#[test] +fn fixture_dry_run_does_not_create_files() { + let Some(iso) = fixture_path() else { + return; + }; + + let tmp = tempfile::TempDir::new().expect("tempdir"); + let dest = tmp.path(); + + let mut opts = ConvertOptions { + trim: TrimMode::PreserveLayout, + game_title: None, + dry_run: true, + progress: None, + should_abort: None, + }; + + let report = convert_iso(&iso, dest, &mut opts).expect("dry-run convert"); + assert!(report.part_count >= 1); + + let entries: Vec<_> = fs::read_dir(dest) + .expect("readdir") + .filter_map(|e| e.ok()) + .collect(); + assert!( + entries.is_empty(), + "dry_run should not write anything; found {:?}", + entries.iter().map(|e| e.path()).collect::>() + ); +} + +#[test] +fn fixture_extracts_expected_title_id() { + // XellLaunch2_retail's TitleID is 0xFFFF011D (homebrew/dev range). + // If this assertion fires, either the fixture changed or the XEX + // parser drifted. + let Some(iso) = fixture_path() else { + return; + }; + + let tmp = tempfile::TempDir::new().expect("tempdir"); + let mut opts = ConvertOptions { + dry_run: true, + ..Default::default() + }; + + let report = convert_iso(&iso, tmp.path(), &mut opts).expect("dry-run convert"); + assert_eq!( + report.title_id, 0xFFFF011D, + "expected XellLaunch2_retail TitleID; fixture may have changed" + ); +} + +#[test] +fn streams_fixture_into_fatx_volume() { + let Some(iso) = fixture_path() else { + return; + }; + let (_tmp, mut vol) = common::create_fatx_image(8); + + let mut opts = ConvertOptions { + trim: TrimMode::PreserveLayout, + game_title: Some("XellLaunch2 fixture"), + dry_run: false, + progress: None, + should_abort: None, + }; + + let report = convert_iso_to_fatx(&iso, &mut vol, "/", &mut opts).expect("convert_iso_to_fatx"); + + assert!(report.title_id != 0); + assert!(report.part_count >= 1); + + // The Title-ID tree should live at the FATX root. + let title_dir = format!("/{:08X}", report.title_id); + let content_dir = format!("{}/{:08X}", title_dir, report.content_type as u32); + let media_id_hex = if matches!( + report.content_type, + fatxlib::iso::god::ContentType::XboxOriginal + ) { + format!("{:08X}", report.title_id) + } else { + format!("{:08X}", report.media_id) + }; + let con_header_path = format!("{}/{}", content_dir, media_id_hex); + let data_dir = format!("{}/{}.data", content_dir, media_id_hex); + let first_part_path = format!("{}/Data0000", data_dir); + + let header_bytes = vol + .read_file_by_path(&con_header_path) + .expect("read CON header from FATX"); + assert_eq!( + header_bytes.len(), + 0xB000, + "CON header should be 45 056 bytes" + ); + assert_eq!( + &header_bytes[..4], + b"LIVE", + "CON header missing LIVE magic; got {:?}", + &header_bytes[..4] + ); + + let first_part_bytes = vol + .read_file_by_path(&first_part_path) + .expect("read Data0000 from FATX"); + assert!( + !first_part_bytes.is_empty(), + "Data0000 should be non-empty on FATX" + ); +} + +#[test] +fn compact_mode_converts_fixture_into_valid_god_package() { + let Some(iso) = fixture_path() else { + return; + }; + + let tmp = tempfile::TempDir::new().expect("tempdir"); + let dest = tmp.path(); + + let mut opts = ConvertOptions { + trim: TrimMode::Compact, + game_title: Some("XellLaunch2 fixture"), + dry_run: false, + progress: None, + should_abort: None, + }; + + let report = convert_iso(&iso, dest, &mut opts).expect("compact convert_iso"); + assert!(report.title_id != 0); + assert!(report.part_count >= 1); + + let title_hex = format!("{:08X}", report.title_id); + let ctype_hex = format!("{:08X}", report.content_type as u32); + let media_hex = format!("{:08X}", report.media_id); + let con_header_path = dest.join(&title_hex).join(&ctype_hex).join(&media_hex); + assert!(con_header_path.exists(), "compact CON header missing"); +} + +#[test] +fn compact_mode_streams_fixture_into_fatx_volume() { + let Some(iso) = fixture_path() else { + return; + }; + + let (_tmp, mut vol) = common::create_fatx_image(8); + let mut opts = ConvertOptions { + trim: TrimMode::Compact, + game_title: Some("XellLaunch2 fixture"), + dry_run: false, + progress: None, + should_abort: None, + }; + + let report = + convert_iso_to_fatx(&iso, &mut vol, "/", &mut opts).expect("compact convert_iso_to_fatx"); + let data_path = format!( + "/{:08X}/{:08X}/{:08X}.data/Data0000", + report.title_id, report.content_type as u32, report.media_id + ); + assert!( + !vol.read_file_by_path(&data_path) + .expect("read compact Data0000") + .is_empty(), + "compact Data0000 should be non-empty on FATX" + ); +} + +#[test] +fn streaming_dry_run_writes_nothing_to_fatx() { + let Some(iso) = fixture_path() else { + return; + }; + let (_tmp, mut vol) = common::create_fatx_image(4); + + let initial_free = vol.stats().expect("stats").free_clusters; + + let mut opts = ConvertOptions { + dry_run: true, + ..Default::default() + }; + let report = + convert_iso_to_fatx(&iso, &mut vol, "/", &mut opts).expect("dry-run convert_iso_to_fatx"); + assert!(report.part_count >= 1); + + let final_free = vol.stats().expect("stats").free_clusters; + assert_eq!( + final_free, initial_free, + "dry-run must not allocate any clusters" + ); +} + +#[test] +fn trim_ignores_appended_tail_padding_for_file_output() { + let Some((_tmp, iso)) = padded_fixture_path() else { + return; + }; + + let out = tempfile::TempDir::new().expect("tempdir"); + let mut opts = ConvertOptions { + trim: TrimMode::PreserveLayout, + ..Default::default() + }; + + let report = convert_iso(&iso, out.path(), &mut opts).expect("convert padded iso"); + assert_eq!(report.part_count, 1, "fixture should stay single-part"); + + let title_hex = format!("{:08X}", report.title_id); + let ctype_hex = format!("{:08X}", report.content_type as u32); + let media_hex = format!("{:08X}", report.media_id); + let data_path = out + .path() + .join(title_hex) + .join(ctype_hex) + .join(format!("{}.data/Data0000", media_hex)); + + let actual = fs::metadata(&data_path).expect("stat data part").len(); + let expected = expected_part_len(report.data_size); + assert_eq!( + actual, expected, + "trimmed conversion should ignore appended bytes beyond the XDVDFS payload" + ); +} + +#[test] +fn trim_ignores_appended_tail_padding_for_fatx_output() { + let Some((_tmp, iso)) = padded_fixture_path() else { + return; + }; + + let (_img_tmp, mut vol) = common::create_fatx_image(64); + let mut opts = ConvertOptions { + trim: TrimMode::PreserveLayout, + ..Default::default() + }; + + let report = + convert_iso_to_fatx(&iso, &mut vol, "/", &mut opts).expect("convert padded iso to fatx"); + assert_eq!(report.part_count, 1, "fixture should stay single-part"); + + let data_path = format!( + "/{:08X}/{:08X}/{:08X}.data/Data0000", + report.title_id, report.content_type as u32, report.media_id + ); + let actual = vol + .read_file_by_path(&data_path) + .expect("read data part") + .len() as u64; + let expected = expected_part_len(report.data_size); + assert_eq!( + actual, expected, + "streaming FATX conversion should ignore appended bytes beyond the XDVDFS payload" + ); +} diff --git a/fatxlib/tests/xiso_reader.rs b/fatxlib/tests/xiso_reader.rs index 46bee7e..ebf64f5 100644 --- a/fatxlib/tests/xiso_reader.rs +++ b/fatxlib/tests/xiso_reader.rs @@ -5,7 +5,7 @@ mod common; use std::fs::File; use std::io::Cursor; -use fatxlib::xiso::XisoImage; +use fatxlib::iso::image::XisoImage; // --------------------------------------------------------------------------- // Negative paths — always runnable @@ -61,8 +61,8 @@ fn walks_fixture_image() { ); let names: Vec<&str> = files.iter().map(|f| f.path.as_str()).collect(); assert!( - names.iter().any(|n| n.ends_with("default.xbe")), - "expected default.xbe in fixture; got {:?}", + names.iter().any(|n| n.ends_with("default.xex")), + "expected default.xex (XellLaunch2_retail) in fixture; got {:?}", names ); } @@ -160,6 +160,21 @@ fn extract_fixture_into_fatx_volume() { } } +#[test] +fn fixture_title_info_returns_xellaunch() { + let Some(mut img) = open_fixture() else { + return; + }; + let info = img + .title_info() + .expect("title_info should succeed on fixture") + .expect("fixture has a Default.xex, so title_info must be Some"); + assert_eq!( + info.execution_info.title_id, 0xFFFF011D, + "fixture default.xex is XellLaunch2_retail (TitleID 0xFFFF011D)" + ); +} + #[test] fn streams_invokes_progress_callback() { let Some(mut img) = open_fixture() else { diff --git a/src/main.rs b/src/main.rs index 998046b..0de69cc 100644 --- a/src/main.rs +++ b/src/main.rs @@ -172,6 +172,50 @@ enum Commands { #[arg(long)] no_save: bool, }, + /// Extract every file from an Xbox / Xbox 360 XISO disc image to a + /// local directory. Useful for inspecting an ISO's contents or for + /// feeding loose game files to alt dashboards. + Extract { + /// Source XISO file + iso: PathBuf, + /// Destination directory (created if missing) + dest: PathBuf, + /// Skip the `$SystemUpdate` folder (dashboard update payload that + /// alt dashboards never run). On by default; pass + /// `--keep-systemupdate` to write it out anyway. + #[arg(long, action = clap::ArgAction::SetTrue)] + keep_systemupdate: bool, + /// Print what would be extracted without writing anything. + #[arg(long)] + dry_run: bool, + }, + /// Convert an Xbox 360 XISO into a Games-on-Demand package in a local + /// directory. Writes `///{,.data/}`. + God { + /// Source XISO file + iso: PathBuf, + /// Destination directory (the title-id tree lands underneath) + dest: PathBuf, + /// How much of the source partition to pack: + /// `compact` (default) — rebuild a dense XDVDFS image first, then + /// convert that compact image into GoD. + /// `preserve-layout` — walk the file tree, pack only + /// through the highest used extent while preserving mastered + /// holes inside the XDVDFS layout. + /// `none` — pack everything from the start of the data partition + /// to the end of the source file. + #[arg(long, value_parser = ["compact", "preserve-layout", "none"], default_value = "compact")] + trim: String, + /// Print the parsed metadata (TitleID, MediaID, data_size, part_count) + /// without writing anything. + #[arg(long)] + dry_run: bool, + /// Override the human-readable title written into the CON header. + /// Defaults to the catalog name for the parsed TitleID, or blank + /// if the catalog doesn't know it. + #[arg(long)] + game_title: Option, + }, } // =========================================================================== @@ -1087,5 +1131,385 @@ fn main() { } } } + + Some(Commands::Extract { + iso, + dest, + keep_systemupdate, + dry_run, + }) => run_extract(&iso, &dest, keep_systemupdate, dry_run, json), + + Some(Commands::God { + iso, + dest, + trim, + dry_run, + game_title, + }) => run_god(&iso, &dest, &trim, dry_run, game_title.as_deref(), json), + } +} + +// =========================================================================== +// `xtafkit extract` — XISO → local directory +// =========================================================================== + +fn run_extract( + iso: &std::path::Path, + dest: &std::path::Path, + keep_systemupdate: bool, + dry_run: bool, + json: bool, +) { + use std::io::BufWriter; + use std::time::Instant; + + let file = match File::open(iso) { + Ok(f) => f, + Err(e) => { + cli_error(json, &format!("open {}: {}", iso.display(), e)); + return; + } + }; + let mut img = match fatxlib::iso::image::XisoImage::open(file) { + Ok(i) => i, + Err(e) => { + cli_error(json, &format!("parse {}: {}", iso.display(), e)); + return; + } + }; + let plan = match fatxlib::iso::manifest::build_manifest( + &mut img, + fatxlib::iso::manifest::IsoFilterPolicy { keep_systemupdate }, + ) { + Ok(plan) => plan, + Err(e) => { + cli_error(json, &format!("walk {}: {}", iso.display(), e)); + return; + } + }; + let total_files = plan.kept_files(); + let total_bytes = plan.kept_bytes; + let skipped_files = plan.skipped_files(); + let skipped_bytes = plan.skipped_bytes; + + if dry_run { + if json { + println!( + "{}", + serde_json::json!({ + "iso": iso.display().to_string(), + "layout": plan.layout, + "files": total_files, + "bytes": total_bytes, + "skipped_files": skipped_files, + "skipped_bytes": skipped_bytes, + "entries": plan.entries.iter().map(|e| { + serde_json::json!({ + "path": e.file.path, + "offset": e.file.offset, + "size": e.file.size, + "skipped": e.skipped, + }) + }).collect::>(), + }) + ); + } else { + println!("ISO: {}", iso.display()); + println!("Layout: {}", plan.layout); + println!("Files: {} ({})", total_files, format_size(total_bytes)); + if skipped_files > 0 { + println!( + "Skipped: {} files in $SystemUpdate ({})", + skipped_files, + format_size(skipped_bytes) + ); + } + println!(); + for e in &plan.entries { + let tag = if e.skipped { "skip " } else { "keep " }; + println!( + " {} {:48} @0x{:010X} {}", + tag, + e.file.path, + e.file.offset, + format_size(e.file.size) + ); + } + println!(); + println!("(dry-run; nothing written)"); + } + return; + } + + if let Err(e) = std::fs::create_dir_all(dest) { + cli_error(json, &format!("create_dir_all {}: {}", dest.display(), e)); + return; + } + + let started = Instant::now(); + let mut files_done = 0usize; + let mut bytes_done: u64 = 0; + let last_progress = std::cell::Cell::new(Instant::now()); + + for e in plan.kept() { + let normalized = e.path.replace('\\', "/"); + let local = dest.join(&normalized); + if let Some(parent) = local.parent() + && let Err(err) = std::fs::create_dir_all(parent) + { + cli_error( + json, + &format!("create_dir_all {}: {}", parent.display(), err), + ); + return; + } + let out = match File::create(&local) { + Ok(f) => BufWriter::new(f), + Err(err) => { + cli_error(json, &format!("create {}: {}", local.display(), err)); + return; + } + }; + let mut out = out; + let bytes_done_ref = &mut bytes_done; + let last_progress_ref = &last_progress; + let mut cb = |read: u64, _total: u64| { + // Throttled per-file byte progress for stderr. + if !json && last_progress_ref.get().elapsed().as_millis() > 250 { + eprint!( + "\r [{}/{}] {} ({}/{}) ", + files_done + 1, + total_files, + short_name(&normalized), + format_size(*bytes_done_ref + read), + format_size(total_bytes), + ); + let _ = io::stderr().flush(); + last_progress_ref.set(Instant::now()); + } + }; + let written = match img.read_into(e, &mut out, None, Some(&mut cb)) { + Ok(n) => n, + Err(err) => { + if !json { + eprintln!(); + } + cli_error(json, &format!("read {}: {}", e.path, err)); + return; + } + }; + if let Err(err) = out.flush() { + cli_error(json, &format!("flush {}: {}", local.display(), err)); + return; + } + bytes_done += written; + files_done += 1; + } + let elapsed = started.elapsed(); + if !json { + eprint!("\r{:80}\r", ""); + } + if json { + println!( + "{}", + serde_json::json!({ + "iso": iso.display().to_string(), + "dest": dest.display().to_string(), + "files": files_done, + "bytes": bytes_done, + "skipped_files": skipped_files, + "skipped_bytes": skipped_bytes, + "elapsed_secs": elapsed.as_secs_f64(), + }) + ); + } else { + println!( + "Extracted {} files ({}) → {} in {:?}", + files_done, + format_size(bytes_done), + dest.display(), + elapsed, + ); + if skipped_files > 0 { + println!( + "Skipped {} files in $SystemUpdate ({})", + skipped_files, + format_size(skipped_bytes) + ); + } + } +} + +fn short_name(p: &str) -> &str { + p.rsplit('/').next().unwrap_or(p) +} + +// =========================================================================== +// `xtafkit god` — XISO → local Games-on-Demand package +// =========================================================================== + +fn run_god( + iso: &std::path::Path, + dest: &std::path::Path, + trim: &str, + dry_run: bool, + game_title: Option<&str>, + json: bool, +) { + use std::time::Instant; + + let trim_mode = match trim { + "preserve-layout" => fatxlib::iso::god::TrimMode::PreserveLayout, + "none" => fatxlib::iso::god::TrimMode::None, + "compact" => fatxlib::iso::god::TrimMode::Compact, + other => { + cli_error(json, &format!("invalid --trim {:?}", other)); + return; + } + }; + + // Catalog-fill the game title from the dry-run report, unless the + // caller passed --game-title explicitly. + let mut dry_opts = fatxlib::iso::god::ConvertOptions { + trim: trim_mode, + dry_run: true, + ..Default::default() + }; + let report = match fatxlib::iso::god::convert_iso(iso, dest, &mut dry_opts) { + Ok(r) => r, + Err(e) => { + cli_error(json, &format!("parse {}: {}", iso.display(), e)); + return; + } + }; + let resolved_name = fatxlib::titles::lookup(report.title_id).map(|t| t.name); + let effective_title = game_title.or(resolved_name); + + if dry_run { + if json { + println!( + "{}", + serde_json::json!({ + "iso": iso.display().to_string(), + "title_id": format!("{:08X}", report.title_id), + "media_id": format!("{:08X}", report.media_id), + "name": resolved_name.unwrap_or("(unknown)"), + "content_type": format!("{:?}", report.content_type), + "data_size": report.data_size, + "block_count": report.block_count, + "part_count": report.part_count, + }) + ); + } else { + println!("ISO: {}", iso.display()); + println!("Title ID: {:08X}", report.title_id); + println!("Media ID: {:08X}", report.media_id); + println!( + "Name: {}", + resolved_name.unwrap_or("(unknown — catalog miss)") + ); + println!("Content: {:?}", report.content_type); + println!( + "Data size: {} bytes ({})", + report.data_size, + format_size(report.data_size) + ); + println!("Block count: {}", report.block_count); + println!("Part count: {}", report.part_count); + println!(); + println!("(dry-run; nothing written)"); + } + return; + } + + if let Err(e) = std::fs::create_dir_all(dest) { + cli_error(json, &format!("create_dir_all {}: {}", dest.display(), e)); + return; + } + + let started = Instant::now(); + let last_progress = std::cell::Cell::new(Instant::now()); + let mut last_stage = String::new(); + let mut progress_cb = |stage: &str, current: u64, total: u64| { + let stage_changed = stage != last_stage; + if json { + return; + } + if stage_changed || last_progress.get().elapsed().as_millis() > 250 { + if stage.starts_with("part ") { + eprint!( + "\r [{}] {} / {} ", + stage, + format_size(current), + format_size(total) + ); + } else { + eprint!("\r [{}] {}/{} ", stage, current, total); + } + let _ = io::stderr().flush(); + last_progress.set(Instant::now()); + last_stage = stage.to_string(); + } + }; + + let mut opts = fatxlib::iso::god::ConvertOptions { + trim: trim_mode, + game_title: effective_title, + dry_run: false, + progress: Some(&mut progress_cb), + should_abort: None, + }; + + let result = fatxlib::iso::god::convert_iso(iso, dest, &mut opts); + if !json { + eprint!("\r{:80}\r", ""); + } + let elapsed = started.elapsed(); + match result { + Ok(r) => { + if json { + println!( + "{}", + serde_json::json!({ + "iso": iso.display().to_string(), + "dest": dest.display().to_string(), + "title_id": format!("{:08X}", r.title_id), + "media_id": format!("{:08X}", r.media_id), + "name": resolved_name.unwrap_or("(unknown)"), + "content_type": format!("{:?}", r.content_type), + "data_size": r.data_size, + "block_count": r.block_count, + "part_count": r.part_count, + "elapsed_secs": elapsed.as_secs_f64(), + }) + ); + } else { + let resolved_label = resolved_name.unwrap_or("(unknown — catalog miss)"); + println!( + "ISO: {}\nTitle ID: {:08X}\nMedia ID: {:08X}\nName: {}\nContent: {:?}\nData size: {} bytes ({})\nBlock count: {}\nPart count: {}\nDest: {}\nElapsed: {:?}", + iso.display(), + r.title_id, + r.media_id, + resolved_label, + r.content_type, + r.data_size, + format_size(r.data_size), + r.block_count, + r.part_count, + dest.display(), + elapsed + ); + } + } + Err(e) => cli_error(json, &format!("convert_iso: {}", e)), + } +} + +fn cli_error(json: bool, msg: &str) { + if json { + println!("{}", serde_json::json!({"error": msg})); + process::exit(0); } + eprintln!("Error: {}", msg); + process::exit(1); } diff --git a/src/tui.rs b/src/tui.rs index d235607..458f73d 100644 --- a/src/tui.rs +++ b/src/tui.rs @@ -16,9 +16,13 @@ //! d Download selected file to local disk //! u Upload a local file or directory into current directory. //! If the file parses as an XDVDFS/XISO disc image, the -//! TUI asks whether to extract the contents into a new -//! subfolder (preferred for alt dashboards) or copy the -//! raw bytes as-is. +//! TUI asks how to bring it onto the drive: +//! (x)tract — stream the file tree into // +//! (g)oD — convert to a Games-on-Demand package +//! rooted at //00007000/... +//! (r)aw — copy the source ISO byte-for-byte +//! Default is GoD when cwd is inside `/Content//`, +//! extract otherwise. //! m Create new directory (mkdir) //! D Delete selected file/directory //! r Rename selected file/directory @@ -57,10 +61,10 @@ use ratatui::{ widgets::{Block, Borders, List, ListItem, ListState, Paragraph}, }; +use fatxlib::iso::image::XisoImage; use fatxlib::partition::format_size; use fatxlib::types::FileAttributes; use fatxlib::volume::FatxVolume; -use fatxlib::xiso::XisoImage; // =========================================================================== // Display types @@ -107,6 +111,24 @@ impl DisplayEntry { } } +/// What to do with a local XISO that's being uploaded. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum XisoUploadAction { + /// Walk the XISO and stream each file into `//`. + /// Default when cwd is anywhere other than directly inside an XUID + /// folder (e.g. `/Games/`, `/`, an arbitrary user folder). + Extract, + /// Convert the XISO into a Games-on-Demand package — writes + /// `//00007000/{,.data/}`. Default when cwd + /// is directly inside `/Content//`, since that's exactly + /// where Xbox 360 BC looks for GoD packages. + God, + /// Copy the source ISO byte-for-byte to `/`. Useful + /// when the user wants to preserve the disc image as-is for later + /// extraction or conversion elsewhere. + Raw, +} + /// How to order the directory listing. #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum SortMode { @@ -145,6 +167,15 @@ enum IoCmd { source: PathBuf, dest_dir: String, }, + /// Convert `source` (an XDVDFS image) to a Games-on-Demand package + /// rooted at `dest_dir` on the FATX volume. Writes + /// `//00007000/{,.data/Data0000..N}`. + /// The worker resolves the human-readable game title from + /// [`fatxlib::titles`] before writing the CON header. + ConvertXisoToGod { + source: PathBuf, + dest_dir: String, + }, Mkdir { path: String, }, @@ -223,9 +254,12 @@ enum InputMode { RenameName, ConfirmDelete, ConfirmCleanup, - /// Confirmation prompt after detecting an XISO during upload — y extracts - /// the contents, n falls back to a raw byte copy. - ConfirmExtractXiso, + /// Three-way prompt after detecting an XISO during upload: + /// `x` extracts the contents into a stem-named subfolder, + /// `g` converts to a Games-on-Demand package (Title-ID tree under cwd), + /// `r` falls back to a raw byte copy of the source file. + /// The default action on bare Enter depends on cwd context. + ConfirmXisoUpload, } struct App { @@ -247,9 +281,9 @@ struct App { cancel_flag: Arc, /// Pending cleanup paths awaiting user confirmation. pending_cleanup: Vec<(String, bool, u64)>, - /// Local XISO path stashed between the upload prompt and the - /// "extract or raw copy?" confirmation prompt. - pending_xiso_upload: Option, + /// Local XISO path + default action stashed between the upload prompt + /// and the three-way confirmation prompt (extract / GoD / raw). + pending_xiso_upload: Option<(PathBuf, XisoUploadAction)>, /// Current listing sort order. Toggleable with `s`. sort_mode: SortMode, } @@ -282,18 +316,54 @@ fn is_xiso(path: &std::path::Path) -> bool { } } -/// Image-relative paths we always strip when extracting an XISO. Currently -/// just `$SystemUpdate/*` — dashboard update payload that alt dashboards -/// never launch and that wastes tens to hundreds of MiB on the destination -/// drive. `image_path` is expected to use forward slashes; the match is -/// case-insensitive against the first segment. -fn is_xiso_junk(image_path: &str) -> bool { - let first = image_path - .trim_start_matches('/') - .split('/') - .next() - .unwrap_or(""); - first.eq_ignore_ascii_case("$SystemUpdate") +/// Resolve the destination folder name for an XISO extract by reading the +/// embedded `Default.xex` / `default.xbe` and looking the TitleID up in +/// [`fatxlib::titles`]. Returns the catalog-known game name, sanitized for +/// FATX's filename rules. Returns `None` if the image has no parsable +/// executable, the TitleID isn't in the catalog, or the resulting name is +/// empty after sanitization — callers should fall back to the file stem. +fn xiso_folder_name(path: &std::path::Path) -> Option { + let file = std::fs::File::open(path).ok()?; + let mut img = XisoImage::open(file).ok()?; + let info = img.title_info().ok().flatten()?; + let resolved = fatxlib::titles::lookup(info.execution_info.title_id)?; + let sanitized = sanitize_fatx_filename(resolved.name); + if sanitized.is_empty() { + None + } else { + Some(sanitized) + } +} + +/// Coerce a free-form string into something FATX will accept as a filename. +/// Replaces characters the filesystem rejects with `-`, collapses runs of +/// whitespace, trims edge punctuation, and truncates to FATX's 42-byte +/// filename limit. +fn sanitize_fatx_filename(raw: &str) -> String { + const MAX_LEN: usize = 42; + // FATX rejects: < > : " / \ | ? * (plus controls). Replace with '-'. + let mut cleaned: String = raw + .chars() + .map(|c| match c { + '<' | '>' | ':' | '"' | '/' | '\\' | '|' | '?' | '*' => '-', + c if c.is_control() => '-', + c => c, + }) + .collect(); + while cleaned.contains(" ") { + cleaned = cleaned.replace(" ", " "); + } + let trimmed = cleaned.trim_matches(['.', ' ']); + if trimmed.len() <= MAX_LEN { + trimmed.to_string() + } else { + trimmed + .chars() + .take(MAX_LEN) + .collect::() + .trim_end_matches(['.', ' ', '-']) + .to_string() + } } fn dirs_or_home() -> PathBuf { @@ -651,8 +721,13 @@ fn io_worker( continue; } }; - let entries = match img.walk_files() { - Ok(v) => v, + let plan = match fatxlib::iso::manifest::build_manifest( + &mut img, + fatxlib::iso::manifest::IsoFilterPolicy { + keep_systemupdate: false, + }, + ) { + Ok(plan) => plan, Err(e) => { let _ = resp_tx.send(IoResp::Error { message: format!("Walk {}: {}", source.display(), e), @@ -669,21 +744,10 @@ fn io_worker( continue; } - // Xbox 360 disc images carry a `$SystemUpdate` folder with - // the dashboard update payload. Alt dashboards never run it, - // and copying it just wastes hundreds of MiB of FATX space — - // so partition them out before counting totals so the per-file - // progress denominator reflects what we'll actually write. - let (kept, skipped): ( - Vec<&fatxlib::xiso::XisoFile>, - Vec<&fatxlib::xiso::XisoFile>, - ) = entries - .iter() - .partition(|e| !is_xiso_junk(&e.path.replace('\\', "/"))); - let total_files = kept.len(); - let total_bytes: u64 = kept.iter().map(|f| f.size).sum(); - let skipped_files = skipped.len(); - let skipped_bytes: u64 = skipped.iter().map(|f| f.size).sum(); + let total_files = plan.kept_files(); + let total_bytes = plan.kept_bytes; + let skipped_files = plan.skipped_files(); + let skipped_bytes = plan.skipped_bytes; let mut files_done = 0usize; let mut bytes_done: u64 = 0; @@ -692,7 +756,7 @@ fn io_worker( let mut cancelled = false; let mut failed = false; - for entry in &kept { + for entry in plan.kept() { if cancel_flag.load(Ordering::Relaxed) { cancelled = true; break; @@ -802,6 +866,159 @@ fn io_worker( } } + IoCmd::ConvertXisoToGod { source, dest_dir } => { + cancel_flag.store(false, Ordering::Relaxed); + + let display_source = source + .file_name() + .map(|n| n.to_string_lossy().to_string()) + .unwrap_or_else(|| source.display().to_string()); + + let upload_dest = dest_dir.trim_end_matches('/').to_string(); + + // Dry-run first so we can resolve the human-readable title + // and announce the destination before the streaming pass. + let mut dry_opts = fatxlib::iso::god::ConvertOptions { + trim: fatxlib::iso::god::TrimMode::Compact, + dry_run: true, + ..Default::default() + }; + let report = match fatxlib::iso::god::convert_iso_to_fatx( + &source, + &mut vol, + &dest_dir, + &mut dry_opts, + ) { + Ok(r) => r, + Err(e) => { + let _ = resp_tx.send(IoResp::Error { + message: format!("Parse {}: {}", source.display(), e), + }); + continue; + } + }; + let resolved_name = fatxlib::titles::lookup(report.title_id).map(|t| t.name); + + let _ = resp_tx.send(IoResp::Progress { + message: format!( + "Converting {} ({}) → {}/{:08X}/00007000/{:08X}...", + display_source, + resolved_name.unwrap_or("unknown title"), + upload_dest, + report.title_id, + report.media_id, + ), + }); + + // Wire progress + cancel hooks. Both closures share the + // same lifetime so they can co-exist in ConvertOptions. + let cancel_flag_inner = cancel_flag.clone(); + let abort_fn = move || cancel_flag_inner.load(Ordering::Relaxed); + let resp_tx_inner = resp_tx.clone(); + let mut last_stage = String::new(); + let mut last_emit_at: Option = None; + let mut last_emit_bytes: u64 = 0; + let mut progress_cb = move |stage: &str, current: u64, total: u64| { + let stage_changed = stage != last_stage; + let now = std::time::Instant::now(); + + // Byte-level stages ("part X/Y"): rate-limit to ~200 ms + // intervals, and compute MiB/s from the delta between + // emits. Stage transitions always emit so the user sees + // each part's first tick immediately. + if stage.starts_with("part ") { + if !stage_changed + && let Some(t) = last_emit_at + && now.duration_since(t).as_millis() < 200 + { + return; + } + let throughput = if !stage_changed { + last_emit_at + .map(|t| { + let dt = now.duration_since(t).as_secs_f64(); + let dbytes = current.saturating_sub(last_emit_bytes); + let rate = (dbytes as f64) / dt / (1024.0 * 1024.0); + format!(" @ {:.1} MiB/s", rate) + }) + .unwrap_or_default() + } else { + String::new() + }; + let msg = format!( + "[{}] {} / {}{}", + stage, + format_size(current), + format_size(total), + throughput + ); + let _ = resp_tx_inner.send(IoResp::Progress { message: msg }); + } else { + // Integer-milestone stages (parts / mht / header): + // keep the 5 % throttle, render the raw count. + let denom = total.max(1); + let stride = (denom / 20).max(1); + if !stage_changed + && current != 0 + && current != total + && !current.is_multiple_of(stride) + { + return; + } + let _ = resp_tx_inner.send(IoResp::Progress { + message: format!("[{}] {}/{}", stage, current, total), + }); + } + + last_stage = stage.to_string(); + last_emit_at = Some(now); + last_emit_bytes = current; + }; + + let mut opts = fatxlib::iso::god::ConvertOptions { + trim: fatxlib::iso::god::TrimMode::Compact, + game_title: resolved_name, + dry_run: false, + progress: Some(&mut progress_cb), + should_abort: Some(&abort_fn), + }; + + match fatxlib::iso::god::convert_iso_to_fatx( + &source, &mut vol, &dest_dir, &mut opts, + ) { + Ok(r) => { + let _ = vol.flush(); + // Rough total: per-part overhead (4 KiB master + + // 4 KiB × subparts) plus the CON header. Reporting + // the source-side data size is close enough. + let _ = resp_tx.send(IoResp::Done { + message: format!( + "Converted {} → {}/{:08X}/00007000/{:08X} ({} parts, ~{})", + display_source, + upload_dest, + r.title_id, + r.media_id, + r.part_count, + format_size(r.data_size), + ), + }); + } + Err(e) => { + let _ = vol.flush(); + let msg = format!("{}", e); + if msg.contains("cancelled") { + let _ = resp_tx.send(IoResp::Cancelled { + message: format!("GoD conversion cancelled ({})", display_source), + }); + } else { + let _ = resp_tx.send(IoResp::Error { + message: format!("GoD convert: {}", msg), + }); + } + } + } + } + IoCmd::Mkdir { path } => match vol.create_directory(&path) { Ok(_) => { let _ = vol.flush(); @@ -1333,9 +1550,8 @@ fn handle_normal_key(app: &mut App, cmd_tx: &mpsc::Sender, key: KeyEvent) } } KeyCode::Char('u') => { - let default = app.download_dir.to_string_lossy().to_string(); app.input_prompt = format!("Upload file/directory to '{}':", app.cwd); - app.input_buffer = default; + app.input_buffer.clear(); app.input_mode = InputMode::UploadPath; } KeyCode::Char('m') => { @@ -1477,14 +1693,35 @@ fn handle_input_key(app: &mut App, cmd_tx: &mpsc::Sender, key: KeyEvent) app.download_dir = path.parent().unwrap_or(&PathBuf::from(".")).to_path_buf(); app.is_busy = true; } else if is_xiso(&path) { - // Detected an Xbox disc image. Ask the user whether to - // extract the contents (preferred — alt dashboards launch - // loose game files directly) or fall back to raw copy. - app.pending_xiso_upload = Some(path.clone()); + // Detected an Xbox disc image. Pick the default action + // based on cwd: inside a per-user Content folder (where + // GoD packages live), default to GoD; everywhere else + // default to extract (works for alt dashboards). + let default = if fatxlib::display::folder_slot(&app.cwd) + == fatxlib::display::FolderSlot::TitleId + { + XisoUploadAction::God + } else { + XisoUploadAction::Extract + }; + let prompt = match default { + XisoUploadAction::Extract => format!( + "Detected XISO '{}'. e(X)tract / (g)oD / (r)aw / Esc:", + filename + ), + XisoUploadAction::God => format!( + "Detected XISO '{}'. e(x)tract / (G)oD / (r)aw / Esc:", + filename + ), + XisoUploadAction::Raw => format!( + "Detected XISO '{}'. e(x)tract / (g)oD / (R)aw / Esc:", + filename + ), + }; + app.pending_xiso_upload = Some((path.clone(), default)); app.download_dir = path.parent().unwrap_or(&PathBuf::from(".")).to_path_buf(); - app.input_mode = InputMode::ConfirmExtractXiso; - app.input_prompt = - format!("Detected XISO '{}'. Extract contents? (Y/n):", filename); + app.input_mode = InputMode::ConfirmXisoUpload; + app.input_prompt = prompt; app.input_buffer.clear(); } else { let fatx_path = app.full_path(&filename); @@ -1497,44 +1734,77 @@ fn handle_input_key(app: &mut App, cmd_tx: &mpsc::Sender, key: KeyEvent) app.is_busy = true; } } else if app.input_prompt.starts_with("Detected XISO") { - // Y / empty (default) → extract; N → fall back to raw byte copy. - let extract = input.is_empty() - || input.eq_ignore_ascii_case("y") - || input.eq_ignore_ascii_case("yes"); - let path = match app.pending_xiso_upload.take() { - Some(p) => p, + let (path, default) = match app.pending_xiso_upload.take() { + Some(pair) => pair, None => { app.set_error("Internal: missing pending XISO path."); return; } }; + let trimmed = input.trim(); + let action = if trimmed.is_empty() { + default + } else { + match trimmed.chars().next().map(|c| c.to_ascii_lowercase()) { + Some('x') => XisoUploadAction::Extract, + Some('g') => XisoUploadAction::God, + Some('r') => XisoUploadAction::Raw, + _ => { + app.set_error(&format!( + "Unknown choice {:?} — expected x, g, or r.", + trimmed + )); + return; + } + } + }; let filename = path .file_name() .map(|n| n.to_string_lossy().to_string()) .unwrap_or_else(|| "iso".to_string()); - if extract { - // Default subfolder name = file stem (no extension); fall - // back to the whole filename if the path has no extension. - let stem = path - .file_stem() - .map(|n| n.to_string_lossy().to_string()) - .unwrap_or_else(|| filename.clone()); - let dest_dir = app.full_path(&stem); - app.set_status(&format!("Extracting '{}' → {}...", filename, dest_dir)); - let _ = cmd_tx.send(IoCmd::ExtractXiso { - source: path, - dest_dir, - }); - app.is_busy = true; - } else { - let fatx_path = app.full_path(&filename); - app.set_status(&format!("Uploading '{}' (raw)...", filename)); - let _ = cmd_tx.send(IoCmd::WriteFile { - local_path: path, - fatx_path, - }); - app.is_busy = true; + match action { + XisoUploadAction::Extract => { + // Prefer a catalog-resolved folder name over the + // local filename stem: a disc named `disc1.iso` + // with TitleID 0x4D5307E6 should land at + // `/Halo 3 [4D5307E6]/` rather than `/disc1/`. + // Falls back to the file stem on catalog miss or + // unreadable XEX/XBE. + let stem = path + .file_stem() + .map(|n| n.to_string_lossy().to_string()) + .unwrap_or_else(|| filename.clone()); + let resolved = xiso_folder_name(&path).unwrap_or(stem); + let dest_dir = app.full_path(&resolved); + app.set_status(&format!("Extracting '{}' → {}...", filename, dest_dir)); + let _ = cmd_tx.send(IoCmd::ExtractXiso { + source: path, + dest_dir, + }); + app.is_busy = true; + } + XisoUploadAction::God => { + let dest_dir = app.cwd.clone(); + app.set_status(&format!( + "Converting '{}' to GoD under {}...", + filename, dest_dir + )); + let _ = cmd_tx.send(IoCmd::ConvertXisoToGod { + source: path, + dest_dir, + }); + app.is_busy = true; + } + XisoUploadAction::Raw => { + let fatx_path = app.full_path(&filename); + app.set_status(&format!("Uploading '{}' (raw)...", filename)); + let _ = cmd_tx.send(IoCmd::WriteFile { + local_path: path, + fatx_path, + }); + app.is_busy = true; + } } } else if app.input_prompt.starts_with("New directory") { // Mkdir @@ -1594,7 +1864,7 @@ fn handle_input_key(app: &mut App, cmd_tx: &mpsc::Sender, key: KeyEvent) KeyCode::Char(c) => { if app.input_mode == InputMode::ConfirmDelete || app.input_mode == InputMode::ConfirmCleanup - || app.input_mode == InputMode::ConfirmExtractXiso + || app.input_mode == InputMode::ConfirmXisoUpload { app.input_buffer = c.to_string(); } else { @@ -1696,12 +1966,17 @@ fn ui(frame: &mut Frame, app: &mut App) { if app.input_mode != InputMode::Normal { let input_text = format!(" {} {}", app.input_prompt, app.input_buffer); let input_bar = Paragraph::new(input_text) - .style(Style::default().fg(Color::LightYellow).bg(Color::Blue)) + .style( + Style::default() + .fg(Color::Yellow) + .bg(Color::DarkGray) + .bold(), + ) .block( Block::default() .title(" Input (Enter to confirm, Esc to cancel) ") .borders(Borders::ALL) - .border_style(Style::default().fg(Color::LightYellow)), + .border_style(Style::default().fg(Color::Yellow).bold()), ); frame.render_widget(input_bar, chunks[2]); @@ -1774,21 +2049,112 @@ mod tests { #[test] fn test_is_xiso_junk_systemupdate() { - assert!(is_xiso_junk("$SystemUpdate")); - assert!(is_xiso_junk("$SystemUpdate/su20076000_00000000")); - assert!(is_xiso_junk("/$SystemUpdate/anything")); + assert!(fatxlib::iso::policy::is_systemupdate_path("$SystemUpdate")); + assert!(fatxlib::iso::policy::is_systemupdate_path( + "$SystemUpdate/su20076000_00000000" + )); + assert!(fatxlib::iso::policy::is_systemupdate_path( + "/$SystemUpdate/anything" + )); } #[test] fn test_is_xiso_junk_case_insensitive() { - assert!(is_xiso_junk("$SYSTEMUPDATE/foo")); - assert!(is_xiso_junk("$systemupdate/foo")); + assert!(fatxlib::iso::policy::is_systemupdate_path( + "$SYSTEMUPDATE/foo" + )); + assert!(fatxlib::iso::policy::is_systemupdate_path( + "$systemupdate/foo" + )); + } + + #[test] + fn test_sanitize_fatx_filename_replaces_illegal_chars() { + assert_eq!( + sanitize_fatx_filename("Halo: Combat Evolved"), + "Halo- Combat Evolved" + ); + assert_eq!( + sanitize_fatx_filename(r"Tom Clancy's R6: Vegas "), + "Tom Clancy's R6- Vegas -DEMO-" + ); + assert_eq!( + sanitize_fatx_filename("path/with\\slashes"), + "path-with-slashes" + ); + } + + #[test] + fn test_sanitize_fatx_filename_truncates_to_42_bytes() { + let long = "A Really Long Game Subtitle That Definitely Will Not Fit"; + let s = sanitize_fatx_filename(long); + assert!(s.len() <= 42, "got {:?} ({} bytes)", s, s.len()); + // Truncation should be at a word boundary or just under 42 bytes; + // never end with a dangling separator. + assert!(!s.ends_with(' ')); + assert!(!s.ends_with('-')); + assert!(!s.ends_with('.')); + } + + #[test] + fn test_sanitize_fatx_filename_collapses_runs_of_whitespace() { + assert_eq!( + sanitize_fatx_filename("Halo 3 Anniversary"), + "Halo 3 Anniversary" + ); + } + + #[test] + fn test_sanitize_fatx_filename_trims_edges() { + assert_eq!(sanitize_fatx_filename(" Halo 3 "), "Halo 3"); + assert_eq!(sanitize_fatx_filename("...Halo 3..."), "Halo 3"); + } + + #[test] + fn test_xiso_upload_default_god_inside_xuid_folder() { + // cwd directly inside /Content// should default to GoD, + // because that's where Xbox 360 BC looks for title-id folders. + assert_eq!( + fatxlib::display::folder_slot("/Content/0000000000000000"), + fatxlib::display::FolderSlot::TitleId + ); + assert_eq!( + fatxlib::display::folder_slot("/Content/E0001A0BC2E16C4D"), + fatxlib::display::FolderSlot::TitleId + ); + } + + #[test] + fn test_xiso_upload_default_extract_elsewhere() { + // Anywhere outside `/Content//` should default to extract. + assert_ne!( + fatxlib::display::folder_slot("/"), + fatxlib::display::FolderSlot::TitleId + ); + assert_ne!( + fatxlib::display::folder_slot("/Games"), + fatxlib::display::FolderSlot::TitleId + ); + assert_ne!( + fatxlib::display::folder_slot("/Content"), + fatxlib::display::FolderSlot::TitleId + ); + // Deeper than the XUID folder: we're inside a title-id folder + // already, so children are content-type folders — extract default. + assert_ne!( + fatxlib::display::folder_slot("/Content/0000000000000000/4D530002"), + fatxlib::display::FolderSlot::TitleId + ); } #[test] fn test_is_xiso_junk_does_not_match_substring() { - assert!(!is_xiso_junk("default.xbe")); - assert!(!is_xiso_junk("Media/$SystemUpdate")); // not the first segment - assert!(!is_xiso_junk("MyGame$SystemUpdate/foo")); + assert!(!fatxlib::iso::policy::is_systemupdate_path("default.xbe")); + assert!(!fatxlib::iso::policy::is_systemupdate_path( + "Media/$SystemUpdate" + )); // not the first segment + assert!(!fatxlib::iso::policy::is_systemupdate_path( + "MyGame$SystemUpdate/foo" + )); } }