Skip to content

Archive Mode

Jump to bottom
asdfmonster261 edited this page May 4, 2026 · 5 revisions

Archive Mode

Steam-side mirroring pipeline. Watch one or more appids, download every new build to a 7z archive, optionally crack + upload + post a forum announcement — fully automated. Requires the optional archive extra:

pip install -e ".[archive]"

Without the extra, archive subcommands print a friendly error pointing at the install line.

Concepts

Term Meaning
appid Steam application ID (e.g. 730 for CS:GO). PatchForge polls these.
depot Steam content depot — typically one per platform (windows / linux / macos) per app.
manifest_gid Globally-unique ID for a specific build of a specific depot. Sticking a (depot_id, manifest_gid) tuple into Steam's CDN gets you the exact files for that build, forever (until Steam GCs old manifests).
buildid Steam's per-app build counter. Bumps every time the developer publishes an update. PatchForge polls this to detect changes.
branch A named release channel (public, beta, etc.). Public is the default; betas may be password-locked.
.xarchive PatchForge's project file for archive mode. JSON; lists watched apps, default platform, BBCode template, persisted run-time options, and the per-app manifest_history.

Login

patchforge archive login

Walks you through the Steam login flow:

  1. Asks for a username.
  2. Tries QR-code login first (open the Steam mobile app, scan the code).
  3. Falls back to password + 2FA if QR fails.
  4. Saves a long-lived refresh token to archive_credentials.json in your config dir (~/.config/patchforge/ on Linux, %APPDATA%\patchforge\ on Windows). File mode is 0600 on POSIX.

Subsequent archive download / archive depot / archive show-app runs reuse the token without prompting.

patchforge archive logout      # clears the refresh token (keeps web API key + notify creds)

The login token is Steam-account-specific. PatchForge can only download depots licensed to that account. Free-to-play games and any games the account owns are fair game; everything else returns a "missing license" error during the manifest fetch.

Project file (.xarchive)

A .xarchive is plain JSON listing what to watch and how to handle each update. Minimal example:

{
  "schema_version":   1,
  "name":             "my-mirror",
  "default_platform": "windows",
  "default_branch":   "public",
  "output_dir":       "/srv/patchforge/out",
  "restart_delay":    1800,
  "batch_size":       10,
  "max_retries":      3,
  "compression":      9,
  "max_concurrent_uploads": 2,
  "delete_archives":  false,
  "notify_mode":      "delay",
  "bbcode_template":  "[size=150]{APP_NAME}[/size]\nBuild: {BUILDID}...",
  "apps": [
    {"app_id": 730,  "branch": "public", "current_buildid": "16203481"},
    {"app_id": 4048390, "branch": "public"}
  ]
}

Manage it via the CLI:

patchforge archive new-project --output my-mirror.xarchive
patchforge archive add-app     my-mirror.xarchive 730
patchforge archive add-app     my-mirror.xarchive 4048390 --branch beta --platform linux
patchforge archive remove-app  my-mirror.xarchive 730
patchforge archive show-project my-mirror.xarchive

add-app / remove-app mutate the existing apps[] array and save back through the normal project loader, so manifest_history, buildids, BBCode template, and run-time options are all preserved. You can also edit the JSON by hand, but the subcommands are schema-aware (duplicate add → exit 3, missing remove → exit 3).

Each apps[] entry tracks one game. When PatchForge processes a new build for that app, it shifts the entry's current_buildid into previous_buildid and appends a ManifestRecord per (branch, platform, depot_id, manifest_gid) to manifest_history so the build can be re-pulled later via archive depot.

The full schema is in src/core/archive/project.py:ArchiveProject — most fields mirror the CLI flags below.

Single-pass downloads

patchforge archive download \
    --app-ids 730,4048390 \
    --output-dir ./out \
    --platform all

Logs in once, fetches product info for every listed app, downloads any depot whose buildid changed, compresses each platform's depot tree to <safename>.<buildid>.<platform>.<branch>.7z, and exits.

Add --project FILE to load defaults from a .xarchive:

patchforge archive download --project my-mirror.xarchive

In project mode the per-app list comes from apps[]; CLI --app-ids is optional and only used for ad-hoc one-off pulls.

Useful flags

Flag Notes
--platform {windows,linux,macos,all} Default: project's default_platform, falling back to windows. all downloads every platform listed in the depot's oslist.
--branch NAME Public branch (default), or a beta branch name.
--branch-password PW Required only when --branch is a password-protected beta.
--workers N Per-depot parallel download greenlets. Default 8.
--compression N 7z compression level 0–9. Default 9.
--max-retries N Retry budget for CM timeouts and SessionDead recovery (see below).
--archive-password PW Optional 7z archive password.
--volume-size 4g Split each archive into multi-volume parts at this size.
--language LANG Goldberg-emu deploy language (only used with --crack gse).
--force-download Ignore unchanged buildids — download anyway.
--no-progress Disable the live multi-line progress display (use plain log lines).
--log FILE Tee stdout + stderr to FILE with ANSI codes stripped.

Polling mode

Set --restart-delay N (or restart_delay in the project) to turn archive download into an infinite poll loop:

patchforge archive download --project my-mirror.xarchive --restart-delay 1800

Each iteration:

  1. Fetch product info for every app in the project.
  2. Compare buildid against the project's stored current_buildid.
  3. For every changed app: download → compress → optionally crack → optionally upload → optionally post → optionally notify.
  4. Save the project (so the new buildid lands on disk before the next sleep).
  5. Sleep restart_delay seconds. Use gevent.sleep, not time.sleep, so the Steam CM heartbeat keeps firing across the wait.

The countdown loop renders a single-line next poll cycle in Xs (Ctrl-C to stop) ticker on a TTY; on a non-TTY it prints one sleeping Xs line and waits silently. The GUI ticks the same countdown into a Qt label.

--batch-size N

For accounts that own hundreds of games, fetching product info for everything in a single PICS request can hit Steam's response-size cap. --batch-size N chunks the queries into separate requests of N apps each. Within each batch the streaming response is yielded one app at a time, so the progress bar ticks per-app regardless of batch size.

--max-retries N

Bounds two retry loops:

  • PICS timeout retry — when a client.get_product_info call times out, retry up to N times before raising SessionDead.
  • Mid-run relogin — when a SessionDead propagates out of run_one_app, the runner disconnects the dead client, calls cm_login(tokens) again, and retries the same app on the new client. Up to N consecutive failures are tolerated; the counter resets on each clean cycle.

If both retry budgets are exhausted within a single poll cycle, the runner waits one full restart_delay and tries again with a fresh budget. After two consecutive outage windows fail, an alert is sent through any configured notify channels (see "Outage recovery" below) — useful for the weekly Tuesday Steam CM maintenance.

Outage recovery

PatchForge's archive mode is designed to ride through Steam's weekly maintenance window. The recovery shape:

  1. Mid-cycle SessionDead — disconnect, relogin, retry the failing app. Bounded by max_retries.
  2. Burst exhausted — if the relogin loop runs out, sleep restart_delay (using gevent so heartbeats keep firing) and retry the whole cycle with a fresh max_retries budget.
  3. Two consecutive outage cycles — if the burst-then-wait pattern fires twice in a row, fire a one-shot alert through notify.send_alert(creds, ...) to whichever of Telegram / Discord is configured. Alert resets after the next clean cycle so future outages re-arm it.

The runner never gives up unless the user aborts (Ctrl-C / GUI Stop button). This shifts "babysit the script through Tuesday maintenance" off the user's plate.

Crack pipeline

patchforge archive download --project my-mirror.xarchive --crack coldclient
patchforge archive download --project my-mirror.xarchive --crack gse
patchforge archive download --project my-mirror.xarchive --crack all

Three pre-packaged crack flows:

  • coldclient — drops the ColdClientLoader emulator + start_<game>.exe launcher into the depot tree before compressing. Windows only — Linux/macOS runs skip this engine.
  • gse — runs the full Goldberg Steam Emulator deploy: SteamStub unpack via the vendored unpacker tree, DLL surgery to insert the right interface versions, achievement DB fetch via Steamworks Web API (requires web_api_key in credentials), DLC DB save/load, and all the CrackIdentity prompts (steam_id / username / language / listen_port).
  • all — generate both engines side-by-side. On Windows produces both subdirs; on Linux/macOS collapses to gse only because ColdClient has no non-Windows variant.

Output layout

Both engines share one combined output directory gse_config_<appid>/ so a single archive carries both choices. Inside, each engine gets its own subdir:

gse_config_<appid>/
  emulator/      ← Goldberg deploy (gse)
  coldclient/    ← ColdClientLoader deploy

End users pick whichever subtree they want and merge it onto their game install. --crack gse populates only emulator/; --crack coldclient only coldclient/; --crack all populates both.

When --crack all runs both engines for the same app, the canonical steam_settings/ payload (DLCs, achievements, header, language list, configs.user.ini) is fetched exactly once and copied into each engine's expected location. No duplicate API calls, no double prompts.

Both modes integrate with the SteamStub unpacker tree under src/core/archive/crack/unstub/ (vendored from upstream SteamArchiver — not modified here). --keepbind / --keepstub / --dumppayload / --dumpdrmp / --realign / --recalcchecksum map to the unstub options.

The chosen crack mode persists in the project file as crack_mode so a .xarchive carrying "crack_mode": "all" runs the dual-crack pipeline by default without --crack on the CLI.

Per-app override

Each AppEntry can carry its own crack_mode field that overrides the project default for that app only:

{
  "apps": [
    {"app_id": 730,     "branch": "public"},
    {"app_id": 4048390, "branch": "public", "crack_mode": "all"},
    {"app_id": 12345,   "branch": "public", "crack_mode": "off"}
  ],
  "crack_mode": "gse"
}
AppEntry.crack_mode Effect
"" (default) Inherit project.crack_mode (and --crack flag)
"off" Skip the crack pipeline for this app even when the project default sets one
"gse" / "coldclient" / "all" Use that engine for this app regardless of project default

Resolution precedence: AppEntry.crack_mode > --crack CLI flag > project.crack_mode > none.

Set the override at append time via the CLI:

patchforge archive add-app my-mirror.xarchive 4048390 --crack all
patchforge archive add-app my-mirror.xarchive 12345   --crack off

Or edit the Crack column dropdown in the GUI Apps tab.

Upload (MultiUp + PrivateBin)

patchforge archive download --project my-mirror.xarchive \
    --upload-username NAME --upload-password PW \
    --binurl https://privatebin.example/ --binpass paste-password

Or save the credentials once (also persisted to archive_credentials.json):

patchforge archive download --project my-mirror.xarchive \
    --upload-username NAME --upload-password PW
# Future runs only need the username; PW is loaded from creds.

Behaviour:

  • Multi-part archives sharing a stem (game.7z.001 / .002 / ...) are grouped into one MultiUp project, so users see a single canonical short URL per build.
  • If --binurl is set, per-stem download links are consolidated into a single PrivateBin paste; the paste URL replaces the per-stem .txt sidecar.
  • --max-concurrent-uploads N controls the per-project parallelism (gevent green pool, not real threads, so the live-display redraw greenlet stays alive).
  • --delete-archives removes the local .7z files after a successful upload — saves disk on a long-running mirror.

BBCode templates

Generated forum posts use a placeholder-substituting template stored in the project's bbcode_template. The GUI's BBCode tab provides a full editor:

  • Format toolbar — B / I / U / S, Quote, Code, List / List=, [*], Img, URL, Size dropdown, Spoiler / Spoiler="title", YouTube
  • 120-swatch colour picker (vertical strip on the right of the editor)
  • Placeholder chips that insert at cursor: {APP_NAME}, {APPID}, {BUILDID}, {PREVIOUS_BUILDID}, {DATE}, {DATETIME}, {STEAMDB_URL}, {WINDOWS_LINK}, {LINUX_LINK}, {MACOS_LINK}, {ALL_LINKS}, {PLATFORMS}, {MANIFESTS}, {HEADER_IMAGE}
  • Live preview pane with a Raw text / Forum post toggle — the forum-rendered preview converts the BBCode subset to HTML and renders it in QTextEdit's rich-text mode

When the runner has both the upload links and the rendered template, a <safename>.<buildid>.post.txt is written next to the archives. Per-stem sidecar .txt files are removed in favour of the consolidated post.

Notifications (Telegram + Discord)

Both channels are opt-in. Set them via CLI flags or in archive_credentials.json:

patchforge archive download --project my-mirror.xarchive \
    --telegram-bot-token TOKEN --telegram-chat-id 1234567890 \
    --discord-webhook https://discord.com/api/webhooks/...

Modes (--notify flag, or notify_mode in project):

Mode Pre-download notify Post-download notify
none — —
pre yes (no upload links) —
delay (default when MultiUp set) — yes (with upload links)
both yes yes

Pre-notify can't include upload links (the upload hasn't happened yet); post-notify always includes them when the upload step ran.

In addition to per-build notifications, the outage alert (see above) uses the same channels for "Steam CM unreachable" plain-text warnings.

Historical builds (archive depot)

patchforge archive depot \
    --app 4048390 --depot 4048391 --manifest 5520155637093182018 \
    --output-dir ./out

Pulls one specific (app, depot, manifest_gid) triple — the same shape DepotDownloader takes. The triples that worked at any point on this account are recorded in your .xarchive's manifest_history, so:

patchforge archive show-project my-mirror.xarchive

gives you a printable index of every replayable build. Useful when a Steam update breaks something and you want to roll a friend's install back to last week's build.

The output layout is <output_dir>/<app>_<depot>_<gid>/... for the file tree plus <output_dir>/depotcache/<depot>_<gid>.manifest for the serialised manifest. Crack / upload / notify do not run on this code path — archive depot is purely a content fetch. If you want crack + upload + post for a historical build, run archive download --force-download against the live depot instead.

GUI

The PatchForge GUI exposes the archive workflow as a third top-level tab. Layout:

  • Apps — table of appid / branch / platform / per-app crack / current buildid / previous buildid / app name with add/remove/import buttons (the same edits are also reachable via patchforge archive add-app / remove-app)
  • Credentials — Steam refresh-token status (with expiry colour), web API key, MultiUp / PrivateBin / Telegram / Discord fields
  • BBCode — the template editor described above
  • Run options — output dir, default platform, restart_delay, batch_size, max_retries, compression, language, archive password, volume size, max-concurrent-uploads, crack mode, force-download, log file
  • Manifest history — flat sortable table of every ManifestRecord accumulated across all apps (build / branch / platform / depot / manifest GID / time). Filter dropdown narrows to one app. "Pull selected build" fires the equivalent of archive depot --app X --depot Y --manifest Z in-process and routes the live depot pull through the same Live run tab the regular download uses
  • Run — start/stop button, stage strip (Poll / Download / Crack / Compress / Upload / Paste / Notify), per-file progress, app-info progress bar, build log

The Run pane subscribes to the same DownloadEvent stream the CLI's live display uses, so progress is identical between the two. File and upload panes clear automatically on each (app, platform) boundary so multi-app runs don't interleave.

The GUI runs the archive worker on a QThread, so the main UI thread stays responsive (cancel / abort works). Steam CM heartbeats fire on the worker thread's gevent hub, exactly like the CLI.

Home

Clone this wiki locally