Skip to content

fix: compress case backups + demo-data archive (DEFLATE)#481

Merged
SeaCelo merged 3 commits into
EAPD-DRB:mainfrom
SeaCelo:fix/backupcase-zip-compression
Jun 15, 2026
Merged

fix: compress case backups + demo-data archive (DEFLATE)#481
SeaCelo merged 3 commits into
EAPD-DRB:mainfrom
SeaCelo:fix/backupcase-zip-compression

Conversation

@SeaCelo

@SeaCelo SeaCelo commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Three related changes that cut ZIP/demo size and keep setup robust.

1. Compress case backups. backupCase opened the archive with ZipFile(zippedFile, 'w'), which defaults to ZIP_STORED — no compression. Case backups are all JSON/CSV/text (model inputs, solver results, pivot views) and compress 15–30×, so backups were far larger than needed: the demo case backs up to 48 MB and a real country case (Lao PDR CLEWS+WESM) to ~944 MB. The fix is one line — open with compression=ZIP_DEFLATED. DEFLATE is the standard zip method, so every reader (Python zipfile, unzip, OS archivers, upstream MUIO) handles it, and the extract/restore code is untouched; existing uncompressed backups still restore.

2. Recompress the demo-data archive. The published CLEWs.Demo.zip release asset was also stored uncompressed (48.7 MB). Recompressed with DEFLATE it's 3.2 MB — same files, byte-for-byte identical content (verified across all 226 entries). This bumps the pinned DEMO_DATA_ARCHIVE_SHA256 in setup_dev.py and the README SHA to the new archive's hash (db92d380…).

3. Self-heal a stale demo cache. Setup verifies the cached assets/demo-data/CLEWs.Demo.zip against the pin, but --force-demo-data only clears the extracted dirs, not the cached archive — so after a hash change a reinstall would fail the checksum on the old cached file. Setup now drops a cached archive whose hash doesn't match the pin and re-downloads, so reinstalls recover automatically.

Tested: /backupCase on a throwaway case returns 200 with every entry DEFLATED, lp.lp excluded, paths relative; the recompressed demo verified to extract identical content to WebAPP/DataStorage/CLEWs Demo/; the stale-cache self-heal verified to re-download and reinstall (offline). Full suite 49/49, ruff clean.

Rollout: the new compressed CLEWs.Demo.zip needs to be uploaded to the demo-data release. The SHA is checked only on a fresh download — existing installs short-circuit on the marker, and a stale cache now self-heals — so the only thing to coordinate is the same-name swap: upload the asset close to merging this PR so a fresh setup never sees the old asset against the new pin (or vice-versa). CI doesn't download demo data, so the PR's own checks are unaffected by upload timing.

@github-actions github-actions Bot added the needs-intake-fix PR intake structure needs maintainer follow-up label Jun 12, 2026
@SeaCelo SeaCelo changed the title fix: compress case backup archives (DEFLATE) instead of storing uncompressed fix: compress case backups + demo-data archive (DEFLATE) Jun 12, 2026
@SeaCelo

SeaCelo commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator Author

tested locally. Merging

@SeaCelo SeaCelo merged commit 5cb5911 into EAPD-DRB:main Jun 15, 2026
4 checks passed
@SeaCelo SeaCelo deleted the fix/backupcase-zip-compression branch June 15, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-intake-fix PR intake structure needs maintainer follow-up

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant