fix: compress case backups + demo-data archive (DEFLATE)#481
Merged
Conversation
…-> 3.2MB), update pinned SHA-256
Collaborator
Author
|
tested locally. Merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related changes that cut ZIP/demo size and keep setup robust.
1. Compress case backups.
backupCaseopened the archive withZipFile(zippedFile, 'w'), which defaults toZIP_STORED— no compression. Case backups are all JSON/CSV/text (model inputs, solver results, pivot views) and compress 15–30×, so backups were far larger than needed: the demo case backs up to 48 MB and a real country case (Lao PDR CLEWS+WESM) to ~944 MB. The fix is one line — open withcompression=ZIP_DEFLATED. DEFLATE is the standard zip method, so every reader (Pythonzipfile,unzip, OS archivers, upstream MUIO) handles it, and the extract/restore code is untouched; existing uncompressed backups still restore.2. Recompress the demo-data archive. The published
CLEWs.Demo.ziprelease asset was also stored uncompressed (48.7 MB). Recompressed with DEFLATE it's 3.2 MB — same files, byte-for-byte identical content (verified across all 226 entries). This bumps the pinnedDEMO_DATA_ARCHIVE_SHA256insetup_dev.pyand the README SHA to the new archive's hash (db92d380…).3. Self-heal a stale demo cache. Setup verifies the cached
assets/demo-data/CLEWs.Demo.zipagainst the pin, but--force-demo-dataonly clears the extracted dirs, not the cached archive — so after a hash change a reinstall would fail the checksum on the old cached file. Setup now drops a cached archive whose hash doesn't match the pin and re-downloads, so reinstalls recover automatically.Tested:
/backupCaseon a throwaway case returns 200 with every entry DEFLATED,lp.lpexcluded, paths relative; the recompressed demo verified to extract identical content toWebAPP/DataStorage/CLEWs Demo/; the stale-cache self-heal verified to re-download and reinstall (offline). Full suite 49/49, ruff clean.Rollout: the new compressed
CLEWs.Demo.zipneeds to be uploaded to thedemo-datarelease. The SHA is checked only on a fresh download — existing installs short-circuit on the marker, and a stale cache now self-heals — so the only thing to coordinate is the same-name swap: upload the asset close to merging this PR so a fresh setup never sees the old asset against the new pin (or vice-versa). CI doesn't download demo data, so the PR's own checks are unaffected by upload timing.