Skip to content

Performance: incremental/delta snapshots — only store changed files #10

Description

@piranhap

Summary

Every push creates a full .tar.zst snapshot of all tracked files regardless of what changed. For users tracking large directories (~/.config/nvim/, bin/) this means uploading the same unchanged files every time, wasting bandwidth and storage — especially with the GitHub backend where each snapshot is a committed binary blob.

Proposed Design

Store a base snapshot (full) plus delta snapshots (changed files only). The manifest tracks which deltas to apply to reconstruct any version:

{
  "latest_version": 7,
  "snapshots": [
    { "version": 1, "type": "full",  "file": "v001.tar.zst" },
    { "version": 2, "type": "delta", "file": "v002.tar.zst", "base": 1 },
    { "version": 7, "type": "delta", "file": "v007.tar.zst", "base": 1 }
  ]
}

A delta snapshot contains only files whose checksums changed since the base. Reconstruction: extract base, overlay delta files.

Full snapshots are created every N versions (configurable, default 10) to bound reconstruction cost.

Implementation Notes

  • snapshot::create_delta_snapshot(home_dir, changed_files) — tar only the changed CollectedFiles
  • snapshot::apply_delta(base_data, delta_data) -> Vec<u8> — extract base, overlay delta entries
  • BackendManifest gains snapshot_type: SnapshotType and base_version: Option<u64> fields
  • cmd_push computes which files changed (already has checksums) and calls the appropriate snapshot creator
  • cmd_pull reconstructs by fetching base + all deltas up to target version

Backwards Compatibility

Existing full snapshots continue to work. The type field defaults to "full" when absent so old manifests parse correctly.

Affected Files

  • src/snapshot.rscreate_delta_snapshot(), apply_delta()
  • src/backend/mod.rsSnapshotMeta gains snapshot_type, base_version
  • src/sync.rscmd_push(), cmd_pull()

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions