wallet: cooperative-yield sync (incremental persist + sentinel)

[saroupille]

Problem

tzel-wallet sync scans [scanned, tree_size) sequentially and persists
wallet.json.scanned only at the end. While running it holds an exclusive
lock on wallet.json.lock, so every other wallet op (shield, transfer,
unshield, balance, sync, …) blocks for the full duration. On a long-running
rollup that can be 5–45 min, which is unacceptable in daemon-driven flows
where users want to act now.

Design

Cooperative yield via a sentinel file (<wallet>.yield):

1. Incremental persist. cmd_rollup_sync now scans in batches of
   CHECKPOINT_EVERY = 50 commits, each batch ending in a save_wallet
   that records w.scanned = batch_end plus any newly-recovered notes.
   The atomic-rename invariant in save_wallet (temp-file + fsync +
   rename + parent-dir fsync) means a kill -9 between fsync and rename
   leaves the previous wallet.json intact: at most one batch (~50 commits,
   ~2 s of scan work) is lost, never corrupted.
2. Sentinel check at every checkpoint. After persisting a batch, the
   sync stat()s <wallet>.yield. Present ⇒ exit 0 cleanly. The next sync
   resumes from the persisted cursor. The companion change in
   trilitech/tzel-infra teaches the daemon to touch the sentinel before
   admitting a slow CLI op and to remove it after, so a long initial sync
   no longer starves slow-lane ops (shield/transfer/unshield/withdraw).
3. Final flush at end-of-loop. Nullifier prune + pending-pool prune
   run once over the cumulative cm set accumulated across batches —
   preserves the existing apply_scan_feed semantics exactly. (The
   existing helper is split into apply_scan_feed_recover_batch and
   apply_scan_feed_finalize; legacy apply_scan_feed is kept for
   in-tree tests and back-compat.)

Choice of K = 50

K	persist overhead (1 save_wallet / K HTTP fetches)	loss-on-kill-9	per-batch sentinel cost
1	~33%	<1 commit	sub-ms (negligible)
10	~6%	<1 s	sub-ms
50	~0.7%	~2 s	sub-ms
200	~0.2%	~10 s	sub-ms
500	<0.1%	~25 s	sub-ms

50 is the empirical sweet spot. Don't drop below ~10 (overhead dominates)
or push past ~500 (the loss becomes user-visible during a long initial
sync). Documented inline.

Failure-mode analysis

• kill -9 between fsync and rename: previous wallet.json wins; one
  batch lost. Confirmed by the existing atomic-rename invariant — no
  change to save_wallet, just called more often.
• kill -9 mid-batch HTTP fetch: the in-RAM w is dropped, prior
  on-disk state is the last committed checkpoint. Same as above.
• Sentinel races a checkpoint boundary: stat() succeeds late ⇒ next
  checkpoint catches it. Worst case: one extra batch runs (~2 s).
• fs error on stat(): treated as "not present" so a flaky FS does
  NOT cause spurious exits. Correctness comes from the daemon's
  best-effort sentinel removal post-op.

Tests

Three new tests in network_profile_tests:

1. cmd_rollup_sync_exits_at_first_checkpoint_when_sentinel_present
   — pre-create the sentinel; assert sync stops after exactly one
   checkpoint and does NOT remove the caller's sentinel.
2. cmd_rollup_sync_resumes_from_checkpointed_cursor — drive the
   sentinel via the set_sync_checkpoint_hook test hook, run sync
   twice. Notes before the yield boundary are recovered in run 1;
   notes after are recovered in run 2.
3. cmd_rollup_sync_intermediate_wallet_is_always_parseable —
   concurrent reader thread parses wallet.json while the scan runs;
   every read must serde-parse (no half-written file).

Result: cargo +nightly-2025-07-14 test -p tzel-wallet-app --lib
→ 102 passed (99 baseline + 3 new), 0 failed.

Companion change

trilitech/tzel-infra branch feat/sync-cooperative-yield (PR linked
once opened). The daemon dispatcher there wraps slow-lane task
execution with a touch + RAII-removed sentinel, so a long-running
sync yields the lock for shield/transfer/unshield/withdraw and
resumes after.

🤖 Generated with Claude Code