Unified ingestion workflow: streaming ingestion daemon#795

Draft

chowbao wants to merge 32 commits into

feature/full-historyfrom

streaming-ingestion-daemon

Commits on Jun 18, 2026

feat(fullhistory/streaming): meta-store catalog + one write protocol + sweeps
chowbao
committed
feat(fullhistory/streaming): processChunk + catchupSource
chowbao
committed
feat(fullhistory/streaming): tx-hash rolling rebuild + coverage protocol
chowbao
committed
feat(fullhistory/streaming): postcondition resolver + executor
chowbao
committed
refactor(fullhistory): collapse per-chunk hot stores into one multi-CF DB; atomic WriteBatch/ledger (decision a)
chowbao
committed
feat(fullhistory/streaming): derived progress (deriveWatermark/deriveCompleteThrough)
chowbao
committed
feat(fullhistory/streaming): hot-DB ingestion loop
chowbao
committed
feat(fullhistory/streaming): lifecycle tick + loop (clean shutdown)
chowbao
committed
feat(fullhistory/streaming): startup orchestration (startStreaming)
chowbao
committed
test(fullhistory/streaming): close derived-progress coverage gaps
chowbao
committed
feat(fullhistory/streaming): config schema + validateConfig + single-process locking
chowbao
committed
feat(fullhistory/streaming): retention widen/shorten + reader-retention contract
chowbao
committed
feat(fullhistory/streaming): surgical recovery + hot-volume-loss handling
chowbao
committed
feat(fullhistory/streaming): audit command (INV-1..4 invariant walks)
chowbao
committed
fix(full-history): audit completes (not aborts) when an INV-2 two-frozen window also has hot/txhash keys
chowbao
committed
feat(fullhistory/streaming): runnable streaming-daemon entrypoint + config wiring
chowbao
committed
fix(fullhistory/streaming): honor storage-path overrides in the data path
chowbao
committed
feat(fullhistory/streaming): observability — metrics + structured logging
chowbao
committed
fix(fullhistory/streaming): real steady-state liveness signal + log/lag test coverage
chowbao
committed
test(fullhistory/streaming): crash-injection + convergence suite (INV-1..4)
chowbao
committed
test(fullhistory/streaming): document convergence caveats; verify case-4 tick is a no-op
chowbao
committed
test(fullhistory/streaming): end-to-end daemon integration (in-process)
chowbao
committed
test/docs(fullhistory/streaming): tx-hash format alignment + perf expectations
chowbao
committed
refactor(fullhistory/streaming): rename lfs->ledgers, catch_up->backfill, meta_store->catalog (design c586667a)
chowbao
committed
refactor(fullhistory/streaming): align execution layer to design c586667a
chowbao
committed
test(fullhistory/streaming): address review -- preserve assertion intent
chowbao
committed
fix(fullhistory/streaming): join lifecycle goroutine per supervise iteration; widen e2e boundary-cross budget
chowbao
committed
docs(fullhistory/streaming): address review -- comment accuracy + widen clean-shutdown budget
chowbao
committed
docs(full-history): add implementation issue breakdown + status traceability (reference)
chowbao
committed

Commits on Jun 19, 2026