Add streaming precompute engine: sketch-based windowed aggregation over Prometheus/VictoriaMetrics remote write#162

Closed

zzylol wants to merge 38 commits into

mainfrom

156-discussion-requirements-and-design-choices-for-a-sketch-based-streaming-pre-compute

Commits on Mar 9, 2026

Add standalone precompute engine to replace Arroyo streaming pipeline

zzylol
and
claude
committed
Restore standalone precompute engine binary with fixed imports

zzylol
and
claude
committed
Wire up DatasketchesKLL in accumulator factory and add E2E test

zzylol
and
claude
committed
Add raw-only mode to PrecomputeEngine that skips aggregation

zzylol
and
claude
committed
Add debug-level tracing spans for e2e latency measurement in precompute engine

zzylol
and
claude
committed
Add 1000-sample batch latency test to E2E precompute test

zzylol
and
claude
committed
Add 10M-sample throughput test to E2E precompute test

zzylol
and
claude
committed
update
zzylol
committed
Fix sliding window aggregation: feed samples into all overlapping windows

zzylol
and
claude
committed
Add late data handling for closed windows and precompute engine design doc

zzylol
and
claude
committed
Minor wording fix in precompute engine design doc

zzylol
and
claude
committed
Clarify single-machine multi-threaded architecture in design doc

zzylol
and
claude
committed
Add multi-connector ingest support (Prometheus + VictoriaMetrics)

zzylol
and
claude
committed
Run cargo fmt

zzylol
and
claude
committed
Add pane-based incremental sliding window computation

zzylol
and
claude
committed
Add windowed aggregation benchmarks and fix build issues

zzylol
and
claude
committed
Add scalability benchmark varying worker count (1-16)

zzylol
and
claude
committed
Make throughput test multi-threaded for data generation and sending

zzylol
and
claude
committed
Increase throughput test drain timeout from 60s to 120s

zzylol
and
claude
committed
Fix PR405 reorg integration for ingest decode and config APIs
zzylol
committed
Fix precompute engine clippy warnings
zzylol
committed
Fix clippy lints in test_e2e_precompute
zzylol
committed
update
zzylol
committed
Add CapturingOutputSink and worker correctness unit tests

zzylol
and
claude
committed
Update precompute engine design doc

zzylol
and
claude
committed
doc: explain sliding window behaviour with cross-worker GROUP BY

zzylol
and
claude
committed
doc: document accumulator lifecycle and ownership in worker section

zzylol
and
claude
committed
doc: document load balancing trade-offs and alternatives for SeriesRouter

zzylol
and
claude
committed
Fix misleading WindowManager struct comment

zzylol
and
claude
committed
doc: add fault tolerance TODO section; fix rustfmt formatting in worker tests

zzylol
and
claude
committed
Fix Clippy len_without_is_empty warning
zzylol
committed
test: add worker test that loads aggregation config from streaming_config YAML

zzylol
and
claude
committed