tune reservoir async buffer defaults per storage engine

[Polliog]

Context

The async buffer introduced in #205 (@logtide/reservoir buffered submodule) ships with a single set of defaults tuned empirically for TimescaleDB:

• RESERVOIR_BUFFER_SHARDS=8
• RESERVOIR_BUFFER_MAX_BATCH_SIZE=500
• RESERVOIR_BUFFER_MAX_BATCH_AGE_MS=1000

These produce excellent results on Timescale (p95 drops from 2471 ms to 24 ms on the in-memory transport) but regress on out-of-process engines.

Benchmark evidence

k6, 100 req/s constant arrival rate for 3 min, batch size 10:

Engine	Buffer OFF	Buffer ON memory	Buffer ON redis
TimescaleDB	p95 2471 ms	p95 24 ms	p95 623 ms
ClickHouse	p95 3305 ms	p95 4129 ms	p95 6078 ms
MongoDB	p95 675 ms	p95 844 ms	p95 756 ms

On ClickHouse and MongoDB the buffer loses; the bottleneck is the flush side (HTTP INSERT for CH, bulkWrite for Mongo), which the current defaults hammer with 8 concurrent consumers doing 10-record batches. Storage-engine-native batching prefers fewer, larger inserts.

Proposal

Define engine-specific defaults applied automatically when the backend detects STORAGE_ENGINE. Starting point for discussion:

Engine	Shards	Max batch size	Max batch age	Rationale
timescale	8	500	1000 ms	current defaults, proven on in-process pg pool
clickhouse	2	5000	2000 ms	favor wide INSERTs, let ClickHouse's MergeTree merge
mongodb	4	1000	1500 ms	mongo driver bulkWrite is efficient at this size

Implementation sketch: a getDefaultFlushConfig(engine: EngineType) helper in the backend's reservoir.ts that seeds the config passed to ReservoirBuffered, with env vars still overriding when set.

Acceptance criteria

• Per-engine defaults defined and documented in docs/async-buffer/
• Env vars continue to override per-engine defaults
• Re-run buffered-vs-sync.js on all three engines and update the benchmark table in the docs; the ClickHouse and MongoDB numbers must not regress vs current buffer-OFF baseline (they should ideally improve)
• No change in test coverage requirements
• Startup warning from Reservoir async buffer #205 stays in place until the ClickHouse/MongoDB numbers actually win

Related

• PR Reservoir async buffer #205 (original experiment and current defaults)
• Warning at startup introduced in Reservoir async buffer #205 for non-Timescale engines (remove once this issue is resolved)