compose: FlowKV aged-history compression + drafter residency fix — 1.72x vs disk-cache baseline at <=64K

[dusterbloom]

TL;DR

On current main (RTX 3090 24GB, which includes the PR #364 scoped disk prefix cache), enabling FlowKV aged-history compression on top of the disk cache now delivers:

	main (disk cache alone)	this PR (compose)	delta
7-turn agentic session wall (N=3 mean)	527.5s	306.7s	1.72x
worst-turn fresh prefill (26K tok)	370 tok/s (73-77s)	396 tok/s (66-73s)	parity+
decode @63k context	8.1-9.1 tok/s	10.6-14.5 tok/s	+16-30%
tool-call validity	16/21	18/21	held

Benchmark: goldgate_fix trace (real multi-turn agentic session, 34K-64K prompt tokens per turn), N=3 interleaved A/B on the same binary, same thermal window.

Summary

PR #364 made warm agentic turns cheap by restoring a stable token prefix from disk. The remaining cost on long sessions is the aged conversation history that still has to be prefilled fresh whenever the prefix diverges, and the per-turn growth beyond the cached boundary. This PR composes FlowKV aged-history compression with that cache: messages older than a hot window are compressed (drafter-scored, anchor-preserving) while the system prompt stays verbatim as the cache anchor, so the disk-cache key remains stable across turns. A unified gate keeps the three paths exclusive — turn-1 verbatim, FlowKV on continuations, whole-prompt pFlash otherwise — and with compression disabled the request path is byte-identical to main.

Two fixes found during benchmarking turned the compose from a wash into the 1.72x above:

• Drafter residency. The pflash scoring drafter (~2GB BF16) staying resident through the target's large prefill collapses prefill throughput 370 -> 121 tok/s on 24GB cards (allocator pressure at ~61K KV; verified by A/B, not capacity — the same prefill runs at full rate with the drafter released). The drafter is now released after its scoring pass and lazily reloaded (~2s). This is the auto residency default; --draft-residency persistent keeps the old behavior for >=32GB cards.
• Admission ordering. The ingress context-length check rejected prompt+max_tokens > max_ctx before compression could run. Oversized requests are now admitted when compression will run, and the hard limit is enforced on the post-compress effective size instead.

Changes

• FlowKV aged-history compression composed with the feat(server): add scoped disk prefix cache policy #364 scoped disk cache behind a unified gate (http_server.cpp); compress off keeps main's behavior byte-identical.
• auto draft residency releases the pflash drafter after compress scoring (placement/draft_residency.h).
• Pure admission helper should_reject_oversized() + post-compress effective-size gate (server/admission.h).
• Skip-park guard: --prefill-skip-park downgraded on <32GB GPUs at max_ctx>65536 (VMM VA-fragmentation crash class) (placement/skip_park_guard.h).
• ee7 early-exit drafter, anchor-transitive cascade with expansion throttles, tail-capture guard for the chunk-boundary assert.
• Tests: 1926-assertion unit suite green; standalone suites for admission (7), skip-park guard (6), anchor cascade, early-exit score range, warm-path regression. ~55% of the diff is tests.

Limitations

• Cold >64K prompts at max_ctx=131072 still fail during verbatim-turn prefill (VMM pool growth with target+decode-draft resident; root-caused, follow-up scoped: decode-draft release during large prefills / lazy-KV).
• Compression keeps ~93% on dense code (anchor-dominated) — known, separate lever.
• GGML_CUDA_NO_VMM=1 as an environment variable is a no-op (compile-time option in this fork); scripts relying on it were never protected.

History

1. 731561d1 compose FlowKV with feat(server): add scoped disk prefix cache policy #364 scoped cache; 0efdc33c gate compression as fallback so compose can't regress main; 6a848058 unified gate (FlowKV reachable + scoped save preserved).
2. cefa3caf ee7 early-exit drafter + anchor-transitive cascade + tail-capture guard.
3. 3fc6882f drafter auto-release after compress scoring (the 1.72x).
4. 2ae98c0f compress-aware admission.
5. 637fbdaf comment trim (-133 LOC, no logic changes).
6. 1c562eb4 skip-park footprint guard.

[cubic-dev-ai]

8 issues found across 24 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/server/freeze_history.h">

<violation number="1" location="server/src/server/freeze_history.h:10">
P3: Unused include: `<vector>` is not used by any declaration in this header. Remove it to keep dependencies minimal.</violation>
</file>

<file name="server/src/qwen35/qwen35_backend.cpp">

<violation number="1" location="server/src/qwen35/qwen35_backend.cpp:541">
P2: Skip-park safety guard reads VRAM from current CUDA device instead of the configured placement GPU, making guard decisions incorrect on multi-GPU setups.</violation>
</file>

<file name="server/src/server/http_server.cpp">

<violation number="1" location="server/src/server/http_server.cpp:1904">
P1: FlowKV message-part extraction reads unvalidated JSON as strings; non-string `text` values can throw uncaught exceptions in the worker loop.</violation>
</file>

<file name="server/src/qwen3/anchor_scan.cpp">

<violation number="1" location="server/src/qwen3/anchor_scan.cpp:27">
P1: `search_end` clamping to 0 causes one invalid n-gram comparison when `body_end < ngram`, risking out-of-bounds reads and boundary violations.</violation>

<violation number="2" location="server/src/qwen3/anchor_scan.cpp:103">
P1: Transitive cascade loop exits early due to comparing `forced` against an immediately copied snapshot, so subsequent expansion/rescan iterations are skipped.</violation>
</file>

<file name="server/src/qwen3/anchor_scan.h">

<violation number="1" location="server/src/qwen3/anchor_scan.h:18">
P2: max_forced_count hard cap is checked only inside the cascade loop, but not after pass-1. If pass-1 alone already pushes forced chunks above max_forced_count, the cap is never enforced — the result can exceed the limit.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: search_end clamping to 0 causes one invalid n-gram comparison when body_end < ngram, risking out-of-bounds reads and boundary violations.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen3/anchor_scan.cpp, line 27:

<comment>`search_end` clamping to 0 causes one invalid n-gram comparison when `body_end < ngram`, risking out-of-bounds reads and boundary violations.</comment>

<file context>
@@ -0,0 +1,164 @@
+{
+    const int n_chunks = (int)forced.size();
+    const int ngram    = cfg.ngram;
+    const int search_end = std::max(0, body_end - ngram);
+
+    for (int qi = 0; qi + ngram <= (int)query_pool.size(); ++qi) {
</file context>

[cubic-dev-ai]

P1: Transitive cascade loop exits early due to comparing forced against an immediately copied snapshot, so subsequent expansion/rescan iterations are skipped.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen3/anchor_scan.cpp, line 103:

<comment>Transitive cascade loop exits early due to comparing `forced` against an immediately copied snapshot, so subsequent expansion/rescan iterations are skipped.</comment>

<file context>
@@ -0,0 +1,164 @@
+    // Cascade loop: expand pool with tokens from newly-forced chunks and re-scan.
+    std::vector<uint8_t> prev_forced;
+    for (int it = 0; it < max_iters; ++it) {
+        prev_forced = forced;
+
+        // Rare-token worklist: catches multi-hop cascades within a single outer iteration.
</file context>

[cubic-dev-ai]

P1: FlowKV message-part extraction reads unvalidated JSON as strings; non-string text values can throw uncaught exceptions in the worker loop.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/server/http_server.cpp, line 1904:

<comment>FlowKV message-part extraction reads unvalidated JSON as strings; non-string `text` values can throw uncaught exceptions in the worker loop.</comment>

<file context>
@@ -1798,6 +1808,233 @@ void HttpServer::worker_loop() {
+                                        const std::string ptype = part.value("type", "");
+                                        if (ptype == "text" || ptype == "input_text" ||
+                                            ptype == "output_text")
+                                            msg_content += part.value("text", "");
+                                    }
+                                }
</file context>

Suggested change

	msg_content += part.value("text", "");
	if (part.contains("text") && part["text"].is_string()) msg_content += part["text"].get<std::string>();

[cubic-dev-ai]

P2: Skip-park safety guard reads VRAM from current CUDA device instead of the configured placement GPU, making guard decisions incorrect on multi-GPU setups.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen35/qwen35_backend.cpp, line 541:

<comment>Skip-park safety guard reads VRAM from current CUDA device instead of the configured placement GPU, making guard decisions incorrect on multi-GPU setups.</comment>

<file context>
@@ -534,7 +535,22 @@ bool Qwen35Backend::handle_compress(const std::string & line, const DaemonIO & i
+    {
+        size_t total_vram = 0;
+        int dev = 0;
+        cudaGetDevice(&dev);
+        cudaDeviceProp prop{};
+        if (cudaGetDeviceProperties(&prop, dev) == cudaSuccess)
</file context>

[cubic-dev-ai]

P2: max_forced_count hard cap is checked only inside the cascade loop, but not after pass-1. If pass-1 alone already pushes forced chunks above max_forced_count, the cap is never enforced — the result can exceed the limit.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen3/anchor_scan.h, line 18:

<comment>max_forced_count hard cap is checked only inside the cascade loop, but not after pass-1. If pass-1 alone already pushes forced chunks above max_forced_count, the cap is never enforced — the result can exceed the limit.</comment>

<file context>
@@ -0,0 +1,42 @@
+    int ngram = 4;
+    int rare_token_max_freq = 8;        // tokens appearing <= this many times in body count as rare
+    int cascade_min_anchor_count = 0;   // skip cascade if pass-1 forced >= this many chunks (0 = always cascade)
+    int max_forced_count = INT_MAX;     // hard cap on total forced chunks
+};
+
</file context>

[dusterbloom]

Review disposition (all 8 cubic findings verified against code + tests before fixing):

finding	verdict	action
http_server.cpp:2175 raw size on full-cache hits	confirmed	fixed in e542e90 (effective_prompt_overflows helper, failing-test-first)
disk_prefix_cache.h:50 scope override drops compress	confirmed	fixed in e542e90 (apply_request_scope_override, failing-test-first)
http_server.cpp:1904 JSON throw	refuted	json::value(key, default) is type-safe (returns default on type mismatch, no throw); the .get<std::string>() at :1897 is guarded by is_string()
anchor_scan.cpp:27 / :103 / anchor_scan.h:18	confirmed in the utility library	not reachable from production (the shipping drafter uses its own inline scan; these functions are exercised by tests only). Fixes queued as a follow-up batch rather than blocking this PR
qwen35_backend.cpp:541 current-device VRAM query	partial	latent multi-GPU-only issue; single-GPU is the only shipped config. 1-line fix queued with the follow-up batch
freeze_history.h:10 unused include	confirmed	queued with follow-up batch

Suite after fixes: 1939 assertions green; admission standalone 12/12.

[cubic-dev-ai]

1 issue found across 6 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/qwen35/qwen35_backend.cpp">

<violation number="1" location="server/src/qwen35/qwen35_backend.cpp:541">
P2: Skip-park safety guard reads VRAM from current CUDA device instead of the configured placement GPU, making guard decisions incorrect on multi-GPU setups.</violation>
</file>

<file name="server/src/server/http_server.cpp">

<violation number="1" location="server/src/server/http_server.cpp:1904">
P1: FlowKV message-part extraction reads unvalidated JSON as strings; non-string `text` values can throw uncaught exceptions in the worker loop.</violation>
</file>

<file name="server/src/qwen3/anchor_scan.cpp">

<violation number="1" location="server/src/qwen3/anchor_scan.cpp:27">
P1: `search_end` clamping to 0 causes one invalid n-gram comparison when `body_end < ngram`, risking out-of-bounds reads and boundary violations.</violation>

<violation number="2" location="server/src/qwen3/anchor_scan.cpp:103">
P1: Transitive cascade loop exits early due to comparing `forced` against an immediately copied snapshot, so subsequent expansion/rescan iterations are skipped.</violation>
</file>

<file name="server/src/qwen3/anchor_scan.h">

<violation number="1" location="server/src/qwen3/anchor_scan.h:18">
P2: max_forced_count hard cap is checked only inside the cascade loop, but not after pass-1. If pass-1 alone already pushes forced chunks above max_forced_count, the cap is never enforced — the result can exceed the limit.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[dusterbloom]

cubic P2 (http_server.cpp:1798 sentinel conflation): a zero-length full-cache entry is not constructible today (admission floor + kept anchors guarantee non-empty compressed prompts), but the conflation class is now removed outright — sentinel is -1, gate treats >=0 as a hit. Failing-test-first (test_effective_overflows_zero_length_hit_is_a_hit), suite 13/13 + 1939 asserts green. Fixed in 26e0ee3.