engine: VDF reader mis-decodes arrayed standalone graphical-function ("lookup-only") descriptor columns (Ref.vdf)

[bpowers]

Summary

The VDF reader (src/simlin-engine/src/vdf/record_results.rs, src/simlin-engine/src/vdf.rs) mis-decodes the reference columns for arrayed standalone graphical-function ("lookup-only") variables in test/xmutil_test_models/Ref.vdf.

These variables are saved by Vensim only as a graphical-function descriptor record (no separate consumer-owner record). Such a descriptor's f[11] holds a section-6 lookup-record index (a forward link), not an OT-block start. The reader currently emits the f[11]-as-OT-start ghost slot (a class-0x08 stock slot holding 0 / a constant / wrong-magnitude data) instead of following the forward link lookup_record[f[11]].word[10] to the real evaluated-output series.

This is purely a reference-side / test-fixture reader artifact, NOT an engine bug.

Relationship to the landed scalar fix

The scalar subset of this bug was fixed in commit d69754bc ("engine: fix VDF reader decode of standalone graphical-function columns") via the pure functional-core standalone_descriptor_rebinds (follow lookup_record[f[11]].word[10], gated to scalar, non-overlapping descriptors on stock-ghost slots). Proven byte-exact for the scalar case "Global Emissions from graph LOOKUP" -> OT 1612.

The arrayed subset was deliberately deferred in that commit (its message: "an arrayed descriptor re-bound to one forward OT would scalarize and lose its element columns -- deferred"). This issue tracks the deferred arrayed half.

Why the arrayed case is hard (may be infeasible from the VDF alone)

Correctly decoding the arrayed case needs element-order information that the VDF format spec says is NOT stored on disk (docs/design/vdf.md; record_results.rs module docs ~lines 9-15 and ~230-235):

• A single descriptor record maps to a forward-link block of OT columns.
• The block's internal element order does not match the declared dimension order.
• There is an 8-vs-7 slot-count mismatch and a discriminator that is absent from the file.

So a correct general decode may require cross-referencing the model's dimension definitions, or may be infeasible from Ref.vdf alone. This needs a design decision before implementation, not a quick patch.

Affected bases (all proven ENGINE-CORRECT)

For every base below, the engine's gf(Time) output matches the model tables byte-for-byte, and applied consumers match Ref.vdf exactly. The divergence is entirely in the reference-side reader's column decode.

Arrayed (13): historical_forestry_lookup, historical_gdp_lookup, rs_ch4, rs_co2_ff, rs_gdp_in_trillions, rs_hfc125, rs_hfc134a, rs_hfc143a, rs_hfc152a, rs_hfc227ea, rs_hfc23, rs_hfc245ca, rs_hfc32.

Related but distinct sub-case -- 4 SCALAR lookup-only bases where Vensim stores NO distinct saved column matching the engine's lookup-only output (their descriptor forward-link points to Time or a shared/foreign consumer OT). These may be genuinely unrecoverable from Ref.vdf (the data simply is not there), distinct from the arrayed element-order problem:

• ref_global_emissions_from_graph_lookup -- word[10] = 0 / Time
• ozone_precursor_forcings and oc,_bc,_and_bio_aerosol_forcings -- forward-link to a shared "User other forcings" OT differing >1%
• other_forcings_smooth_plus_rcp85 -- shared OT, 131/251 cells match

Impact

These ~17 bases remain in EXPECTED_VDF_RESIDUAL (src/simlin-engine/tests/simulate.rs) as documented VDF-decode-artifact carve-outs, guarded for exactness by the clearn_residual_exactness gate. They are excluded honestly with sourced reasons; the engine is correct. Fixing (or proving infeasible) the reader decode would let the C-LEARN residual carve-out shrink further -- it is currently the dominant remaining contributor to that carve-out.

Components affected

• src/simlin-engine/src/vdf/record_results.rs:
  • identify_descriptor_records (~lines 252-364)
  • the descriptor-overlap gate (~lines 298-357)
  • standalone_descriptor_rebinds (~lines 384-460, currently scalar-only)
• src/simlin-engine/src/vdf.rs: VdfSection6LookupRecord, to_results_via_records
• Format spec: docs/design/vdf.md
• Test fixtures: test/xmutil_test_models/Ref.vdf, test/xmutil_test_models/C-LEARN v77 for Vensim.mdl
• src/simlin-engine/tests/simulate.rs -- EXPECTED_VDF_RESIDUAL / clearn_residual_exactness

Possible approaches

1. Extend standalone_descriptor_rebinds to the arrayed case by mapping a descriptor to its forward-link OT block and recovering per-element columns. Blocked on resolving the element-order ambiguity above -- likely requires cross-referencing the model's dimension definitions, since the discriminator is absent from the file.
2. For the 4 scalar no-distinct-column bases, determine whether the data is recoverable at all; if not, document them as a permanent carve-out (the reference does not contain a matching saved series).
3. Decide explicitly whether a fully general VDF-alone decode is feasible; if not, narrow the goal to "recover what is recoverable; document the rest."

Severity

Test-fixture / reference-reader accuracy. NOT an engine correctness bug -- the engine is verified correct for all affected bases. It limits how tightly the C-LEARN residual gate can be shrunk.

Discovery context

Identified during C-LEARN residual Phase 4 (branch clearn-residual). The scalar half is fixed (d69754bc); this issue tracks the deferred arrayed half plus the 4 no-distinct-column scalar cases.

Relationship to existing issues (DISTINCT)

• engine: C-LEARN data/graph-lookup variables import as 0+0 (zeroed) instead of their lookup/data values #590 (C-LEARN data/graph-lookup vars import as 0+0) frames these same arrayed bases as an engine 0+0-stub import bug with an undetermined root cause (candidate (a) importer defect vs (b) missing external data). This issue corrects that framing: the engine is proven correct (gf(Time) matches the tables byte-for-byte), and the residual is a VDF reference-reader decode artifact in a different component (the VDF reader, not the MDL importer).
• C-LEARN: residual Ref.vdf numeric divergence after element-cycle-resolution (SAMPLE UNTIL / INIT-:NA: / numeric tail / NaN-vs-NA) #591 (C-LEARN residual: SAMPLE UNTIL / INIT-:NA: / numeric tail / NaN-vs-NA) is the umbrella for the other residual clusters; this reader-decode artifact is not among them.
• Tech-debt equation editing improvements #19 (Rust VDF Parser Boundary Parity) is about porting vdf_xray.py boundary cases (empty ref streams, slot-table detection, sparse-block bitmap sizing, raw-zero OT decoding); build(deps): bump xmldom from 0.4.0 to 0.5.0 #33 (VDF 0x53 Result-Family Tail) is a different result-family container. Neither is the standalone graphical-function descriptor forward-link decode for arrayed lookups.