engine: metasd macro corpus SIMULATION tier is empty -- enable it (Vensim DSS reference outputs starting with Theil_2011 + per-model non-macro blocker triage)

[bpowers]

Problem

Phase 7 Task 2 (macros.AC6.4) of the Vensim macro support epic added a tiered metasd macro corpus harness (src/simlin-engine/tests/metasd_macros.rs) over the 17 macro-using .mdl files / 14 directories under test/metasd/.

• The expansion tier (the in-scope macro validation: every model expands its macros with no macro-attributable diagnostic) is fully green for all 17 models after the in-flight delayn fix. The one genuine macro-attributable failure (thyroid-2008-d.mdl, a engine: macro whose body wraps a same-canonical-name intrinsic (INIT = INITIAL(x)) causes false recursion cycle, blocking C-LEARN macro expansion #554-class delayn -> delayn false-positive recursion) is explicitly documented as KNOWN_MACRO_BUG in the harness and asserted by an inverse guard test; it is a engine: macro whose body wraps a same-canonical-name intrinsic (INIT = INITIAL(x)) causes false recursion cycle, blocking C-LEARN macro expansion #554 follow-up and out of scope here.
• The simulation tier (full VM run compared to a Vensim DSS reference output) currently has ZERO eligible models. SimTier::Eligible is intentionally unconstructed and metasd_simulation_tier runs zero models. This is by design and self-documented in the harness, but the deferred enablement should be tracked so the corpus simulation validation can be completed later.

The simulation tier is empty for two non-macro reasons, annotated per-model in CORPUS (metasd_macros.rs):

1. Missing checked-in Vensim reference outputs (a documented prerequisite)

This is a documented setup task ("Test prerequisites" note in the macro design; docs/implementation-plans/2026-05-13-macros/phase_07.md line 108, test-requirements.md line 303), not engine work. Most metasd macro directories ship only .mdl + PROVENANCE.md with no sibling reference.

The cleanest case: test/metasd/theil-statistics/Theil_2011.mdl COMPILES AND SIMULATES with ZERO errors (the THEIL multi-output macro materializes + simulates -- pinned end-to-end by simulate.rs::corpus_theil_multi_output_materializes_and_simulates). It is not simulation-tier-eligible only because no sibling Vensim reference output (.vdf / output.tab / .dat) is checked in. Authoring/obtaining a Vensim DSS reference output for Theil_2011 would immediately unlock the simulation tier for it (a one-line CORPUS SimTier::Eligible edit -- the harness machinery is already in place).

2. Unrelated, non-macro blockers on the other models

Each is annotated in CORPUS and is NOT a macro defect (the corresponding macro expands cleanly in every case):

• FREE/FREE6/FREE6-original/free 6.mdl -- heavy non-macro MDL-parse / dimension errors (ExtraToken / UnrecognizedToken / MismatchedDimensions / ArrayReferenceNeedsExplicitSubscripts) on main-model variables, despite a sibling all_data2.vdf.
• social-network-valuation/groupon {1,2,3}.mdl -- data_* variables are unresolved external data (EmptyEquation / UnrecognizedToken) despite sibling .vdf files.
• covid19-us-homer/homer v8/Covid19US v8.mdl -- *_data variables are unresolved GET DIRECT / GET XLS DATA references (no DataProvider supplied). Already tracked as engine: metasd COVID model not simulation-tier-eligible -- unsatisfied GET DIRECT/GET XLS DATA external data blocks full SSTATS reference validation (Phase 7 corpus scope) #555 -- cross-reference, do not duplicate.
• scientific-revolution/scirev{7,8}.mdl, wonderland/Wonderland3.mdl, industrial-dynamics/IDch15/IDch15d.mdl, interpolating-arrays/InterpolatingArrays.mdl -- main-model UnknownBuiltin / UnknownDependency / UnrecognizedToken / ExtraToken / CantSubscriptScalar / Generic errors.
• pink-noise/PinkNoise2010.mdl, bathtub-statistics/integration3.mdl, critical-slowing/critical-slowing.mdl, early-warnings-catastrophe/catastropeWarning2.mdl, beer-game/RealBeer4-Sterman13.mdl -- the pink_noise macro body uses the engine-unsupported RANDOM NORMAL builtin (UnknownBuiltin); beer-game additionally has a peak -> peak model-logic CircularDependency and main-model dimension errors. The RANDOM NORMAL engine gap is referenced inline in the harness but has no standalone tracking issue today; it is the dominant blocker for the pink_noise-family models.

Why it matters

The macro EXPANSION tier (the in-scope macro validation for the epic) is fully green; this is purely the deferred simulation-tier enablement. The metasd macro simulation tier is intentionally empty until these non-macro prerequisites are met. Without explicit tracking, the corpus's most thorough validation -- end-to-end numeric comparison of macro-using real-world models against Vensim DSS output -- silently never happens, and the "those without unrelated blockers match Vensim DSS reference output" half of macros.AC6.4 stays unrealized with no follow-up trail.

Components affected

• src/simlin-engine/tests/metasd_macros.rs (the tiered corpus harness; CORPUS annotations, SimTier::Eligible currently unconstructed, metasd_simulation_tier runs zero models)
• test/metasd/** (missing checked-in Vensim reference outputs, esp. theil-statistics/)
• src/simlin-engine engine gaps for the per-model blockers (MDL parser/dimension handling, external-data resolution, RANDOM NORMAL builtin)

Recommended remediation (not to perform now)

1. Author/obtain Vensim DSS reference outputs for the clean-compiling metasd macro models, starting with Theil_2011.mdl (it is already green end-to-end; adding its reference is a one-line CORPUS SimTier::Eligible edit and immediately yields the first simulation-tier-validated macro corpus model).
2. Triage the per-model unrelated non-macro blockers, cross-referencing existing issues rather than duplicating:
   • COVID GET DIRECT / GET XLS DATA: track under engine: metasd COVID model not simulation-tier-eligible -- unsatisfied GET DIRECT/GET XLS DATA external data blocks full SSTATS reference validation (Phase 7 corpus scope) #555.
   • delayn -> delayn macro false recursion (thyroid-2008-d): engine: macro whose body wraps a same-canonical-name intrinsic (INIT = INITIAL(x)) causes false recursion cycle, blocking C-LEARN macro expansion #554 follow-up (already the KNOWN_MACRO_BUG).
   • RANDOM NORMAL engine gap (pink_noise family): a separate engine-feature gap -- file/locate a dedicated issue and cross-reference rather than duplicating here.
   • The remaining MDL-parse / dimension / unresolved-external-data blockers (FREE, groupon, scirev, wonderland, IDch15, InterpolatingArrays): triage individually.
3. As references + fixes land, promote each model into the simulation tier by flipping its CORPUS entry from SimTier::Skip(reason) to SimTier::Eligible { reference } (the harness metasd_simulation_tier machinery already consumes it -- no new harness code needed).

Context

Identified during Phase 7 Task 2 of the Vensim macro support epic (docs/implementation-plans/2026-05-13-macros/), while building the tiered metasd macro corpus harness. The simulation-tier emptiness is by design (documented in the harness module docs and per-model CORPUS annotations) and is explicitly deferred per the design's "Test prerequisites" note (phase_07.md line 108 directs filing the unrelated blockers and annotating non-eligible models). This umbrella issue is broader than #555 (which scopes only the single COVID model / SSTATS validation): it tracks the overall deferred enablement -- the missing reference outputs (Theil_2011 first), the cross-referenced per-model blockers, and the promotion path -- so the metasd macro simulation validation is visibly tracked rather than silently empty.

Related: #555 (COVID GET DIRECT/GET XLS DATA, subset of blocker 2), #554 (INIT/recursion false-positive class; the thyroid delayn KNOWN_MACRO_BUG is a follow-up), #349 (C-LEARN macro expansion blocks 1 simulation test).

[bpowers]

Broader follow-up: no standing numeric-fidelity oracle/policy for Vensim-imported models

Filing this as a comment on the natural umbrella issue (per a macro-epic retrospective) rather than a duplicate, since #561 already tracks the specific metasd simulation-tier prerequisite. The broader, recurring problem:

There is no standing policy or CI-available oracle for numeric-fidelity validation of Vensim-imported models. Because no licensed Vensim DSS is available to generate ground-truth .vdf/output.tab series, every Vensim-compat epic re-litigates this and ends up down-scoping acceptance criteria. Concretely, in the just-completed macro epic:

• The metasd simulation-tier eligible-set is empty (this issue, engine: metasd macro corpus SIMULATION tier is empty -- enable it (Vensim DSS reference outputs starting with Theil_2011 + per-model non-macro blocker triage) #561).
• macros.AC6.3 was gated against formula-derived output.tab arithmetic (hand-computed from the verbatim macro body + the engine's own STEP/RAMP/forward-Euler semantics) rather than DSS-validated series — see the C-LEARN-derived SAMPLE UNTIL fixture rationale in commit 7cba4ad2.

This is a policy decision, not a code defect, and it is broader than #561 (which scopes the metasd corpus specifically). Options to decide between (enumerated here so a future session need not re-derive them):

• (a) A one-time licensed-Vensim-DSS run to bank a reference .vdf corpus that is then checked in and reused indefinitely (Theil_2011 is the cheapest first target per engine: metasd macro corpus SIMULATION tier is empty -- enable it (Vensim DSS reference outputs starting with Theil_2011 + per-model non-macro blocker triage) #561 — already green end-to-end, one-line CORPUS edit once a reference exists).
• (b) A CI-available alternative oracle (e.g. PySD) with documented fidelity caveats (integration-method and builtin-coverage differences vs. Vensim DSS must be stated).
• (c) A formal written policy that accepts formula-derived references with a stated precision bound, so future epics cite the policy instead of silently down-scoping ACs per-epic.

Recommend resolving with an explicit decision among (a)/(b)/(c) and recording it in the repo (e.g. docs/dev/ or the test-models README) so it is durable across epics.

Context: surfaced during the Vensim macro epic retrospective (PR #564, branch macros). Not requesting any change now — tracking the policy gap so it stops being re-litigated.