diff --git a/docs/README.md b/docs/README.md index 2364a21c8..472210613 100644 --- a/docs/README.md +++ b/docs/README.md @@ -27,11 +27,14 @@ - [design-plans/2026-05-06-ltm-482-variable-level-loop-enumeration.md](design-plans/2026-05-06-ltm-482-variable-level-loop-enumeration.md) -- Tiered LTM loop enumeration: variable-level Johnson first, expand only the cross-element subgraph - [design-plans/2026-05-09-ltm-503-cross-element-agg.md](design-plans/2026-05-09-ltm-503-cross-element-agg.md) -- Cross-element LTM scoring: per-element arrayed-target partials, element-level cross-element loops, array reducers as aggregate nodes - [design-plans/2026-05-11-ltm-arrays-hardening.md](design-plans/2026-05-11-ltm-arrays-hardening.md) -- Arrayed/cross-element LTM hardening: unify the reference-site walkers behind one classification IR (#520), then layer eight fixes (#487, #511, #510, #514, #515, #483, #502, #492) + - [design-plans/2026-05-13-macros.md](design-plans/2026-05-13-macros.md) -- Vensim macro support: macros as a data-driven generalization of the stdlib module mechanism, persisted via a `MacroSpec` marker on `Model`; 7 implementation phases - [plans/](plans/README.md) -- Implementation plans (active and completed) - [test-plans/](test-plans/) -- Human verification plans for completed features - [implementation-plans/](implementation-plans/) -- Detailed phase-by-phase implementation plans - [implementation-plans/2026-04-05-server-rewrite/](implementation-plans/2026-04-05-server-rewrite/) -- 8-phase plan to build `simlin-serve` and refactor `simlin-mcp` into a shared core library - [test-requirements.md](implementation-plans/2026-04-05-server-rewrite/test-requirements.md) -- AC-to-test mapping for execution validation + - [implementation-plans/2026-05-13-macros/](implementation-plans/2026-05-13-macros/) -- 7-phase plan implementing Vensim `:MACRO:` support: datamodel/serialization foundation, MDL & XMILE import/export, compile-time expansion, multi-output materialization, and hero-corpus validation + - [test-requirements.md](implementation-plans/2026-05-13-macros/test-requirements.md) -- AC-to-test mapping for execution validation ## Security @@ -40,6 +43,7 @@ ## Domain Knowledge - [reference/xmile-v1.0.html](reference/xmile-v1.0.html) -- XMILE interchange format specification +- [reference/vensim-macros.md](reference/vensim-macros.md) -- Vensim macros (`:MACRO:`): definition/call syntax, semantics (per-invocation stock state, locality, recursion), XMILE `` representation, xmutil's mapping, and implementation implications - [reference/ltm--loops-that-matter.md](reference/ltm--loops-that-matter.md) -- Loops That Matter technique: link scores, loop scores, algorithm reference - [array-design.md](array-design.md) -- Array/subscript design notes diff --git a/docs/design-plans/2026-05-13-macros.md b/docs/design-plans/2026-05-13-macros.md new file mode 100644 index 000000000..492f6f3fa --- /dev/null +++ b/docs/design-plans/2026-05-13-macros.md @@ -0,0 +1,262 @@ +# Vensim Macro Support Design + +## Summary + +Vensim's `.mdl` format and the XMILE standard both let modelers define **macros** -- reusable, parameterized templates of equations (including stocks and flows) that other parts of a model invoke like function calls. Today Simlin's engine parses these definitions and then throws them away: a macro-bearing model loses its macros on import, and the writers refuse to emit them. This design makes macros a first-class, persistent concept that survives the full import -> store -> simulate -> export lifecycle. + +The key insight is that Simlin already has a macro system and doesn't know it. The engine's stock-and-flow builtins (`SMTH1`, `DELAY3`, `TREND`, `NPV`) are implemented as tiny models that the compiler instantiates as **modules** -- self-contained sub-model instances with their own state -- whenever it sees the corresponding function-call syntax. This design generalizes that machinery from a fixed, compiled-in standard library to a per-project, data-driven registry. A macro definition becomes an ordinary model carrying one new marker field, a `MacroSpec`, that records its parameters and outputs. A single-output macro invocation needs no new representation at all: it stays as plain function-call equation text (`y = MYMACRO(a, b)`), exactly like a builtin call, and the synthetic module that runs it is created only at compile time. Only Vensim's multi-output `:` call syntax -- which returns several named values at once and so can't be expressed as equation text -- is materialized explicitly at import. The simulation compiler and VM are left untouched, because they already handle modules generically: per-instance state, recursive nesting, arrayed (apply-to-all) unrolling, and global-time access all work for free. The work is therefore concentrated in the front end -- parsing, the import/export converters, and a unified call-name resolver -- plus the datamodel and protobuf changes needed to persist the new marker. + +## Definition of Done + +Vensim macros become a first-class, persistent concept in the Simlin engine -- instead of being silently parsed and discarded as they are today. + +1. **Parsing & representation.** The engine parses the full Vensim `:MACRO:` syntax -- including the `:`-separated multi-output form that no parser handles today -- and the XMILE `` element. Macros are represented in the datamodel and protobuf so macro identity persists across import -> storage -> export: a macro-bearing model survives a round-trip with its macros intact, and the MDL and XMILE writers emit macros instead of rejecting them. + +2. **Simulation.** Models that invoke macros simulate with faithful Vensim semantics: per-invocation independent stock state, positional argument binding, multi-equation bodies with macro-local helper variables, arrayed and expression-valued arguments, `$`-suffixed global-time access, multiple outputs, and macro-to-macro nesting. A macro whose name shadows a builtin resolves to the macro, not the builtin. Recursive macros -- which Vensim's definition-before-use rule effectively precludes and no test model exercises -- are resolved in the design as either a supported case or an explicit, documented limitation. + +3. **Validation.** The 6 existing `test/test-models/tests/macro_*` fixtures are wired into the active test suite and pass; C-LEARN's three invoked macros (`SAMPLE UNTIL`, `SSHAPE`, `RAMP FROM TO`) compute correct values against Vensim DSS reference output; and the macro-using metasd models validate against Vensim DSS reference outputs. + +**Out of scope.** C-LEARN's non-macro blockers (circular dependencies, dimension mismatches, unit errors) and full end-to-end C-LEARN simulation -- tracked separately. Native authoring or interactive editing of macros in the Simlin UI. Fixing xmutil's C++ macro parser -- the native engine MDL parser is the path forward. (The diagram must still open macro-bearing models without crashing -- that floor is in scope -- but dedicated macro visualization or editing UX is not.) + +## Acceptance Criteria + +### macros.AC1: Macro definitions parse and represent faithfully +- **macros.AC1.1 Success:** A `.mdl` `:MACRO:` block imports as a macro-marked `Model` whose `MacroSpec.parameters` matches the header's input list in order, with body variables matching the body equations. +- **macros.AC1.2 Success:** A `:MACRO:` header with a `:` output list (`add3(a,b,c : minval, maxval)`) imports with `MacroSpec.additional_outputs` populated in order. +- **macros.AC1.3 Success:** An XMILE `` element imports as a macro-marked `Model`; an expression-form `` is normalized into a macro-named body variable. +- **macros.AC1.4 Success:** A macro-bearing project round-trips losslessly through protobuf and through JSON -- `MacroSpec` and macro body are identical after deserialize. +- **macros.AC1.5 Success:** `$`-suffixed time references in a macro body (`TIME STEP$`, `Time$`) import as the canonical time identifiers. +- **macros.AC1.6 Failure:** A `:MACRO:` block with no `:END OF MACRO:` reports a clear parse error. +- **macros.AC1.7 Edge:** A macro defined but never invoked (C-LEARN's `INIT`) imports as a valid macro-marked model and is preserved. + +### macros.AC2: Single-output invocations simulate with correct Vensim semantics +- **macros.AC2.1 Success:** A stockless single-output macro invocation (`macro_expression`) simulates and matches the fixture's `output.tab`. +- **macros.AC2.2 Success:** A stock-bearing macro invocation (`macro_stock`) simulates with correct per-invocation integration, matching the 11-step `output.tab`. +- **macros.AC2.3 Success:** The same macro invoked at multiple call sites produces independent per-invocation state -- two stock-bearing invocations don't share a stock. +- **macros.AC2.4 Success:** A macro invoked with an expression-valued argument (`MYMACRO(a + b, t)`) simulates correctly, with the argument evaluated in the caller's context. +- **macros.AC2.5 Success:** A multi-equation macro body with macro-local helpers (`macro_multi_expression`) simulates correctly; helper names don't leak into the caller's namespace. +- **macros.AC2.6 Success:** A macro that calls another macro (`macro_cross_reference`) expands recursively and simulates correctly. +- **macros.AC2.7 Success:** A macro body referencing global time via the `$` escape simulates with the global time values. +- **macros.AC2.8 Edge:** A macro invocation nested inside a larger expression (`y = c + MYMACRO(x, t)`) expands and simulates correctly. + +### macros.AC3: Multi-output and arrayed invocation +- **macros.AC3.1 Success:** A multi-output invocation (`total = add3(a,b,c : minv, maxv)`) materializes as a module instance; `total` receives the primary output and `minv`/`maxv` become model variables holding the additional outputs. +- **macros.AC3.2 Success:** `minv` and `maxv` are referenceable by subsequent equations and carry the correct values. +- **macros.AC3.3 Success:** The metasd multi-output models (`THEIL`, `SSTATS`) expand and simulate. +- **macros.AC3.4 Success:** An arrayed invocation (`y[Dim] = MYMACRO(x[Dim], …)`) expands into one independent macro instance per dimension element. +- **macros.AC3.5 Edge:** An arrayed invocation of a stock-bearing macro gives each element its own persistent stock. + +### macros.AC4: Round-trip and export +- **macros.AC4.1 Success:** A macro-bearing `.mdl` file round-trips with definitions emitted as `:MACRO:` blocks and invocations preserved. +- **macros.AC4.2 Success:** A macro-bearing XMILE file round-trips with `` elements and `simlin:` extensions; the `` header option is emitted. +- **macros.AC4.3 Success:** A multi-output macro round-trips through `.mdl` with the `:` call syntax reconstructed. +- **macros.AC4.4 Success:** A cross-format conversion (`.mdl` → datamodel → `.xmile`) preserves macro definitions and invocations. +- **macros.AC4.5 Edge:** A single-output-only model exports as standards-clean XMILE with no extensions; multi-output triggers the `simlin:` extension. + +### macros.AC5: Error handling and edge cases +- **macros.AC5.1 Failure:** A macro invoked with the wrong number of arguments reports an arity-mismatch diagnostic naming the macro. +- **macros.AC5.2 Failure:** A directly or mutually recursive macro is rejected with a cycle-detection error rather than expanding without termination. +- **macros.AC5.3 Failure:** Two macros sharing a name, or a macro name colliding with a model name, report a registry-build diagnostic. +- **macros.AC5.4 Success:** A macro shadowing a builtin (`SSHAPE`, `RAMP FROM TO`) resolves to the macro; the builtin is not invoked. +- **macros.AC5.5 Success:** A macro defined after its first use (`macro_trailing_definition`) still resolves and simulates. +- **macros.AC5.6 Failure:** A call to a name that is neither a macro, a stdlib function, nor a builtin reports an "unknown function or macro" error. + +### macros.AC6: Validation corpus and consumer floor +- **macros.AC6.1 Success:** All six `test/test-models/tests/macro_*` fixtures are wired into the active test suite and pass. +- **macros.AC6.2 Success:** C-LEARN's macros (`SAMPLE UNTIL`, `SSHAPE`, `RAMP FROM TO`, `INIT`) parse, register, and expand with no macro-specific errors. +- **macros.AC6.3 Success:** Focused models invoking C-LEARN's `SAMPLE UNTIL`, `SSHAPE`, and `RAMP FROM TO` with known inputs match Vensim DSS reference output. +- **macros.AC6.4 Success:** All 14 macro-using metasd models pass the expansion tier; those without unrelated blockers match Vensim DSS reference output. +- **macros.AC6.5 Success:** The diagram opens every macro-bearing fixture without crashing. +- **macros.AC6.6 Success:** Macro-marked models don't appear as standalone, navigable models in the diagram's model list. + +## Glossary + +- **Vensim**: A widely used commercial system dynamics modeling tool; its native model format is the `.mdl` text format. "Vensim DSS" is the professional edition, used here to generate reference simulation outputs. +- **XMILE**: The open XML standard for interchanging system dynamics models (`docs/reference/xmile-v1.0.html`); Simlin reads and writes it alongside `.mdl`. +- **Macro**: A reusable, parameterized template of equations -- potentially including stocks and flows -- that a model invokes like a function call. Defined in `.mdl` with a `:MACRO: ... :END OF MACRO:` block or in XMILE with a `` element. +- **Stock / flow / aux**: The three core system dynamics variable kinds. A stock accumulates over time (integrates its flows); a flow is a rate that changes a stock; an aux (auxiliary) is a computed intermediate value. In the engine these are the variants of `datamodel::Variable`. +- **Module**: A self-contained instance of a sub-model embedded in a larger model, with its own independent per-instance state. Represented by `Variable::Module` and `ModuleReference`. Macros expand into modules at compile time. +- **Stdlib (standard library)**: The engine's built-in stock-and-flow functions (`SMTH1`, `DELAY3`, `TREND`, `NPV`), each implemented as a small model (`stdlib/*.stmx`, transpiled to `stdlib.gen.rs`) and instantiated as a module by the compiler. Macros generalize exactly this mechanism. +- **Builtin**: A function recognized natively by the engine. A "module-function" builtin (the stdlib functions above) expands into a module instance; this design adds a rule that a project macro shadows a builtin of the same name. +- **`BuiltinVisitor`**: The compiler pass (`src/simlin-engine/src/builtins_visitor.rs`) that walks equation ASTs, recognizes function-call syntax for module-functions, and rewrites each call into a synthetic module instance plus hoisted argument variables. +- **MacroSpec**: The one new datamodel field this design adds to `Model`. It marks a model as a callable macro template and records its formal `parameters`, `primary_output`, and `:`-list `additional_outputs`. +- **Datamodel**: Simlin's in-memory, serializable representation of a project, model, and variables (`src/simlin-engine/src/datamodel.rs`), independent of any file format. Both `.mdl` and XMILE import to and export from it. +- **Apply-to-all**: A system dynamics construct where one equation applies element-wise across every element of a dimension (subscript). The engine "unrolls" it into per-element computation; arrayed macro invocation rides this existing path. +- **Multi-output (`:`) call syntax**: Vensim's macro-call form `total = add3(a, b, c : minv, maxv)`, where outputs after the `:` bind additional named values. It can't be expressed as plain equation text, so it is materialized as an explicit module plus binding variables. +- **`$`-time escape**: Vensim macro-body syntax (e.g. `Time$`, `TIME STEP$`) that reaches the caller's global time variables instead of any macro-local shadow. It needs only a front-end translation to canonical time identifiers because the VM already resolves global time at any module nesting depth. +- **C-LEARN**: The "hero" model driving this work -- a real macro-using Vensim model whose macros (`SAMPLE UNTIL`, `SSHAPE`, `RAMP FROM TO`, `INIT`) the implementation must parse, expand, and simulate correctly. +- **metasd corpus**: A collection of 14 macro-using community models used as a broader validation corpus beyond the hero model and the bundled fixtures. +- **`test/test-models/`**: The integration-test directory of model files paired with expected simulation outputs (`output.tab`); the six `macro_*` fixtures live here. +- **Round-trip**: Importing a model to the datamodel and exporting it back out without losing information -- a core correctness property for macro persistence. +- **Protobuf**: Protocol Buffers, the binary serialization format Simlin uses for persisted projects (`project_io.proto`). Changes must be additive for backward compatibility because serialized instances exist in a database. +- **serde**: The Rust serialization framework; here, specifically the conversion layer (`src/simlin-engine/src/serde.rs`) between the datamodel and the protobuf representation. +- **AST**: Abstract syntax tree -- the parsed, structured form of an equation that the parser produces and `BuiltinVisitor` rewrites. +- **Arity**: The number of arguments a macro or function expects; an "arity-mismatch" diagnostic is reported when a call passes the wrong count. +- **xmutil**: Simlin's existing C++-derived Vensim-to-XMILE converter (test-only); its macro parser is explicitly not the path forward -- the native Rust MDL parser is. +- **`$⁚` prefix**: The naming convention `BuiltinVisitor` uses for compiler-generated module instances and hoisted argument variables, which are never serialized. + +## Architecture + +Macros reuse Simlin's existing module machinery instead of adding a new primitive. The engine already implements its stock-and-flow builtins -- `SMTH1`, `DELAY3`, `TREND`, `NPV` -- as small models that the compiler instantiates as modules when it sees function-call syntax. That mechanism is already a macro system. This design generalizes it from a fixed, build-time stdlib registry to a per-project, data-driven one. + +The representation has two asymmetric halves. + +A **macro definition** becomes an ordinary `datamodel::Model` -- its body stocks, flows, and auxes are just `Model.variables` -- plus one new optional field, a `MacroSpec`: + +```rust +struct MacroSpec { + parameters: Vec, // formal parameters, in positional calling order + primary_output: String, // body variable the call-site LHS receives + additional_outputs: Vec, // the ':'-list outputs, usually empty +} +``` + +Macro-definition models live in the existing flat `project.models` list, distinguished only by carrying a `MacroSpec`. This mirrors how stdlib models are already just entries in that list. + +A **macro invocation**, in the common case, is nothing but function-call equation text. `y = MYMACRO(a, b)` is stored as an ordinary `Aux`/`Stock`/`Flow` equation, exactly like `y = SMTH1(x, 5)` today. It's never materialized in the datamodel; the synthetic module instance that runs it is a compile-time artifact that's never serialized. So a single-output invocation round-trips for free. The exception is Vensim's multi-output `:` syntax -- `total = add3(a, b, c : minv, maxv)` -- where equation-text sugar can't return multiple named values. A multi-output invocation is materialized at import as an explicit `Variable::Module` plus auxiliary variables that read the module's declared outputs. Both forms converge on the same module-instance mechanism at compile time; they differ only in whether the instance is compile-time sugar or materialized at import. + +The components: + +- **Module-function resolver.** Unifies the stdlib lookup and a per-project macro registry. Given a call name, it answers either "not a module-function" or `{ target model, ordered parameter ports, primary output }`. The `BuiltinVisitor` pass and the `model.rs` sites that consult the hardcoded `is_stdlib_module_function`/`stdlib_args` today all route through it. +- **Macro registry.** Built at compile time from the macro-marked models in `project.models`. It carries name-resolution precedence: a project macro shadows a stdlib function or builtin of the same name. This is Vensim's rule, and it fixes a current silent-wrong-result bug where C-LEARN's `RAMP FROM TO` resolves to an arity-compatible builtin. +- **Compile-time expansion.** `BuiltinVisitor` turns `MYMACRO(a, b)` into a synthetic module instance, hoists expression-valued arguments into synthetic auxes, and replaces the call with a reference to the instance's output. This is the existing stdlib transformation, reused unchanged -- only the lookup is generalized. +- **Import and export.** `mdl/convert` builds macro-marked models from the already-parsed `MacroDef` AST; the empty `xmile::Macro` stub is fleshed out. The MDL and XMILE writers emit `:MACRO:` blocks and `` elements. + +System boundaries: the datamodel gains exactly one optional field on `Model`; `Variable::Module` and `Project` are unchanged. The simulation compiler and VM are unchanged -- they already handle modules generically, with per-instance state, recursive nesting, and apply-to-all unrolling for arrayed invocation. Global time (`time`, `dt`, `initial_time`, `final_time`) resolves inside any module at any nesting depth through absolute VM slots, so Vensim's `$`-time escape needs only a front-end translation, not an engine change. + +## Existing Patterns + +This design follows the engine's existing stdlib mechanism closely. It doesn't introduce a new architectural pattern. + +- **Stdlib-as-modules.** `SMTH1`, `DELAY3`, `TREND`, and `NPV` are defined as small models (`stdlib/*.stmx`, transpiled to `src/simlin-engine/src/stdlib.gen.rs`) and instantiated as modules by `BuiltinVisitor` (`src/simlin-engine/src/builtins_visitor.rs`) when it sees function-call syntax. Macros generalize exactly this; the structural change is that the function-name/port-name lookup becomes data-driven instead of a hardcoded table. +- **Materializing definition-models into `project.models`.** `Project::ensure_referenced_stdlib_models()` (`src/simlin-engine/src/datamodel.rs`) already copies embedded stdlib models into the flat `project.models` list on demand. Macro-definition models live in that same list, distinguished by their `MacroSpec`. +- **The `$⁚` synthetic-variable convention.** `BuiltinVisitor` already names compiler-generated module instances and hoisted argument auxes with a `$⁚` prefix and never serializes them. Macro expansion reuses this untouched. +- **`Variable::Module`, `ModuleReference`, and the recursive module VM** are reused as-is for per-invocation state, nesting, and arrayed invocation. The hand-maintained `NPV` stdlib model confirms that global time resolves inside a module body. + +The one new element -- the `MacroSpec` marker on `Model` -- has a clear precedent: stdlib models are already "a special kind of model," distinguished today only by a name prefix. This design makes "this model is a callable template" an explicit, serialized property instead of a convention. + +## Implementation Phases + +Seven phases. The ordering front-loads what the C-LEARN hero model needs -- single-output macros with stocks, `$`-time access, and arrayed invocation, across Phases 2-4 -- then follows with the remaining formats and the validation corpus. + + +### Phase 1: Datamodel and serialization foundation +**Goal:** Represent macros in the datamodel and persist them losslessly. + +**Components:** +- `MacroSpec` type and an optional field on `datamodel::Model` (`src/simlin-engine/src/datamodel.rs`) +- An additive `MacroSpec` field on the `Model` protobuf message (`src/simlin-engine/src/project_io.proto`, next free field number) and its serde conversion (`src/simlin-engine/src/serde.rs`) +- The mirrored optional field and JSON converters in `src/core/datamodel.ts`, the `src/engine` JSON types, and `src/pysimlin` dataclasses + +**Dependencies:** None. + +**Covers:** macros.AC1 (representation and round-trip). + +**Done when:** `MacroSpec` round-trips losslessly through protobuf and JSON; types compile across all consumer packages; serde round-trip tests pass. + + + +### Phase 2: MDL parsing and import +**Goal:** Parse the full `:MACRO:` syntax and import macro definitions from `.mdl` files. + +**Components:** +- MDL parser support for the `:` separator in `:MACRO:` headers and in call-site argument lists; the call AST node gains an optional output-binding list (`src/simlin-engine/src/mdl/parser.rs`, `src/simlin-engine/src/mdl/ast.rs`) +- `mdl/convert`: each parsed `MacroDef` becomes a macro-marked `Model`; `$`-suffixed time references translate to canonical time idents; single-output invocations remain as equation text (`src/simlin-engine/src/mdl/convert/`) + +**Dependencies:** Phase 1 (`MacroSpec` to populate). + +**Covers:** macros.AC1 (MDL definitions). + +**Done when:** parser unit tests cover the `:` header and call forms; the six `test/test-models/tests/macro_*` `.mdl` fixtures import into datamodels with correct `MacroSpec`s and macro bodies; `$`-time translation is verified. + + + +### Phase 3: Compile-time expansion and single-output simulation +**Goal:** Resolve and expand macro calls so single-output macros simulate correctly. + +**Components:** +- The module-function resolver, unifying the stdlib lookup and the macro registry (`src/simlin-engine/src/builtins.rs`, `src/simlin-engine/src/builtins_visitor.rs`) +- The generalized `BuiltinVisitor` expansion; the `model.rs` sites (`equation_is_stdlib_call`, `collect_module_idents`) routed through the resolver +- Name-resolution precedence (a macro shadows a stdlib function or builtin), arity-mismatch diagnostics, and recursion cycle detection + +**Dependencies:** Phase 2 (macro-marked models to register and expand). + +**Covers:** macros.AC2 (single-output simulation), macros.AC5 (errors and edge cases). + +**Done when:** the single-output `.mdl` fixtures simulate and match their `output.tab`, wired into `simulate.rs`; new focused fixtures pass for a macro at multiple call sites, expression-valued arguments, `$`-time access, and builtin-name shadowing; resolver, shadowing, arity, and recursion-detection unit tests pass. + + + +### Phase 4: Multi-output and arrayed invocation +**Goal:** Support the `:` multi-output call form and subscripted macro invocation. + +**Components:** +- Import-time materialization of a multi-output invocation as a `Variable::Module` plus binding auxiliary variables that read the module's declared outputs (`src/simlin-engine/src/mdl/convert/`) +- Verification that arrayed invocation (`y[Dim] = MYMACRO(x[Dim], …)`) rides the existing apply-to-all per-element unrolling once the resolver recognizes macro calls + +**Dependencies:** Phase 3 (expansion mechanism), Phase 2 (the `:` parser). + +**Covers:** macros.AC3. + +**Done when:** new multi-output fixtures pass (none exist today); an arrayed-invocation fixture passes; the metasd multi-output models (`THEIL`, `SSTATS`) expand and simulate; C-LEARN's four macros -- including the arrayed `[COP]` `SAMPLE UNTIL` invocations -- expand without macro-specific errors. + + + +### Phase 5: XMILE import and export +**Goal:** Round-trip macros through XMILE, including multi-output via a Simlin extension. + +**Components:** +- A fleshed-out `xmile::Macro` (`name`, `parm`s, `eqn`, `variables`, optional `sim_specs`) plus `simlin:`-namespaced extension elements for additional outputs and multi-output bindings (`src/simlin-engine/src/xmile/mod.rs`) +- XMILE reader producing macro-marked `Model`s; the expression-form `` normalized into a macro-named body variable +- XMILE writer emitting `` elements, the `simlin:` extensions, and the `` header option + +**Dependencies:** Phases 1, 3, and 4. + +**Covers:** macros.AC1 (XMILE definitions), macros.AC4 (XMILE round-trip). + +**Done when:** the `.xmile` macro fixtures import and simulate; XMILE round-trip preserves macro definitions and invocations; a cross-format `.mdl` → datamodel → `.xmile` test preserves them. + + + +### Phase 6: MDL export and round-trip +**Goal:** Emit `:MACRO:` blocks from `.mdl` and close the round-trip loop. + +**Components:** +- MDL writer support: emit each macro-marked model as a `:MACRO: … :END OF MACRO:` block; reconstruct single-output invocations (already equation text) and multi-output invocations (from the materialized module plus its binding auxes) (`src/simlin-engine/src/mdl/writer.rs`) +- Macro models wired into the `mdl_roundtrip.rs` harness, which excludes them today + +**Dependencies:** Phase 2 (import, to round-trip against), Phase 4 (multi-output to reconstruct). + +**Covers:** macros.AC4 (MDL round-trip). + +**Done when:** `.mdl` round-trip preserves every macro fixture; multi-output reconstruction is verified; the `mdl_roundtrip.rs` macro exclusion is removed. + + + +### Phase 7: Hero validation, corpus, and consumer integration +**Goal:** Validate against C-LEARN and the metasd corpus; make consumers macro-aware. + +**Components:** +- C-LEARN macro validation: its macros parse, register, and expand with no macro-specific errors; focused models invoking `SAMPLE UNTIL`, `SSHAPE`, and `RAMP FROM TO` with known inputs, checked against Vensim DSS reference output +- A metasd corpus harness for the 14 macro-using models, tiered: macro expansion succeeds, then full simulation matches Vensim DSS reference output for models without unrelated blockers +- `src/diagram`: filter macro-marked models out of the model-navigation and `getAvailableModels` lists; confirm macro-bearing models open without crashing + +**Dependencies:** Phases 1-6. + +**Covers:** macros.AC6. + +**Done when:** C-LEARN's macros validate against reference output; the corpus harness passes at the expansion tier for all 14 models and at the simulation tier for those without unrelated blockers; the diagram opens every macro fixture without crashing, and macro-marked models don't appear as standalone models in navigation. + + +## Additional Considerations + +**Documented limitations.** Three macro features are deliberately left out of full support, each because no model in the test corpus exercises it: +- *Recursive macros* are rejected with a cycle-detection error rather than supported. Vensim's "definition before use" rule already makes self-recursion inexpressible in `.mdl`; XMILE permits it, but a per-call-site instantiation strategy can't terminate for genuine recursion. +- *XMILE per-macro ``* -- a macro running with its own dt and stop time -- isn't supported. It behaves more like a nested simulation per step than a module. +- *The non-time `$` escape* (`FOO$` for a non-time model variable) is wired as an implicit module input but deprioritized; the corpus only exercises `$`-time, which resolves for free. + +**Definition-order leniency.** Vensim requires a macro to be defined before it's used. The engine collects all top-level items before conversion, so this design is lenient: it accepts the `macro_trailing_definition` fixture and trailing definitions generally. This is a deliberate, compatibility-favoring deviation from Vensim's stated rule. + +**Test prerequisites.** The metasd corpus validation depends on Vensim DSS reference outputs generated for the 14 macro-using models -- a setup task, not implementation work. The `.stmx` variants of the six fixtures have pre-existing, unrelated parse issues and carry no `` element; they're out of scope. + +**Pre-existing writer gaps.** The MDL writer currently writes only the main model and drops all module variables. Phase 6 adds macro-related writer support; a general overhaul of MDL module export is out of scope. Two unrelated serialization gaps found during investigation are tracked separately as GitHub #538 and #539. diff --git a/docs/implementation-plans/2026-05-13-macros/phase_01.md b/docs/implementation-plans/2026-05-13-macros/phase_01.md new file mode 100644 index 000000000..1b882d3be --- /dev/null +++ b/docs/implementation-plans/2026-05-13-macros/phase_01.md @@ -0,0 +1,367 @@ +# Vensim Macro Support — Phase 1: Datamodel and serialization foundation + +**Goal:** Represent macros in the datamodel with a new `MacroSpec` marker and persist it losslessly through protobuf and JSON across every consumer package. + +**Architecture:** Add a `MacroSpec` struct and an `Option` field to the `Model` datamodel type, then mirror it through the five existing serialization layers exactly as the `LoopMetadata` nested-struct precedent does: the Rust protobuf path (`project_io.proto` + `serde.rs`), the Rust JSON path (`json.rs`, which also drives the auto-generated JSON schema), the TypeScript mirrors (`json-types.ts` + `datamodel.ts`), and the Python mirrors (`json_types.py` + `json_converter.py`). No import/export or simulation logic is touched in this phase — only representation and round-trip. + +**Tech Stack:** Rust (prost protobuf codegen, serde, schemars JSON-schema derive), TypeScript (Jest), Python (cattrs, pytest). + +**Scope:** 7 phases from the original design (`docs/design-plans/2026-05-13-macros.md`); this is phase 1 of 7. + +**Codebase verified:** 2026-05-14 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### macros.AC1: Macro definitions parse and represent faithfully +- **macros.AC1.4 Success:** A macro-bearing project round-trips losslessly through protobuf and through JSON -- `MacroSpec` and macro body are identical after deserialize. + +**Note on scope within AC1:** Phase 1 establishes the *representation* and proves the *round-trip* half of `macros.AC1.4`. It does so by constructing `MacroSpec` values directly in test code (not via import). The remaining AC1 cases — `AC1.1`, `AC1.2`, `AC1.5`, `AC1.6`, `AC1.7` (MDL import) and `AC1.3` (XMILE import) — are completed in Phases 2 and 5, which build on the representation this phase adds. In Phase 1 the "macro body" referenced by AC1.4 is simply the model's ordinary `variables`, which already round-trips; the new persisted element is `MacroSpec`. + +--- + +## Background: the two Rust `Model` representations + +There are **two** distinct Rust structs involved, and `MacroSpec` must persist through **both**: + +1. **`datamodel::Model`** (`src/simlin-engine/src/datamodel.rs`) — the in-memory model. `src/simlin-engine/src/serde.rs` converts it to/from **`project_io::Model`** (the prost-generated protobuf type). This is the **protobuf** persistence path (the project DB stores serialized protobuf, so changes must be additive). + +2. **`json::Model`** (`src/simlin-engine/src/json.rs`) — a separate serde struct (`#[serde(rename_all = "camelCase")]`). `json.rs` converts it to/from `datamodel::Model`. `json::Model` is the single source of truth for (a) the TypeScript `JsonModel` shape in `src/engine/src/json-types.ts`, (b) the Python `Model` dataclass in `src/pysimlin/simlin/json_types.py`, and (c) the JSON Schema at `docs/simlin-project.schema.json`, which is **auto-generated** by a test in `json_proptest.rs` (gated `#[cfg(all(test, feature = "schema"))]`; `schema` is a default feature, so plain `cargo test -p simlin-engine` regenerates it). + +**Generated files — never hand-edit:** +- `src/simlin-engine/src/project_io.gen.rs` — regenerate with `pnpm build:gen-protobufs`. +- `docs/simlin-project.schema.json` — regenerates itself when `cargo test -p simlin-engine` runs the `generate_and_write_schema` test. + +**The `LoopMetadata` precedent.** `LoopMetadata` is a nested struct on `Model` that already threads through all five layers. Every task below points at the exact `LoopMetadata` lines to mirror. `MacroSpec` differs from `LoopMetadata` in one way: it is a *singular optional* field (`Option`), not a `Vec`. For the singular-optional shape, the precedent is `Model.sim_specs: Option` (in `datamodel.rs`) and `json::Model.sim_specs` (`#[serde(skip_serializing_if = "Option::is_none", default)]` in `json.rs`). + +The `MacroSpec` shape (from the design plan): + +```rust +pub struct MacroSpec { + pub parameters: Vec, // formal parameters, in positional calling order + pub primary_output: String, // body variable the call-site LHS receives + pub additional_outputs: Vec, // the ':'-list outputs, usually empty +} +``` + +All three fields are `Eq`-able (no `f64`), so `MacroSpec` derives `Eq`. `datamodel::Model` itself derives only `Clone, PartialEq` (it transitively contains `f64`); adding an `Option` field does not change that. + +--- + + +## Subcomponent A: Rust protobuf persistence + + +### Task 1: Define `MacroSpec` and add `Model.macro_spec` + +**Verifies:** None (type definition; the round-trip it enables is tested in Tasks 2-5). + +**Files:** +- Modify: `src/simlin-engine/src/datamodel.rs` (add `MacroSpec` struct near the `Model` struct, ~line 783; add field to `Model`, ~lines 783-792) +- Modify: every other file in the workspace that constructs a `datamodel::Model { ... }` literal — these are discovered by the compiler, not guessed (see Step 2). Known sites include `src/simlin-engine/src/serde.rs` (`From for Model`, ~line 2069), `src/simlin-engine/src/json.rs` (`From for datamodel::Model`, ~line 1185), `src/simlin-engine/src/mdl/convert/variables.rs` (~line 88), the XMILE reader under `src/simlin-engine/src/xmile/`, `src/simlin-engine/src/test_common.rs` / `testutils.rs`, and inline `#[cfg(test)]` fixtures in `serde.rs` and `json.rs`. + +**Implementation:** + +Step 1 — In `datamodel.rs`, immediately before the `pub struct Model` definition, add the `MacroSpec` struct. Mirror the derive attributes of the neighboring `LoopMetadata` struct (`#[cfg_attr(feature = "debug-derive", derive(Debug))]` + `#[derive(Clone, PartialEq, Eq)]`). Give it concise rustdoc explaining that `Some` on `Model.macro_spec` marks the model as a callable macro template, that the model's `variables` are the macro body, and what each field means: + +```rust +/// Marks a [`Model`] as a callable macro template rather than an ordinary +/// model, and records its calling convention. A macro definition is an +/// ordinary model whose `variables` are the macro body; this spec names which +/// body variables are the formal parameters and which are the outputs. +/// `Model.macro_spec` is `None` for every non-macro model. +#[cfg_attr(feature = "debug-derive", derive(Debug))] +#[derive(Clone, PartialEq, Eq)] +pub struct MacroSpec { + /// Formal parameter names, in positional calling order. Each names a body + /// variable that a macro invocation binds an argument to. + pub parameters: Vec, + /// The body variable whose value the call-site left-hand side receives. + pub primary_output: String, + /// Additional named outputs from Vensim's `:`-list multi-output call + /// syntax, in declaration order. Empty for ordinary single-output macros. + pub additional_outputs: Vec, +} +``` + +Step 2 — Add the field to `Model`, after `pub groups: Vec,`: + +```rust + /// `Some` if this model is a callable macro template. See [`MacroSpec`]. + pub macro_spec: Option, +``` + +Step 3 — Make the workspace compile again. Adding a field to `Model` breaks every struct-literal construction of `datamodel::Model`. Run `cargo build --workspace --all-targets` and, for each `error[E0063]: missing field \`macro_spec\``, add `macro_spec: None` to that `Model { ... }` literal. Repeat until the build is clean. Do **not** use `..Default::default()` — `datamodel::Model` does not derive `Default`, and the surrounding code uses explicit field lists. `cargo build --workspace` includes the `pysimlin` Rust crate, so if it constructs any `datamodel::Model` literal the compiler flags it here — there is **no** separate Rust-side `pysimlin` datamodel mirror to hand-edit (pysimlin consumes the engine through the JSON bridge, which Task 5 handles on the Python side). + +**Testing:** None. This task only defines a type and adds a field; no behavior to test yet. The compiler is the verifier. + +**Verification:** +Run: `cargo build --workspace --all-targets` +Expected: compiles with no errors. + +**Commit:** `engine: add MacroSpec type to datamodel` + + + +### Task 2: `MacroSpec` protobuf message and serde conversion + +**Verifies:** macros.AC1.4 (protobuf round-trip). + +**Files:** +- Modify: `src/simlin-engine/src/project_io.proto` (add `message MacroSpec`; add field 7 to `message Model`, ~lines 317-325) +- Regenerate: `src/simlin-engine/src/project_io.gen.rs` (and any TypeScript server protobuf the script touches) — via `pnpm build:gen-protobufs`, never by hand +- Modify: `src/simlin-engine/src/serde.rs` (add `From` impls for `MacroSpec` near the `LoopMetadata` impls ~line 1989; wire `macro_spec` into both `Model` conversion impls ~lines 1951-1987 and ~2069-2091; add a round-trip test near `test_model_with_loop_metadata_roundtrip` ~line 2093) + +**Implementation:** + +Step 1 — In `project_io.proto`, add a top-level `message MacroSpec` next to `message LoopMetadata` (~line 299). Nested messages in this file are declared top-level, not inside `Model`: + +```proto +message MacroSpec { + repeated string parameters = 1; + string primary_output = 2; + repeated string additional_outputs = 3; +} +``` + +Then add a field to `message Model`. Fields 1,3,4,5,6 are in use; field 2 was historically skipped; **use field 7** (the next sequential number — consistent with how this file has grown): + +```proto + MacroSpec macro_spec = 7; +``` + +A singular message field in proto3 is presence-tracked, so prost generates it as `::core::option::Option` — no `optional` keyword needed. Verify this in the regenerated `.gen.rs` in Step 2. + +Step 2 — Regenerate the bindings: `pnpm build:gen-protobufs`. Then run `git status` and stage every file it changed (at minimum `project_io.gen.rs`; the script also runs `pnpm format`). Confirm `project_io.gen.rs` now has `pub macro_spec: ::core::option::Option` on `Model` and a `MacroSpec` message with `#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]`. + +Step 3 — In `serde.rs`, add the two `From` impls converting `datamodel::MacroSpec` ⇄ `project_io::MacroSpec`, mirroring the `LoopMetadata` impls at ~line 1989. The body is a field-by-field move (all three fields have identical names and types on both sides). + +Step 4 — Wire `macro_spec` into the two `Model` conversion impls: +- In `From for project_io::Model` (~lines 1951-1987): add `macro_spec: model.macro_spec.map(project_io::MacroSpec::from),` to the constructed `project_io::Model`. +- In `From for Model` (~lines 2069-2091): change the placeholder `macro_spec: None` (added in Task 1) to `macro_spec: model.macro_spec.map(MacroSpec::from),`. + +**Testing:** +Add `test_model_with_macro_spec_roundtrip` next to `test_model_with_loop_metadata_roundtrip` (`serde.rs` ~line 2093). It must verify **macros.AC1.4 (protobuf half)**: construct a `Model` with `macro_spec: Some(MacroSpec { ... })` populated with non-empty `parameters`, a non-empty `primary_output`, and non-empty `additional_outputs`, plus at least one body `Variable` (so the "macro body" is also exercised). Round-trip it via `Model::from(project_io::Model::from(expected.clone()))` and `assert_eq!`. Follow the exact `cases: &[Model]` + loop shape of the neighboring tests. Also add one case (or a second test) with `macro_spec: None` to confirm a non-macro model still round-trips. + +**Verification:** +Run: `cargo test -p simlin-engine serde` +Expected: all `serde` tests pass, including `test_model_with_macro_spec_roundtrip`. + +**Commit:** `engine: persist MacroSpec through protobuf` + + + + +## Subcomponent B: Rust JSON persistence and schema + + +### Task 3: `json::MacroSpec`, JSON serde, and schema regeneration + +**Verifies:** macros.AC1.4 (JSON round-trip). + +**Files:** +- Modify: `src/simlin-engine/src/json.rs` (add `json::MacroSpec` struct near `json::LoopMetadata` ~line 551; add `macro_spec` field to `json::Model` ~lines 455-481; add `From` impls both ways near the `LoopMetadata` impls ~lines 1263 and ~1921; wire `macro_spec` into `From for datamodel::Model` ~line 1185 and `From for json::Model` ~line 1818; extend the `json::Model` test fixtures and `test_model_roundtrip` ~line 2693) +- Regenerate: `docs/simlin-project.schema.json` — by running `cargo test -p simlin-engine` (the `generate_and_write_schema` test rewrites it), never by hand +- Possibly modify: `src/simlin-engine/src/json_proptest.rs` (only if it has a `json::Model` generation strategy — see Step 5) + +**Implementation:** + +Step 1 — In `json.rs`, add the `json::MacroSpec` struct next to `json::LoopMetadata` (~line 551). Mirror `json::LoopMetadata`'s attributes exactly: `#[cfg_attr(feature = "debug-derive", derive(Debug))]`, `#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]`, `#[cfg_attr(feature = "schema", derive(JsonSchema))]`, `#[serde(rename_all = "camelCase")]`. Use the file's existing `is_empty_vec` helper for `additional_outputs`: + +```rust +#[cfg_attr(feature = "debug-derive", derive(Debug))] +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[cfg_attr(feature = "schema", derive(JsonSchema))] +#[serde(rename_all = "camelCase")] +pub struct MacroSpec { + pub parameters: Vec, + pub primary_output: String, + #[serde(skip_serializing_if = "is_empty_vec", default)] + pub additional_outputs: Vec, +} +``` + +Step 2 — Add the field to `json::Model` (~lines 455-481), after `groups`. Use the *optional-singular* serde idiom from `json::Model.sim_specs`: + +```rust + #[serde(skip_serializing_if = "Option::is_none", default)] + pub macro_spec: Option, +``` + +Step 3 — Add the two `From` impls converting `json::MacroSpec` ⇄ `datamodel::MacroSpec`, mirroring the `json::LoopMetadata` impls at ~lines 1263 and ~1921. Field-by-field move. + +Step 4 — Wire `macro_spec` into the two `Model` conversion impls: +- In `From for datamodel::Model` (~lines 1185-1215): change the placeholder `macro_spec: None` (from Task 1) to `macro_spec: model.macro_spec.map(|m| m.into()),`. +- In `From for json::Model` (~lines 1818-1856): add `macro_spec: model.macro_spec.map(|m| m.into()),`. +- Adding the field to `json::Model` breaks every `json::Model { ... }` literal in `json.rs`'s own `#[cfg(test)]` fixtures. Run `cargo build -p simlin-engine --all-targets` and add `macro_spec: None` to each one the compiler flags (the one populated test case is set in the Testing step below). + +Step 5 — Check `src/simlin-engine/src/json_proptest.rs` for a strategy/generator that builds a `json::Model` (e.g. a `prop_model()` function or a `proptest!` macro over `json::Model`). If one exists, extend it to populate `macro_spec` — a `proptest::option::of(...)` over a `MacroSpec` strategy is ideal, but `Just(None)` is acceptable as a minimal start. If `json::Model` is generated via a derive-based strategy that picks up new fields automatically, no change is needed; note that in the commit message. + +**Testing:** +Verify **macros.AC1.4 (JSON half)**. `json.rs`'s `test_model_roundtrip` (~line 2693) currently sets `loop_metadata: vec![]` and does not exercise the optional nested struct. Add a populated `macro_spec` to a `json::Model` round-trip test: either extend `test_model_roundtrip` with a `macro_spec: Some(MacroSpec { ... })` (non-empty `parameters` and `primary_output`, non-empty `additional_outputs`) plus a body variable, or add a focused `test_macro_spec_roundtrip` that goes `json::Model` → `datamodel::Model` → `json::Model` → `serde_json` string → `json::Model` and `assert_eq!`s. Match the existing `test_model_roundtrip` structure. + +**Verification:** +Run: `cargo test -p simlin-engine json` +Expected: all `json` tests pass. + +Run: `cargo test -p simlin-engine generate_and_write_schema` +Expected: passes; `git diff docs/simlin-project.schema.json` now shows a `"MacroSpec"` entry under `$defs` and a `"macroSpec"` property on the `Model` definition. + +**Commit:** `engine: persist MacroSpec through JSON and regenerate schema` (stage `json.rs`, the regenerated `docs/simlin-project.schema.json`, and `json_proptest.rs` if changed) + + + + +## Subcomponent C: Consumer mirrors (TypeScript and Python) + + +### Task 4: TypeScript mirrors + +**Verifies:** macros.AC1.4 (TypeScript JSON round-trip). + +**Files:** +- Modify: `src/engine/src/json-types.ts` (add `JsonMacroSpec` interface near `JsonLoopMetadata` ~line 325; add `macroSpec?` to `JsonModel` ~lines 355-368) +- Modify: `src/core/datamodel.ts` (add `MacroSpec` interface near `LoopMetadata` ~line 1112; add `macroSpecFromJson`/`macroSpecToJson` near `loopMetadataFromJson`/`loopMetadataToJson` ~line 1119; add `macroSpec?` to the `Model` interface ~lines 1181-1187; wire into `modelFromJson` ~line 1189 and `modelToJson` ~line 1208; the `JsonMacroSpec` import joins the existing `@simlin/engine` import block ~lines 11-41) +- Modify: `src/core/tests/datamodel.test.ts` (add a `MacroSpec` round-trip `describe` block mirroring the `LoopMetadata` block ~lines 557-592; add the converter imports ~lines 30-31 and the `MacroSpec` type import ~line 57) + +**Implementation:** + +`json-types.ts` is the hand-maintained TypeScript mirror of `json.rs` — keep the field names and optionality identical to the `json::MacroSpec` you wrote in Task 3. + +Step 1 — In `json-types.ts`, add `JsonMacroSpec` mirroring `JsonLoopMetadata`'s style (camelCase keys, optional for skip-serialized fields): + +```typescript +/** + * Marks a model as a callable macro template and records its calling convention. + */ +export interface JsonMacroSpec { + parameters: string[]; + primaryOutput: string; + additionalOutputs?: string[]; +} +``` + +Then add `macroSpec?: JsonMacroSpec;` to the `JsonModel` interface. + +Step 2 — In `datamodel.ts`, add the rich (readonly) `MacroSpec` interface and its two converters, mirroring `LoopMetadata` / `loopMetadataFromJson` / `loopMetadataToJson`: + +```typescript +export interface MacroSpec { + readonly parameters: readonly string[]; + readonly primaryOutput: string; + readonly additionalOutputs: readonly string[]; +} + +export function macroSpecFromJson(json: JsonMacroSpec): MacroSpec { + return { + parameters: json.parameters, + primaryOutput: json.primaryOutput, + additionalOutputs: json.additionalOutputs ?? [], + }; +} + +export function macroSpecToJson(spec: MacroSpec): JsonMacroSpec { + const result: JsonMacroSpec = { + parameters: [...spec.parameters], + primaryOutput: spec.primaryOutput, + }; + if (spec.additionalOutputs.length > 0) { + result.additionalOutputs = [...spec.additionalOutputs]; + } + return result; +} +``` + +Step 3 — Add `readonly macroSpec?: MacroSpec;` to the `Model` interface. In `modelFromJson`, add `macroSpec: json.macroSpec ? macroSpecFromJson(json.macroSpec) : undefined,`. In `modelToJson`, add (mirroring how `loopMetadata` is conditionally attached) `if (model.macroSpec) { result.macroSpec = macroSpecToJson(model.macroSpec); }`. + +**Testing:** +Verify **macros.AC1.4 (TypeScript half)**. Add a `describe('MacroSpec', ...)` block to `datamodel.test.ts` mirroring the `describe('LoopMetadata', ...)` block (~lines 557-592): a "should roundtrip correctly" test that builds a `MacroSpec` with non-empty `parameters`, `primaryOutput`, and `additionalOutputs`, runs `macroSpecToJson` then `macroSpecFromJson`, and asserts equality; and a "should omit empty additionalOutputs" test confirming `macroSpecToJson` leaves `additionalOutputs` undefined when the array is empty and `macroSpecFromJson` restores it to `[]`. + +**Verification:** +Run: `pnpm build` (rebuilds `@simlin/engine` so `@simlin/core` sees the new `JsonMacroSpec` export, then builds all TS packages) +Run: `pnpm tsc` +Expected: type-checks with no errors. +Run: `pnpm --filter @simlin/core test` +Expected: all `@simlin/core` tests pass, including the new `MacroSpec` block. + +(Note: `json-types.ts` is an existing file being edited, not a new file, so the `lib.browser/` `.d.ts` regeneration caveat for *new* engine files does not apply — `pnpm build` covers it.) + +**Commit:** `core: mirror MacroSpec in the TypeScript datamodel` + + + +### Task 5: Python mirrors + +**Verifies:** macros.AC1.4 (Python JSON round-trip). + +**Files:** +- Modify: `src/pysimlin/simlin/json_types.py` (add `MacroSpec` dataclass near `LoopMetadata` ~line 298; add `macro_spec` field to the `Model` dataclass ~lines 308-319) +- Modify: `src/pysimlin/simlin/json_converter.py` (add a `structure_macro_spec` hook near `structure_loop_metadata` ~line 723; register it; read `macroSpec` in `structure_model` ~lines 765-787; add `MacroSpec` to `additional_type_required_fields` ~line 805) +- Modify: the relevant test file under `src/pysimlin/tests/` (add a `MacroSpec` round-trip test — see Testing) + +**Implementation:** + +The Python dataclasses are not self-serializing; serialization is centralized in `json_converter.py` via `cattrs`. JSON keys are camelCase; dataclass fields are snake_case; the converter bridges them (`structure_*` reads camelCase keys, `_make_omit_default_hook` emits camelCase via `_to_camel_case`). + +Step 1 — In `json_types.py`, add the `MacroSpec` dataclass mirroring `LoopMetadata`'s style (all fields defaulted): + +```python +@dataclass +class MacroSpec: + """Marks a model as a callable macro template and records its calling convention.""" + + parameters: list[str] = field(default_factory=list) + primary_output: str = "" + additional_outputs: list[str] = field(default_factory=list) +``` + +Then add `macro_spec: MacroSpec | None = None` to the `Model` dataclass (mirroring `sim_specs: SimSpecs | None = None`). + +Step 2 — In `json_converter.py`, add a structure hook next to `structure_loop_metadata` (~line 723) and register it: + +```python +def structure_macro_spec(d: dict[str, Any], _: type) -> MacroSpec: + return MacroSpec( + parameters=d.get("parameters", []), + primary_output=d.get("primaryOutput", ""), + additional_outputs=d.get("additionalOutputs", []), + ) + +conv.register_structure_hook(MacroSpec, structure_macro_spec) +``` + +Step 3 — In `structure_model` (~lines 765-787), structure the optional `macroSpec` key and pass it to the `Model(...)` constructor (mirror how `sim_specs` is handled): + +```python + macro_spec = conv.structure(d["macroSpec"], MacroSpec) if d.get("macroSpec") else None +``` + +Step 4 — Add `MacroSpec` to the `additional_type_required_fields` dict (~line 805) so `parameters` and `primary_output` always serialize even at their defaults (keys are snake_case dataclass field names): `MacroSpec: {"parameters", "primary_output"}`. The generic `_make_omit_default_hook` then handles unstructure (emitting `macroSpec`, `primaryOutput`, `additionalOutputs` in camelCase). + +Step 5 — Ensure `MacroSpec` is exported wherever `LoopMetadata` is (check `json_types.py`'s `__all__` if it has one, and any re-export in `simlin/__init__.py`). + +**Testing:** +Verify **macros.AC1.4 (Python half)**. Find the existing `Model`/`LoopMetadata` round-trip test in `src/pysimlin/tests/` (likely `test_json_types.py`) and add a `MacroSpec` round-trip test in the same style: build a `Model` with a populated `macro_spec` (non-empty `parameters`, `primary_output`, `additional_outputs`), unstructure it to a dict via the `json_converter`, structure it back, and assert the `MacroSpec` is identical. Confirm the unstructured dict uses the camelCase keys `macroSpec`, `primaryOutput`, `additionalOutputs`. + +**Verification:** +Run: `cd src/pysimlin && uv run pytest tests/test_json_types.py -x` +Expected: all tests pass, including the new `MacroSpec` round-trip test. + +**Commit:** `pysimlin: mirror MacroSpec in the Python json types` + + + +--- + +## Phase 1 completion check + +When all five tasks are committed, the following hold: +- `MacroSpec` is a first-class datamodel type with an `Option` field on `Model`. +- A macro-bearing `Model` round-trips losslessly through protobuf (`serde.rs` test), through Rust JSON (`json.rs` test), through the TypeScript mirror (`@simlin/core` test), and through the Python mirror (pysimlin test) — satisfying **macros.AC1.4**. +- `docs/simlin-project.schema.json` describes `MacroSpec`. +- `cargo build --workspace`, `pnpm tsc`, and the pre-commit hook all pass. + +No import, export, resolution, or simulation behavior changes in this phase — those build on this representation in Phases 2-7. diff --git a/docs/implementation-plans/2026-05-13-macros/phase_02.md b/docs/implementation-plans/2026-05-13-macros/phase_02.md new file mode 100644 index 000000000..3cce3ef33 --- /dev/null +++ b/docs/implementation-plans/2026-05-13-macros/phase_02.md @@ -0,0 +1,384 @@ +# Vensim Macro Support — Phase 2: MDL parsing and import + +**Goal:** Parse the full `:MACRO:` syntax — including the `:` multi-output separator no parser handles today — and import every macro definition from a `.mdl` file as a macro-marked `datamodel::Model`. + +**Architecture:** Two halves. (a) **Parser**: the MDL lexer/reader/parser already recognize `:MACRO:` ... `:END OF MACRO:` and build a `MacroDef` AST node, but they have no notion of the `:` multi-output separator. Phase 2 adds an output list to `MacroDef`, an output-binding list to `Expr::App`, and the parser logic to populate both. (b) **Convert**: `MacroDef`s are currently parsed and then silently discarded by `collect_symbols` in `convert/mod.rs`. Phase 2 makes `mdl/convert` turn each `MacroDef` into a macro-marked `datamodel::Model` (carrying the `MacroSpec` from Phase 1) by reusing the existing `ConversionContext` conversion pipeline scoped to the macro body, and translates `$`-suffixed time references in macro bodies to canonical engine time idents. + +**Tech Stack:** Rust — the native MDL parser (`src/simlin-engine/src/mdl/`). No external dependencies. The Vensim `:MACRO:` syntax reference is in-repo at `docs/reference/vensim-macros.md` and `docs/reference/xmile-v1.0.html`. + +**Scope:** 7 phases from the original design (`docs/design-plans/2026-05-13-macros.md`); this is phase 2 of 7. + +**Codebase verified:** 2026-05-14 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### macros.AC1: Macro definitions parse and represent faithfully +- **macros.AC1.1 Success:** A `.mdl` `:MACRO:` block imports as a macro-marked `Model` whose `MacroSpec.parameters` matches the header's input list in order, with body variables matching the body equations. +- **macros.AC1.2 Success:** A `:MACRO:` header with a `:` output list (`add3(a,b,c : minval, maxval)`) imports with `MacroSpec.additional_outputs` populated in order. +- **macros.AC1.5 Success:** `$`-suffixed time references in a macro body (`TIME STEP$`, `Time$`) import as the canonical time identifiers. +- **macros.AC1.6 Failure:** A `:MACRO:` block with no `:END OF MACRO:` reports a clear parse error. +- **macros.AC1.7 Edge:** A macro defined but never invoked (C-LEARN's `INIT`) imports as a valid macro-marked model and is preserved. + +(`macros.AC1.3` is XMILE import — Phase 5. `macros.AC1.4` is round-trip — completed in Phase 1.) + +--- + +## Current state (verified 2026-05-14) + +**Already works** — the lexer, reader, and parser fully handle the *single-output* `:MACRO:` form: +- `src/simlin-engine/src/mdl/lexer.rs` — `RawToken::Macro` (`:MACRO:`), `RawToken::EndOfMacro` (`:END OF MACRO:`), and `RawToken::Colon` (bare `:`) are all tokenized (`colon_keyword()`, lexer.rs:518-570). `$` is a symbol char (`is_symbol_char`, lexer.rs:343-351) and `symbol()` (lexer.rs:355-399) does **not** strip a trailing `$`, so `Time$` lexes as one `Symbol("Time$")` and `TIME STEP$` as `Symbol("TIME STEP$")`. +- `src/simlin-engine/src/mdl/parser.rs` — the macro header is parsed in `parse_full_eq_with_units` (parser.rs:518-545), producing `SectionEnd::MacroStart(name, args, loc)`; `:END OF MACRO:` → `SectionEnd::MacroEnd(loc)`. Argument lists go through `parse_expr_list` (parser.rs:1361-1399), which separates only on `Comma` and `Semicolon` — **there is no `Colon` arm anywhere in expression parsing.** `Token::Colon` is meaningful only as the subscript-definition operator immediately after an LHS symbol. +- `src/simlin-engine/src/mdl/reader.rs` — `MacroState` (reader.rs:60-70) accumulates body equations; `handle_parse_result` (reader.rs:273-332) builds `MacroDef` and emits `MdlItem::Macro(Box)`. **An unterminated `:MACRO:` block already produces `ReaderError::EofInsideMacro`** ("unexpected end of file inside macro"); a stray `:END OF MACRO:` produces `ReaderError::UnmatchedMacroEnd`. Nested `:MACRO:` blocks are rejected. +- `src/simlin-engine/src/mdl/ast.rs` — `MacroDef` (ast.rs:483-493): `{ name: Cow, args: Vec, equations: Vec, loc: Loc }`. `Expr::App(Cow, Vec, Vec, CallKind, Loc)` (ast.rs:128-134). `CallKind` is `Builtin | Symbol` (ast.rs:98-108). + +**Does not work / missing:** +- No `:` multi-output separator support — not in `MacroDef` (no input/output split on `args`), not in `Expr::App`, not in the parser. +- Macros are **discarded**: `collect_symbols` in `convert/mod.rs:248` has `MdlItem::Macro(_) | MdlItem::EqEnd(_) => {}` — every parsed `MacroDef` is dropped and never reaches the datamodel. (Note: this line declines to register macros as *main-model symbols*, which is correct; the `MdlItem::Macro` entries remain in `self.items`. Phase 2 adds a new conversion step that consumes them — it does not need to change line 248.) +- No `$`-time translation anywhere — `Time$` survives as an `Expr::Var` named `"Time$"` that nothing maps to a canonical ident. + +**Key types and entry points:** +- `ConversionContext<'input>` (`convert/mod.rs:45-88`) — holds `items: Vec`, `symbols: HashMap`, `dimensions`, `sim_specs`, `formatter: XmileFormatter`, `data_provider`. Built by `new_with_data` (`convert/mod.rs:98-141`, drains an `EquationReader` into `self.items`). +- `convert()` (`convert/mod.rs:144-197`) runs these steps in order: `collect_symbols`, `build_dimensions`, set-subrange-names, `mark_variable_types`, `scan_for_extrapolate_lookups`, `link_stocks_and_flows`, `build_project` — six numbered passes (the in-code comments label them Pass 1 through Pass 6) plus the set-subrange-names half-pass (Pass 2.5). +- `build_project` (`convert/variables.rs:28-106`) iterates `self.symbols`, builds variables via `build_variable` / `build_variable_with_elements` / `build_equation`, and constructs exactly one `Model { name: "main", ... }` wrapped in a `Project`. **After Phase 1, this construction site already includes `macro_spec: None`.** +- `build_variable` (`convert/variables.rs:798-854`) — `SymbolInfo` + `FullEquation` → `datamodel::Variable` (Stock/Flow/Aux). `build_equation` (`convert/variables.rs:859-939`) — `MdlEquation` → `(datamodel::Equation, Compat, Option)`, RHS formatted via `self.formatter.format_expr(expr)`. +- `ConvertError` (`convert/types.rs:12-26`): `Reader | View | InvalidRange | CyclicDimensionDefinition | Import | Other`. New macro-conversion errors use `ConvertError::Other(String)` or a new variant. +- Test-only entry point `convert_mdl(source: &str) -> Result` (`convert/mod.rs:315-318`); production `open_vensim(&str) -> common::Result` (`compat.rs:62-74`). +- Canonical engine time idents (`builtins.rs:146-149`, `354-367`): `time`, `time_step` (alias `dt`), `initial_time` (alias `starttime`), `final_time` (alias `stoptime`). +- Parser test idiom (`parser.rs:2343-2359`, `test_parse_macro_start`): build a `TokenNormalizer` from `&str`, `collect_tokens`, call public `parse(&tokens)`, assert on the `(Equation, Option, SectionEnd)` tuple. Reader test idiom (`reader.rs:1304-1320`, `test_macro_followed_by_equation`): `EquationReader::new(input).next_item()`. + +**The 6 fixtures** (`test/test-models/tests/macro_*/`) — all single-output, all `.mdl` + `output.tab`; none uses the `:` separator or `$`-time, so those two features need synthetic test inputs: +- `macro_expression` — `:MACRO: EXPRESSION MACRO(input, parameter)` / body `EXPRESSION MACRO = input * parameter` (one aux). Invoked `macro output = EXPRESSION MACRO(macro input,macro parameter)`. +- `macro_multi_expression` — same header; 2-equation body (`EXPRESSION MACRO = input * intermediate` + helper `intermediate = parameter * 3`). +- `macro_stock` — body is a stock: `EXPRESSION MACRO = INTEG(input, parameter)`. +- `macro_cross_reference` — two macros; `EXPRESSION MACRO` body calls `SECOND MACRO(input,parameter)` (a macro call inside a macro body — stays as equation text in Phase 2). +- `macro_multi_macros` — two independent macros. +- `macro_trailing_definition` — macro defined *after* its call site. + +--- + + +## Subcomponent A: Parser support for the `:` multi-output syntax + + +### Task 1: AST fields for macro outputs and call-site output bindings + +**Verifies:** None (AST structural change; the `:`-form ACs are verified in Tasks 5-7). + +**Files:** +- Modify: `src/simlin-engine/src/mdl/ast.rs` (`MacroDef` ~lines 483-493; `Expr::App` ~lines 128-134; the `Expr` `loc()` / walk helpers that `match` on `App`) +- Modify: every other site in `src/simlin-engine/` that constructs or non-wildcard-matches `Expr::App` — discovered by the compiler. Known sites: `parser.rs` (3 `App` construction sites at `parser.rs:1283/1306/1348`, plus non-wildcard `match` arms in its `#[cfg(test)]` module), `reader.rs` (4 non-wildcard `Expr::App` match arms at `reader.rs:747/768/787/810`), `xmile_compat.rs`, `convert/helpers.rs:66` (`is_top_level_integ`), `convert/stocks.rs`, `convert/variables.rs`, `writer.rs`. Known `MacroDef` construction sites: `reader.rs` `handle_parse_result`, the inline tests `ast.rs:1026` (`test_macro_def`) and `reader.rs:1304` (`test_macro_followed_by_equation`). + +**Implementation:** + +Step 1 — Add an `outputs` field to `MacroDef`, after `args`: + +```rust +pub struct MacroDef<'input> { + pub name: Cow<'input, str>, + /// Formal input parameters (the comma-separated list before any `:`), + /// parsed as expressions (typically just `Var` nodes). + pub args: Vec>, + /// Additional named outputs from the `:`-list in the header + /// (`:MACRO: add3(a,b,c : minval, maxval)`). Empty for ordinary + /// single-output macros. + pub outputs: Vec>, + /// Equations within the macro body. + pub equations: Vec>, + pub loc: Loc, +} +``` + +Step 2 — Add a 6th field to `Expr::App` for call-site output bindings. Update its doc comment: + +```rust + /// Function/macro application: name, subscripts, args, kind, output bindings. + /// + /// `output_bindings` holds the post-`:` output-binding expressions of a + /// Vensim multi-output macro call (`add3(a, b, c : minv, maxv)`); it is + /// empty for every ordinary call. The bindings are parsed as expressions + /// (expected to be `Var` nodes naming caller variables). + App( + Cow<'input, str>, + Vec>, + Vec>, + CallKind, + Vec>, + Loc, + ), +``` + +Step 3 — Make `src/simlin-engine` compile again. Run `cargo build -p simlin-engine --all-targets`. For every error: +- `Expr::App` construction sites: add `vec![]` for the new `output_bindings` field (the parser will populate it in Task 3; everything else leaves it empty). +- `Expr::App` non-wildcard `match` arms: add a binding (or `_`) for the new field. In AST traversal helpers (`loc()`, `walk`), the new `Vec` should be walked like `args` so later passes see nested expressions — but for Task 1, matching it as `_` is acceptable where the helper does not recurse into `args` either; match the surrounding code's behavior. +- The MDL-writer formatter `format_call_ctx` (`src/simlin-engine/src/mdl/xmile_compat.rs`, the `Expr::App` arm ~lines 219-498): a multi-output `App` cannot be represented as plain equation text. Add `debug_assert!(output_bindings.is_empty(), "multi-output macro invocations must be materialized by the converter before formatting -- see Phase 4")` at the top of that arm. Phase 4 materializes multi-output invocations in the converter *before* anything formats them, so this assertion never trips at runtime; it exists to catch a regression where a multi-output invocation reaches the text formatter and its output bindings would be silently dropped. +- `MacroDef` construction sites: add `outputs: vec![]`. + +Do not change `SectionEnd::MacroStart` or `MacroState` in this task — they keep their current shape, and `reader.rs` constructs `MacroDef { ..., outputs: vec![] }`. Tasks 2 and 3 populate the new fields. + +**Testing:** None. Structural change only; the compiler is the verifier. The existing `test_macro_def` (ast.rs) and `test_macro_followed_by_equation` (reader.rs) must still pass after their literals are updated. + +**Verification:** +Run: `cargo build -p simlin-engine --all-targets` +Expected: compiles with no errors. +Run: `cargo test -p simlin-engine mdl::` +Expected: existing MDL tests still pass (behavior unchanged). + +**Commit:** `engine: add AST fields for macro multi-output syntax` + + + +### Task 2: Parse the `:` separator in `:MACRO:` headers + +**Verifies:** None directly (parser support; `macros.AC1.2` is completed at import level in Task 5). Required by the design's "Done when: parser unit tests cover the `:` header ... forms". + +**Files:** +- Modify: `src/simlin-engine/src/mdl/parser.rs` (the macro-header path in `parse_full_eq_with_units` ~lines 518-545; the `SectionEnd::MacroStart` variant — find its definition and add the `outputs` field) +- Modify: `src/simlin-engine/src/mdl/reader.rs` (`MacroState::InMacro` ~lines 60-70; `handle_parse_result` ~lines 273-332) +- Test: `src/simlin-engine/src/mdl/parser.rs` (inline `#[cfg(test)]` module, near `test_parse_macro_start` ~line 2343); `src/simlin-engine/src/mdl/reader.rs` (inline tests near `test_macro_followed_by_equation` ~line 1304) + +**Implementation:** + +Step 1 — Extend `SectionEnd::MacroStart` to carry the output list. Change `MacroStart(name, args, loc)` to `MacroStart(name, args, outputs, loc)` (or add an `outputs: Vec` field). Update its construction in the parser and its consumption in `reader.rs`. + +Step 2 — In the macro-header path (`parse_full_eq_with_units`, parser.rs:518-545): after parsing the input `args` via `parse_expr_list()?.into_exprs()` and **before** `expect(TokenKind::RParen, ...)`, check for an optional `:` separator. `parse_expr_list` naturally stops at `:` (it separates only on `Comma`/`Semicolon`), so the token after the input list is either `RParen` or `Colon`: + +```rust +let outputs = if self.peek_kind() == Some(TokenKind::Colon) { + self.advance_pos(); // consume ':' + self.parse_expr_list()?.into_exprs() +} else { + vec![] +}; +self.expect(TokenKind::RParen, "')' to close macro arguments")?; +``` + +Pass `outputs` into `SectionEnd::MacroStart`. + +Step 3 — Thread `outputs` through the reader. Add `outputs: Vec>` to `MacroState::InMacro`; set it from `MacroStart` in `handle_parse_result`; include it in the `MacroDef { ... }` built on `MacroEnd`. + +**Testing:** +Parser/reader unit tests (parser-structural, following the `test_parse_macro_start` idiom): +- `:MACRO: add3(a, b, c : minval, maxval)` → `SectionEnd::MacroStart` with `args.len() == 3`, `outputs.len() == 2`, output names `minval`/`maxval` in order. +- `:MACRO: MYFUNC(arg1, arg2)` (no `:`) → `outputs` is empty (regression: the single-output form is unchanged). +- Reader-level: a full `:MACRO: add3(a,b,c : minval, maxval) ... :END OF MACRO:` block → `MdlItem::Macro` whose `MacroDef.outputs` has the two names in order. +- A `:` header with an empty output list (`add3(a : )`) — `parse_expr_list` calls `parse_expr` unconditionally first (it has no leading-emptiness tolerance), so a `:` immediately followed by `)` produces a **clean parse error** on the `)`. Assert that error is returned (not a panic, not a silently-empty `outputs`), and lock the behavior in with the test. + +**Verification:** +Run: `cargo test -p simlin-engine mdl::parser` +Run: `cargo test -p simlin-engine mdl::reader` +Expected: all pass, including the new `:`-header tests. + +**Commit:** `engine: parse the ':' separator in :MACRO: headers` + + + +### Task 3: Parse the `:` separator in macro call-site argument lists + +**Verifies:** None directly (parser support; multi-output *invocation* is materialized in Phase 4). Required by the design's "Done when: parser unit tests cover the ... call forms". + +**Files:** +- Modify: `src/simlin-engine/src/mdl/parser.rs` (the `parse_atom` Symbol-call branch ~lines 1264-1294) +- Test: `src/simlin-engine/src/mdl/parser.rs` (inline `#[cfg(test)]` module) + +**Implementation:** + +A multi-output macro invocation (`total = add3(a, b, c : minv, maxv)`) lands in the `parse_atom` Symbol branch, because a macro name used in call position is tokenized as a `Symbol` and produces `Expr::App(..., CallKind::Symbol, ...)`. (A `:` in a `CallKind::Builtin` call is not valid Vensim, so only the Symbol branch needs this; add a brief comment in the code stating that.) + +In the Symbol branch (parser.rs:1264-1294), after `let args = self.parse_expr_list()?.into_exprs();` and **before** `self.expect(TokenKind::RParen, "')' to close call")?`, parse the optional `:` output bindings exactly as in Task 2: + +```rust +let output_bindings = if self.peek_kind() == Some(TokenKind::Colon) { + self.advance_pos(); // consume ':' + self.parse_expr_list()?.into_exprs() +} else { + vec![] +}; +let close = self.expect(TokenKind::RParen, "')' to close call")?; +``` + +Pass `output_bindings` as the new 6th field of `Expr::App`. The existing "symbol call requires at least one argument" check stays as-is (it guards `args`, not the output list). + +**Testing:** +Parser unit tests (following the `test_parse_macro_start` idiom — parse a full equation and inspect the RHS `Expr`): +- `total = add3(a, b, c : minv, maxv)` → RHS is `Expr::App` with `args.len() == 3`, `output_bindings.len() == 2`, names `minv`/`maxv` in order, `CallKind::Symbol`. +- `y = MYMACRO(a, b)` (no `:`) → `Expr::App` with `output_bindings` empty (regression). +- A `:` call nested in a larger expression (`y = c + add3(a, b, c : minv, maxv)`) → the `App` inside the `Op2` carries the `output_bindings`. + +**Verification:** +Run: `cargo test -p simlin-engine mdl::parser` +Expected: all pass, including the new `:`-call tests. + +**Commit:** `engine: parse the ':' separator in macro call sites` + + + + +## Subcomponent B: Convert each `MacroDef` into a macro-marked `Model` + + +### Task 4: Make the conversion pipeline reusable for a scoped sub-model + +**Verifies:** None (behavior-preserving refactor; verified by existing tests still passing). + +**Files:** +- Modify: `src/simlin-engine/src/mdl/convert/mod.rs` (`new_with_data` ~lines 98-141; `convert()` ~lines 144-197) +- Modify: `src/simlin-engine/src/mdl/convert/variables.rs` (`build_project` ~lines 28-106) + +**Background — why this refactor:** A macro body is a little model with its own variable namespace. `macro_cross_reference` defines two macros that both have body references to `input`/`parameter`; `macro_stock` has a body stock (`INTEG`) needing the stock/flow-linking pass. The body must therefore be converted with a *scoped* symbol table, not the global one. The existing `ConversionContext` pipeline already converts "a list of `MdlItem`s into a `Model`" — it is only hardwired to (a) drain its items from an `EquationReader` and (b) emit exactly one model named `"main"`. This task removes those two hardwirings without changing any observable behavior. + +**Implementation:** + +Investigate `convert/mod.rs` and `convert/variables.rs` in full, then make these two behavior-preserving changes: + +1. **Injectable item list.** Factor the reader-draining out of `new_with_data` so a `ConversionContext` can be constructed over a caller-supplied `Vec` (e.g. add `ConversionContext::new_from_items(items, dimensions, data_provider, formatter, ...)` and have `new_with_data` call it after draining the reader). The macro path (Task 5) builds a sub-context from a macro's body equations this way. **The sub-context shares the parent's already-built `dimensions`, `data_provider`, and `formatter` — it does not rebuild or independently re-derive them** (the macro body defines no dimensions of its own, and the formatter's subrange-name state must match the parent's). Whether "shares" is a borrow, an `Rc`, or a cheap clone is at the implementor's discretion as long as the sub-context uses the *same* dimensions/data_provider/formatter the parent built — Task 5 relies on this. + +2. **Parameterized model name.** Extract from `build_project` a method `build_model(&self, name: &str) -> Result` that builds `variables` / `views` / `groups` into a `Model` (with `sim_specs: None`, `macro_spec: None`). `build_project` becomes `build_model("main")` followed by the `Project` assembly. This lets the macro path produce a `Model` named after the macro. + +The exact mechanics (helper signatures, which fields move where) are at the implementor's discretion **as long as**: the `"main"` model conversion is byte-for-byte unchanged, the public `convert_mdl` / `convert_mdl_with_data` signatures are unchanged, and a sub-context can be built from a `Vec` plus the parent's shared `dimensions`, `data_provider`, and `formatter`, and have `collect_symbols` / `mark_variable_types` / `scan_for_extrapolate_lookups` / `link_stocks_and_flows` / `build_model` run on it. + +**Testing:** None new. This is a refactor — its correctness is "every existing test still passes." + +**Verification:** +Run: `cargo test -p simlin-engine mdl::` +Run: `cargo test -p simlin-engine --test mdl_equivalence` +Run: `cargo test -p simlin-engine --test simulate simulates_except_basic_mdl` +Expected: all pass unchanged — the refactor introduced no behavior change. + +**Commit:** `engine: make MDL conversion pipeline reusable for sub-models` + + + +### Task 5: Convert each `MacroDef` into a macro-marked `Model` + +**Verifies:** macros.AC1.1, macros.AC1.2, macros.AC1.7. + +**Files:** +- Modify: `src/simlin-engine/src/mdl/convert/mod.rs` (add a macro-conversion step in `convert()`, after `build_project`) and/or `src/simlin-engine/src/mdl/convert/variables.rs` +- Create or modify: a `convert/` module file for the macro-conversion routine (e.g. a new `src/simlin-engine/src/mdl/convert/macros.rs`, declared `mod macros;` in `convert/mod.rs`, following the file-per-concern layout of `stocks.rs` / `variables.rs` / `dimensions.rs`) +- Modify: `src/simlin-engine/src/mdl/convert/helpers.rs` (a helper to extract a canonical ident from a `MacroDef` arg `Expr::Var`, if one does not already exist) +- Test: the inline `#[cfg(test)] mod tests` in `convert/mod.rs` (near `test_mdl_group_variables_after_macro` ~line 616) + +**Implementation:** + +Add a conversion step that runs after the `"main"` model is built (the `MdlItem::Macro` entries are still in `self.items` — `collect_symbols` leaves them there). For each `MdlItem::Macro(macro_def)`: + +1. **Build the macro's body model.** Wrap `macro_def.equations` (a `Vec`) as `MdlItem::Equation` entries, build a scoped `ConversionContext` over them (Task 4's `new_from_items`, sharing the parent's `dimensions` and `data_provider`), run `collect_symbols` / `mark_variable_types` / `scan_for_extrapolate_lookups` / `link_stocks_and_flows`, then `build_model(¯o_name)`. This handles single-aux bodies (`macro_expression`), multi-equation bodies with helpers (`macro_multi_expression`'s `intermediate`), and stock bodies (`macro_stock`'s `INTEG`) for free, because those are exactly what the existing passes do. + +2. **Synthesize port variables for the formal parameters.** A macro's formal parameters (`input`, `parameter`) are referenced by the body equations but are not themselves defined by body equations — yet the engine's module machinery requires the sub-model to contain an actual `Variable` for every input port. This is exactly how stdlib models work: `stdlib⁚smth1` defines `input`, `delay_time`, `initial_value` as ordinary variables with placeholder-constant equations (`"0"`, `"1"`, `"NAN"`), and `stdlib⁚delay1` defines its `input` port as a `Flow`. So, after `build_model`, append a synthesized port `Variable` to the macro `Model.variables` for each name in `MacroSpec.parameters` that is not already a variable in the model: + - **Kind:** if the parameter name appears in any body stock's `inflows` or `outflows` list, synthesize a `Variable::Flow`; otherwise synthesize a `Variable::Aux`. (`macro_stock`'s `input` is the `INTEG` rate → `Flow`, mirroring `stdlib⁚delay1`; its `parameter` is the initial value → `Aux`, mirroring `stdlib⁚smth1`'s `input`.) + - **Equation:** a placeholder constant `Equation::Scalar("0")`. The placeholder only ever runs if the port is left unwired; a macro invocation always overrides it with the call-site argument. + - **`compat`:** `Compat { can_be_module_input: true, ..Compat::default() }`. This flag is **required**: it is how `collect_module_idents` recognizes the port as a module-input slot (so `PREVIOUS(port)` inside the body compiles correctly). It is the same flag an ordinary XMILE submodel sets via `access="input"`. Without it, a macro model — which is registered as an ordinary, non-`stdlib⁚`-prefixed sub-model — can mis-compile. + - This makes the macro `Model` a complete, self-contained, compilable sub-model — structurally identical to an ordinary XMILE submodel — and the design's intent that `MacroSpec.parameters` name body variables becomes literally true. + - If running the scoped conversion on the body alone causes `link_stocks_and_flows` to error on the undefined parameter references (rather than leaving them as resolved-at-compile-time names), the alternative is to prepend a synthetic ` = 0` `MdlItem::Equation` for each parameter *before* the scoped conversion so the pipeline assigns kinds itself, then set `can_be_module_input = true` on the results. Investigate `link_stocks_and_flows` and pick whichever yields a correct, compilable macro `Model`; the verification below is the safety net. + +3. **Build the `MacroSpec`.** On the resulting `Model`, set `macro_spec: Some(MacroSpec { parameters, primary_output, additional_outputs })`: + - `parameters` — one canonical ident per `Expr` in `macro_def.args`, in order. Extract the name from each `Expr::Var` and canonicalize it with the **same** function the body-equation formatter uses for `Var` identifiers (so `MacroSpec.parameters` entries are byte-identical to how the body equations reference them, and to the synthesized port-variable idents — verify this with the test below). If an `args` entry is not a `Var`, that is a malformed macro header — return `ConvertError::Other` naming the macro. + - `primary_output` — the macro's own name, canonicalized the same way as a variable ident (e.g. `quoted_space_to_underbar`). Vensim requires the body to contain an equation whose LHS is the macro name; Phase 2 sets `primary_output` to the canonicalized macro name without validating that the equation exists (a missing primary-output equation surfaces later, in Phase 3's compiler). + - `additional_outputs` — one canonical ident per `Expr` in `macro_def.outputs`, in order (empty for the 6 fixtures; populated for the `:`-header form from Task 2). Unlike `parameters`, additional outputs are computed by the body and so already have body equations — do **not** synthesize placeholders for them. + +4. **Push the macro `Model` into `project.models`**, alongside `"main"`. + +**Steps 2 and 3 must be a single named, reusable function** — Phase 5's XMILE reader reuses exactly this port-synthesis + `MacroSpec`-construction logic. Implement it as a `pub(crate)` function, e.g. `Model::new_macro(macro_name: &str, parameters: &[String], additional_outputs: &[String], body_variables: Vec) -> Model`, placed in `datamodel.rs` (as an associated function on `Model`) or a small shared module — somewhere callable from **both** `mdl/convert/` and `xmile/`. It takes an already-built body variable list, synthesizes the port variables (step 2), builds and attaches the `MacroSpec` (step 3), and returns the complete macro-marked `Model`. The format-specific part — step 1, building `body_variables` from `MacroDef.equations` via the scoped `ConversionContext` — stays in `mdl/convert/`; it produces the `body_variables` that get passed to the shared function. (Phase 5 Task 1 calls this same function with `body_variables` built from XMILE ``/``.) + +Single-output macro *invocations* (`macro output = EXPRESSION MACRO(macro input,macro parameter)`, and the body-level `EXPRESSION MACRO = SECOND MACRO(input,parameter)` in `macro_cross_reference`) are **not** materialized here — they remain ordinary equation text in their `Aux`/`Stock`/`Flow`, exactly as a `SMTH1(...)` call is today. They are resolved and expanded in Phase 3. + +Definition order does not matter: `macro_trailing_definition` defines the macro after its call site, but the reader collects all `MdlItem`s before `convert()` runs, so the macro `Model` is produced regardless of where the `:MACRO:` block appears (this is the design's deliberate "definition-order leniency"). + +**Testing:** +Tests in `convert/mod.rs`'s inline test module, using `convert_mdl(source)`: +- **macros.AC1.1:** convert a small `.mdl` with `:MACRO: EXPRESSION MACRO(input, parameter)` / `EXPRESSION MACRO = input * parameter` plus a `main`-model invocation. Assert `project.models` contains a model whose `macro_spec` is `Some` with `parameters == ["input", "parameter"]` and `primary_output == "expression_macro"`; assert the macro model's `variables` contains the body aux `expression_macro` and that its equation text references `input` and `parameter` byte-identically to the `parameters` entries; assert the macro model also contains synthesized port variables `input` and `parameter`, each with `compat.can_be_module_input == true`. Assert the `"main"` model still has `macro_spec: None` and that the invocation `macro output = EXPRESSION MACRO(...)` is preserved as equation text. +- **macros.AC1.2:** convert a `.mdl` with `:MACRO: add3(a, b, c : minval, maxval)` and a small body. Assert the macro model's `MacroSpec.additional_outputs == ["minval", "maxval"]` in order, and `parameters == ["a", "b", "c"]`; assert port variables `a`, `b`, `c` were synthesized (with `can_be_module_input == true`) but `minval`/`maxval` were **not** synthesized as ports (they are body-computed outputs). +- **macros.AC1.7:** convert a `.mdl` that defines a macro but never invokes it. Assert the macro-marked `Model` is still present in `project.models` with the correct `MacroSpec` and synthesized port variables. +- A stock-bodied macro (`EXPRESSION MACRO = INTEG(input, parameter)`): assert the macro model's body variable `expression_macro` is a `Variable::Stock`, that the synthesized `input` port is a `Variable::Flow` and `parameter` a `Variable::Aux` (both with `can_be_module_input == true`), and that the stock's inflow resolves to `input` (directly, or via a synthetic flow whose equation is `input`). + +**Verification:** +Run: `cargo test -p simlin-engine convert` +Expected: all pass, including the new macro-conversion tests. + +**Commit:** `engine: import MDL macro definitions as macro-marked models` + + + +### Task 6: Translate `$`-suffixed time references in macro bodies + +**Verifies:** macros.AC1.5. + +**Files:** +- Modify: `src/simlin-engine/src/mdl/convert/helpers.rs` (add a `pub(super)` `$`-time rewrite helper) +- Modify: the macro-conversion routine from Task 5 (apply the rewrite to each macro body equation's expression tree before it is formatted) +- Test: `convert/helpers.rs` inline tests (helper-level) and `convert/mod.rs` inline tests (import-level) + +**Implementation:** + +A `$`-suffixed time reference in a macro body (`Time$`, `TIME STEP$`, `INITIAL TIME$`, `FINAL TIME$`) is Vensim's escape to reach the caller's global time variables. After lexing, such a reference is an `Expr::Var` whose name *includes* the trailing `$` (e.g. `"Time$"` or `"TIME STEP$"`). The engine already resolves the bare canonical time idents inside a module body at any nesting depth (they are zero-arity builtins — `builtins.rs:354-367`), so the only work is a front-end name translation. + +Add a helper in `convert/helpers.rs` that recursively walks an `Expr` tree and rewrites every `Expr::Var` whose name — lowercased and space-normalized, with the trailing `$` stripped — matches a Vensim time variable, replacing it with the canonical engine ident: +- `time$` → `time` +- `time step$` → `time_step` +- `initial time$` → `initial_time` +- `final time$` → `final_time` +- (`dt$` → `dt` may be included for completeness; the corpus only exercises `Time$` and `TIME STEP$`.) + +The rewrite must recurse through `Op1`, `Op2`, `Paren`, and the `args` (and `output_bindings`) of `App` so a `$`-time reference nested in a larger expression is caught. Apply it in the Task 5 macro-conversion routine to each body equation's expression **before** formatting. Scope it to macro bodies only — do not touch the global `XmileFormatter` or non-macro equations (a `$`-time reference outside a macro body is meaningless and out of scope). + +This is the design's "non-time `$` escape ... deprioritized" boundary: only `$`-*time* is translated; a `$`-suffixed reference to a non-time variable is out of Phase 2 scope. + +**Testing:** +- Helper-level (in `convert/helpers.rs` inline tests): build small `Expr` trees containing `Var("Time$")`, `Var("TIME STEP$")`, `Var("Initial Time$")`, and a nested `c + MYMACRO(Time$, x)` shape; assert the rewrite produces `time`, `time_step`, `initial_time`, and rewrites the nested occurrence; assert a non-time `Var` (`"foo$"`, `"input"`) is left untouched. +- **macros.AC1.5** (import-level, in `convert/mod.rs` tests): `convert_mdl` a `.mdl` whose macro body uses `Time$` and `TIME STEP$` (e.g. `MYMACRO = input * Time$ / TIME STEP$`); assert the imported macro model's body equation text references `time` and `time_step`, not `Time$` / `TIME STEP$`. + +**Verification:** +Run: `cargo test -p simlin-engine convert` +Expected: all pass, including the `$`-time helper and import tests. + +**Commit:** `engine: translate $-time references in macro bodies` + + + + +## Subcomponent C: Fixture import verification + + +### Task 7: Import-verify the 6 macro fixtures and the unterminated-macro error + +**Verifies:** macros.AC1.1, macros.AC1.6, macros.AC1.7. + +**Files:** +- Test: `src/simlin-engine/src/mdl/convert/mod.rs` inline `#[cfg(test)] mod tests` (or a new `src/simlin-engine/src/mdl/convert/macro_tests.rs` declared `#[cfg(test)] mod macro_tests;`, following the project's `*_tests.rs` separate-file convention for larger test sets) + +**Implementation:** Tests only — no production code. Embed each fixture with `include_str!` so a missing fixture is a compile error (the project's stated preference: "if a required test file is missing, fail loudly"). From `src/simlin-engine/src/mdl/convert/mod.rs`, the path to a fixture is `../../../../../test/test-models/tests//.mdl` (five `../` to the repo root). + +**Testing:** +For each of the 6 `test/test-models/tests/macro_*` `.mdl` fixtures, `convert_mdl(include_str!(...))` and assert (**macros.AC1.1**): +- `project.models` contains the expected macro-marked model(s) — `macro_cross_reference` and `macro_multi_macros` produce two macro models each; the rest produce one. +- Each macro model's `macro_spec` is `Some` with `parameters` matching the header input list in order (e.g. `["input", "parameter"]`) and `primary_output` equal to the canonicalized macro name. +- Each macro model's `variables` match the body equations **plus** a synthesized port variable per formal parameter (each with `compat.can_be_module_input == true`): `macro_expression` → body aux `expression_macro` + port vars `input`, `parameter`; `macro_multi_expression` → body auxes `expression_macro` and the `intermediate` helper + port vars `input`, `parameter`; `macro_stock` → body stock `expression_macro` + port `Flow` `input` + port `Aux` `parameter`; `macro_cross_reference`'s `EXPRESSION MACRO` body → body aux `expression_macro` whose equation references `SECOND MACRO` as a call (equation text, not expanded) + port vars `input`, `parameter`. +- The `"main"` model has `macro_spec: None` and preserves each invocation as equation text. +- **macros.AC1.7:** `macro_trailing_definition` — the macro is defined after its call site; assert the macro model is still present and correct, and `main`'s `macro output` invocation is preserved. + +For **macros.AC1.6**: assert that `open_vensim` (or `convert_mdl`) on a `.mdl` containing `:MACRO: BAD(x)` ... with **no** `:END OF MACRO:` returns `Err`; the error must be `ConvertError::Reader(ReaderError::EofInsideMacro)` and its `Display` must be the clear message `"reader error: unexpected end of file inside macro"` (through `open_vensim`, the message is `"Failed to parse MDL: reader error: unexpected end of file inside macro"`). Also assert a stray `:END OF MACRO:` with no opening `:MACRO:` yields `ReaderError::UnmatchedMacroEnd`. + +**Verification:** +Run: `cargo test -p simlin-engine convert` +Expected: all 6 fixture-import tests and both error-case tests pass. + +**Commit:** `engine: import-verify the macro test fixtures` + + + +--- + +## Phase 2 completion check + +When all seven tasks are committed: +- The MDL parser handles the full `:MACRO:` syntax, including the `:` multi-output separator in both headers and call sites — parser unit tests cover both forms (the design's "Done when"). +- Every `.mdl` `:MACRO:` block imports as a macro-marked `datamodel::Model` carrying a correct `MacroSpec` and synthesized port variables for its formal parameters (so the macro `Model` is a complete, compilable sub-model); the 6 `test/test-models/tests/macro_*` `.mdl` fixtures import into datamodels with correct `MacroSpec`s and macro bodies (the design's "Done when"). +- `$`-suffixed time references in macro bodies translate to canonical engine time idents (the design's "Done when"). +- An unterminated `:MACRO:` block reports a clear parse error. +- `macros.AC1.1`, `macros.AC1.2`, `macros.AC1.5`, `macros.AC1.6`, `macros.AC1.7` are verified. + +Macro *invocations* are still plain equation text — resolution and expansion (so macros actually simulate) is Phase 3; multi-output invocation materialization is Phase 4. diff --git a/docs/implementation-plans/2026-05-13-macros/phase_03.md b/docs/implementation-plans/2026-05-13-macros/phase_03.md new file mode 100644 index 000000000..f58e6aa18 --- /dev/null +++ b/docs/implementation-plans/2026-05-13-macros/phase_03.md @@ -0,0 +1,266 @@ +# Vensim Macro Support — Phase 3: Compile-time expansion and single-output simulation + +**Goal:** Resolve and expand single-output macro invocations so macro-using models simulate with faithful Vensim semantics — per-invocation independent state, expression-valued arguments, macro-local helpers, macro-to-macro nesting, `$`-time access, and builtin-name shadowing. + +**Architecture:** Generalize the engine's existing **stdlib-as-modules** mechanism. `SMTH1`/`DELAY3`/`TREND`/`NPV` are small models that `BuiltinVisitor` instantiates as `Variable::Module` instances when it sees function-call syntax; the rest of the compiler and VM then handle them generically. Phase 2 already made each macro definition an ordinary `datamodel::Model` (with a `MacroSpec` and synthesized port variables), so a macro *is* structurally just another module-target model. Phase 3 adds: (1) a **module-function resolver** with a per-project **macro registry** that answers "is this call name a macro or a stdlib function, and what is its `{ model_name, parameter_ports, primary_output }` descriptor?" — with macros taking precedence over identically-named builtins; (2) a generalized `BuiltinVisitor` that consults the resolver and expands a macro call into a `Variable::Module` exactly the way it expands a stdlib call; (3) routing of the `model.rs` pre-classification sites through the resolver. The simulation compiler, the module-instantiation machinery, and the VM are **unchanged** — they are already model-origin-agnostic (ordinary user submodels already work as module targets today). + +**Tech Stack:** Rust — the engine compile pipeline (`builtins.rs`, `builtins_visitor.rs`, `model.rs`, `db.rs`, `variable.rs`) and a new `module_functions.rs` module. No external dependencies. + +**Scope:** 7 phases from the original design (`docs/design-plans/2026-05-13-macros.md`); this is phase 3 of 7. + +**Codebase verified:** 2026-05-14 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### macros.AC2: Single-output invocations simulate with correct Vensim semantics +- **macros.AC2.1 Success:** A stockless single-output macro invocation (`macro_expression`) simulates and matches the fixture's `output.tab`. +- **macros.AC2.2 Success:** A stock-bearing macro invocation (`macro_stock`) simulates with correct per-invocation integration, matching the 11-step `output.tab`. +- **macros.AC2.3 Success:** The same macro invoked at multiple call sites produces independent per-invocation state -- two stock-bearing invocations don't share a stock. +- **macros.AC2.4 Success:** A macro invoked with an expression-valued argument (`MYMACRO(a + b, t)`) simulates correctly, with the argument evaluated in the caller's context. +- **macros.AC2.5 Success:** A multi-equation macro body with macro-local helpers (`macro_multi_expression`) simulates correctly; helper names don't leak into the caller's namespace. +- **macros.AC2.6 Success:** A macro that calls another macro (`macro_cross_reference`) expands recursively and simulates correctly. +- **macros.AC2.7 Success:** A macro body referencing global time via the `$` escape simulates with the global time values. +- **macros.AC2.8 Edge:** A macro invocation nested inside a larger expression (`y = c + MYMACRO(x, t)`) expands and simulates correctly. + +### macros.AC5: Error handling and edge cases +- **macros.AC5.1 Failure:** A macro invoked with the wrong number of arguments reports an arity-mismatch diagnostic naming the macro. +- **macros.AC5.2 Failure:** A directly or mutually recursive macro is rejected with a cycle-detection error rather than expanding without termination. +- **macros.AC5.3 Failure:** Two macros sharing a name, or a macro name colliding with a model name, report a registry-build diagnostic. +- **macros.AC5.4 Success:** A macro shadowing a builtin (`SSHAPE`, `RAMP FROM TO`) resolves to the macro; the builtin is not invoked. +- **macros.AC5.5 Success:** A macro defined after its first use (`macro_trailing_definition`) still resolves and simulates. +- **macros.AC5.6 Failure:** A call to a name that is neither a macro, a stdlib function, nor a builtin reports an "unknown function or macro" error. + +--- + +## Current state (verified 2026-05-14) + +**The stdlib-as-modules mechanism — what Phase 3 generalizes:** +- `is_stdlib_module_function` (`builtins.rs:377-380`) is the predicate for "this name expands to a stdlib module"; `stdlib_args` (`builtins_visitor.rs:16-27`) is the **only** declaration of stdlib input-port names/order (`["input", "delay_time", "initial_value"]` for the smooth/delay/trend family, `["stream", "discount_rate", "initial_value", "factor"]` for `npv`). +- `BuiltinVisitor` (`builtins_visitor.rs:124-146`), its `walk()` (`builtins_visitor.rs:377-566`), and `instantiate_implicit_modules` (`builtins_visitor.rs:574-690`, signature `instantiate_implicit_modules(ident, ast, dimensions_ctx, module_idents) -> EquationResult<(Ast, Vec)>`, called from `variable.rs:513`). Inside `walk()`'s `Expr0::App` arm the dispatch order is: (1) `rewrite_alias_module_call` (normalizes `delay`/`delayn`/`smthn`), (2) special routing for `modulo`/`previous`/`init`, (3) `if is_builtin_fn(&func)` → a `BuiltinFn`, (4) else `let Some(ports) = stdlib_args(&func) else { return eqn_err!(UnknownBuiltin, ...) }` then the module rewrite (`builtins_visitor.rs:455-540`). **That `else` branch is exactly where an unrecognized macro call dies today.** +- The module rewrite (`builtins_visitor.rs:455-540`): synthetic instance name `format!("$⁚{}⁚{}⁚{}", self.variable_name, self.n, func)` (+ subscript suffix in A2A context); each argument hoisted into a synthetic `Aux` named `$⁚{var}⁚{n}⁚arg{i}`; one `ModuleReference { src: , dst: format!("{}.{}", module_name, port_name) }` per arg; a `datamodel::Variable::Module { ident: module_name, model_name: format!("stdlib⁚{}", func), references, ... }`; the call expression replaced with `Var(format!("{}·output", module_name))` — the `·output` suffix is **hardcoded**. +- The `model.rs` pre-classification sites: `collect_module_idents` (`model.rs:794-820`) and `equation_is_stdlib_call` (`model.rs:833-848`), both routing through `is_stdlib_module_function`. `collect_module_idents` is called from `ModelStage0::new` (`model.rs:882`, test-only), `ModelStage0::new_cached` (`model.rs:959`, production), and the salsa path `module_ident_context_for_model` (`db.rs:799`). + +**The module machinery is already model-origin-agnostic — Phase 3 needs NO compiler/VM changes:** +- `sync_from_datamodel` (`db.rs:2398-2524`) registers **every** `project.models` entry as a `SourceModel`. Since Phase 2 puts macro-marked models in `project.models`, they are **already registered** — no new registration step is needed. +- A `Variable::Module` compiles to `Expr::EvalModule(ident, model_name, input_set, inputs)` (`compiler/mod.rs:686-707`). `enumerate_modules_inner` (`model.rs:741-780`) resolves `model_name` against the project's models (`model_err!(BadModelName, ...)` if absent) and builds the `input_set: BTreeSet>` from the `ModuleReference.dst` idents. A sub-model variable is overridden as an input port purely by name-membership: `Var::new` (`compiler/mod.rs:2313-2324`) does `ctx.inputs.iter().find(|n| n.as_str() == var.ident())` → `AssignCurr(offset, ModuleInput(off))`. **No flag, no naming convention, no special equation** — the port variable just has to exist with that ident (Phase 2 synthesizes it) and be named in a `ModuleReference.dst`. +- `is_stdlib`/`implicit` is derived **solely** from the `"stdlib⁚"` name prefix (`db.rs:1710`, `project.rs:110-112`); `implicit` drives three stdlib-only behaviors (all-names→`module_idents`, parse-cache bypass, skip unit-inference). A macro model is an ordinary, non-prefixed `SourceModel` and **must not** get the `implicit` treatment — it compiles correctly as an ordinary sub-model because Phase 2 flags its port variables `can_be_module_input: true` (the same flag an ordinary XMILE submodel sets via `access="input"`). Ordinary user models are already instantiated as module targets today — `test/modules_hares_and_foxes/modules_hares_and_foxes.stmx`, `simulates_modules()` at `simulate.rs:378-379`. + +**Builtin-name shadowing must be resolved at COMPILE time:** `SSHAPE` and `RAMP FROM TO` are in the MDL `BUILTINS` table (`mdl/builtins.rs:332-333`) → the MDL parser classifies them `CallKind::Builtin` (→ `Token::Function`) **at parse time**, with no knowledge of a same-named macro. `SAMPLE UNTIL` is *not* in the table → `CallKind::Symbol`. So the resolver must consult the macro registry **before** honoring the builtin dispatch, for *both* `CallKind` values — `CallKind` cannot be trusted to detect a shadowing macro. + +**Errors:** `ErrorCode` (`common.rs:51-105`) has `UnknownBuiltin` (the current catch-all for an unrecognized call name), `BadBuiltinArgs` (the natural code for an arity mismatch), `CircularDependency` (for a recursion cycle), `DuplicateVariable` (the closest existing code for a duplicate-name collision). `EquationError { start: u16, end: u16, code: ErrorCode }` attaches a code to a byte span within an equation; `Error { kind, code, details: Option }` is the model/project-level error. Construction macros: `eqn_err!(code, start, end)`, `model_err!(code, str)`, `sim_err!(code [, str])`. Salsa path: `Diagnostic { model, variable, error: DiagnosticError, severity }`, `DiagnosticError::{Equation(EquationError), Model(Error), Assembly(String)}`. `compile_project_incremental` (`db.rs:5904-5913`) maps a returned `Err` → `sim_err!(NotSimulatable, msg)`. + +**Compile pipeline:** `sync_from_datamodel` → parse (`parse_source_variable_with_module_context`, threading `module_idents` from `module_ident_context_for_model`/`collect_module_idents` at `db.rs:799`) → `instantiate_implicit_modules` → `BuiltinVisitor` → synthetic `Variable::Module` + hoisted `Aux`es land as `implicit_vars` → `model_implicit_var_info` extracts `ImplicitVarMeta { is_module, model_name }` → dep graph → `enumerate_module_instances` → `assemble_module` (`dep_graph.has_cycle()` → `DiagnosticError::Assembly`) → `assemble_simulation` → `compile_project_incremental`. + +**Test surfaces:** `convert_mdl(source) -> Result` (`convert/mod.rs:315-318`, `#[cfg(test)]`, in-crate only). `open_vensim(&str) -> common::Result` (public). `simulate.rs`: `compile_vm(project)` (lines 96-102), `simulate_mdl_path(path)` (lines 277-296), `simulate_path` (lines 185-251). The three macro `.xmile` fixtures are commented out at `simulate.rs:27-29`; `#[ignore] simulates_clearn()` is at `simulate.rs:908-938`. New MDL fixtures wire in as `#[test] fn simulates__mdl() { simulate_mdl_path("../../test/.../X.mdl"); }`. `builtins_visitor.rs`'s `mod tests` holds the stdlib expansion tests (`test_arrayed_smooth1`, `test_npv_basic`, `test_arrayed_delayn_unsupported_order` which asserts `UnknownBuiltin`). `TestProject` (`testutils.rs`) with `assert_vm_result` / `assert_compile_error_vm` is the in-crate compile/run assertion surface. + +**Documented limitation — the non-time `$` escape.** Vensim's `$` escape has two uses: the *time* form (`Time$` — a macro body's access to global simulation time, AC2.7, fully supported; Phase 2 translates it) and the *non-time* form (`FOO$` — referencing some other non-time model variable from inside a macro body). Per the design, the non-time `$` form is **deprioritized and not supported** in this implementation. A non-time `$` reference is therefore *expected to fail*: it surfaces as an ordinary unknown-variable / unresolved-reference diagnostic at compile time (there is no macro-specific error code for it), and that failure is an **accepted documented limitation**, not a macro-handling bug. Phase 7's C-LEARN expansion assertion accounts for this — such a diagnostic is *not* macro-attributable. + +--- + + +## Subcomponent A: The module-function resolver and macro registry + + +### Task 1: `ModuleFunctionDescriptor`, `MacroRegistry`, and registry-build validation + +**Verifies:** macros.AC5.2, macros.AC5.3 (the detection logic — end-to-end surfacing is verified in Task 2). + +**Files:** +- Create: `src/simlin-engine/src/module_functions.rs` +- Modify: `src/simlin-engine/src/lib.rs` (add `mod module_functions;`) +- Modify: `src/simlin-engine/src/builtins_visitor.rs` (make `stdlib_args` `pub(crate)`, or move it into `module_functions.rs` — the implementor's choice; it should be the single source of truth for stdlib port names) + +**Implementation:** + +This task is a pure functional core — it takes datamodel values and returns a registry or an error, with no I/O and no compiler-pipeline plumbing (that is Task 2). Define in `module_functions.rs`: + +1. **`ModuleFunctionDescriptor`** — the unified answer for "what does this module-function expand into," serving both stdlib functions and macros: + ```rust + pub(crate) struct ModuleFunctionDescriptor { + /// The `datamodel::Model.name` of the target model — `"stdlib⁚smth1"` + /// for a stdlib function, or the macro's canonical model name. + pub model_name: String, + /// Ordered input-port variable names; call argument `i` wires to port `i`. + pub parameter_ports: Vec, + /// The body variable whose value the call expression is replaced with. + pub primary_output: String, + /// `:`-list additional output ports (empty for stdlib and for + /// single-output macros; consumed in Phase 4). + pub additional_outputs: Vec, + /// True for project macros (strict arity — argument count must equal + /// `parameter_ports.len()`); false for stdlib functions, which permit + /// fewer arguments than ports (trailing ports are optional). + pub is_macro: bool, + } + ``` + +2. **`stdlib_descriptor(name: &str) -> Option`** — builds a descriptor for a stdlib module-function. It is called *after* `rewrite_alias_module_call` has normalized aliases, so `name` is already a canonical stdlib model name. For a `name` where `stdlib_args(name)` is `Some`: `model_name = format!("stdlib⁚{name}")`, `parameter_ports = stdlib_args(name).to_vec()`, `primary_output = "output"`, `additional_outputs = vec![]`, `is_macro = false`. Otherwise `None`. (This preserves the existing stdlib behavior exactly — it just bundles the previously-scattered facts into one struct. Do **not** fold the `rewrite_alias_module_call` alias normalization into this — that stays a separate pre-step in `walk()`.) + +3. **`MacroRegistry`** — built once per project from all of its models: + ```rust + pub(crate) struct MacroRegistry { + /// canonical macro name -> descriptor + macros: HashMap, + } + ``` + - **`MacroRegistry::build(models: &[datamodel::Model]) -> Result`** — iterate `models`; a model is a macro iff `model.macro_spec.is_some()`. For each macro model, build a `ModuleFunctionDescriptor` from its `MacroSpec` (`model_name = model.name`, `parameter_ports = macro_spec.parameters`, `primary_output = macro_spec.primary_output`, `additional_outputs = macro_spec.additional_outputs`, `is_macro = true`). Keyed by the canonical macro name. + - **`MacroRegistry::resolve_macro(&self, call_name: &str) -> Option<&ModuleFunctionDescriptor>`** — canonicalize `call_name` and look it up. + - **Validation (returns `Err` from `build`):** + - **macros.AC5.3 — duplicate macro name:** two macro-marked models with the same canonical name → `Err` (use `model_err!(DuplicateVariable, ...)` or, if you add a dedicated `ErrorCode::DuplicateMacroName`, add it at the end of the enum — confirm `ErrorCode` is a runtime type not part of `project_io.proto` first, via grep; it is, so additions are safe). The message names the duplicated macro. + - **macros.AC5.3 — macro/model name collision:** a macro's canonical name equals a non-macro model's canonical name → `Err`, message naming the collision. + - **macros.AC5.2 — recursion cycle:** build the macro call graph — for each macro model, parse each body variable's equation text (`Expr0::new(text, LexerType::Equation)`), walk the AST for `App(name, ...)` nodes whose canonicalized `name` is another macro in the set, and add an edge `this_macro -> called_macro`. Run cycle detection over the graph (DFS, or reuse an existing cycle-detection utility if one is exposed in the crate). A cycle → `Err` with code `CircularDependency` and a message naming the cycle path (e.g. `"recursive macro: A -> B -> A"`). + +**Testing:** Pure-function unit tests in `module_functions.rs`'s inline `#[cfg(test)] mod tests`. Build small `Vec` fixtures by hand (or with `TestProject` if it is convenient — but hand-built `datamodel::Model`s with `macro_spec: Some(...)` are fine and direct): +- `stdlib_descriptor("smth1")` returns the expected ports/output; `stdlib_descriptor("not_a_thing")` returns `None`. +- `MacroRegistry::build` over a project with one macro → `resolve_macro` returns its descriptor; `resolve_macro` of a non-macro name → `None`. +- A macro whose name equals a stdlib name → `resolve_macro` still returns the macro descriptor (the *precedence* is enforced in Task 3's `walk()` ordering, but confirm the registry itself stores and returns the macro). +- **macros.AC5.3:** two macro models named `FOO` → `build` returns `Err` naming `FOO`. A macro named `main` alongside a `main` model → `build` returns `Err` naming the collision. +- **macros.AC5.2:** a macro `A` whose body calls `A` → `build` returns `Err` with `CircularDependency`. A macro `A` calling `B` calling `A` → `build` returns `Err`. A macro `A` calling `B` (no cycle, the `macro_cross_reference` shape) → `build` succeeds. + +**Verification:** +Run: `cargo test -p simlin-engine module_functions` +Expected: all resolver/registry/validation unit tests pass. + +**Commit:** `engine: add the module-function resolver and macro registry` + + + + +## Subcomponent B: Wire the resolver into compilation + + +### Task 2: Build the registry during compilation; route the `model.rs` classification sites + +**Verifies:** macros.AC5.2, macros.AC5.3 (end-to-end: a recursive or name-colliding macro `.mdl` produces a clear compile error). + +**Files:** +- Modify: `src/simlin-engine/src/model.rs` (`collect_module_idents` ~lines 794-820, `equation_is_stdlib_call` ~lines 833-848, `ModelStage0::new` ~line 853, `ModelStage0::new_cached` ~line 932) +- Modify: `src/simlin-engine/src/db.rs` (`module_ident_context_for_model` ~lines 789-811; the compile entry `compile_project_incremental` ~lines 5904-5913) +- Modify: `src/simlin-engine/src/project.rs` (`Project::from_datamodel` / `from_salsa` — the non-incremental compile entry) + +**Implementation:** + +1. **Build the registry once per compile.** At the compile entry points, build `MacroRegistry::build(&project_models)` and make it available to the parse/classification stages. In the salsa path this is ideally a salsa-tracked query keyed on the macro-marked `SourceModel`s; in the non-incremental path (`Project::from_datamodel`/`from_salsa`) build it once. Thread `&MacroRegistry` to wherever `module_idents` is computed — `module_ident_context_for_model` (`db.rs:799`) and `ModelStage0::new`/`new_cached` (`model.rs`). + +2. **Surface registry-build errors as compile failures.** If `MacroRegistry::build` returns `Err` (AC5.2 cycle, AC5.3 duplicate/collision), surface it as a compile failure with that error's message — a project-level error is the natural fit since registry-build precedes per-model processing (propagate the `Err`; `compile_project_incremental` maps it to `sim_err!(NotSimulatable, msg)`). If the salsa diagnostic accumulator is reachable at registry-build time, emit a `Diagnostic` instead. Either way the surfaced message must clearly identify the offending macro(s) / cycle. + +3. **Route the `model.rs` pre-classification through the resolver.** `equation_is_stdlib_call` currently returns `is_stdlib_module_function(&canonicalize(&func))`. Change it to also return true when `func` resolves to a macro: take `&MacroRegistry` as a parameter and return `is_stdlib_module_function(name) || registry.resolve_macro(name).is_some()`. (Consider renaming it `equation_is_module_call` since it is no longer stdlib-only — optional cleanup.) Thread `&MacroRegistry` into `collect_module_idents` and on to `equation_is_stdlib_call`. This makes a caller variable whose equation is `y = MYMACRO(...)` get pre-classified into `module_idents`, exactly as a `y = SMTH1(...)` variable is — required so that `PREVIOUS(y)` rewrites correctly. + + **Scope note — `Equation::Arrayed`:** `equation_is_stdlib_call` inspects only `Equation::Scalar` and `Equation::ApplyToAll`; it returns `false` immediately for `Equation::Arrayed` (the per-element-equation form). This is sufficient for Phase 4's apply-to-all `macro_arrayed` fixture, whose arrayed invocation `out[Region] = SCALE(inp[Region], factor)` is stored as `Equation::ApplyToAll`. A macro invocation written as a *per-element* `Equation::Arrayed` would not be pre-classified through this path — but that exactly matches the **pre-existing** behavior for arrayed stdlib calls, so it is not a macro-specific regression and is out of Phase 3 scope; an executing engineer should not chase a per-element-equation macro call's pre-classification miss as a bug. + +Do **not** thread the registry into `instantiate_implicit_modules` / `BuiltinVisitor` yet — that is Task 3. (A threaded-but-unused parameter would trip `clippy -D warnings`.) After Task 2: no-macro projects are byte-for-byte unchanged; a valid macro project classifies macro-call variables correctly but its macro calls still hit `UnknownBuiltin` in `BuiltinVisitor` (fixed in Task 3); an *invalid* macro project (cycle / duplicate / collision) fails the compile with a clear error. + +**Testing:** +- A focused in-crate unit test of `equation_is_stdlib_call` (its new signature) in `model.rs`'s `#[cfg(test)]` module: with a `MacroRegistry` containing a macro `MYMACRO`, assert it returns `true` for an equation `MYMACRO(a, b)` and `true` for `SMTH1(x, 5)` and `false` for `a + b`. +- **macros.AC5.2** (end-to-end): `convert_mdl` a `.mdl` with a directly recursive macro (its body calls itself) and a `main` invocation, compile it, and assert the compile fails with a cycle-detection error (`CircularDependency`) whose message names the macro. Repeat for a mutually recursive `A`/`B` pair. +- **macros.AC5.3** (end-to-end): `convert_mdl` a `.mdl` with two `:MACRO:` blocks of the same name → assert the compile fails with a duplicate-name error naming the macro. A `.mdl` with a macro named `main` → assert the compile fails with a collision error. +- Existing engine tests still pass unchanged (no-macro projects are unaffected). + +**Verification:** +Run: `cargo test -p simlin-engine model::` +Run: `cargo test -p simlin-engine --test simulate` +Expected: all pass, including the new cycle / duplicate / collision tests; existing simulation tests unchanged. + +**Commit:** `engine: build the macro registry during compilation` + + + +### Task 3: Generalize `BuiltinVisitor` to expand macro calls + +**Verifies:** macros.AC2.1 (end-to-end smoke), macros.AC5.1, macros.AC5.4, macros.AC5.6. + +**Files:** +- Modify: `src/simlin-engine/src/builtins_visitor.rs` (`BuiltinVisitor` struct ~lines 124-146; `walk()`'s `App` arm ~lines 377-566; the module-rewrite region ~lines 455-540; `instantiate_implicit_modules` ~lines 574-690) +- Modify: `src/simlin-engine/src/variable.rs` (the `parse_var_with_module_context` parse path that calls `instantiate_implicit_modules` ~line 513) +- Modify: `src/simlin-engine/src/model.rs`, `src/simlin-engine/src/db.rs` (extend the Task 2 registry threading the rest of the way down to the `instantiate_implicit_modules` call) +- Test: `src/simlin-engine/src/builtins_visitor.rs` inline `#[cfg(test)] mod tests` (or a new `src/simlin-engine/src/macro_expansion_tests.rs` declared `#[cfg(test)] mod macro_expansion_tests;` if the set grows large) + +**Implementation:** + +1. **Thread the registry to `BuiltinVisitor`.** Add a `macro_registry: &'a MacroRegistry` field to the `BuiltinVisitor` struct (alongside `module_idents`). Add it as a parameter to `instantiate_implicit_modules` and extend the Task 2 threading from `ModelStage0` through `variable.rs`'s parse path so the registry reaches the `instantiate_implicit_modules` call at `variable.rs:513`. + +2. **Generalize the module rewrite into a descriptor-driven helper.** Extract the existing stdlib module-rewrite (`builtins_visitor.rs:455-540`) into a helper that takes a `&ModuleFunctionDescriptor` and the call's args/loc and produces the synthetic `Variable::Module` + hoisted arg `Aux`es + the replacement expression. Changes vs. the current hardcoded code: + - `model_name` comes from `descriptor.model_name` (was `format!("stdlib⁚{func}")`). + - The `ModuleReference.dst` port names come from `descriptor.parameter_ports` (was `stdlib_args`). + - The replacement expression is `Var(format!("{}·{}", module_name, descriptor.primary_output))` (was hardcoded `·output`). For stdlib descriptors `primary_output` is `"output"`, so stdlib expansion is byte-for-byte unchanged. **Separator layering (do not confuse with Phase 4):** the `·` (U+00B7) here is the *compile-time AST* form — `BuiltinVisitor` rewrites an `Expr0::Var` node directly (post-parse), so it uses the already-canonical separator. This is a *distinct layer* from Phase 4's *import-time datamodel* materialization, where a multi-output binding aux's `Equation::Scalar` text is written with an ASCII period (`{module}.{output}`) that `canonicalize()` converts to `·` only later, when that equation string is parsed during compilation. Both are correct — this phase operates on the AST, the Phase 4 MDL converter operates on datamodel equation strings. + - **Arity check:** if `descriptor.is_macro` and `args.len() != descriptor.parameter_ports.len()`, return `eqn_err!(BadBuiltinArgs, loc.start, loc.end)`. For stdlib descriptors (`is_macro == false`), preserve the current lenient behavior exactly — do not add an arity check to the stdlib path (stdlib functions legitimately accept fewer args than ports; e.g. `SMTH1` with the `initial_value` port unwired). + The synthetic-name (`$⁚{var}⁚{n}⁚{func}`), arg-hoisting, and A2A subscript-suffix logic are reused unchanged. + +3. **Consult the resolver in `walk()`'s `App` arm.** Make two changes to the dispatch: + - **At the very top of the `App` arm** (before `rewrite_alias_module_call`, before the `modulo`/`previous`/`init` routing, before `is_builtin_fn`): `if let Some(desc) = self.macro_registry.resolve_macro(&func) { return }`. This is the macro-shadows-everything precedence — a project macro named `SSHAPE` is expanded as the macro even though `SSHAPE` parsed as `CallKind::Builtin`. + - **At the existing stdlib branch** (the `stdlib_args` `else` that currently emits `UnknownBuiltin`): replace `let Some(ports) = stdlib_args(&func) else { eqn_err!(UnknownBuiltin) }` with `let Some(desc) = stdlib_descriptor(&func) else { return eqn_err!(UnknownBuiltin, ...) }` and expand via the same descriptor-driven helper. (`UnknownBuiltin` still fires for a name that is neither a macro, nor an `is_builtin_fn` builtin, nor a stdlib module — satisfying AC5.6. Optionally update the user-facing message text to say "unknown function or macro.") + +4. **Make `contains_stdlib_call` macro-aware.** `contains_stdlib_call` (`builtins_visitor.rs:30-56`) is a separate recognition predicate that gates the `Ast::ApplyToAll` / `Ast::Arrayed` per-element expansion paths inside `instantiate_implicit_modules` — it currently recognizes only `is_stdlib_module_function` names (plus `init`/`previous`). Thread `&MacroRegistry` into it (it is called from `instantiate_implicit_modules`, which now has the registry) and have it also return `true` when an `App`'s name resolves to a macro. Without this, a *scalar* macro call would expand (via the `walk()` change above) but an *arrayed* macro invocation would never enter the per-element expansion path. This is the change that lets Phase 4's arrayed-invocation work "ride the existing apply-to-all path for free"; Phase 4 adds the arrayed fixtures and the AC3.4/AC3.5 verification. + +After Task 3 a macro call expands into a `Variable::Module` pointing at the macro's model. Because Phase 2 made the macro `Model` a complete sub-model (port variables flagged `can_be_module_input: true`), and `sync_from_datamodel` already registers it, and the module machinery is origin-agnostic, the call now compiles and simulates end-to-end — no further wiring is needed. + +**Testing:** Tests build a macro-bearing `datamodel::Project` with `convert_mdl(inline_mdl_str)` (in-crate, test-only) and compile/run it via the in-crate compile path (the same path `TestProject`'s `assert_vm_result` / `assert_compile_error_vm` use — if `TestProject` cannot wrap a `convert_mdl`-produced `Project`, add a minimal `TestProject::from_datamodel`-style constructor, a small general test helper). +- **Structural** (via `instantiate_implicit_modules` directly, or by inspecting the compiled project): a macro call `y = MYMACRO(a, b)` produces a synthetic `Variable::Module` whose `model_name` is the macro's model, with one `ModuleReference` per parameter (`dst` ports matching `MacroSpec.parameters`), and the caller equation replaced by a reference to `·`. +- **`contains_stdlib_call` macro-awareness (structural, non-simulation):** a focused unit test for item 4's change, so the apply-to-all gate's contract is verified *in Phase 3* rather than deferred entirely to Phase 4's arrayed fixtures. With a `MacroRegistry` containing a macro `MYMACRO`, call `contains_stdlib_call` directly on parsed `Expr0` ASTs and assert it returns `true` for an arrayed macro `App` (`MYMACRO(x[Dim], k)`), `true` for a stdlib call (`SMTH1(x, 5)`), and `false` for a plain arithmetic expression (`a + b`). Phase 4's arrayed fixtures then exercise the gate end-to-end, but the predicate's behavior is no longer entirely deferred to Phase 4. +- **macros.AC5.1:** a macro declared with 2 parameters, invoked with 3 args (and separately with 1 arg) → compile fails with `ErrorCode::BadBuiltinArgs`; assert the error span covers the macro call (so the macro is identified in context). +- **macros.AC5.4:** a `.mdl` defining `:MACRO: SSHAPE(x, p)` with body `SSHAPE = x + p`, invoked `y = SSHAPE(3, 4)` → compiles and simulates with `y == 7` (the macro's definition), proving the macro shadowed the `SSHAPE` builtin. Repeat for a `RAMP FROM TO` macro. +- **macros.AC5.6:** a `.mdl` with `y = NOTAFUNCTION(x)` (a name that is neither macro, stdlib, nor builtin) → compile fails with `ErrorCode::UnknownBuiltin`. +- **macros.AC2.1 smoke (end-to-end):** `convert_mdl` a trivial single-output macro (`:MACRO: M(a, b)` / `M = a * b`, invoked `y = M(5, 1.1)`), compile, run the VM, assert `y == 5.5` — confirming the full expansion → registration → compilation → VM path works for macros. + +**Verification:** +Run: `cargo test -p simlin-engine builtins_visitor` +Run: `cargo test -p simlin-engine --test simulate` (existing stdlib-module tests must still pass — stdlib expansion is unchanged) +Expected: all pass, including the new macro-expansion tests. + +**Commit:** `engine: expand macro invocations through BuiltinVisitor` + + + + +## Subcomponent C: Single-output simulation fixtures and edge cases + + +### Task 4: Wire the macro `.mdl` fixtures into `simulate.rs` and add focused simulation tests + +**Verifies:** macros.AC2.1, macros.AC2.2, macros.AC2.3, macros.AC2.4, macros.AC2.5, macros.AC2.6, macros.AC2.7, macros.AC2.8, macros.AC5.4, macros.AC5.5. + +**Files:** +- Modify: `src/simlin-engine/tests/simulate.rs` (add dedicated `#[test] fn`s; the three macro `.xmile` fixtures commented out at lines 27-29 stay commented — they are wired in Phase 5, because the XMILE reader does not turn their `` elements into macro-marked models until Phase 5 Task 1) + +**Implementation:** Tests only — no production code (if a test surfaces a real wiring gap, fix it, but the expectation after Task 3 is that single-output macros simulate). Two groups: + +**Group 1 — the six bundled `.mdl` fixtures**, each wired as a dedicated test calling `simulate_mdl_path` (which does `open_vensim` → `compile_vm` → run → `ensure_results` against the fixture's `output.tab`): +- `simulates_macro_expression_mdl` → `../../test/test-models/tests/macro_expression/test_macro_expression.mdl` — **macros.AC2.1** (stockless single-output). +- `simulates_macro_stock_mdl` → `.../macro_stock/test_macro_stock.mdl` — **macros.AC2.2** (stock-bearing, the 11-step `output.tab`). +- `simulates_macro_multi_expression_mdl` → `.../macro_multi_expression/test_macro_multi_expression.mdl` — **macros.AC2.5** (multi-equation body; the `intermediate` helper must not leak into `main`'s namespace — `ensure_results` already only checks expected columns, but additionally assert the `main` model has no `intermediate` variable). +- `simulates_macro_cross_reference_mdl` → `.../macro_cross_reference/test_macro_cross_reference.mdl` — **macros.AC2.6** (macro calls another macro). +- `simulates_macro_multi_macros_mdl` → `.../macro_multi_macros/test_macro_multi_macros.mdl` — two independent macros. +- `simulates_macro_trailing_definition_mdl` → `.../macro_trailing_definition/test_macro_trailing_definition.mdl` — **macros.AC5.5** (macro defined after its first use). + +**Group 2 — focused tests** for the four behaviors with no bundled fixture. Each uses an inline `.mdl` string passed to `open_vensim`, compiled via `compile_vm`, run, with **hand-computed** expected values asserted against the `Results` (use the `Results` API directly, or build a small in-code expected `Results` and call `ensure_results` — the implementor picks the lighter mechanism). Keep each macro trivial so the hand computation is reliable; document the arithmetic in a comment. +- **macros.AC2.3** — independent per-invocation state: a stock-bearing macro `M(rate, init) = INTEG(rate, init)` invoked twice with different arguments (`x = M(1, 0)`, `y = M(2, 10)`); over the run, assert `x` integrates `1`/step from `0` and `y` integrates `2`/step from `10` — they do not share a stock. +- **macros.AC2.4** — expression-valued argument: `M(in, p) = in * p` invoked `y = M(a + b, t)` with constants `a`, `b`, `t`; assert `y == (a + b) * t`, with the argument evaluated in the caller's context. +- **macros.AC2.7** — `$`-time access: a macro body referencing `Time$` (e.g. `M(x) = x + Time$`) invoked `y = M(10)`; assert `y == 10 + time` at each step. (This exercises Phase 2's `$`-time translation plus Phase 3's global-time-resolves-inside-a-module behavior.) +- **macros.AC2.8** — nested invocation: `y = c + M(x, t)` (a macro call inside a larger expression); assert `y` equals `c` plus the macro's value. +- **macros.AC5.4** (simulation-level confirmation): a macro shadowing `SSHAPE` invoked in a model that also uses other builtins; assert the simulated value matches the macro's definition, not the `SSHAPE` builtin's. (Task 3 verifies AC5.4 at the expansion level; this confirms it end-to-end in `simulate.rs`, matching the design's "Done when" listing builtin-name shadowing among the focused fixtures.) + +**Verification:** +Run: `cargo test -p simlin-engine --test simulate` +Expected: all six fixture tests and all focused tests pass. + +**Commit:** `engine: wire macro fixtures and focused single-output simulation tests` + + + +--- + +## Phase 3 completion check + +When all four tasks are committed: +- The module-function resolver unifies stdlib and macro lookup behind one `ModuleFunctionDescriptor`; the macro registry is built per-compile with duplicate-name, name-collision, and recursion-cycle validation. +- `BuiltinVisitor` expands a macro call into a `Variable::Module` exactly as it expands a stdlib call; a project macro shadows an identically-named builtin or stdlib function. +- The six single-output `.mdl` macro fixtures simulate and match their `output.tab`, wired into `simulate.rs`; focused tests cover a macro at multiple call sites, expression-valued arguments, `$`-time access, builtin-name shadowing, and nested invocation (the design's "Done when"). +- Arity-mismatch, recursion-cycle, registry-collision, and unknown-name errors all report clear diagnostics. +- `macros.AC2.1`–`AC2.8` and `macros.AC5.1`–`AC5.6` are verified. +- The simulation compiler, module-instantiation machinery, and VM are unchanged — no compiler/VM edits were needed, only the front-end resolver and `BuiltinVisitor` generalization. + +Multi-output (`:`-list) invocation and arrayed invocation are Phase 4; XMILE round-trip is Phase 5; MDL export is Phase 6. diff --git a/docs/implementation-plans/2026-05-13-macros/phase_04.md b/docs/implementation-plans/2026-05-13-macros/phase_04.md new file mode 100644 index 000000000..3ff9b4754 --- /dev/null +++ b/docs/implementation-plans/2026-05-13-macros/phase_04.md @@ -0,0 +1,164 @@ +# Vensim Macro Support — Phase 4: Multi-output and arrayed invocation + +**Goal:** Support Vensim's `:` multi-output macro call form (`total = add3(a, b, c : minv, maxv)`) by materializing it at MDL-import time as an explicit `Variable::Module` plus binding auxiliary variables, and confirm that arrayed (apply-to-all) macro invocation rides the engine's existing per-element unrolling. + +**Architecture:** The two halves are asymmetric. **(a) Multi-output** is the only macro form that cannot be expressed as plain equation text (a call returns several named values at once), so it is *materialized at import* in `src/simlin-engine/src/mdl/convert/`: a multi-output invocation becomes an explicit `datamodel::Variable::Module` (pointing at the macro's model, with the call arguments wired as input `ModuleReference`s) plus one binding `Variable::Aux` per output — the LHS aux reads `.`, and each `:`-list aux reads `.`. The module-output reference resolves through the existing `get_submodel_offset` machinery, which is fully general — a binding aux's equation text `module_instance.output_name` (ASCII period at the datamodel layer — see the authoritative Separator note in Current state below) canonicalizes to the `·` (U+00B7) form and resolves like any other reference. **(b) Arrayed invocation** needs no new mechanism: Phase 3 already made `BuiltinVisitor`'s recognition (`walk()` and `contains_stdlib_call`) macro-aware, and the `instantiate_implicit_modules` apply-to-all path builds one independent synthetic module per dimension element regardless of whether the module-function is a stdlib function or a macro. Phase 4 therefore *verifies* arrayed invocation with fixtures rather than building new machinery. + +**Tech Stack:** Rust — the MDL converter (`src/simlin-engine/src/mdl/convert/`), with fixture files under `test/test-models/tests/`. No external dependencies. + +**Scope:** 7 phases from the original design (`docs/design-plans/2026-05-13-macros.md`); this is phase 4 of 7. + +**Codebase verified:** 2026-05-14 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### macros.AC3: Multi-output and arrayed invocation +- **macros.AC3.1 Success:** A multi-output invocation (`total = add3(a,b,c : minv, maxv)`) materializes as a module instance; `total` receives the primary output and `minv`/`maxv` become model variables holding the additional outputs. +- **macros.AC3.2 Success:** `minv` and `maxv` are referenceable by subsequent equations and carry the correct values. +- **macros.AC3.3 Success:** The metasd multi-output models (`THEIL`, `SSTATS`) expand and simulate. +- **macros.AC3.4 Success:** An arrayed invocation (`y[Dim] = MYMACRO(x[Dim], …)`) expands into one independent macro instance per dimension element. +- **macros.AC3.5 Edge:** An arrayed invocation of a stock-bearing macro gives each element its own persistent stock. + +--- + +## Current state (verified 2026-05-14) + +**The `·` module-output reference convention is fully general (the crux of multi-output materialization):** +- `canonicalize()` (`common.rs:321`, the replacement at `common.rs:343-346`) turns a `.` in an identifier segment into U+00B7. The MDL lexer (`lexer/mod.rs:307`) accepts `.` as an identifier-continue char, so `module.output` lexes as one identifier and lowers (`ast/expr1.rs:73`) to a canonicalized `Var`. So a `datamodel::Equation::Scalar("module_instance.output_name")` is a valid reference — the period auto-canonicalizes to `·`. +- `get_submodel_offset` (`compiler/context.rs:405-455`) and `get_submodel_metadata` (`compiler/context.rs:372-403`) split on U+00B7 and recurse into the named submodule — **for any module, regardless of how it was created** (stdlib-synthesized or explicitly authored). `get_offset` (`context.rs:124`) → `get_submodel_offset`. Dependency resolution also handles `·`: `all_deps` (`model.rs:365-414`), `module_output_deps` (`model.rs:203-248`). So a binding `Aux { equation: Scalar("add3_call.minval") }` resolves to the `add3_call` module's `minval` output. + +**Separator (authoritative for Phases 4-6).** The materialized binding-aux equation text uses an **ASCII period** — `format!("{}.{}", module_ident, output)` — and the datamodel layer stores exactly that ASCII `.` form. `canonicalize()` converts it to U+00B7 (`·`) only later, when the equation string is parsed during compilation. So: the *datamodel* (`Equation::Scalar` text — what `convert_mdl` produces and what `convert/`-level tests inspect) uses `.`; the *compiled AST / VM* layer uses `·`. Phase 3's `BuiltinVisitor` rewrite emits `·` directly because it operates on the post-parse AST — a distinct layer, not a contradiction. Phase 5 (XMILE round-trip) and Phase 6 (MDL reconstruction) both read the datamodel form, so both match against `.`. + +**The datamodel `Module` shape and the `BuiltinVisitor` precedent:** +- `datamodel.rs:344-370`: `ModuleReference { src: String, dst: String }`; `Module { ident, model_name, documentation, units, references: Vec, ai_state, uid, compat }`; `Variable::Module(Module)`. +- `BuiltinVisitor` (`builtins_visitor.rs:389-541`) is the precedent for minting a `Variable::Module` programmatically: input `ModuleReference`s have `dst = format!("{}.{}", module_ident, input_name)` and `src =` the caller-side argument (a hoisted-arg ident). `parse_var` (`variable.rs:644-669`) turns every `ModuleReference` into a `ModuleInput`. +- **`ModuleReference`s are inputs-only by compile time** — `build_module_inputs` (`db.rs:3232-3256`) strips any reference whose `dst` does not start with the module's own prefix (i.e. output-direction ``s are dropped). A parent reads a submodule's *output* purely via `·`-equation text, never via an output reference. **So Phase 4's materialized `Variable::Module` carries only input `ModuleReference`s; the outputs are realized as separate binding `Aux`es whose equation text is `.`.** + +**The MDL convert path:** +- `MdlItem::Macro` is currently ignored at `convert/mod.rs:248` (Phase 2 changes this — each `MacroDef` becomes a macro-marked `datamodel::Model` with a `MacroSpec` and synthesized port variables, added to `project.models`). +- A regular invocation equation: `build_project` (`convert/variables.rs:28-106`) → `build_variable` (`convert/variables.rs:798-854`) → `build_equation` (`convert/variables.rs:859-939`), which formats the RHS via `self.formatter.format_expr(expr)`. The `XmileFormatter`'s `App` arm — `format_call_ctx` (`xmile_compat.rs:219-498`) — currently formats a multi-arg `CallKind::Symbol` call as plain text `MACRONAME(arg1, arg2, ...)` (`xmile_compat.rs:479-497`); the lookup-invocation special case is `args.len() == 1` only. After Phase 2 adds the 6th `output_bindings` field to `Expr::App`, Phase 2 places a `debug_assert!(output_bindings.is_empty(), ...)` in `format_call_ctx`'s `App` arm — **so Phase 4 must materialize multi-output invocations in the converter *before* anything formats them.** `build_variable` returns `Result, ConvertError>` (one variable) — multi-output materialization produces *several* datamodel variables, so it cannot be a plain `build_variable` call. +- `ConvertError` (`convert/types.rs:11-26`): `Reader | View | InvalidRange | CyclicDimensionDefinition | Import | Other`. No macro-specific variant; Phase 4 uses `Other(String)` (or adds a variant, updating the `Display`/`Error`/`From` impls in the same file). + +**The apply-to-all path is generic (arrayed invocation rides it for free):** +- `instantiate_implicit_modules` (`builtins_visitor.rs:574-690`): the `Ast::ApplyToAll` arm (`:588-621`) and `Ast::Arrayed` arm (`:622-688`) build one `BuiltinVisitor::new_with_subscript_context` per subscript, so each dimension element gets its own subscript-suffixed synthetic module (`$⁚{var}⁚{n}⁚{func}⁚{subscript_suffix}`). The only stdlib-specific gate was `contains_stdlib_call` (`builtins_visitor.rs:30-56`) → `is_stdlib_module_function` — **Phase 3 (Task 3) makes `contains_stdlib_call` macro-aware**, so an arrayed macro invocation now enters the per-element expansion path. Nothing else in the A2A path is stdlib-coupled. Existing arrayed-module tests: `test_arrayed_smooth1`, `test_arrayed_delay1_numerical_values` in `builtins_visitor.rs`'s `mod tests`. + +**The real-world multi-output / arrayed corpus models (verified by a repo-wide scan):** +- **Multi-output `:` form** — exactly two `.mdl` files in the whole repo use it: `test/metasd/theil-statistics/Theil_2011.mdl` — `:MACRO: THEIL(historical,simulated:R2,MAPE,RMSPE,RMSE,MSE,SSE,Dif Mea,Dif Var,Dif Cov,Um,Us,Uc,Count)` (2 inputs, 13 outputs), invoked at `Theil_2011.mdl:518-519`; and `test/metasd/covid19-us-homer/homer v8/Covid19US v8.mdl` — `:MACRO: SSTATS(historical,simulated:R2,MAE,MAE over Mean,MAPE,RMSE,MSE,Um,Us,Uc,Count)` (2 inputs, 10 outputs), with two invocations. The six bundled `test/test-models/tests/macro_*` fixtures are **all single-output** — Phase 4 must author its own focused multi-output fixture. +- **Arrayed invocation** — C-LEARN (`test/xmutil_test_models/C-LEARN v77 for Vensim.mdl`) has ~8 arrayed `[COP]` / `[COP,Target]` macro invocations of `SAMPLE UNTIL`, `RAMP FROM TO`, `SSHAPE` (some nested inside `IF THEN ELSE`); all four C-LEARN macros are single-output. C-LEARN exercises arrayed invocation but not the `:` multi-output form. + +**Test surfaces (recap):** `convert_mdl(source) -> Result` (`convert/mod.rs:315-318`, in-crate test-only). `open_vensim(&str) -> common::Result` (public). `simulate.rs`: `compile_vm`, `simulate_mdl_path`. New MDL fixtures wire in as `#[test] fn simulates__mdl() { simulate_mdl_path("../../test/.../X.mdl"); }`. Large-model tests must be `#[ignore]`d with a documented opt-in command per `docs/dev/rust.md` test-time-budget rules. + +--- + + +## Subcomponent A: Multi-output materialization and arrayed-invocation verification + + +### Task 1: Materialize multi-output invocations in the MDL converter + +**Verifies:** macros.AC3.1, macros.AC3.2. + +**Files:** +- Modify: `src/simlin-engine/src/mdl/convert/variables.rs` (the variable-building loop in `build_project` ~lines 28-106, and/or `build_variable` ~lines 798-854) — and/or a new `convert/` step +- Possibly modify: `src/simlin-engine/src/mdl/convert/types.rs` (a `ConvertError` variant, if `Other(String)` is not preferred) +- Create: `test/test-models/tests/macro_multi_output/test_macro_multi_output.mdl` and `test/test-models/tests/macro_multi_output/output.tab` +- Modify: `src/simlin-engine/tests/simulate.rs` (wire the new fixture) +- Test: `src/simlin-engine/src/mdl/convert/` inline `#[cfg(test)]` tests (the same module Phase 2's macro-conversion tests live in) + +**Implementation:** + +A multi-output invocation `total = add3(a, b, c : minv, maxv)` is parsed (Phase 2 Task 3) into an equation whose RHS is an `Expr::App` with a non-empty `output_bindings` field (`[minv, maxv]`). Phase 4 detects this and, instead of letting `build_equation` format it as text, materializes it. + +1. **Ordering.** The materialization needs the called macro's `MacroSpec` (its `parameters`, `primary_output`, `additional_outputs`). Phase 2 builds every macro-marked model into `project.models` (with `model.macro_spec` populated). Build a `name -> &MacroSpec` lookup over those macro-marked models, and run multi-output materialization at a point where both the lookup and the main model's variable list are available — either by ordering Phase 2's macro-model-building step before the main-model variable loop, or by running materialization as a step after `build_project`. The implementor picks the cleanest insertion given the actual `convert/` pass structure; the contract is: every multi-output invocation in the main model is materialized, and no multi-output `Expr::App` ever reaches `build_equation`/the formatter. + +2. **Detect.** For each main-model symbol whose equation's top-level RHS is an `Expr::App` with non-empty `output_bindings`: this is a multi-output invocation. (A multi-output call cannot legally be a sub-expression — it is always the whole RHS.) Look up the called name in the macro `MacroSpec` map. If it is not a known macro → `ConvertError` ("multi-output call to unknown macro ``"). Arity: `args.len()` must equal `MacroSpec.parameters.len()` and `output_bindings.len()` must equal `MacroSpec.additional_outputs.len()` — otherwise a `ConvertError` naming the macro and the expected/actual counts. + +3. **Materialize.** Emit, in place of the plain `total` aux: + - **One `Variable::Module`** — a deterministic, collision-safe, *serialization-stable* `ident`: **`{lhs}_macro`** — the LHS variable's canonical ident with a literal `_macro` suffix. If `{lhs}_macro` already collides with an existing symbol in the model, append the lowest numeric disambiguator that makes it unique (`{lhs}_macro_2`, `{lhs}_macro_3`, …). This is deliberately *not* the `$⁚` compile-time-synthetic prefix — this module is materialized at import, so it is serialized and round-tripped, and its ident must be stable and human-readable. `model_name` is the called macro's `Model.name`. `references` is one input `ModuleReference` per call argument: `dst = format!("{}.{}", module_ident, MacroSpec.parameters[i])`, `src =` the argument. For a simple `Var` argument, `src` is that variable's canonical name directly. For an expression-valued argument, hoist it into a synthetic `Aux` with a deterministic, serialization-safe name and use that aux's ident as `src`. (The real-world corpus — `THEIL`, `SSTATS` — uses only simple-`Var` arguments, so the hoisting path is for generality; if it proves involved, it may be deferred to a tracked follow-up, but simple-`Var` arguments must work.) + - **The primary-output binding aux** — `Variable::Aux { ident: , equation: Scalar(format!("{}.{}", module_ident, MacroSpec.primary_output)), .. }`. This replaces the would-be plain `total` aux; `total` now reads the module's primary output. + - **One additional-output binding aux per `:`-list entry** — for each `i`, `Variable::Aux { ident: , equation: Scalar(format!("{}.{}", module_ident, MacroSpec.additional_outputs[i])), .. }`. The call-site name (`output_bindings[i]`, e.g. `minv`) becomes the variable ident; the macro's internal output name (`MacroSpec.additional_outputs[i]`, e.g. `minval`) is what it reads from the module. These are brand-new model variables not present in `self.symbols`. + +4. **Author the fixture.** `test/test-models/tests/macro_multi_output/test_macro_multi_output.mdl`: a **stockless** multi-output macro so the expected output is constant over time and the `output.tab` is trivial to hand-compute. For example a macro `:MACRO: ADD3(a, b, c : minval, maxval)` with body `ADD3 = a + b + c`, `minval = MIN(a, MIN(b, c))`, `maxval = MAX(a, MAX(b, c))`; an invocation `total = ADD3(in1, in2, in3 : the min, the max)`; constant inputs (`in1`, `in2`, `in3`); and a **downstream equation** that references the additional outputs, e.g. `spread = the max - the min` (this is what proves AC3.2). Hand-compute `output.tab` with `total`, `the min`, `the max`, `spread` (all constant). Document the arithmetic in the fixture's `README.md` or a comment. + +5. **Wire into `simulate.rs`** as `#[test] fn simulates_macro_multi_output_mdl() { simulate_mdl_path("../../test/test-models/tests/macro_multi_output/test_macro_multi_output.mdl"); }`. + +**Testing:** +- **macros.AC3.1** (`convert/`-level structure test, `convert_mdl(include_str!(".../test_macro_multi_output.mdl"))`): assert the main model contains exactly one `Variable::Module` whose `model_name` is the `ADD3` macro's model, with input `ModuleReference`s wiring `in1`/`in2`/`in3` to the `a`/`b`/`c` ports; assert `total` is a `Variable::Aux` whose equation reads `.` (ASCII period — the datamodel form, per the authoritative separator note above); assert `the min` and `the max` are `Variable::Aux`es reading `.minval` and `.maxval` respectively. +- **macros.AC3.2** (simulation, via the wired `simulate.rs` fixture test): the fixture's `simulate_mdl_path` test passes — `total`, `the min`, `the max`, and the downstream `spread` all match the hand-computed `output.tab`, proving `the min`/`the max` are referenceable by a subsequent equation and carry correct values. +- Arity-mismatch: a `convert/`-level test that `convert_mdl` of a `.mdl` invoking `ADD3` with the wrong number of arguments or the wrong number of `:`-outputs returns a `ConvertError` naming `ADD3`. + +**Verification:** +Run: `cargo test -p simlin-engine convert` +Run: `cargo test -p simlin-engine --test simulate simulates_macro_multi_output_mdl` +Expected: all pass. + +**Commit:** `engine: materialize multi-output macro invocations at import` + + + +### Task 2: Verify arrayed macro invocation + +**Verifies:** macros.AC3.4, macros.AC3.5. + +**Files:** +- Create: `test/test-models/tests/macro_arrayed/test_macro_arrayed.mdl` and `test/test-models/tests/macro_arrayed/output.tab` +- Modify: `src/simlin-engine/tests/simulate.rs` (wire the new fixture; add the focused arrayed-stock test) +- Possibly modify: `src/simlin-engine/src/builtins_visitor.rs` — only if a gap is found (see below) + +**Implementation:** This task is primarily **verification**. Phase 3 made both `BuiltinVisitor::walk()` and `contains_stdlib_call` macro-aware, and the `instantiate_implicit_modules` apply-to-all path (`builtins_visitor.rs:588-688`) builds one independent subscript-suffixed synthetic module per dimension element with no stdlib coupling beyond that recognition step. So an arrayed macro invocation should already expand into one independent macro instance per element. Author the fixtures and tests; **if a test surfaces a real gap** (something arrayed-macro-specific that does not ride the existing path), fix it in `builtins_visitor.rs` and document what was missing. + +1. **Author the arrayed fixture.** `test/test-models/tests/macro_arrayed/test_macro_arrayed.mdl`: a **stockless** macro invoked apply-to-all over a small dimension — e.g. a dimension `Region` with 2-3 elements, a macro `:MACRO: SCALE(x, k)` with body `SCALE = x * k`, an arrayed input `inp[Region]` with a different constant per element, and an arrayed invocation `out[Region] = SCALE(inp[Region], factor)`. Stockless ⇒ constant output ⇒ trivial hand-computed `output.tab` with one column per `out[Region]` element. Wire into `simulate.rs` as `simulates_macro_arrayed_mdl`. + +2. **Author the arrayed-stock focused test (inline `.mdl`).** AC3.5's per-element-independent-stock case is hard to express as a trivial `output.tab` file, so test it as an inline-`.mdl` test in `simulate.rs` (`open_vensim` + `compile_vm` + run + hand-computed assertions): a stock-bearing macro `:MACRO: ACCUM(rate)` with body `ACCUM = INTEG(rate, 0)`, an arrayed invocation `total[Region] = ACCUM(rate[Region])` where `rate[Region]` differs per element. Over the run, assert each `total[element]` integrates *its own* `rate[element]` independently (e.g. with `rate = [1, 3]` over 4 steps at dt=1: `total[r1] = 0,1,2,3,4` and `total[r2] = 0,3,6,9,12`) — proving each element got its own persistent stock. + +**Testing:** +- **macros.AC3.4** (via the wired `simulate.rs` fixture test): `simulates_macro_arrayed_mdl` passes — each `out[Region]` element equals `inp[element] * factor`, confirming one independent macro instance per dimension element. Additionally, a `convert/`- or expansion-level assertion that the arrayed invocation produced one synthetic `Variable::Module` per element (subscript-suffixed idents), not a single shared instance. +- **macros.AC3.5** (the inline arrayed-stock test): each element's stock integrates independently, as hand-computed — proving per-element persistent state. + +**Verification:** +Run: `cargo test -p simlin-engine --test simulate macro_arrayed` +Run: `cargo test -p simlin-engine --test simulate` (full file — confirm no regression) +Expected: all pass. + +**Commit:** `engine: verify arrayed macro invocation` + + + +### Task 3: Early validation against the multi-output and arrayed corpus models + +**Verifies:** macros.AC3.3. + +**Files:** +- Modify: `src/simlin-engine/tests/simulate.rs` (add the metasd `THEIL` / `SSTATS` and C-LEARN macro-expansion tests) + +**Implementation:** Tests only — early validation that the Phase 4 mechanisms work on real-world models (the comprehensive 14-model corpus harness and Vensim-reference-output comparison is Phase 7; this task is a focused early gate). These models are large, so `#[ignore]` the heavy ones with a documented opt-in command (`// Run with: cargo test --release -- --ignored `) per `docs/dev/rust.md` test-time-budget rules; measure `Theil_2011.mdl` and keep it as a regular test only if it compiles+runs within the per-test budget. + +**Testing:** +- **macros.AC3.3 — `THEIL`:** import `test/metasd/theil-statistics/Theil_2011.mdl` via `open_vensim`; assert the `THEIL` multi-output invocation materialized (the datamodel has a `Variable::Module` for the `THEIL` macro plus binding `Aux`es for the primary output and all 13 `:`-list outputs); compile via `compile_vm` and run to the end. The model "expands and simulates." +- **macros.AC3.3 — `SSTATS`:** same for `test/metasd/covid19-us-homer/homer v8/Covid19US v8.mdl` (note the spaces in the path). Assert both `SSTATS` invocations materialized (each: a `Variable::Module` + 1 primary + 10 additional binding auxes). Compile and run. If this large real-world COVID model turns out to have *unrelated, non-macro* blockers that prevent it reaching a runnable VM, narrow the assertion to "the `SSTATS` multi-output materialization succeeded and produced no macro-specific compile diagnostics" and file the unrelated blocker via the `track-issue` agent (it is then in scope for Phase 7's tiered corpus harness, not Phase 4). +- **C-LEARN macro expansion** (the design's Phase 4 "Done when: C-LEARN's four macros ... expand without macro-specific errors"): import `test/xmutil_test_models/C-LEARN v77 for Vensim.mdl`; assert its four macros (`SAMPLE UNTIL`, `SSHAPE`, `RAMP FROM TO`, `INIT`) imported as macro-marked `Model`s with correct `MacroSpec`s; compile far enough to confirm macro expansion succeeds — assert that the compile diagnostics (if any) contain **no macro-specific errors** (`UnknownBuiltin` for a macro name, `BadModelName` for a macro's model, an arity error on a macro call), even though C-LEARN has known unrelated blockers (circular dependencies, dimension mismatches, unit errors — explicitly out of scope per the design). This confirms the arrayed `[COP]` `SAMPLE UNTIL` / `RAMP FROM TO` / `SSHAPE` invocations expand. `#[ignore]` this test (C-LEARN is ~53k lines / 1.4 MB) with a documented opt-in command. Full C-LEARN reference-output validation is Phase 7. + +**Verification:** +Run: `cargo test -p simlin-engine --test simulate -- --ignored` (the gated corpus tests) +Run: `cargo test -p simlin-engine --test simulate` (the non-ignored subset, e.g. `Theil_2011.mdl` if it fits the budget) +Expected: the multi-output macros in `THEIL`/`SSTATS` materialize and the models expand and simulate; C-LEARN's macros expand with no macro-specific errors. + +**Commit:** `engine: early-validate multi-output and arrayed macros against corpus models` + + + +--- + +## Phase 4 completion check + +When all three tasks are committed: +- A multi-output invocation (`total = add3(a,b,c : minv, maxv)`) materializes at import as an explicit `Variable::Module` plus binding `Aux`es — `total` receives the primary output, and the `:`-list names become model variables holding the additional outputs, referenceable by subsequent equations (the design's "Done when": new multi-output fixtures pass). +- An arrayed macro invocation expands into one independent macro instance per dimension element, including per-element persistent stocks (the design's "Done when": an arrayed-invocation fixture passes). +- The metasd multi-output models `THEIL` and `SSTATS` expand and simulate; C-LEARN's four macros — including the arrayed `[COP]` invocations — expand with no macro-specific errors (the design's "Done when"). +- `macros.AC3.1`–`AC3.5` are verified. + +XMILE round-trip is Phase 5; MDL export (including reconstructing the `:` call syntax from the materialized module) is Phase 6; the full corpus harness and Vensim-reference-output validation are Phase 7. diff --git a/docs/implementation-plans/2026-05-13-macros/phase_05.md b/docs/implementation-plans/2026-05-13-macros/phase_05.md new file mode 100644 index 000000000..2a55f2562 --- /dev/null +++ b/docs/implementation-plans/2026-05-13-macros/phase_05.md @@ -0,0 +1,209 @@ +# Vensim Macro Support — Phase 5: XMILE import and export + +**Goal:** Round-trip macros through the XMILE format — flesh out the `xmile::Macro` type, make the XMILE reader produce macro-marked `datamodel::Model`s from `` elements, and make the XMILE writer emit `` elements, the `` header option, and `simlin:`-namespaced extensions for the multi-output forms that standard XMILE cannot express. + +**Architecture:** XMILE handling is **hybrid and asymmetric**: reading uses `quick_xml::de` + serde `Deserialize` derives (so a `` sibling of `` deserializes into the already-present `xmile::File.macros` field once `xmile::Macro` has fields); writing uses a hand-written `ToXml` trait + `quick_xml::Writer` (the `Serialize` derives are not used for XMILE output). A macro is a top-level `` element, a sibling of `` — so the bridge between `xmile::File.macros` and the macro-marked entries of `datamodel::Project.models` lives at the `File ↔ Project` level, while the macro *body* (variables/views) reuses the existing per-`Model` conversion. Standard XMILE `` expresses a single-output macro (`` inputs, ``/`` body); Vensim's `:`-multi-output form has no XMILE equivalent, so a multi-output macro's additional-output ports and a multi-output invocation's bindings round-trip through `simlin:`-namespaced extension elements. A single-output-only project therefore exports as standards-clean XMILE with no extensions. + +**Tech Stack:** Rust — `src/simlin-engine/src/xmile/`. The XMILE `` specification is in-repo at `docs/reference/xmile-v1.0.html` (section 4.8). No external dependencies. + +**Scope:** 7 phases from the original design (`docs/design-plans/2026-05-13-macros.md`); this is phase 5 of 7. + +**Codebase verified:** 2026-05-14 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### macros.AC1: Macro definitions parse and represent faithfully +- **macros.AC1.3 Success:** An XMILE `` element imports as a macro-marked `Model`; an expression-form `` is normalized into a macro-named body variable. + +### macros.AC4: Round-trip and export +- **macros.AC4.2 Success:** A macro-bearing XMILE file round-trips with `` elements and `simlin:` extensions; the `` header option is emitted. +- **macros.AC4.4 Success:** A cross-format conversion (`.mdl` → datamodel → `.xmile`) preserves macro definitions and invocations. +- **macros.AC4.5 Edge:** A single-output-only model exports as standards-clean XMILE with no extensions; multi-output triggers the `simlin:` extension. + +(`macros.AC4.1` and `macros.AC4.3` are MDL round-trip — Phase 6.) + +--- + +## Current state (verified 2026-05-14) + +**`xmile::Macro` is an empty stub, but `File` is already wired to deserialize `` siblings:** +- `xmile::Macro` (`xmile/mod.rs:334-338`) is `pub struct Macro { /* TODO */ }` — empty, with `Deserialize`/`Serialize` derived but no fields and no `ToXml` impl. +- `xmile::File` (`xmile/mod.rs:56-77`) already has `#[serde(rename = "macro", default)] pub macros: Vec` — so `` elements already deserialize into `file.macros` once `Macro` has fields. + +**XMILE is hybrid — read via serde derive, write via hand-written `ToXml`:** +- Reading: `project_from_reader` (`xmile/mod.rs:824-834`) → `quick_xml::de::from_reader` into `File` → `convert_file_to_project` → `datamodel::Project::from(file)`. +- Writing: `project_to_xmile` (`xmile/mod.rs:802-822`) → `project.clone().into()` (`From for File`) → `file.write_xml()` (the `ToXml` trait, `xmile/mod.rs:25-32`). Write helpers: `write_tag_start`, `write_tag_start_with_attrs`, `write_tag_end`, `write_tag`, `write_tag_with_attrs` (`xmile/mod.rs:398-445`). +- `compat.rs`: `open_xmile` (`:76-78`) and `to_xmile` (`:32-34`) are thin pass-throughs. + +**The XMILE `` element** (`docs/reference/xmile-v1.0.html` §4.8): a top-level child of ``, sibling of ``. **Required:** a `name` attribute and one `` child. **Optional children:** `` (a formal parameter; optional `default` attribute; ``s appear before ``), ``, ``, `` (only valid with ``), `` (same content model as a ``'s ``), ``; optional `namespace` attribute. §4.8.1 — expression-form: the body is a single `` over the ``s. §4.8.2 — with-variables: extra stocks/flows/auxes in ``, each macro invocation getting its own independent instances. A macro is *invoked* like a builtin function call: `MACRONAME(arg1, arg2)`. XMILE has **no** native multi-output call syntax — the `:`-form is Vensim-specific. + +**The four write-side gaps Phase 5 must fill:** +1. `File::write_xml` (`xmile/mod.rs:79-126`) does not iterate `self.macros` — hook point is between the `for model in self.models.iter()` loop (~line 121) and `write_tag_end(writer, "xmile")` (~line 125). `xmlns:simlin` is **already** declared on the root `` (line 96). +2. `Header::write_xml` (`xmile/mod.rs:447-474`) emits only ``/``/`` — it writes **no** ``/features at all. Emitting `` is net-new code. +3. `From for File` (`xmile/mod.rs:188-247`) hard-codes `macros: vec![]` (~line 244). +4. `From for datamodel::Project` (`xmile/mod.rs:129-186`) never reads `file.macros` — `datamodel::Project.models` is populated solely from `file.models`. + +**The `simlin:` extension pattern** (read with a serde rename to the namespace-*stripped* local name — quick-xml strips the `simlin:` prefix on deserialize; write by hand with the `simlin:`-prefixed tag). Reference implementations to mirror: `XmileLoopMetadata` (`xmile/model.rs:20-36`, with `#[serde(rename = "@name")]` etc.; written at `xmile/model.rs:106-119` via `write_tag_with_attrs(writer, "simlin:loop-metadata", ...)`) and the more complete `DataSourceElement` (`xmile/variables.rs:21-102` — has `from_datamodel`/`to_datamodel`/`write_xml`, deserialized via `#[serde(rename = "data_source", ...)]`). + +**`Feature::UsesMacros`** (`xmile/mod.rs:510-513`) exists: `UsesMacros { recursive_macros: Option, option_filters: Option }`, read inside `Options.features` (`#[serde(rename = "$value")]`, `xmile/mod.rs:542`) on `Header.options`. **Latent bug:** the two fields lack `#[serde(rename = "@...")]`, so they currently deserialize as child *elements*, not the spec-required *attributes* — Phase 5 (Task 2) fixes this as part of emitting `` correctly. + +**The per-`Model` bridges** (`xmile/model.rs`): `From for datamodel::Model` (`:126-205`) and `From for xmile::Model` (`:207-289`) — reusable for the macro *body* (a macro body is structurally an ordinary model's `variables`/`views`), but a macro's `name`/``s/``/`MacroSpec`/additional-output extensions live on the `Macro` type, not `Model`. After Phase 1, `datamodel::Model` has a `macro_spec: Option` field these impls will need to default/populate. + +**Fixture reality (corrected):** the `.xmile` macro fixtures **do** contain `` elements. `grep -rl "` elements), `.../macro_stock/test_macro_stock.xmile` (a stock-bearing macro body). The `.stmx` files and the `macro_cross_reference`/`macro_trailing_definition` directories have **no** `` element (xmutil dropped it / those dirs have no `.xmile`). The xmutil-emitted `` shape has *both* `` children and a `` body, with `` holding the macro name (e.g. `macro_expression.xmile` lines 47-77: `EXPRESSION MACROinputparameterinput*parameter...`). The real `` element also carries a `` child and a `` child (a macro-body diagram) — `` is modeled by `xmile::Macro` (Task 1), `` is **deliberately not** (see Task 1's "No `views` field" note). **No fixture anywhere uses ``.** Of these four files, `simulate.rs`'s `TEST_MODELS` static array (`tests/simulate.rs:22-36`) already lists exactly **three** as commented-out entries (lines 27-29: `macro_expression`, `macro_multi_expression`, `macro_stock`) — `macro_multi_macros` is **not** present in the array at all. So Task 4 *uncomments* three and *adds* a fourth new entry. + +**Round-trip safety net:** `simulate_path_with` (`tests/simulate.rs:189-251`) already performs a byte-stable XMILE round-trip assertion — it serializes the project via `project_to_xmile`, re-parses, re-serializes, and `assert_eq!`s the two serializations. So **wiring a macro `.xmile` fixture into `simulate.rs`'s `TEST_MODELS` automatically exercises an XMILE→datamodel→XMILE byte-stable round-trip.** There is no standalone XMILE round-trip harness (unlike `mdl_roundtrip.rs`); `tests/roundtrip.rs` only parses XMILE, never re-emits. MDL→XMILE conversion is untested anywhere. + +**Documented limitation (per the design):** XMILE per-macro `` — a macro running with its own dt/stop time — is **not supported**. `xmile::Macro` carries an optional `sim_specs` field for parse-completeness/round-trip, but a `` with a non-empty `` is rejected with a clear error at conversion time. + +--- + + +## Subcomponent A: XMILE macro read/write + + +### Task 1: Flesh out `xmile::Macro` and the XMILE reader + +**Verifies:** macros.AC1.3. + +**Files:** +- Modify: `src/simlin-engine/src/xmile/mod.rs` (replace the empty `Macro` stub ~lines 334-338; extend `From for datamodel::Project` ~lines 129-186) +- Possibly modify: `src/simlin-engine/src/mdl/convert/` and/or `src/simlin-engine/src/datamodel.rs` (extract Phase 2's port-variable-synthesis logic into a shared helper if it is not already callable from outside `mdl/convert/`) + +**Implementation:** + +1. **Define `xmile::Macro`** as a serde-`Deserialize` struct mirroring the established `xmile::Model`/`xmile::Var` serde patterns. **First, drop the `Eq` derive from the existing stub:** it currently derives `#[derive(Clone, PartialEq, Eq, Deserialize, Serialize)]` — it can derive `Eq` only because it is empty. The new `variables`/`sim_specs` fields transitively contain `f64` (which is not `Eq`), so keeping `Eq` will not compile. Change the derive to match `xmile::Model` exactly — `#[derive(Clone, PartialEq, Deserialize, Serialize)]` (no `Eq`), keeping the existing `#[cfg_attr(feature = "debug-derive", derive(Debug))]` line above it. Then add the fields: + - `#[serde(rename = "@name")] name: String` + - `#[serde(rename = "parm", default)] parms: Vec` — a `Parm` struct with `#[serde(rename = "$text")] name: String` and `#[serde(rename = "@default", skip_serializing_if = "Option::is_none", default)] default: Option` + - `#[serde(rename = "eqn")] eqn: Option` — the expression-form body / primary-output expression (``) + - `variables: Option` — the multi-equation body (reuses the existing `xmile::Variables` type) + - `sim_specs: Option` — for parse-completeness; a non-empty value is rejected at conversion (the documented limitation) + - `doc: Option`, `#[serde(rename = "@namespace")] namespace: Option` — round-tripped for fidelity + - **No `views` field — deliberate.** The xmutil-emitted `` carries a `` child (a macro-body diagram), but macro models are non-navigable (AC6.6), so a macro body's views are inert. `xmile::Macro` deliberately does **not** model ``: `quick_xml::de` silently ignores the unknown element on read, and the Task 2 writer never emits one. This is a documented intentional non-round-trip — it does **not** break Task 4's byte-stable round-trip (the `` is dropped on the *first* `project_to_xmile` too, so both serializations in `simulate_path_with`'s `assert_eq!` agree), but an engineer diffing a wired fixture's emitted output against the original `.xmile` file will see the macro `` is gone, and that is expected. + - The `simlin:`-namespaced additional-output extension field is added in **Task 3** (leave it out here; Task 3's serde field uses `#[serde(default)]` so Task 1's reader compiles without it). + +2. **Bridge `xmile::Macro` → a macro-marked `datamodel::Model`.** In `From for datamodel::Project`, after building `models` from `file.models`, convert each `file.macros` entry to a macro-marked `datamodel::Model` and append it to `project.models`. Conversion: + - **Body:** if `` is present, the body is those variables (reuse the existing `xmile::Variables` → `datamodel::Variable` conversion). If `` is absent (expression-form, §4.8.1), **normalize the `` into a body variable named after the macro** — a `Variable::Aux { ident: , equation: Scalar(), .. }` (this is the AC1.3 "expression-form `` is normalized into a macro-named body variable" requirement). + - **`primary_output`:** the canonical macro name. (For the with-`` xmutil shape where `` is literally the macro name, this is consistent. If `` is an expression that does *not* name an existing body variable, also normalize it into a macro-named body variable as above and use that as the primary output.) + - **Port variables + `MacroSpec`:** **call the shared `Model::new_macro` function that Phase 2 Task 5 created** (the named, reusable `pub(crate)` helper in `datamodel.rs` or a small shared module — `Model::new_macro(macro_name, parameters, additional_outputs, body_variables) -> Model`). The XMILE reader's job is just to build the inputs: the macro name (the `` attribute, canonicalized), the parameter names (the `` names, canonical, in order), `additional_outputs` (empty here — Task 3 populates it from the `simlin:` extension), and `body_variables` (built from ``/`` per the **Body** bullet above). `Model::new_macro` then synthesizes the port variables (placeholder `Equation::Scalar("0")`, `compat.can_be_module_input = true`, kind by body usage) and attaches the `MacroSpec` — identically to the MDL path. Do not re-implement port synthesis in `xmile/`. + - **Per-macro ``:** if `macro.sim_specs` is `Some` and non-empty, return a clear error (per-macro `` is the documented unsupported limitation) — use the XMILE layer's existing error type. + +**Testing:** +- **macros.AC1.3** (`xmile/` inline `#[cfg(test)]` tests, or a focused test using `open_xmile` on an in-test XMILE string): an XMILE string with a `aba * b` (expression-form, no ``) imports as a `datamodel::Project` containing a macro-marked `Model` whose `MacroSpec.parameters == ["a", "b"]`, with a body variable named `mymacro` carrying the equation `a * b` (the normalized ``) and synthesized port variables `a`/`b` (`can_be_module_input == true`). +- A `` with a `` body imports with the `` as the body and the ``-named variable as `primary_output`. +- A `` with a non-empty `` returns the documented-limitation error. + +**Verification:** +Run: `cargo test -p simlin-engine xmile` +Expected: all pass, including the new `` reader tests. + +**Commit:** `engine: read XMILE elements as macro-marked models` + + + +### Task 2: XMILE writer for `` elements and the `` header option + +**Verifies:** macros.AC4.2 (definition + header-option emission), macros.AC4.5 (single-output standards-clean). + +**Files:** +- Modify: `src/simlin-engine/src/xmile/mod.rs` (`impl ToXml for Macro`; `File::write_xml` ~lines 79-126; `Header::write_xml` ~lines 447-474 — or the header path in `File::write_xml`; `From for File` ~lines 188-247; `Feature::UsesMacros` ~lines 510-513) + +**Implementation:** + +1. **`impl ToXml for Macro`** — hand-written, mirroring `Model::write_xml` and the write-helper style. Emit ``, then the `` children (with `default` attributes where present), then ``, then `` (looping `Var::write_xml` like `Model::write_xml` does), then `` if present, then ``. The `simlin:` additional-output extension emission is added in Task 3. + +2. **`File::write_xml`** — add `for macro in self.macros.iter() { macro.write_xml(writer)?; }` between the models loop and `write_tag_end(writer, "xmile")`. + +3. **`From for File`** — replace `macros: vec![]` with a real population: partition `project.models` by `model.macro_spec.is_some()`; macro-marked models become `file.macros` entries (`datamodel::Model` → `xmile::Macro`: name from `model.name`, ``s from `MacroSpec.parameters`, `` from the body variables *excluding the synthesized port variables* — the port variables are reconstructed from the ``s on re-import, so emitting them in `` too would be redundant and break round-trip stability; `` = the `MacroSpec.primary_output` name), non-macro models stay in `file.models`. + +4. **Emit ``** — when `project.models` contains at least one macro, emit the `` header option (Simlin does not support recursive macros, and emits both attributes as fixed `"false"` — a deterministic emission, which keeps the byte-stable round-trip stable). This requires writing the `
`'s `` block, which `Header::write_xml` does not do today — add the minimal options-writing code. **Also fix the `Feature::UsesMacros` serde field renames** to `#[serde(rename = "@recursive_macros")]` / `#[serde(rename = "@option_filters")]` so the *reader* parses the spec-correct attribute form that the writer now emits (without this, read and write disagree and the round-trip is lossy). + +5. **AC4.5 — standards-clean for single-output:** a macro with empty `MacroSpec.additional_outputs` emits a plain standard `` with **no** `simlin:`-namespaced child elements. (The `simlin:` additional-output extension — Task 3 — is emitted only when `additional_outputs` is non-empty.) + +**Testing:** +- **macros.AC4.2** (definition): a `datamodel::Project` with a single-output macro-marked `Model` (built in-test, or imported from a `.xmile`/`.mdl` fixture) → `to_xmile` → assert the output string contains a `` element with the expected ``s and body, and a `` header option. +- **macros.AC4.5** (single-output standards-clean): assert the `to_xmile` output of a single-output-only macro project contains **no** `simlin:`-prefixed macro-extension element (it may still contain other pre-existing `simlin:` elements like `simlin:loop-metadata`; assert specifically that no macro-additional-output extension is present). +- Round-trip: `to_xmile` → `open_xmile` → assert the macro-marked model survives with the same `MacroSpec` and body. + +**Verification:** +Run: `cargo test -p simlin-engine xmile` +Expected: all pass. + +**Commit:** `engine: write XMILE elements and the uses_macros option` + + + +### Task 3: `simlin:` extensions for multi-output macros — read and write + +**Verifies:** macros.AC4.2 (`simlin:` extensions), macros.AC4.5 (multi-output triggers the extension). + +**Files:** +- Modify: `src/simlin-engine/src/xmile/mod.rs` (`xmile::Macro` — add the additional-output extension field; `Macro` reader bridge and `ToXml` impl; the multi-output-invocation extension) +- Possibly create: a small `simlin:`-extension type (mirroring `DataSourceElement` in `xmile/variables.rs`) + +**Implementation:** + +Standard XMILE `` has no concept of multiple *output* ports, and no concept of a multi-output *invocation*. Both round-trip through `simlin:`-namespaced extension elements (the established pattern: deserialize via a serde rename to the namespace-stripped local name; serialize by hand with the `simlin:`-prefixed tag). Design the two extension elements; the contract is round-trip fidelity and that single-output projects never emit them. + +1. **Additional-output ports on a macro definition.** A multi-output macro imported from `.mdl` has a non-empty `MacroSpec.additional_outputs`. Add a `simlin:`-namespaced child element on `` (e.g. `` listing the additional output port names in order). Wire it: a serde field on `xmile::Macro` (`#[serde(rename = "additional-outputs", default)]`), emitted in `Macro::write_xml` (`write_tag_with_attrs(writer, "simlin:additional-outputs", ...)`), populated from / into `MacroSpec.additional_outputs` in the `Macro ↔ Model` bridge. Emitted **only** when `additional_outputs` is non-empty. + +2. **Multi-output invocations.** After Phase 4, a multi-output invocation `total = add3(a,b,c : minv, maxv)` is materialized in the datamodel as a `Variable::Module` (with input-only `ModuleReference`s, `model_name` = the macro's model) plus binding `Variable::Aux`es (the LHS aux reads `.`, each `:`-list aux reads `.` — ASCII period, the datamodel separator form per Phase 4's authoritative separator note; the XMILE layer reads the datamodel, so it matches against `.`). Standard XMILE `` references a ``, not a `` — so this materialized cluster round-trips through a `simlin:`-namespaced extension that records the invocation (the macro name, the argument wiring, and the output bindings) faithfully enough that the reader reconstructs exactly the same `Variable::Module` + binding `Aux`es. Wire it read + write. Emitted **only** for modules whose `model_name` resolves to a macro-marked model. + +3. The reader side of both extensions reconstructs the datamodel shape (the macro `Model`'s `additional_outputs`; the `Variable::Module` + binding auxes) so an XMILE→datamodel→XMILE round-trip is byte-stable. + +**Testing:** +- **macros.AC4.5** (multi-output triggers the extension): a `datamodel::Project` with a multi-output macro (non-empty `MacroSpec.additional_outputs`) and a multi-output invocation → `to_xmile` → assert the output contains the `simlin:` additional-outputs extension on the `` and the `simlin:` multi-output-invocation extension; and (contrast) confirm a single-output project does **not**. +- **macros.AC4.2** (`simlin:` extension round-trip): a multi-output macro project → `to_xmile` → `open_xmile` → assert the round-tripped project has the same macro `MacroSpec.additional_outputs` and the same materialized `Variable::Module` + binding `Aux`es as the original; and a second `to_xmile` of the round-tripped project is byte-identical to the first (byte-stable). +- Multi-output cross-format: `open_vensim` a multi-output `.mdl` (a focused `:`-form fixture, or `test/metasd/theil-statistics/Theil_2011.mdl`) → `to_xmile` → `open_xmile` → assert the multi-output macro and its invocation survived. + +**Verification:** +Run: `cargo test -p simlin-engine xmile` +Expected: all pass, including the multi-output `simlin:`-extension round-trip tests. + +**Commit:** `engine: round-trip multi-output macros through simlin: XMILE extensions` + + + + +## Subcomponent B: Fixture wiring and cross-format verification + + +### Task 4: Wire the `.xmile` macro fixtures and the cross-format test + +**Verifies:** macros.AC1.3 (the fixtures import and simulate), macros.AC4.2 (round-trip), macros.AC4.4 (cross-format). + +**Files:** +- Modify: `src/simlin-engine/tests/simulate.rs` (uncomment the three commented-out macro `.xmile` fixtures at `TEST_MODELS` lines 27-29 and add a fourth new entry for `macro_multi_macros`; add a cross-format test) + +**Implementation:** Tests only — if a test surfaces a real gap, fix it in `xmile/`, but after Tasks 1-3 the expectation is that the `.xmile` macro fixtures import, simulate, and round-trip. + +1. **Wire all four `.xmile` macro fixtures into `TEST_MODELS`** (`simulate.rs` lines 22-36). Three are *already present* as commented-out entries at lines 27-29 — **uncomment** them: `test/test-models/tests/macro_expression/test_macro_expression.xmile`, `.../macro_multi_expression/test_macro_multi_expression.xmile`, `.../macro_stock/test_macro_stock.xmile`. The fourth, `test/test-models/tests/macro_multi_macros/test_macro_multi_macros.xmile` (two `` elements), is **not** in the array at all — **add** it as a new entry alongside the other macro fixtures. Because `simulate_path_with` runs each `TEST_MODELS` entry through import → simulate → protobuf round-trip → **byte-stable XMILE round-trip** (`tests/simulate.rs:189-251`), wiring these four fixtures in exercises, for each: the Task 1 reader (the `` imports as a macro-marked model), Phase 3's expansion (the invocation simulates and matches `output.tab`), and the Task 2 writer's byte-stability (`project_to_xmile` is stable across a re-parse). The `.stmx` variants stay out (no `` element); `macro_cross_reference`/`macro_trailing_definition` have no `.xmile`. + +2. **Cross-format `.mdl` → datamodel → `.xmile` test (macros.AC4.4):** add a test that `open_vensim`s a single-output macro `.mdl` fixture (e.g. `test/test-models/tests/macro_expression/test_macro_expression.mdl`), converts the resulting `datamodel::Project` to XMILE via `to_xmile`, re-imports via `open_xmile`, and asserts the macro definition (the macro-marked `Model` + its `MacroSpec`) and the invocation are preserved — i.e. the cross-format-round-tripped project's macro models and invocation equations match those of the directly-imported `.mdl` project. + +**Testing:** +- **macros.AC1.3 / macros.AC4.2:** the four wired `.xmile` fixture tests pass — each imports, simulates to its `output.tab`, and round-trips byte-stably through XMILE. +- **macros.AC4.4:** the cross-format test passes — `.mdl` → datamodel → `.xmile` → datamodel preserves the macro definitions and invocations. + +**Verification:** +Run: `cargo test -p simlin-engine --test simulate` +Expected: all pass, including the four newly-wired macro `.xmile` fixtures and the cross-format test. + +**Commit:** `engine: wire XMILE macro fixtures and the cross-format conversion test` + + + +--- + +## Phase 5 completion check + +When all four tasks are committed: +- `xmile::Macro` is a real type; an XMILE `` element imports as a macro-marked `datamodel::Model`, and an expression-form `` is normalized into a macro-named body variable (the design's "Done when": the `.xmile` macro fixtures import). +- The XMILE writer emits `` elements, the `` header option, and `simlin:`-namespaced extensions for multi-output additional outputs and invocation bindings; a single-output-only project exports as standards-clean XMILE with no extensions. +- A macro-bearing XMILE file round-trips byte-stably (the four wired fixtures, via `simulate_path_with`'s round-trip assertion); a cross-format `.mdl` → datamodel → `.xmile` conversion preserves macro definitions and invocations (the design's "Done when"). +- `macros.AC1.3`, `macros.AC4.2`, `macros.AC4.4`, `macros.AC4.5` are verified. + +MDL export (the `:MACRO:` writer and reconstructing the `:` call syntax) is Phase 6; the corpus harness is Phase 7. diff --git a/docs/implementation-plans/2026-05-13-macros/phase_06.md b/docs/implementation-plans/2026-05-13-macros/phase_06.md new file mode 100644 index 000000000..66206f140 --- /dev/null +++ b/docs/implementation-plans/2026-05-13-macros/phase_06.md @@ -0,0 +1,182 @@ +# Vensim Macro Support — Phase 6: MDL export and round-trip + +**Goal:** Make the MDL writer emit `:MACRO: … :END OF MACRO:` blocks for macro-marked models and reconstruct multi-output invocations back into the `:` call syntax, then wire the macro fixtures into the `mdl_roundtrip.rs` harness so the full `.mdl` → datamodel → `.mdl` loop is verified. + +**Architecture:** Three pieces. **(a)** `project_to_mdl` currently rejects any project with more than one model and any `Variable::Module`; both gates relax to allow macro-marked models and macro-module instances (while still rejecting ordinary multi-model projects and ordinary submodule instances — a general MDL module-export overhaul is out of scope). **(b)** The MDL writer emits each macro-marked `datamodel::Model` as a `:MACRO:` block — header from `MacroSpec`, body from the model's variables *minus* the synthesized port variables (the ports are reconstructed from the header parameter list, so emitting them as body equations would be redundant and would lose the `can_be_module_input` flag on re-import). Single-output invocations need no special work — they are already plain equation text, and the writer's RHS formatter passes unknown function calls through verbatim. **(c)** A multi-output invocation, materialized by Phase 4 as a `Variable::Module` plus binding `Aux`es, is detected and reconstructed into `total = add3(a, b, c : minv, maxv)` text; the cluster's module and binding auxes are suppressed from normal per-variable emission. + +**Tech Stack:** Rust — `src/simlin-engine/src/mdl/writer.rs`, `src/simlin-engine/src/mdl/mod.rs`, and `src/simlin-engine/tests/mdl_roundtrip.rs`. The `:MACRO:` block syntax is documented in-repo at `docs/reference/vensim-macros.md`. No external dependencies. + +**Scope:** 7 phases from the original design (`docs/design-plans/2026-05-13-macros.md`); this is phase 6 of 7. + +**Codebase verified:** 2026-05-14 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### macros.AC4: Round-trip and export +- **macros.AC4.1 Success:** A macro-bearing `.mdl` file round-trips with definitions emitted as `:MACRO:` blocks and invocations preserved. +- **macros.AC4.3 Success:** A multi-output macro round-trips through `.mdl` with the `:` call syntax reconstructed. + +(`macros.AC4.2`, `macros.AC4.4`, `macros.AC4.5` are XMILE round-trip — Phase 5.) + +--- + +## Current state (verified 2026-05-14) + +**`project_to_mdl` rejects everything macro-related.** `src/simlin-engine/src/mdl/mod.rs:43-66`: +```rust +pub fn project_to_mdl(project: &Project) -> Result { + if project.models.len() != 1 { + return Err(... "MDL format supports only a single model" ...); + } + let model = &project.models[0]; + for var in &model.variables { + if matches!(var, Variable::Module(_)) { + return Err(... "MDL format does not support Module variables" ...); + } + } + let writer = MdlWriter::new(); + writer.write_project(project) +} +``` +Both gates must relax for macro-marked models/modules. `compat::to_mdl` (`compat.rs:36-38`) is a one-line wrapper over `project_to_mdl` — fixing `project_to_mdl` fixes both. + +**Pre-existing MDL-writer gaps (out of scope, already tracked).** The MDL writer today writes only the main model and drops *all* `Variable::Module` instances (`write_variable_entry`'s `Variable::Module(_) => return`). Phase 6 adds *macro-specific* writer support — `:MACRO:` blocks and multi-output reconstruction — but a general overhaul of MDL module export is **out of scope**. Two unrelated MDL serialization gaps found during the design investigation are tracked separately as GitHub **#538** and **#539**. An engineer who encounters one of these while wiring the round-trip harness should recognize it as pre-existing and tracked — *not* a regression introduced by macro support. + +**The MDL writer (`src/simlin-engine/src/mdl/writer.rs`):** +- `write_project` (`writer.rs:2459-2470`) writes `{UTF-8}\n`, then `write_equations_section(model, project)` for `project.models[0]` only, then sketch + settings. `&datamodel::Project` is in scope here. +- `write_equations_section` (`writer.rs:2526-2595`) assembles: dimension defs → grouped variables (with `****` markers) → ungrouped variables (alphabetical) → `.Control` header + sim specs → `\\\---///` terminator. **No `:MACRO:` emission point exists.** `&Project` is in scope here too. +- `write_variable_entry` (`writer.rs:851-921`) dispatches per variable: `Variable::Stock` → `write_stock_variable`; `Variable::Module(_) => return` (line 867, emits nothing); `Flow`/`Aux` → `write_single_entry`. **It receives only `var` + `display_names` — not `&Project`/`&Model`** — so it cannot look up `module.model_name → macro_spec`. Multi-output reconstruction must therefore happen in `write_equations_section` (which has `&Project`). +- The RHS formatter: `equation_to_mdl` (`writer.rs:738-760`) parses the equation string to `Expr0` and walks it with `MdlPrintVisitor` (`expr0_to_mdl`, `writer.rs:622-736`). The `App` arm (`writer.rs:645-671`) → `xmile_to_mdl_function_name` (`writer.rs:194-216`), whose fall-through `_ => underbar_to_space(xmile_name).to_uppercase()` passes **unknown function calls through verbatim** (uppercased, `_`→space). +- Variable-entry emitters: `write_single_entry` (`writer.rs:1239-1292`), `write_stock_variable`/`write_stock_entry` (`writer.rs:978-1067`), `write_units_and_comment` (`writer.rs:1357-1366`, the `\n\t~\tunits\n\t~\tcomment\n\t|` trailer). + +**Single-output macro invocations already round-trip.** A single-output invocation is an ordinary `Variable::Aux` with `equation: Scalar("mymacro(a, b)")`. `write_variable_entry` → `write_single_entry` → `equation_to_mdl` parses it (the parser parses *any* `ident(...)` as `Expr0::App` — no builtin-table validation) → `MdlPrintVisitor` emits `MYMACRO(a, b)` (uppercased). Pinned today by the `function_unknown_uppercased` writer test. So Phase 6 needs **no** special single-output-invocation code — only the `:MACRO:` definition emission and the relaxed `project_to_mdl` gate. + +**The `:MACRO:` block syntax** (`docs/reference/vensim-macros.md`): a top-level block in the equation section. Single-output header `:MACRO: macroname(in1, in2)`; multi-output header `:MACRO: macroname(in1, in2, in3 : out1, out2)`; body = ordinary `name = rhs ~ units ~ comment |` equations; `:END OF MACRO:` terminator. Every well-formed `macro_*` fixture places its `:MACRO:` blocks **immediately after the `{UTF-8}` line, before the main model's equations and the `.Control` section** — so the writer should emit all `:MACRO:` blocks right after `{UTF-8}\n`. Multiple back-to-back `:MACRO:` blocks are allowed. + +**The materialized multi-output cluster** (Phase 4, per `phase_04.md` Task 1): `total = add3(a, b, c : minv, maxv)` becomes a `Variable::Module` (deterministic serialization-stable `ident`, `model_name` = the macro's `Model.name`, input-only `ModuleReference`s with `dst = "{module_ident}.{param}"`, `src` = the arg) plus binding `Aux`es — a primary-output aux `Aux { ident: , equation: Scalar("{module_ident}.{primary_output}") }` and one additional-output aux per `:`-list entry `Aux { ident: , equation: Scalar("{module_ident}.{additional_output}") }`. Reconstruction is unambiguous **only with the macro model's `MacroSpec` as a cross-reference** (to classify each binding aux as primary vs additional and to recover positional arg order — `Module.references` ordering is not guaranteed positional, so each `dst`'s post-`.` segment must be matched against `MacroSpec.parameters[i]`). `Module` and `ModuleReference` are at `datamodel.rs:344-361`. + +**`mdl_roundtrip.rs`:** +- Header comment (`mdl_roundtrip.rs:12-22`) lists excluded categories, first being `macros (:MACRO:) -- the writer rejects them`. +- `TEST_MDL_MODELS` (`mdl_roundtrip.rs:23-82`) — a flat `static &[&str]` of repo-relative paths; `resolve_path` prepends `../../`. +- `mdl_to_mdl_roundtrip()` (`mdl_roundtrip.rs:245-295`) drives `parse_mdl → project_to_mdl → parse_mdl → assert_semantic_equivalence`. +- `assert_semantic_equivalence` / `assert_model_equivalence` (`mdl_roundtrip.rs:134-214`) compares: `sim_specs`, `dimensions`, model **count**, and per-model (paired by **zip index**) variable count + per-variable (sorted by ident) name + equation text. **It does NOT compare `Model.macro_spec`, `Model.name`, groups, views, variable type, `Compat`, or `ModuleReference`s** — so after Phase 1 it neither breaks on the new `macro_spec` field nor verifies it. +- Writer tests live in `src/simlin-engine/src/mdl/writer_tests.rs` (`assert_mdl` for equation-text tests; `make_aux`/`make_model`/`make_project` for full-project tests). `project_to_mdl_rejects_multiple_models` and `project_to_mdl_rejects_module_variable` (`writer_tests.rs:1280-1314`) assert the current reject-everything behavior and must be rewritten to reject only the *non-macro* cases. + +**Fixtures:** all 6 `test/test-models/tests/macro_*` directories have a `.mdl` file. Phase 4 authors `test/test-models/tests/macro_multi_output/test_macro_multi_output.mdl` (the `:`-multi-output fixture) and `test/test-models/tests/macro_arrayed/test_macro_arrayed.mdl` (a stockless apply-to-all invocation — note: an arrayed *single-output* invocation stays as `ApplyToAll` equation text, not a materialized module, so it round-trips via the ordinary equation-text path, not Task 2's reconstruction). + +--- + + +## Subcomponent A: MDL macro export and round-trip + + +### Task 1: Emit `:MACRO:` blocks for macro-marked models + +**Verifies:** macros.AC4.1 (single-output: definitions emitted as `:MACRO:` blocks, invocations preserved). + +**Files:** +- Modify: `src/simlin-engine/src/mdl/mod.rs` (`project_to_mdl` ~lines 43-66 — relax the `models.len()` gate) +- Modify: `src/simlin-engine/src/mdl/writer.rs` (`write_project` ~lines 2459-2470 and/or `write_equations_section` ~lines 2526-2595 — add `:MACRO:` block emission) +- Modify: `src/simlin-engine/src/mdl/writer_tests.rs` (rewrite `project_to_mdl_rejects_multiple_models` ~line 1280; add `:MACRO:`-emission writer tests) + +**Implementation:** + +1. **Relax the `models.len()` gate in `project_to_mdl`.** Instead of rejecting `project.models.len() != 1`, reject only when there is more than one **non-macro** model (`project.models.iter().filter(|m| m.macro_spec.is_none()).count() != 1`). Ordinary multi-model XMILE projects are still rejected (MDL has no general multi-model representation); macro-marked extra models are allowed. + +2. **Emit `:MACRO:` blocks.** In `write_project` (or at the top of `write_equations_section`), after `{UTF-8}\n` and **before** the dimension definitions / main-model variables, iterate `project.models` in order and, for each model with `macro_spec.is_some()`, emit a `:MACRO:` block: + - **Header:** `:MACRO: (, , ...)` from `MacroSpec.parameters`. If `MacroSpec.additional_outputs` is non-empty, emit the multi-output header form `:MACRO: ( : )`. Use the MDL display-name form for the macro name and parameter names (the writer's existing `format_mdl_ident` / `display_name_for_ident` / `underbar_to_space` helpers — match how the main model's variable names are emitted). + - **Body:** for each variable in the macro `Model.variables` whose ident is **not** in `MacroSpec.parameters`, emit it via the existing `write_variable_entry`. This reuses the existing Stock/Flow/Aux emission verbatim — a macro-body stock round-trips the same way a main-model stock does; a macro body that calls another macro is an `Aux` whose equation is plain call text and round-trips like any unknown-function call. **Excluding the `MacroSpec.parameters` variables is required:** those are Phase 2's synthesized port variables; they are reconstructed from the header on re-import, and emitting them as ` = 0` body equations would make the re-imported macro treat them as ordinary body variables (losing `compat.can_be_module_input` and diverging from the original). + - **Terminator:** `:END OF MACRO:`. + - Emit the macro blocks in a **deterministic order** (`project.models` order — stable across passes, which the round-trip harness's zip-index model pairing relies on). + - Single-output invocations in the main model need no special handling — they are already equation text and the existing per-variable emission handles them. + +3. **Rewrite `project_to_mdl_rejects_multiple_models`** so it asserts an ordinary two-*non-macro*-model project is still rejected, and add a positive case: a project with one `main` model + one macro-marked model is accepted and produces a `:MACRO:` block. + +**Testing:** +- **macros.AC4.1** (writer unit tests in `writer_tests.rs`, full-project style): build a `datamodel::Project` with a `main` model and a single-output macro-marked `Model` (with `MacroSpec` and synthesized port variables) → `project_to_mdl` → assert the output contains a `:MACRO: ()` header, the body equation(s), `:END OF MACRO:`, and that the synthesized port variables do **not** appear as body equations; assert the main model's invocation equation is preserved as call text. +- A macro with a multi-equation body (a helper variable) and a macro with a body stock both emit correct `:MACRO:` blocks. +- The rewritten rejection test passes (ordinary multi-model still rejected; macro-marked extra model accepted). + +**Verification:** +Run: `cargo test -p simlin-engine mdl::writer` +Expected: all pass, including the new `:MACRO:`-emission tests. + +**Commit:** `engine: emit :MACRO: blocks from the MDL writer` + + + +### Task 2: Reconstruct multi-output invocations + +**Verifies:** macros.AC4.3. + +**Files:** +- Modify: `src/simlin-engine/src/mdl/mod.rs` (`project_to_mdl` ~lines 43-66 — relax the `Variable::Module` gate) +- Modify: `src/simlin-engine/src/mdl/writer.rs` (`write_equations_section` ~lines 2526-2595 — detect and reconstruct the multi-output cluster) +- Modify: `src/simlin-engine/src/mdl/writer_tests.rs` (rewrite `project_to_mdl_rejects_module_variable` ~line 1300; add multi-output-reconstruction tests) + +**Implementation:** + +1. **Relax the `Variable::Module` gate in `project_to_mdl`.** Reject a `Variable::Module` only when its `model_name` does **not** resolve to a macro-marked model in `project.models`. An ordinary submodule instance is still rejected (a general MDL module-export overhaul is out of scope); a macro-module instance materialized by Phase 4 is allowed. + +2. **Detect and reconstruct the multi-output cluster** in `write_equations_section` (it has `&Project`). Before the per-variable emission loop: + - Build a `name → &MacroSpec` lookup over `project.models` (the macro-marked models). + - For each `Variable::Module` in the main model whose `model_name` resolves to a macro-marked model: look up its `MacroSpec`. Recover the call arguments — for each `ModuleReference`, the `dst` is `{module_ident}.{param_name}`; match each `param_name` against `MacroSpec.parameters[i]` to get positional order, and the `src` is the argument's name. Collect the binding auxes — scan the main model's `Aux` variables for a `Scalar` equation that is exactly a module-output reference into this module instance (`{module_ident}.{output}` — an **ASCII period**: per Phase 4's authoritative separator note, the materialized binding-aux equation text is the datamodel `.` form, *not* the canonical `·`, since `project_to_mdl` operates on datamodel equation strings). Classify each binding aux: the one whose referenced output equals `MacroSpec.primary_output` is the LHS variable; the ones matching `MacroSpec.additional_outputs[j]` are the `:`-list output bindings in order `j`. + - Emit the reconstructed invocation: ` = (, ..., : , ..., )` followed by the standard `~ units ~ comment |` trailer (use the primary-output binding aux's units/documentation). Use the existing MDL ident-formatting helpers. + - **Suppress** the `Variable::Module` and all of its binding auxes from the normal per-variable emission loop (collect their idents into a skip-set the loop checks). `write_variable_entry`'s existing `Variable::Module(_) => return` stays as a harmless fallback. + +3. **Rewrite `project_to_mdl_rejects_module_variable`** so it asserts an *ordinary* (non-macro) `Variable::Module` is still rejected, and add a positive case: a `Variable::Module` whose `model_name` is a macro-marked model is accepted and reconstructed as a `:` invocation. + +**Testing:** +- **macros.AC4.3** (writer unit tests, full-project style): build a `datamodel::Project` with a multi-output macro-marked `Model` (non-empty `MacroSpec.additional_outputs`) and a materialized multi-output cluster (a `Variable::Module` + a primary-output binding aux + additional-output binding auxes, exactly as Phase 4 produces) → `project_to_mdl` → assert the output contains the reconstructed ` = ( : )` `:` call syntax with the arguments in positional order and the output bindings in `:`-list order, and that the `Variable::Module` and binding auxes do **not** appear as separate ``/aux entries. +- The rewritten rejection test passes (ordinary module still rejected; macro-module accepted). + +**Verification:** +Run: `cargo test -p simlin-engine mdl::writer` +Expected: all pass, including the multi-output-reconstruction tests. + +**Commit:** `engine: reconstruct multi-output macro invocations in the MDL writer` + + + +### Task 3: Wire macro fixtures into the `mdl_roundtrip.rs` harness + +**Verifies:** macros.AC4.1, macros.AC4.3 (the full `.mdl` → datamodel → `.mdl` round-trip). + +**Files:** +- Modify: `src/simlin-engine/tests/mdl_roundtrip.rs` (remove the macro-exclusion comment line; add the 8 fixtures to `TEST_MDL_MODELS`; extend `assert_model_equivalence`) + +**Implementation:** Tests + harness only — if the round-trip surfaces a real writer gap, fix it in `writer.rs`/`mod.rs`, but after Tasks 1-2 the expectation is that macro fixtures round-trip. + +1. **Extend `assert_model_equivalence`** (`mdl_roundtrip.rs`) to also compare `Model.macro_spec` (via `PartialEq` — `MacroSpec` derives it) and `Model.name`. Without this, the harness would pass even if the writer dropped a `:MACRO:` block entirely, because the body variables live in the macro model and a missing model would only change the model *count* (which is checked) — but a *malformed* `MacroSpec` would slip through. Comparing `macro_spec` makes the macro round-trip actually verified. Confirm the existing non-macro fixtures still pass (the `main` model is named `"main"` and has `macro_spec: None` on both passes, so the new comparisons are no-ops for them). + +2. **Remove** the `// - macros (:MACRO:) -- the writer rejects them` line from the `TEST_MDL_MODELS` header comment (`mdl_roundtrip.rs:12-22`). + +3. **Add the 8 macro fixtures to `TEST_MDL_MODELS`:** the 6 bundled `test/test-models/tests/macro_*/test_macro_*.mdl` fixtures plus the 2 Phase-4-authored fixtures `test/test-models/tests/macro_multi_output/test_macro_multi_output.mdl` and `test/test-models/tests/macro_arrayed/test_macro_arrayed.mdl`. Each then flows through `mdl_to_mdl_roundtrip()`'s `parse_mdl → project_to_mdl → parse_mdl → assert_semantic_equivalence` automatically. + +**Testing:** +- **macros.AC4.1:** the 6 `macro_*` fixtures plus `macro_arrayed` round-trip — `parse_mdl → project_to_mdl → parse_mdl` yields a semantically equivalent project (same models, same `MacroSpec`s, same variables/equations). `macro_cross_reference` (one macro calls another) and `macro_multi_macros` (two macros) exercise multiple `:MACRO:` blocks; `macro_stock` exercises a macro-body stock; `macro_trailing_definition` exercises a macro block emitted before its invocation. +- **macros.AC4.3:** `macro_multi_output` round-trips — the `:` multi-output call syntax is reconstructed and re-parses to the same materialized cluster (`Variable::Module` + binding auxes) and the same macro `MacroSpec` with non-empty `additional_outputs`. + +**Verification:** +Run: `cargo test -p simlin-engine --test mdl_roundtrip` +Expected: `mdl_to_mdl_roundtrip` passes with all 8 macro fixtures included; no regression in the existing non-macro fixtures. + +**Commit:** `engine: wire macro fixtures into the MDL round-trip harness` + + + +--- + +## Phase 6 completion check + +When all three tasks are committed: +- `project_to_mdl` accepts macro-bearing projects (macro-marked extra models and macro-module instances) while still rejecting ordinary multi-model projects and ordinary submodule instances. +- The MDL writer emits each macro definition as a `:MACRO: … :END OF MACRO:` block (header from `MacroSpec`, body from the model's non-port variables); single-output invocations are preserved as equation text; multi-output invocations are reconstructed into the `:` call syntax from the materialized `Variable::Module` + binding auxes. +- The `mdl_roundtrip.rs` macro exclusion is removed, all 6 bundled `macro_*` fixtures plus the 2 Phase-4 fixtures round-trip, and `assert_model_equivalence` now verifies `macro_spec` (the design's "Done when": `.mdl` round-trip preserves every macro fixture; multi-output reconstruction is verified; the exclusion is removed). +- `macros.AC4.1` and `macros.AC4.3` are verified. + +The hero-model and corpus validation, and the diagram/consumer integration, are Phase 7. diff --git a/docs/implementation-plans/2026-05-13-macros/phase_07.md b/docs/implementation-plans/2026-05-13-macros/phase_07.md new file mode 100644 index 000000000..2ec333e12 --- /dev/null +++ b/docs/implementation-plans/2026-05-13-macros/phase_07.md @@ -0,0 +1,166 @@ +# Vensim Macro Support — Phase 7: Hero validation, corpus, and consumer integration + +**Goal:** Validate the macro implementation against the C-LEARN hero model and the metasd corpus, confirm the bundled fixture suite is wired and green, and make the `src/diagram` consumer macro-aware (filter macro-marked models out of the model-reference list; confirm macro-bearing projects open without crashing). + +**Architecture:** Three independent strands. **(a) C-LEARN:** C-LEARN's four macros must parse, register, and expand with no macro-specific errors (verified by compiling C-LEARN and inspecting diagnostics — C-LEARN's *non-macro* blockers are explicitly out of scope), and focused isolation models invoking each invoked macro (`SAMPLE UNTIL`, `SSHAPE`, `RAMP FROM TO`) with known inputs must compute the macro's defined behavior. **(b) metasd corpus:** a tiered harness over the 14 macro-using metasd models — an *expansion tier* (every model expands its macros with no macro-attributable diagnostics, runnable for all 14 with no prerequisites) and a *simulation tier* (full simulation matches Vensim DSS reference output, runnable only for models that both have a checked-in reference output and have no unrelated blockers). **(c) diagram:** macro-marked models are ordinary entries in `project.models` after import, so the diagram receives them automatically through `projectFromJson` — the work is to skip them in `getAvailableModels` (so they never appear as a selectable module-reference target) and to confirm a macro-bearing project opens and renders without throwing. + +**Tech Stack:** Rust — `src/simlin-engine/tests/`. TypeScript — `src/diagram/`. The Vensim DSS reference outputs for newly-authored fixtures are a documented prerequisite (a setup task, not implementation work — per the design's "Test prerequisites" note); C-LEARN's reference output (`test/xmutil_test_models/Ref.vdf`) already exists in-repo. + +**Scope:** 7 phases from the original design (`docs/design-plans/2026-05-13-macros.md`); this is phase 7 of 7 — the final phase. + +**Codebase verified:** 2026-05-14 + +--- + +## Acceptance Criteria Coverage + +This phase implements and tests: + +### macros.AC6: Validation corpus and consumer floor +- **macros.AC6.1 Success:** All six `test/test-models/tests/macro_*` fixtures are wired into the active test suite and pass. +- **macros.AC6.2 Success:** C-LEARN's macros (`SAMPLE UNTIL`, `SSHAPE`, `RAMP FROM TO`, `INIT`) parse, register, and expand with no macro-specific errors. +- **macros.AC6.3 Success:** Focused models invoking C-LEARN's `SAMPLE UNTIL`, `SSHAPE`, and `RAMP FROM TO` with known inputs match Vensim DSS reference output. +- **macros.AC6.4 Success:** All 14 macro-using metasd models pass the expansion tier; those without unrelated blockers match Vensim DSS reference output. +- **macros.AC6.5 Success:** The diagram opens every macro-bearing fixture without crashing. +- **macros.AC6.6 Success:** Macro-marked models don't appear as standalone, navigable models in the diagram's model list. + +--- + +## Current state (verified 2026-05-14) + +**C-LEARN:** `test/xmutil_test_models/` contains `C-LEARN v77 for Vensim.mdl` (1.4 MB; macros `SAMPLE UNTIL`, `SSHAPE`, `RAMP FROM TO` — all invoked, some arrayed over `[COP]` — plus `INIT`, defined but never invoked) **and `Ref.vdf` (1.86 MB, the Vensim reference output)**. `simulates_clearn()` (`tests/simulate.rs:908-938`) is `#[ignore]`d; it already does the full path — `open_vensim` → `compile_vm` → `Vm::run_to_end` → `VdfFile::parse("../../test/xmutil_test_models/Ref.vdf").to_results_via_records()` → `ensure_vdf_results` — and its `#[ignore]` comment says the blocker is macro expansion. The design explicitly scopes C-LEARN's *non-macro* blockers (circular dependencies, dimension mismatches, unit errors) and *full* end-to-end C-LEARN simulation **out**, "tracked separately." + +**metasd:** `test/metasd/` has 15 subdirectories; the 14 that contain `:MACRO:`-using `.mdl` files (per a repo-wide `:MACRO:` grep) are `bathtub-statistics`, `beer-game`, `covid19-us-homer`, `critical-slowing`, `early-warnings-catastrophe`, `FREE`, `industrial-dynamics`, `interpolating-arrays`, `pink-noise`, `scientific-revolution`, `social-network-valuation`, `theil-statistics`, `thyroid-dynamics`, `wonderland`. **Most have only `.mdl` + `PROVENANCE.md` — no checked-in reference output.** Only `social-network-valuation/` (6 `.vdf` files), `FREE/` (`all_data2.vdf`), and `WRLD3-03/` (the non-macro 15th dir) carry `.vdf`s. The only existing `test/metasd/` test is `simulates_wrld3_03` (`simulate.rs:~865`); there is no corpus loop. + +**Corpus-harness patterns:** the established template is a `static &[&str]` path list + a loop test that accumulates `Vec<(String, String)> failures` and asserts it empty — `incremental_compilation_covers_all_models` (`simulate.rs:~970`), `mdl_to_mdl_roundtrip` (`mdl_roundtrip.rs:245`). Failing/excluded entries are commented out with multi-line reason comments (`TEST_SDEVERYWHERE_MODELS`, `simulate.rs:~531`). Comparison: `ensure_results` / `ensure_vdf_results` (`tests/test_helpers.rs`) — absolute `2e-3`, or relative `max_val * 5e-6` floored at `2e-3` for Vensim-sourced data; `$⁚`-prefixed implicit-module internals are skipped. The `../../test/...` path prefix is relative to `src/simlin-engine/`. + +**Diagnostic API:** `collect_all_diagnostics(&db, &sync) -> Vec` (`db.rs:~2255`); `Diagnostic { model, variable: Option, error: DiagnosticError, severity }`; `DiagnosticSeverity` ∈ `{ Error, Warning }`. `roundtrip.rs` is the precedent for "compile, collect diagnostics, inspect them." There is no macro-specific `ErrorCode` variant — after Phases 1-6 a *correctly* macro-using model produces **zero** macro-attributable diagnostics, so a macro-attributable diagnostic is identified by its `variable`/equation referencing a known macro name or macro-instance, or by being a registry-build error. + +**`src/diagram` model list:** `getAvailableModels(project, currentModelName)` (`src/diagram/module-details-utils.ts:69-112`) returns `{ projectModels, stdlibModels }` — two flat `string[]` lists. Its loop (lines 91-109) over `project.models.keys()` already filters self-references, cycle-creating models, and `STDLIB_PREFIX`-prefixed names — **but has no macro filter**. Its **sole consumer** is `ModuleDetails.tsx:109`, which renders the ``. + +**Verification:** +Run: `pnpm build` (rebuilds `@simlin/core`/`@simlin/engine` so the `macroSpec` field is visible to `@simlin/diagram`) +Run: `pnpm --filter @simlin/diagram test` +Run: `pnpm tsc` +Expected: all pass — macro-marked models are filtered from `getAvailableModels`, and a macro-bearing project opens without crashing. + +**Commit:** `diagram: filter macro-marked models from the model list` + + + +--- + +## Phase 7 completion check + +When all three tasks are committed: +- C-LEARN's four macros parse, register, and expand with no macro-specific errors; focused models invoking `SAMPLE UNTIL`, `SSHAPE`, and `RAMP FROM TO` with known inputs compute the macros' defined behavior (the design's "Done when": C-LEARN's macros validate against reference output). +- A tiered metasd corpus harness passes the expansion tier for all 14 macro-using models and the simulation tier for every model with a checked-in reference output and no unrelated blockers; non-eligible models are annotated with documented reasons (the design's "Done when"). +- All six bundled `test/test-models/tests/macro_*` fixtures are confirmed wired into `simulate.rs` and `mdl_roundtrip.rs` and passing. +- The diagram filters macro-marked models out of `getAvailableModels` so they never appear as a selectable module-reference target, and a macro-bearing project opens without crashing (the design's "Done when"). +- `macros.AC6.1`–`AC6.6` are verified. + +This is the final phase. With Phases 1-7 complete, Vensim macros are a first-class, persistent concept in the Simlin engine: parsed, represented, persisted, simulated, round-tripped through both `.mdl` and XMILE, and surfaced safely to the diagram — meeting the design's Definition of Done. Out-of-scope items (C-LEARN's non-macro blockers and full end-to-end C-LEARN simulation; native macro authoring/editing UX; fixing xmutil's C++ macro parser) remain tracked separately. diff --git a/docs/implementation-plans/2026-05-13-macros/test-requirements.md b/docs/implementation-plans/2026-05-13-macros/test-requirements.md new file mode 100644 index 000000000..c2ffdd056 --- /dev/null +++ b/docs/implementation-plans/2026-05-13-macros/test-requirements.md @@ -0,0 +1,384 @@ +# Vensim Macro Support — Test Requirements + +**Generated:** 2026-05-14 +**Source:** `docs/design-plans/2026-05-13-macros.md` (the authoritative acceptance-criteria list) and the seven implementation-plan phase files `phase_01.md` … `phase_07.md` in this directory (the per-task test-to-AC mapping the planning worked out). + +This file maps every acceptance criterion in the macro design (`macros.AC1.1` … `macros.AC6.6`) to the automated test(s) that verify it — test type, expected file path, the test name or fixture the plan names, the phase + task that produces it, and the `cargo test` / `pnpm test` verification command — or to documented human verification where a criterion genuinely cannot be automated. It is consumed by the `test-analyst` agent during execution to validate coverage. + +**AC count discrepancy noted up front.** The task brief refers to "36 acceptance criteria," but the design plan's `## Acceptance Criteria` section enumerates **37**: AC1 has 7 cases (1.1–1.7), AC2 has 8 (2.1–2.8), AC3 has 5 (3.1–3.5), AC4 has 5 (4.1–4.5), AC5 has 6 (5.1–5.6), AC6 has 6 (6.1–6.6) — 7+8+5+5+6+6 = 37. This file covers all 37; the count is called out again in the coverage summary. This is a judgment call: the design's literal enumeration is treated as authoritative over the brief's round number. + +**Project testing conventions (recap).** +- Engine tests are Rust. Unit tests are inline `#[cfg(test)]` modules in the source file under test; integration tests are files under `src/simlin-engine/tests/` (`simulate.rs`, `mdl_roundtrip.rs`, and the new `metasd_macros.rs`). Run via `cargo test -p simlin-engine …`. +- Diagram tests are TypeScript (Jest, under `src/diagram/tests/`). Run via `pnpm --filter @simlin/diagram test`. +- TypeScript datamodel-mirror tests live in `src/core/tests/`; run via `pnpm --filter @simlin/core test`. Python-mirror tests live in `src/pysimlin/tests/`; run via `uv run pytest` from `src/pysimlin/`. +- Fixture-driven integration tests compare simulation output against checked-in `output.tab` / `.vdf` reference files. Wiring a fixture into `simulate.rs`'s `TEST_MODELS` array runs it through `simulate_path_with`, which asserts: import → simulate against `output.tab` → protobuf round-trip → byte-stable XMILE round-trip. So for an `.xmile` fixture, "the automated test" is the `TEST_MODELS` entry plus `simulate_path_with`'s built-in assertions — not a separately-named function. +- Some tests are `#[ignore]`d (large corpus / hero models that exceed the per-test time budget) with a documented opt-in command (`cargo test … -- --ignored`). These are noted as **automated-but-gated**. + +--- + +## macros.AC1: Macro definitions parse and represent faithfully + +### macros.AC1.1 +**AC text:** *Success: A `.mdl` `:MACRO:` block imports as a macro-marked `Model` whose `MacroSpec.parameters` matches the header's input list in order, with body variables matching the body equations.* + +**Automated test:** Integration / converter-level, two layers. +- **Primary:** inline `#[cfg(test)]` tests in `src/simlin-engine/src/mdl/convert/mod.rs` (or a sibling `convert/macro_tests.rs`), produced by **Phase 2, Task 5** and **Phase 2, Task 7**. Task 5 adds a focused test converting an inline `:MACRO: EXPRESSION MACRO(input, parameter)` `.mdl` via `convert_mdl(...)` and asserting `MacroSpec.parameters == ["input", "parameter"]`, `primary_output == "expression_macro"`, body aux `expression_macro` present with equation text byte-identical to the parameter names, plus synthesized port variables `input`/`parameter` with `compat.can_be_module_input == true`. Task 7 extends this to all six `test/test-models/tests/macro_*` `.mdl` fixtures via `convert_mdl(include_str!(...))`, asserting each macro-marked model's `MacroSpec` and body. +- Verification command: `cargo test -p simlin-engine convert`. + +### macros.AC1.2 +**AC text:** *Success: A `:MACRO:` header with a `:` output list (`add3(a,b,c : minval, maxval)`) imports with `MacroSpec.additional_outputs` populated in order.* + +**Automated test:** Integration / converter-level, with supporting parser-unit tests. +- **Parser support:** inline `#[cfg(test)]` tests in `src/simlin-engine/src/mdl/parser.rs` near `test_parse_macro_start`, and reader tests in `src/simlin-engine/src/mdl/reader.rs`, produced by **Phase 2, Task 2** — assert `:MACRO: add3(a, b, c : minval, maxval)` parses to a `SectionEnd::MacroStart` with `outputs.len() == 2` (names `minval`/`maxval` in order) and the reader builds a `MacroDef.outputs` with those names. Command: `cargo test -p simlin-engine mdl::parser` and `cargo test -p simlin-engine mdl::reader`. +- **Import-level (the AC itself):** inline `#[cfg(test)]` test in `src/simlin-engine/src/mdl/convert/mod.rs`, produced by **Phase 2, Task 5** — converts a `.mdl` with `:MACRO: add3(a, b, c : minval, maxval)` and asserts the macro model's `MacroSpec.additional_outputs == ["minval", "maxval"]` in order, `parameters == ["a", "b", "c"]`, and that `minval`/`maxval` are *not* synthesized as port variables (they are body-computed outputs). Command: `cargo test -p simlin-engine convert`. + +### macros.AC1.3 +**AC text:** *Success: An XMILE `` element imports as a macro-marked `Model`; an expression-form `` is normalized into a macro-named body variable.* + +**Automated test:** Integration / XMILE-reader-level, plus fixture wiring. +- **Reader unit test:** inline `#[cfg(test)]` test in `src/simlin-engine/src/xmile/mod.rs` (or a focused test using `open_xmile` on an in-test XMILE string), produced by **Phase 5, Task 1** — an XMILE string with `aba * b` imports as a `datamodel::Project` containing a macro-marked `Model` with `MacroSpec.parameters == ["a", "b"]`, a body variable `mymacro` carrying the normalized `` `a * b`, and synthesized port variables `a`/`b`. A second test covers a ``-body macro; a third asserts a non-empty `` returns the documented-limitation error. Command: `cargo test -p simlin-engine xmile`. +- **Fixture wiring:** **Phase 5, Task 4** wires the four `.xmile` macro fixtures into `src/simlin-engine/tests/simulate.rs`'s `TEST_MODELS` array — `test/test-models/tests/macro_expression/test_macro_expression.xmile`, `.../macro_multi_expression/test_macro_multi_expression.xmile`, `.../macro_stock/test_macro_stock.xmile` (uncommented at lines 27-29) and `.../macro_multi_macros/test_macro_multi_macros.xmile` (added new). `simulate_path_with` then exercises the `` → macro-marked-model import for each. Command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC1.4 +**AC text:** *Success: A macro-bearing project round-trips losslessly through protobuf and through JSON -- `MacroSpec` and macro body are identical after deserialize.* + +**Automated test:** Unit / serde round-trip, across all five serialization layers — **Phase 1** in full. +- **Protobuf (Rust):** `test_model_with_macro_spec_roundtrip` in `src/simlin-engine/src/serde.rs`, near `test_model_with_loop_metadata_roundtrip` — **Phase 1, Task 2**. Constructs a `Model` with `macro_spec: Some(MacroSpec{...})` (non-empty `parameters`, `primary_output`, `additional_outputs`) plus a body `Variable`, round-trips via `Model::from(project_io::Model::from(expected.clone()))`, `assert_eq!`. Includes a `macro_spec: None` case. Command: `cargo test -p simlin-engine serde`. +- **JSON (Rust):** an extended `test_model_roundtrip` or focused `test_macro_spec_roundtrip` in `src/simlin-engine/src/json.rs` — **Phase 1, Task 3**. Goes `json::Model` → `datamodel::Model` → `json::Model` → `serde_json` string → `json::Model` and `assert_eq!`s. Also regenerates `docs/simlin-project.schema.json` with a `MacroSpec` entry (verified by `cargo test -p simlin-engine generate_and_write_schema`). Command: `cargo test -p simlin-engine json`. +- **TypeScript mirror:** a `describe('MacroSpec', …)` block in `src/core/tests/datamodel.test.ts` — **Phase 1, Task 4**. A "should roundtrip correctly" test (`macroSpecToJson` → `macroSpecFromJson`, assert equality) and a "should omit empty additionalOutputs" test. Command: `pnpm --filter @simlin/core test`. +- **Python mirror:** a `MacroSpec` round-trip test in `src/pysimlin/tests/test_json_types.py` — **Phase 1, Task 5**. Builds a `Model` with a populated `macro_spec`, unstructures via `json_converter`, structures back, asserts identity; confirms camelCase keys. Command: `cd src/pysimlin && uv run pytest tests/test_json_types.py -x`. + +### macros.AC1.5 +**AC text:** *Success: `$`-suffixed time references in a macro body (`TIME STEP$`, `Time$`) import as the canonical time identifiers.* + +**Automated test:** Unit (helper-level) + integration (import-level) — **Phase 2, Task 6**. +- **Helper-level:** inline `#[cfg(test)]` tests in `src/simlin-engine/src/mdl/convert/helpers.rs` — build `Expr` trees with `Var("Time$")`, `Var("TIME STEP$")`, `Var("Initial Time$")`, and a nested `c + MYMACRO(Time$, x)` shape; assert the `$`-time rewrite produces `time`, `time_step`, `initial_time`, rewrites the nested occurrence, and leaves non-time `Var`s (`"foo$"`, `"input"`) untouched. +- **Import-level (the AC):** inline `#[cfg(test)]` test in `src/simlin-engine/src/mdl/convert/mod.rs` — `convert_mdl` a `.mdl` whose macro body uses `Time$` and `TIME STEP$` (e.g. `MYMACRO = input * Time$ / TIME STEP$`); assert the imported macro model's body equation text references `time` and `time_step`. +- Verification command: `cargo test -p simlin-engine convert`. + +### macros.AC1.6 +**AC text:** *Failure: A `:MACRO:` block with no `:END OF MACRO:` reports a clear parse error.* + +**Automated test:** Integration / error-case — **Phase 2, Task 7**. +- Inline `#[cfg(test)]` test in `src/simlin-engine/src/mdl/convert/mod.rs` (or `convert/macro_tests.rs`) — asserts `open_vensim` / `convert_mdl` on a `.mdl` containing `:MACRO: BAD(x)` … with no `:END OF MACRO:` returns `Err`, specifically `ConvertError::Reader(ReaderError::EofInsideMacro)` whose `Display` is the clear message `"reader error: unexpected end of file inside macro"` (through `open_vensim`: `"Failed to parse MDL: reader error: unexpected end of file inside macro"`). The same test also asserts a stray `:END OF MACRO:` with no opening `:MACRO:` yields `ReaderError::UnmatchedMacroEnd`. +- Verification command: `cargo test -p simlin-engine convert`. + +### macros.AC1.7 +**AC text:** *Edge: A macro defined but never invoked (C-LEARN's `INIT`) imports as a valid macro-marked model and is preserved.* + +**Automated test:** Integration / converter-level — **Phase 2, Task 5** and **Phase 2, Task 7**. +- Task 5 adds a focused test: `convert_mdl` a `.mdl` that defines a macro but never invokes it; assert the macro-marked `Model` is still present in `project.models` with the correct `MacroSpec` and synthesized port variables. +- Task 7 covers the `macro_trailing_definition` fixture (defined after its call site, exercising definition-order leniency) and confirms its preservation. +- The C-LEARN `INIT` case specifically (a real never-invoked macro) is additionally covered by **Phase 7, Task 1**'s C-LEARN macro-expansion test, which asserts all four C-LEARN macros — including `INIT` — import as macro-marked `Model`s with correct `MacroSpec`s (automated-but-gated, `#[ignore]`d). +- Verification command: `cargo test -p simlin-engine convert` (Phase 2); `cargo test -p simlin-engine --test simulate -- --ignored` (Phase 7 C-LEARN). + +--- + +## macros.AC2: Single-output invocations simulate with correct Vensim semantics + +### macros.AC2.1 +**AC text:** *Success: A stockless single-output macro invocation (`macro_expression`) simulates and matches the fixture's `output.tab`.* + +**Automated test:** Integration / fixture simulation — **Phase 3, Task 4** (with an end-to-end smoke in **Phase 3, Task 3**). +- `simulates_macro_expression_mdl` in `src/simlin-engine/tests/simulate.rs` — calls `simulate_mdl_path("../../test/test-models/tests/macro_expression/test_macro_expression.mdl")`, which does `open_vensim` → `compile_vm` → run → `ensure_results` against the fixture's `output.tab`. +- Phase 3 Task 3 additionally adds an inline end-to-end smoke in `src/simlin-engine/src/builtins_visitor.rs`'s `#[cfg(test)] mod tests`: a trivial `:MACRO: M(a, b)` / `M = a * b` invoked `y = M(5, 1.1)`, compiled and run, asserting `y == 5.5`. +- Verification command: `cargo test -p simlin-engine --test simulate` (fixture); `cargo test -p simlin-engine builtins_visitor` (smoke). + +### macros.AC2.2 +**AC text:** *Success: A stock-bearing macro invocation (`macro_stock`) simulates with correct per-invocation integration, matching the 11-step `output.tab`.* + +**Automated test:** Integration / fixture simulation — **Phase 3, Task 4**. +- `simulates_macro_stock_mdl` in `src/simlin-engine/tests/simulate.rs` — `simulate_mdl_path(".../macro_stock/test_macro_stock.mdl")`, comparing against the fixture's 11-step `output.tab`. +- Verification command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC2.3 +**AC text:** *Success: The same macro invoked at multiple call sites produces independent per-invocation state -- two stock-bearing invocations don't share a stock.* + +**Automated test:** Integration / focused inline-`.mdl` simulation — **Phase 3, Task 4** (Group 2). +- A focused `#[test] fn` in `src/simlin-engine/tests/simulate.rs` — an inline `.mdl` defining a stock-bearing macro `M(rate, init) = INTEG(rate, init)` invoked twice (`x = M(1, 0)`, `y = M(2, 10)`); compiled via `compile_vm`, run, with hand-computed assertions that `x` integrates `1`/step from `0` and `y` integrates `2`/step from `10` — proving they do not share a stock. The plan does not pin a function name; expected name follows the file's `simulates_*` convention. +- Verification command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC2.4 +**AC text:** *Success: A macro invoked with an expression-valued argument (`MYMACRO(a + b, t)`) simulates correctly, with the argument evaluated in the caller's context.* + +**Automated test:** Integration / focused inline-`.mdl` simulation — **Phase 3, Task 4** (Group 2). +- A focused `#[test] fn` in `src/simlin-engine/tests/simulate.rs` — an inline `.mdl` with `M(in, p) = in * p` invoked `y = M(a + b, t)` over constants `a`, `b`, `t`; hand-computed assertion `y == (a + b) * t`, confirming the argument is evaluated in the caller's context. +- Verification command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC2.5 +**AC text:** *Success: A multi-equation macro body with macro-local helpers (`macro_multi_expression`) simulates correctly; helper names don't leak into the caller's namespace.* + +**Automated test:** Integration / fixture simulation — **Phase 3, Task 4**. +- `simulates_macro_multi_expression_mdl` in `src/simlin-engine/tests/simulate.rs` — `simulate_mdl_path(".../macro_multi_expression/test_macro_multi_expression.mdl")`. The plan additionally requires the test to assert the `main` model has no `intermediate` variable (the macro-local helper does not leak into the caller's namespace), beyond `ensure_results`'s expected-columns check. +- Verification command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC2.6 +**AC text:** *Success: A macro that calls another macro (`macro_cross_reference`) expands recursively and simulates correctly.* + +**Automated test:** Integration / fixture simulation — **Phase 3, Task 4**. +- `simulates_macro_cross_reference_mdl` in `src/simlin-engine/tests/simulate.rs` — `simulate_mdl_path(".../macro_cross_reference/test_macro_cross_reference.mdl")`. ("Expands recursively" here means a macro body that invokes another macro — `EXPRESSION MACRO` body calls `SECOND MACRO` — not self-recursion, which is rejected per AC5.2.) +- Verification command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC2.7 +**AC text:** *Success: A macro body referencing global time via the `$` escape simulates with the global time values.* + +**Automated test:** Integration / focused inline-`.mdl` simulation — **Phase 3, Task 4** (Group 2). +- A focused `#[test] fn` in `src/simlin-engine/tests/simulate.rs` — an inline `.mdl` with a macro body referencing `Time$` (e.g. `M(x) = x + Time$`) invoked `y = M(10)`; hand-computed assertion `y == 10 + time` at each step. This exercises Phase 2's `$`-time translation plus Phase 3's global-time-resolves-inside-a-module behavior. +- Verification command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC2.8 +**AC text:** *Edge: A macro invocation nested inside a larger expression (`y = c + MYMACRO(x, t)`) expands and simulates correctly.* + +**Automated test:** Integration / focused inline-`.mdl` simulation — **Phase 3, Task 4** (Group 2). +- A focused `#[test] fn` in `src/simlin-engine/tests/simulate.rs` — an inline `.mdl` with `y = c + M(x, t)` (a macro call inside a larger expression); hand-computed assertion that `y` equals `c` plus the macro's value. +- Verification command: `cargo test -p simlin-engine --test simulate`. + +--- + +## macros.AC3: Multi-output and arrayed invocation + +### macros.AC3.1 +**AC text:** *Success: A multi-output invocation (`total = add3(a,b,c : minv, maxv)`) materializes as a module instance; `total` receives the primary output and `minv`/`maxv` become model variables holding the additional outputs.* + +**Automated test:** Integration / converter-level structure test — **Phase 4, Task 1**. +- Inline `#[cfg(test)]` test in `src/simlin-engine/src/mdl/convert/` (the same module as Phase 2's macro-conversion tests) — `convert_mdl(include_str!(".../macro_multi_output/test_macro_multi_output.mdl"))`; asserts the main model contains exactly one `Variable::Module` whose `model_name` is the `ADD3` macro's model with input `ModuleReference`s wiring `in1`/`in2`/`in3` to ports `a`/`b`/`c`; asserts `total` is a `Variable::Aux` reading `.` (ASCII-period datamodel form); asserts `the min` / `the max` are `Variable::Aux`es reading `.minval` / `.maxval`. The fixture `test/test-models/tests/macro_multi_output/test_macro_multi_output.mdl` is authored by this task. +- Verification command: `cargo test -p simlin-engine convert`. + +### macros.AC3.2 +**AC text:** *Success: `minv` and `maxv` are referenceable by subsequent equations and carry the correct values.* + +**Automated test:** Integration / fixture simulation — **Phase 4, Task 1**. +- `simulates_macro_multi_output_mdl` in `src/simlin-engine/tests/simulate.rs` — `simulate_mdl_path(".../macro_multi_output/test_macro_multi_output.mdl")`. The authored fixture includes a **downstream equation** that references the additional outputs (e.g. `spread = the max - the min`); the fixture's hand-computed `output.tab` includes `total`, `the min`, `the max`, and `spread`, so the passing test proves `the min`/`the max` are referenceable by a subsequent equation and carry correct values. +- Verification command: `cargo test -p simlin-engine --test simulate simulates_macro_multi_output_mdl`. + +### macros.AC3.3 +**AC text:** *Success: The metasd multi-output models (`THEIL`, `SSTATS`) expand and simulate.* + +**Automated test:** Integration / corpus-model — **Phase 4, Task 3** (early gate) and **Phase 7, Task 2** (comprehensive tier). Automated-but-gated for the heavy models. +- **Phase 4, Task 3:** focused tests in `src/simlin-engine/tests/simulate.rs` — import `test/metasd/theil-statistics/Theil_2011.mdl` via `open_vensim`, assert the `THEIL` multi-output invocation materialized (a `Variable::Module` plus binding `Aux`es for the primary output and all 13 `:`-list outputs), compile via `compile_vm`, run to the end. Same for `SSTATS` in `test/metasd/covid19-us-homer/homer v8/Covid19US v8.mdl` (a `Variable::Module` + 1 primary + 10 additional binding auxes per invocation). Heavy models are `#[ignore]`d with a documented opt-in command; `Theil_2011.mdl` may stay a regular test if it fits the per-test budget. If the COVID model has unrelated non-macro blockers, the assertion narrows to "the `SSTATS` materialization succeeded with no macro-specific diagnostics" and the blocker is filed via `track-issue`. +- **Phase 7, Task 2:** `THEIL` and `SSTATS` are also covered by the `metasd_macros.rs` tiered corpus harness (expansion tier for all 14 directories; simulation tier where a reference output exists). +- Verification command: `cargo test -p simlin-engine --test simulate -- --ignored` (gated); `cargo test -p simlin-engine --test metasd_macros` (corpus harness, plus `-- --ignored` if gated). + +### macros.AC3.4 +**AC text:** *Success: An arrayed invocation (`y[Dim] = MYMACRO(x[Dim], …)`) expands into one independent macro instance per dimension element.* + +**Automated test:** Integration / fixture simulation + expansion-level structure assertion — **Phase 4, Task 2**, building on **Phase 3, Task 3**. +- `simulates_macro_arrayed_mdl` in `src/simlin-engine/tests/simulate.rs` — `simulate_mdl_path(".../macro_arrayed/test_macro_arrayed.mdl")`. The authored fixture `test/test-models/tests/macro_arrayed/test_macro_arrayed.mdl` is a stockless macro `:MACRO: SCALE(x, k)` invoked apply-to-all `out[Region] = SCALE(inp[Region], factor)` over a small 2-3-element dimension; the hand-computed `output.tab` has one column per `out[Region]` element. +- Additionally, a `convert/`- or expansion-level assertion (per the plan) that the arrayed invocation produced one synthetic `Variable::Module` per element (subscript-suffixed idents), not a single shared instance. +- The supporting `contains_stdlib_call` macro-awareness predicate is unit-tested in **Phase 3, Task 3** (a focused test in `src/simlin-engine/src/builtins_visitor.rs` asserting it returns `true` for an arrayed macro `App`). +- Verification command: `cargo test -p simlin-engine --test simulate macro_arrayed`. + +### macros.AC3.5 +**AC text:** *Edge: An arrayed invocation of a stock-bearing macro gives each element its own persistent stock.* + +**Automated test:** Integration / focused inline-`.mdl` simulation — **Phase 4, Task 2**. +- A focused inline-`.mdl` test in `src/simlin-engine/tests/simulate.rs` — a stock-bearing macro `:MACRO: ACCUM(rate)` with body `ACCUM = INTEG(rate, 0)`, arrayed invocation `total[Region] = ACCUM(rate[Region])` with `rate[Region]` differing per element; `open_vensim` + `compile_vm` + run + hand-computed assertions that each `total[element]` integrates *its own* `rate[element]` independently (e.g. `rate = [1, 3]` over 4 steps at dt=1 → `total[r1] = 0,1,2,3,4` and `total[r2] = 0,3,6,9,12`), proving per-element persistent state. +- Verification command: `cargo test -p simlin-engine --test simulate macro_arrayed`. + +--- + +## macros.AC4: Round-trip and export + +### macros.AC4.1 +**AC text:** *Success: A macro-bearing `.mdl` file round-trips with definitions emitted as `:MACRO:` blocks and invocations preserved.* + +**Automated test:** Unit (writer) + integration (round-trip harness) — **Phase 6, Task 1** and **Phase 6, Task 3**. +- **Writer unit tests:** full-project-style tests in `src/simlin-engine/src/mdl/writer_tests.rs` — **Phase 6, Task 1**. Build a `datamodel::Project` with a `main` model and a single-output macro-marked `Model` → `project_to_mdl` → assert the output contains a `:MACRO: ()` header, the body equation(s), `:END OF MACRO:`, that synthesized port variables do *not* appear as body equations, and that the main model's invocation equation is preserved as call text. Includes the rewritten `project_to_mdl_rejects_multiple_models` (ordinary multi-model still rejected; macro-marked extra model accepted). Command: `cargo test -p simlin-engine mdl::writer`. +- **Round-trip harness (the AC end-to-end):** **Phase 6, Task 3** wires the 6 bundled `macro_*` fixtures plus `macro_arrayed` into `TEST_MDL_MODELS` in `src/simlin-engine/tests/mdl_roundtrip.rs`; `mdl_to_mdl_roundtrip()` drives `parse_mdl → project_to_mdl → parse_mdl → assert_semantic_equivalence` for each. Task 3 also extends `assert_model_equivalence` to compare `Model.macro_spec` and `Model.name`, so the macro round-trip is actually verified (not silently passed). Command: `cargo test -p simlin-engine --test mdl_roundtrip`. + +### macros.AC4.2 +**AC text:** *Success: A macro-bearing XMILE file round-trips with `` elements and `simlin:` extensions; the `` header option is emitted.* + +**Automated test:** Integration / XMILE-writer tests + fixture round-trip — **Phase 5, Tasks 2, 3, 4**. +- **Writer unit tests (definition + header option):** inline `#[cfg(test)]` tests in `src/simlin-engine/src/xmile/mod.rs` — **Phase 5, Task 2**. A `datamodel::Project` with a single-output macro-marked `Model` → `to_xmile` → assert the output contains a `` element with expected ``s and body, and a `` header option. Plus a `to_xmile` → `open_xmile` round-trip asserting the macro-marked model survives with the same `MacroSpec` and body. +- **`simlin:` extension round-trip:** inline `#[cfg(test)]` tests in `src/simlin-engine/src/xmile/mod.rs` — **Phase 5, Task 3**. A multi-output macro project → `to_xmile` → `open_xmile` → assert the round-tripped project has the same `MacroSpec.additional_outputs` and the same materialized `Variable::Module` + binding `Aux`es; a second `to_xmile` is byte-identical to the first. +- **Fixture round-trip (the AC end-to-end):** **Phase 5, Task 4** wires the four `.xmile` macro fixtures into `TEST_MODELS` in `src/simlin-engine/tests/simulate.rs`; `simulate_path_with` performs a byte-stable XMILE round-trip assertion (serialize → re-parse → re-serialize → `assert_eq!`) for each. +- Verification command: `cargo test -p simlin-engine xmile` (writer / extension); `cargo test -p simlin-engine --test simulate` (fixture round-trip). + +### macros.AC4.3 +**AC text:** *Success: A multi-output macro round-trips through `.mdl` with the `:` call syntax reconstructed.* + +**Automated test:** Unit (writer) + integration (round-trip harness) — **Phase 6, Task 2** and **Phase 6, Task 3**. +- **Writer unit tests:** full-project-style tests in `src/simlin-engine/src/mdl/writer_tests.rs` — **Phase 6, Task 2**. Build a `datamodel::Project` with a multi-output macro-marked `Model` and a materialized multi-output cluster (`Variable::Module` + primary-output binding aux + additional-output binding auxes) → `project_to_mdl` → assert the output contains the reconstructed ` = ( : )` `:` call syntax with arguments in positional order and output bindings in `:`-list order, and that the `Variable::Module` / binding auxes do *not* appear as separate entries. Includes the rewritten `project_to_mdl_rejects_module_variable` (ordinary module still rejected; macro-module accepted). Command: `cargo test -p simlin-engine mdl::writer`. +- **Round-trip harness (the AC end-to-end):** **Phase 6, Task 3** adds `test/test-models/tests/macro_multi_output/test_macro_multi_output.mdl` (the Phase-4-authored `:`-multi-output fixture) to `TEST_MDL_MODELS` in `src/simlin-engine/tests/mdl_roundtrip.rs`; the round-trip re-parses the reconstructed `:` syntax to the same materialized cluster and the same `MacroSpec` with non-empty `additional_outputs`. Command: `cargo test -p simlin-engine --test mdl_roundtrip`. + +### macros.AC4.4 +**AC text:** *Success: A cross-format conversion (`.mdl` → datamodel → `.xmile`) preserves macro definitions and invocations.* + +**Automated test:** Integration / cross-format conversion test — **Phase 5, Task 4**. +- A dedicated test in `src/simlin-engine/tests/simulate.rs` — `open_vensim`s a single-output macro `.mdl` fixture (e.g. `test/test-models/tests/macro_expression/test_macro_expression.mdl`), converts the resulting `datamodel::Project` to XMILE via `to_xmile`, re-imports via `open_xmile`, and asserts the macro definition (the macro-marked `Model` + its `MacroSpec`) and the invocation are preserved — the cross-format-round-tripped project's macro models and invocation equations match those of the directly-imported `.mdl` project. (The plan notes "MDL→XMILE conversion is untested anywhere" today; this test is net-new coverage.) The plan does not pin a function name. +- Verification command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC4.5 +**AC text:** *Edge: A single-output-only model exports as standards-clean XMILE with no extensions; multi-output triggers the `simlin:` extension.* + +**Automated test:** Integration / XMILE-writer tests — **Phase 5, Task 2** (standards-clean half) and **Phase 5, Task 3** (multi-output-triggers half). +- **Standards-clean (Phase 5, Task 2):** an inline `#[cfg(test)]` test in `src/simlin-engine/src/xmile/mod.rs` — assert the `to_xmile` output of a single-output-only macro project contains **no** `simlin:`-prefixed macro-extension element (specifically no macro-additional-output extension; other pre-existing `simlin:` elements like `simlin:loop-metadata` may still be present). +- **Multi-output triggers the extension (Phase 5, Task 3):** an inline `#[cfg(test)]` test in `src/simlin-engine/src/xmile/mod.rs` — a `datamodel::Project` with a multi-output macro (non-empty `MacroSpec.additional_outputs`) and a multi-output invocation → `to_xmile` → assert the output contains the `simlin:` additional-outputs extension on `` and the `simlin:` multi-output-invocation extension; and (contrast) confirm a single-output project does not. +- Verification command: `cargo test -p simlin-engine xmile`. + +--- + +## macros.AC5: Error handling and edge cases + +### macros.AC5.1 +**AC text:** *Failure: A macro invoked with the wrong number of arguments reports an arity-mismatch diagnostic naming the macro.* + +**Automated test:** Unit / expansion-level — **Phase 3, Task 3**. +- An inline `#[cfg(test)]` test in `src/simlin-engine/src/builtins_visitor.rs`'s `mod tests` (or `macro_expansion_tests.rs`) — a macro declared with 2 parameters, invoked with 3 args (and separately with 1 arg); compile fails with `ErrorCode::BadBuiltinArgs`; the test asserts the error span covers the macro call so the macro is identified in context. +- Note: the multi-output materialization path also has an arity check — **Phase 4, Task 1** adds a `convert/`-level test that `convert_mdl` of a `.mdl` invoking `ADD3` with the wrong argument or `:`-output count returns a `ConvertError` naming `ADD3`. This is supplementary coverage of the same AC for the multi-output form. +- Verification command: `cargo test -p simlin-engine builtins_visitor` (single-output); `cargo test -p simlin-engine convert` (multi-output path). + +### macros.AC5.2 +**AC text:** *Failure: A directly or mutually recursive macro is rejected with a cycle-detection error rather than expanding without termination.* + +**Automated test:** Unit (registry-build) + integration (end-to-end compile failure) — **Phase 3, Task 1** and **Phase 3, Task 2**. +- **Registry-build unit test:** inline `#[cfg(test)]` tests in `src/simlin-engine/src/module_functions.rs` — **Phase 3, Task 1**. A macro `A` whose body calls `A` → `MacroRegistry::build` returns `Err` with `CircularDependency`; a macro `A` calling `B` calling `A` → `Err`; a macro `A` calling `B` with no cycle (the `macro_cross_reference` shape) → `build` succeeds. Command: `cargo test -p simlin-engine module_functions`. +- **End-to-end compile failure:** inline `#[cfg(test)]` tests in `src/simlin-engine/src/model.rs`'s `#[cfg(test)]` module — **Phase 3, Task 2**. `convert_mdl` a `.mdl` with a directly recursive macro and a `main` invocation, compile, assert the compile fails with a cycle-detection error (`CircularDependency`) whose message names the macro; repeat for a mutually recursive `A`/`B` pair. Command: `cargo test -p simlin-engine model::`. + +### macros.AC5.3 +**AC text:** *Failure: Two macros sharing a name, or a macro name colliding with a model name, report a registry-build diagnostic.* + +**Automated test:** Unit (registry-build) + integration (end-to-end compile failure) — **Phase 3, Task 1** and **Phase 3, Task 2**. +- **Registry-build unit test:** inline `#[cfg(test)]` tests in `src/simlin-engine/src/module_functions.rs` — **Phase 3, Task 1**. Two macro models named `FOO` → `MacroRegistry::build` returns `Err` naming `FOO`; a macro named `main` alongside a `main` model → `build` returns `Err` naming the collision. Command: `cargo test -p simlin-engine module_functions`. +- **End-to-end compile failure:** inline `#[cfg(test)]` tests in `src/simlin-engine/src/model.rs`'s `#[cfg(test)]` module — **Phase 3, Task 2**. `convert_mdl` a `.mdl` with two `:MACRO:` blocks of the same name → assert the compile fails with a duplicate-name error naming the macro; a `.mdl` with a macro named `main` → assert the compile fails with a collision error. Command: `cargo test -p simlin-engine model::`. + +### macros.AC5.4 +**AC text:** *Success: A macro shadowing a builtin (`SSHAPE`, `RAMP FROM TO`) resolves to the macro; the builtin is not invoked.* + +**Automated test:** Unit (expansion-level) + integration (simulation-level confirmation) — **Phase 3, Task 3** and **Phase 3, Task 4**. +- **Expansion-level (Phase 3, Task 3):** an inline `#[cfg(test)]` test in `src/simlin-engine/src/builtins_visitor.rs`'s `mod tests` — a `.mdl` defining `:MACRO: SSHAPE(x, p)` with body `SSHAPE = x + p`, invoked `y = SSHAPE(3, 4)` → compiles and simulates with `y == 7` (the macro's definition), proving the macro shadowed the `SSHAPE` builtin. Repeated for a `RAMP FROM TO` macro. Command: `cargo test -p simlin-engine builtins_visitor`. +- **Simulation-level confirmation (Phase 3, Task 4):** a focused test in `src/simlin-engine/tests/simulate.rs` — a macro shadowing `SSHAPE` invoked in a model that also uses other builtins; assert the simulated value matches the macro's definition, not the builtin's. Command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC5.5 +**AC text:** *Success: A macro defined after its first use (`macro_trailing_definition`) still resolves and simulates.* + +**Automated test:** Integration / fixture simulation — **Phase 3, Task 4** (with import-level coverage in **Phase 2, Task 7**). +- `simulates_macro_trailing_definition_mdl` in `src/simlin-engine/tests/simulate.rs` — `simulate_mdl_path(".../macro_trailing_definition/test_macro_trailing_definition.mdl")`. The fixture defines the macro after its call site; the passing simulation proves it resolves and simulates (definition-order leniency). +- Phase 2 Task 7 additionally covers the import-level half (the macro model is produced regardless of definition order). +- Verification command: `cargo test -p simlin-engine --test simulate`. + +### macros.AC5.6 +**AC text:** *Failure: A call to a name that is neither a macro, a stdlib function, nor a builtin reports an "unknown function or macro" error.* + +**Automated test:** Unit / expansion-level — **Phase 3, Task 3**. +- An inline `#[cfg(test)]` test in `src/simlin-engine/src/builtins_visitor.rs`'s `mod tests` — a `.mdl` with `y = NOTAFUNCTION(x)` (a name that is neither macro, stdlib, nor builtin) → compile fails with `ErrorCode::UnknownBuiltin`. +- Verification command: `cargo test -p simlin-engine builtins_visitor`. + +--- + +## macros.AC6: Validation corpus and consumer floor + +### macros.AC6.1 +**AC text:** *Success: All six `test/test-models/tests/macro_*` fixtures are wired into the active test suite and pass.* + +**Automated test:** Integration / fixture wiring across `simulate.rs` and `mdl_roundtrip.rs` — produced incrementally by **Phases 3, 5, 6** and **confirmed explicitly by Phase 7, Task 2**. +- The six `.mdl` fixtures are wired into `src/simlin-engine/tests/simulate.rs` as `simulates_macro_*_mdl` tests (**Phase 3, Task 4**) and into `src/simlin-engine/tests/mdl_roundtrip.rs`'s `TEST_MDL_MODELS` (**Phase 6, Task 3**). The four `.xmile` fixtures are wired into `simulate.rs`'s `TEST_MODELS` array (**Phase 5, Task 4**). +- **Phase 7, Task 2** is the explicit confirmation step: run the suite and inspect; if any `macro_*` fixture is not wired (a gap left by an earlier phase), wire it. No new harness is needed — AC6.1 is satisfied by the earlier phases' wiring, and Phase 7 makes the satisfaction complete and explicit. +- Verification command: `cargo test -p simlin-engine --test simulate` and `cargo test -p simlin-engine --test mdl_roundtrip`. + +### macros.AC6.2 +**AC text:** *Success: C-LEARN's macros (`SAMPLE UNTIL`, `SSHAPE`, `RAMP FROM TO`, `INIT`) parse, register, and expand with no macro-specific errors.* + +**Automated test:** Integration / hero-model expansion — **Phase 7, Task 1**. Automated-but-gated (`#[ignore]`d; C-LEARN is 1.4 MB). +- An `#[ignore]`d test in `src/simlin-engine/tests/simulate.rs` — `open_vensim`s `../../test/xmutil_test_models/C-LEARN v77 for Vensim.mdl`, asserts its four macros imported as macro-marked `Model`s with correct `MacroSpec`s, syncs + compiles via the salsa path, collects diagnostics via `collect_all_diagnostics`, and asserts **no diagnostic is macro-attributable** (none reference a macro name or macro-instance variable, none are macro-registry-build errors). C-LEARN's known non-macro blockers (circular dependencies, dimension mismatches, unit errors, and any non-time `$` reference) are expected and explicitly allowed. An early-gate version of this expansion check is also in **Phase 4, Task 3**. +- Verification command: `cargo test -p simlin-engine --test simulate -- --ignored`. + +### macros.AC6.3 +**AC text:** *Success: Focused models invoking C-LEARN's `SAMPLE UNTIL`, `SSHAPE`, and `RAMP FROM TO` with known inputs match Vensim DSS reference output.* + +**Automated test:** Integration / focused fixture simulation — **Phase 7, Task 1**. +- Three `simulate_mdl_path` tests in `src/simlin-engine/tests/simulate.rs` (matching `macro_clearn`), one per invoked C-LEARN macro. Each test's fixture is a small focused `.mdl` under `test/test-models/tests/macro_clearn_*/` that defines the macro (its `:MACRO:` block copied verbatim from C-LEARN) and invokes it with known constant inputs, paired with an expected-output file. The plan: prefer a Vensim DSS reference `.vdf` if one is provided (a documented prerequisite setup task) and compare via `ensure_vdf_results`; otherwise the formula-derived `output.tab` (computed by applying the macro's body formula to the known inputs, arithmetic documented in the fixture `README.md`) is the gate. `INIT` is not invoked in C-LEARN and needs no focused model — AC6.2 covers it. +- **Test-prerequisite dependency:** the Vensim DSS reference `.vdf` for these focused fixtures is a documented setup task ("Test prerequisites" in the design), not implementation work. The automated test still exists and gates either way — against the reference `.vdf` if present, or against the formula-derived `output.tab`. So this is fully automated; only the *strength* of the reference (DSS-validated vs formula-derived) depends on the prerequisite. +- Verification command: `cargo test -p simlin-engine --test simulate macro_clearn`. + +### macros.AC6.4 +**AC text:** *Success: All 14 macro-using metasd models pass the expansion tier; those without unrelated blockers match Vensim DSS reference output.* + +**Automated test:** Integration / tiered corpus harness — **Phase 7, Task 2**. Partially automated-but-gated (heavy models `#[ignore]`d). +- A new integration-test file `src/simlin-engine/tests/metasd_macros.rs` with a tiered corpus harness over the 17 macro-using `.mdl` files across the 14 macro-using `test/metasd/` directories (the plan enumerates the exact path list). Two tiers: + - **Expansion tier (all 14 directories / 17 files):** `open_vensim` → sync → compile → `collect_all_diagnostics`; assert no macro-attributable diagnostic; accumulate `(model, macro-diagnostic)` failures and assert the failure vec is empty. Runnable for all 14 with no prerequisites. + - **Simulation tier (subset):** for each model that **both** has a checked-in Vensim reference output **and** has no unrelated blockers, run the VM to completion and compare via `ensure_vdf_results` / `ensure_results`. Every non-eligible model is annotated with its reason (`no reference output checked in` — a documented prerequisite; or `unrelated blocker: ` — filed via `track-issue`). + - Heavy models (e.g. `covid19-us-homer`) are `#[ignore]`d individually or the whole harness gated, measured per `docs/dev/rust.md`. +- **Test-prerequisite dependency:** the simulation tier's coverage depends on Vensim DSS reference outputs being generated for the metasd models — a documented setup task. The expansion tier (the "all 14 pass" half of the AC) is fully automated with no prerequisite; the simulation tier (the "those without unrelated blockers match reference output" half) is automated for whichever models have a checked-in reference, and the harness self-documents which models are not yet simulation-tier-eligible and why. +- Verification command: `cargo test -p simlin-engine --test metasd_macros` (plus `-- --ignored` if heavy models are gated). + +### macros.AC6.5 +**AC text:** *Success: The diagram opens every macro-bearing fixture without crashing.* + +**Automated test:** e2e / diagram component (Jest) — **Phase 7, Task 3**. +- A test in `src/diagram/tests/editor-open-project.test.ts` following the existing `Object.create(Editor.prototype)` + `makeFakeEngine` + `openEngineProject` pattern — constructs a `validProjectJson` whose `models` array includes a macro-marked model (with a `macroSpec` and synthesized port variables) alongside a `main` model; asserts `openEngineProject()` resolves, `state.activeProject` is defined, and no exception is thrown. Optionally a `ModuleDetails` component test (`@testing-library/react`) asserting the macro-marked model name does not appear in the rendered `` DOM and the canvas render), and **in-tool cross-format export +fidelity / cross-tool (Vensim DSS) numeric fidelity**. + +All paths are absolute; all commands run from `/home/bpowers/src/simlin`. + +### Prerequisites + +- `./scripts/dev-init.sh` has been run (idempotent). +- Toolchain confirmed: `cargo --version`, `pnpm --version`, `uv --version` all succeed. +- Baseline green (re-confirm before manual UI work): + - `cargo test -p simlin-engine --features file_io --test simulate --test mdl_roundtrip --test metasd_macros` + → all non-gated macro tests pass; C-LEARN/heavy models `#[ignore]`d. + - `cd src/diagram && npx jest module-details-utils editor-open-project` → green. + +## Phase A — Gated heavy-corpus opt-in runs (automated, human-initiated) + +Exercises the heavy/hero models excluded from CI by the per-test time budget. These ARE automated +coverage; a human must opt in. + +| Step | Action | Expected | +|------|--------|----------| +| A1 | `cargo test -p simlin-engine --release --test simulate -- --ignored corpus_clearn_macros_import` | PASS. C-LEARN's 4 macros (`sample_until`/`sshape`/`ramp_from_to`/`init`) import with the exact parameter lists asserted; zero macro-attributable diagnostics. Runtime ~minutes. | +| A2 | `cargo test -p simlin-engine --release --test simulate -- --ignored corpus_sstats_multi_output_materializes` | PASS. 2 SSTATS module instances, 22 binding auxes; only `*_data` GET-DIRECT diagnostics, no macro-specific ones. | +| A3 | `cargo test -p simlin-engine --release --test metasd_macros -- --ignored metasd_expansion_tier_full` | PASS. All 17 macro-using metasd files. Watch stderr: every line prints `macro_attributable=0`. Confirm **no** line has `macro_attributable>0`. | +| A4 | `cargo test -p simlin-engine --release --test simulate -- --ignored simulates_clearn` | An ignored/failed result is acceptable — this is GH #559 (non-macro C-LEARN blockers, out of scope). Confirm any failure is **not** macro-attributable (no `recursive macro:`, no `DuplicateMacroName`, no macro-model compile error). A macro-attributable error here is a regression — escalate. | + +## Phase B — Diagram UI: macro models absent from the model-reference dropdown (AC6.5/AC6.6 in the real UI) + +| Step | Action | Expected | +|------|--------|----------| +| B1 | Start the local viewer (`pnpm --filter @simlin/serve dev` or the documented local app path); open the served URL. | App loads, no console errors. | +| B2 | Import `/home/bpowers/src/simlin/test/test-models/tests/macro_multi_macros/test_macro_multi_macros.mdl` (two `:MACRO:` blocks → two macro-marked models). | Project opens; the `main` diagram renders (no crash / no blank canvas). `macro output` and `second macro output` appear as ordinary variables. | +| B3 | Add or select a Module element, open `ModuleDetails`, open the model-reference selector (`data-testid="model-ref-select"`). | Dropdown lists ordinary/stdlib models only. **`expression_macro` and `second_macro` MUST NOT appear** as selectable reference targets. | +| B4 | Open `/home/bpowers/src/simlin/test/test-models/tests/macro_multi_output/test_macro_multi_output.mdl`; inspect `main`. | Renders without crash. `total`, `the min`, `the max`, `spread` appear as ordinary auxes; the materialized module is present but `add3` is NOT in the model-reference dropdown. | +| B5 | Navigate the breadcrumb / model list (hamburger → model navigation). | Macro-marked models are NOT navigable standalone entries; only `main` (and any ordinary submodels) are reachable. | + +## Phase C — Cross-tool / cross-format fidelity (visual judgment) + +| Step | Action | Expected | +|------|--------|----------| +| C1 | Open `/home/bpowers/src/simlin/test/test-models/tests/macro_stock/test_macro_stock.mdl`, run the simulation, plot `macro output`. | Series matches stock semantics: 1.1, 6.1, 11.1, … 51.1 over the 11 steps (matches `macro_stock/output.tab`). | +| C2 | Open `/home/bpowers/src/simlin/test/test-models/tests/macro_expression/test_macro_expression.xmile`; export to XMILE via the UI download; export a second time; diff the two downloads. | Two consecutive exports are byte-identical; the `` element and `` header option are present; round-trip preserves the macro. | +| C3 | Export `macro_multi_output.mdl` to XMILE via the UI; inspect the XML. Then export `macro_expression.mdl` to XMILE. | Multi-output XML contains `` and a `simlin:macro-invocation` extension. Single-output XML contains **no** `simlin:` macro extension (AC4.5 in the live tool). | +| C4 | (Requires a licensed Vensim DSS — closes prerequisite GH #561 for these fixtures) Run the 3 `macro_clearn_*` focused fixtures in Vensim DSS, export `.vdf`, compare Simlin's `macro output` series against Vensim for `sample_until`, `sshape`, `ramp_from_to`. | Series match within tolerance. Upgrades AC6.3 from formula-derived to DSS-validated. | + +## Human-verification traceability + +| Criterion | Why manual | Steps | +|-----------|------------|-------| +| AC6.5 (real UI render) | Jest asserts `openEngineProject` with a faked engine; only a human confirms the diagram visually renders without corruption | B2, B4 | +| AC6.6 (rendered dropdown) | Jest asserts pure `getAvailableModels`; the actual `