feat(engine): config groups — one definition fans out routing + strategy to N models#917
Merged
Conversation
…egy to N models
Adds typed config groups: a `models/groups/<name>.toml` definition that a
fan-out of models opts into with `group = "<name>"`. A group supplies a shared
`schema_template` (filled per model from its `[args]`) and `strategy`, so a
one-line change flows to every model in the group.
Resolution precedence is per-model sidecar > group > `_defaults.toml`: a model
can still override anything the group sets, and the group overrides directory
defaults. Group loading rides a side-loader (`load_groups_from_dir`); resolution
threads through new `*_with_groups` variants of `load_model_pair` /
`parse_model_inline` — the bare functions delegate with no groups, so every
existing caller is untouched (zero `ModelConfig` churn; the group fields live on
`RawModelConfig` only). `schema_template` reuses the schema-pattern
`resolve_template`, so there's one template grammar across the codebase.
Wired through BOTH model loaders the compiler uses: `.sql` models via
`load_models_from_dir`, and `.rocky` DSL models via the salsa project loader
(`load_single_rocky_model_with_db`) — verified end-to-end through `rocky
emit-sql` (`compile::compile`) for each, with a `.rocky` regression test so the
DSL path can't silently drop groups.
Guardrails (load-time, GOLD-neutral): an unknown `group` reference fails with
`UnknownGroup`; a `schema_template` placeholder the model doesn't supply fails
with `InvalidGroup`, rather than routing to a literal `mart_{region}` schema.
Unknown keys in a group file are rejected (`deny_unknown_fields`).
This is the declarative-config-groups shape of the governed-materialization
fan-out. Shared tags and per-mode computed keys are follow-ups; the group
reference persists in each sidecar so a later enforcement pass can detect models
that override their group. Documented in reference/model-format.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The declarative shape of the governed-materialization fan-out (the C4 keystone). A
models/groups/<name>.tomldefinition is opted into by N models viagroup = "<name>", supplying shared routing (schema_template, filled per model from its[args]) and strategy — so a one-line change flows to every model in the group.Design
_defaults.toml. A model can still pin its own schema/strategy; the group overrides directory defaults.ModelConfigchurn: group fields live onRawModelConfig; loading rides a side-loader (load_groups_from_dir); resolution threads through new*_with_groupsvariants ofload_model_pair/parse_model_inline. The bare functions delegate withNone, so all existing callers are untouched.schema_templatereuses the schema-patternresolve_template({name}/{name:SEP})..sqlviaload_models_from_dir, and.rockyDSL via the salsa project loader — verified end-to-end throughrocky emit-sql(compile::compile) for each, with a.rockyregression test so the DSL path can't silently drop groups.Guardrails (load-time, deliberately small/neutral)
groupreference →ModelError::UnknownGroup.schema_templateplaceholder the model doesn't supply →ModelError::InvalidGroup(no routing to a literalmart_{region}).deny_unknown_fields).Tests
rocky-core: group supplies strategy + routes schema; sidecar overrides group; group overrides_defaults.toml; unknown group errors; missing template arg errors.rocky-compiler:.rockymodel resolves a group through the salsa loader.Scope / follow-ups
v1 carries
schema_template+strategy. Shared tags (→ Dagster asset metadata) and per-mode computed keys, plus compile-time governance invariants (enforcement: detect a grouped model that overrides its routing), are follow-ups. The group reference persists in each sidecar so enforcement has a hook.cargo fmt + clippy
--all-targetsclean; no codegen/schema drift.