Skip to content

feat(engine): config groups — one definition fans out routing + strategy to N models#917

Merged
hugocorreia90 merged 1 commit into
mainfrom
feat/config-groups
Jun 17, 2026
Merged

feat(engine): config groups — one definition fans out routing + strategy to N models#917
hugocorreia90 merged 1 commit into
mainfrom
feat/config-groups

Conversation

@hugocorreia90

Copy link
Copy Markdown
Contributor

What

The declarative shape of the governed-materialization fan-out (the C4 keystone). A models/groups/<name>.toml definition is opted into by N models via group = "<name>", supplying shared routing (schema_template, filled per model from its [args]) and strategy — so a one-line change flows to every model in the group.

# models/groups/daily_marts.toml
schema_template = "mart_{region}"
[strategy]
type = "merge"
unique_key = ["id"]
update_columns = ["amount", "status"]

# models/fct_orders.toml
group = "daily_marts"
[target]
catalog = "warehouse"     # schema comes from the group template
[args]
region = "emea"           # {region} -> schema "mart_emea"

Design

  • Precedence: per-model sidecar > group > _defaults.toml. A model can still pin its own schema/strategy; the group overrides directory defaults.
  • Zero ModelConfig churn: group fields live on RawModelConfig; loading rides a side-loader (load_groups_from_dir); resolution threads through new *_with_groups variants of load_model_pair/parse_model_inline. The bare functions delegate with None, so all existing callers are untouched.
  • One template grammar: schema_template reuses the schema-pattern resolve_template ({name} / {name:SEP}).
  • Wired through both compiler loaders: .sql via load_models_from_dir, and .rocky DSL via the salsa project loader — verified end-to-end through rocky emit-sql (compile::compile) for each, with a .rocky regression test so the DSL path can't silently drop groups.

Guardrails (load-time, deliberately small/neutral)

  • Unknown group reference → ModelError::UnknownGroup.
  • schema_template placeholder the model doesn't supply → ModelError::InvalidGroup (no routing to a literal mart_{region}).
  • Unknown key in a group file → rejected (deny_unknown_fields).

Tests

rocky-core: group supplies strategy + routes schema; sidecar overrides group; group overrides _defaults.toml; unknown group errors; missing template arg errors. rocky-compiler: .rocky model resolves a group through the salsa loader.

Scope / follow-ups

v1 carries schema_template + strategy. Shared tags (→ Dagster asset metadata) and per-mode computed keys, plus compile-time governance invariants (enforcement: detect a grouped model that overrides its routing), are follow-ups. The group reference persists in each sidecar so enforcement has a hook.

cargo fmt + clippy --all-targets clean; no codegen/schema drift.

…egy to N models

Adds typed config groups: a `models/groups/<name>.toml` definition that a
fan-out of models opts into with `group = "<name>"`. A group supplies a shared
`schema_template` (filled per model from its `[args]`) and `strategy`, so a
one-line change flows to every model in the group.

Resolution precedence is per-model sidecar > group > `_defaults.toml`: a model
can still override anything the group sets, and the group overrides directory
defaults. Group loading rides a side-loader (`load_groups_from_dir`); resolution
threads through new `*_with_groups` variants of `load_model_pair` /
`parse_model_inline` — the bare functions delegate with no groups, so every
existing caller is untouched (zero `ModelConfig` churn; the group fields live on
`RawModelConfig` only). `schema_template` reuses the schema-pattern
`resolve_template`, so there's one template grammar across the codebase.

Wired through BOTH model loaders the compiler uses: `.sql` models via
`load_models_from_dir`, and `.rocky` DSL models via the salsa project loader
(`load_single_rocky_model_with_db`) — verified end-to-end through `rocky
emit-sql` (`compile::compile`) for each, with a `.rocky` regression test so the
DSL path can't silently drop groups.

Guardrails (load-time, GOLD-neutral): an unknown `group` reference fails with
`UnknownGroup`; a `schema_template` placeholder the model doesn't supply fails
with `InvalidGroup`, rather than routing to a literal `mart_{region}` schema.
Unknown keys in a group file are rejected (`deny_unknown_fields`).

This is the declarative-config-groups shape of the governed-materialization
fan-out. Shared tags and per-mode computed keys are follow-ups; the group
reference persists in each sidecar so a later enforcement pass can detect models
that override their group. Documented in reference/model-format.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant