Skip to content

[codex] Add runtime resource overrides#49

Merged
mvallebr merged 3 commits into
mainfrom
codex/execution-overrides-resources
Jun 20, 2026
Merged

[codex] Add runtime resource overrides#49
mvallebr merged 3 commits into
mainfrom
codex/execution-overrides-resources

Conversation

@mvallebr

Copy link
Copy Markdown
Contributor

Summary

This is the third runtime-DI slice for Issue 45.0.

It introduces declared runtime resources on top of the existing ExecutionOverrides work for materializers and observers.

What changed

  • added resources={...} to pipeline(...) / PipelineDef
  • compiled declared resources into the Dag contract as dag.resources
  • changed dependency resolution order so exact-name resources are resolved before upstream step outputs and params
  • added ResourceRegistry to ExecutionOverrides
  • required concrete runtime values for declared resources via ExecutionOverrides.resources
  • made sync and async executors fail loudly when a declared resource is missing at runtime
  • preserved executor blindness to sub-pipelines and namespace structure
  • added build-time validation for resource name collisions with params fields and step names
  • added sync/async tests for resource injection, missing-resource failures, registry key validation, and compiled DAG contract
  • updated README and build-vs-run docs with resource examples

Why

The design already split runtime override into three families:

  • materializers
  • observers
  • resources

Materializers and observers were already implemented. The main missing piece was runtime-only dependencies such as database handles, API clients, or service objects that should not live in params and should not require monkeypatching.

This PR adds that missing contract explicitly.

Contract boundary

This PR keeps the intended boundary explicit:

  • the DAG still compiles topology, dependency edges, and semantic decisions
  • resources are declared at build time as part of the contract
  • concrete resource instances are supplied only at runtime
  • executors still operate on a flat compiled DAG; they only seed runtime inputs from ExecutionOverrides.resources

Example

from synaflow import ExecutionOverrides, Observer, PIPELINE_SCOPE, Scope

sub = Scope("payments")
overrides = ExecutionOverrides.empty(p)

overrides.resources["db"] = FakeDatabase()
overrides.observers[PIPELINE_SCOPE] = [Observer(noop_metrics)]
overrides.observers[sub.scope("validate")] = [Observer(test_recorder)]
overrides.materializers[sub.scope("normalize")] = list

Impact

Pipelines can now declare runtime service dependencies explicitly, validate them at build time, and inject concrete fakes/mocks in tests without smuggling them through params or rebuilding alternative pipeline definitions.

This closes the three-registry shape described in the design doc, even though there is still follow-up work to polish docs and naming (resources vs services).

Validation

  • uv run pytest

@mvallebr mvallebr marked this pull request as ready for review June 20, 2026 07:20
@mvallebr mvallebr merged commit 228cb3f into main Jun 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant