Notebook/Colab API + clean estimator __init__ params#90
Merged
Conversation
Captures the design for a Python-first front door to falcon: flat typed config surface bridged to nested YAML, the product/sum/composite/collection config-shape taxonomy, _target_ resolution unification (Step 0), the init/launch/shutdown Ray lifecycle, the cloudpickle escape hatch for notebook-defined models, JAX process-global-state handling, and the v1 interleaved color-tagged log stream for in-cell output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Step 0 (_target_ unification, net_type → variant classes, NetworkConfig
untangle) deferred: no variant-specific hyperparams exposed yet, churn with
no functional benefit; net_config: dict={} is the escape hatch if needed
- prior list-syntax: existing list-of-lists form serves as Python API too,
no typed-marginal objects needed for v1
- falcon.Simulator base class deferred: duck typing is sufficient for v1
- falcon.session() deferred: not needed before basic API works
- falcon.init(): remove num_cpus/num_gpus, use **ray_init_kwargs passthrough
- falcon.launch(): remove buffer_min_samples etc., model config belongs in
Config/overrides; rename posterior_sample -> auto_sample
- falcon.Sequential: dropped, use _input_ nesting instead
- escape hatch: drop source-extraction to _live_objects.py, placeholder
"<live object: ClassName>" is sufficient for v1
- example notebooks: .py (jupytext) as source of truth, .ipynb as build
artefacts; existing run.py files untouched
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Splits launch_mode into: - _run_pipeline(cfg, *, auto_sample, timeout, stop_check, log_handler, on_graph_ready, summary_sink): pure training pipeline, no Ray lifecycle, no TUI concerns; injectable stop_check and log_handler; returns output_dir - launch_mode: thin CLI frontend; owns Ray init, TUI/shutdown-handler setup, stop_check closure, TUI log handler, status polling thread (via on_graph_ready) CLI behavior is byte-for-byte unchanged. _run_pipeline is now directly callable for the upcoming falcon.launch() Python API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests plain callables, torch simulators, nn.Module subclasses, transitive __main__ deps, global closures (including ~8 MB), and class redefinition. All pass. CUDA tensors stored as instance attributes fail as expected; workaround (store numpy, convert inside forward) confirmed working. Conclusion: cloudpickle + Ray handles all normal notebook simulator patterns. Notebook-defined classes can be passed directly to add_node(); Ray's built-in cloudpickle serializes them transparently to actor processes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- falcon/core/flat_config.py: flat_to_nested, make_flat_signature, apply_flat_signature utilities for prefix-transform config builders - falcon/api.py: Config wrapper + falcon.config() entry point - falcon/estimators/flow.py: _FlowConfigBuilder with synthesised flat signature; Flow.__new__ returns builder when called without positional args, real estimator otherwise - falcon/estimators/gaussian_fullcov.py: rename GaussianPosterior → _GaussianPosterior (implementation detail, not public API) - falcon/estimators/gaussian.py: add deprecation TODO; update import - falcon/estimators/__init__.py: remove GaussianPosterior from exports - falcon/__init__.py: expose falcon.config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- falcon/api.py: add init(), shutdown(), _prepare_config(), launch() launch() resolves Config/path/dict target, lazily inits Ray, calls _run_pipeline, returns load_run(output_dir); wait=False raises NotImplementedError (deferred to Step 7) - falcon/__init__.py: expose init, launch, shutdown via lazy imports - examples/01_minimal/notebook.py: jupytext percent-format notebook covering config load, override, launch, and run inspection (cell story A) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- falcon/core/graph.py: Graph() starts empty (node_list=None default); extract _build(); add add_node() accepting live instances, observed=array, and ray_* kwargs; guard forward_deps.get() for partial graphs - falcon/core/deployed_graph.py: NodeWrapper skips LazyLoader for live simulator instances (isinstance str/type check) - falcon/cli.py: _run_pipeline gains graph=/observations= params; when provided, create_graph_from_config is bypassed - falcon/api.py: _prepare_config handles Graph target by synthesising a default config with _graph_to_config_dict escape-hatch serialization (<live object: ...> / <live array: ...> placeholders); launch() threads prebuilt_graph through to _run_pipeline - falcon/estimators/flow.py: guard OmegaConf.to_container on None embedding - falcon/embeddings/builder.py: instantiate_embedding(None) returns _PassthroughEmbedding (identity, casts to float32) - examples/04_gaussian/notebook.py: jupytext notebook for programmatic API Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Graph._repr_html_: Mermaid flowchart (CDN-loaded) with colour-coded nodes (blue=trainable, green=observed, yellow=deterministic), solid forward edges, dashed evidence edges - _short_cls_name(): module-level helper for compact class display names - Run._repr_html_: inline HTML status card showing per-node final loss and epoch count - Run.plot_metrics(): matplotlib figure of train/val loss curves, one subplot per node Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Generated from jupytext percent-format notebook.py sources. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_launch was async def but contained zero await calls — all blocking was done via ray.get() / ray.wait(). Wrapping it in asyncio.run() broke Jupyter notebooks (which already have a running event loop). Fix: make _launch a plain def and call it directly; remove unused asyncio import. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The name "DatasetManager" caused a crash on the second falcon.launch() call in the same notebook session — Ray rejects duplicate actor names. The name was never used for lookup, so dropping it is safe. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OmegaConf.resolve() was called in-place on the Config's DictConfig,
baking the first run's run_dir into all interpolated paths. A second
launch() call with a different output dir then inherited the resolved
paths from the first run (e.g. paths.graph pointed at run6 even when
output was run7).
Fix: copy via OmegaConf.merge(cfg, {}) before mutating.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Same guard as flow.py: OmegaConf.to_container(None) raises ValueError, so skip it when no embedding config is provided. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When target is a Graph, always set prebuilt_graph regardless of whether a saved config.yml exists. The saved config contains <live object/array> placeholders that create_graph_from_config cannot parse as file paths. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
__init__ is now a pure dataclass (stores flat config kwargs only). setup() is load-bearing: receives runtime objects from NodeWrapper, merges stored config with YAML-sourced config, and wires everything up. BaseEstimator: - __init__(**flat_kwargs): stores self._init_flat_kwargs via flat_to_nested - __init_subclass__: injects per-class __init__ with flat signature when _CONFIG_SECTIONS is declared, giving autocomplete for free - setup() declared as abstract StepwiseEstimator: - __init__ removed (uses BaseEstimator's) - setup() initialises common loop state; subclasses set loop_config first Flow / GaussianFullCov: - _CONFIG_SECTIONS + _CONFIG_EXTRA_PARAMS replace _FlowConfigBuilder - __new__ trick and _FlowConfigBuilder removed entirely - setup() merges defaults < notebook kwargs < YAML/override config NodeWrapper: - Detects BaseEstimator instances (notebook path) vs class/string (YAML) - Always calls .setup() to wire up runtime components Result: Flow(loop_max_epochs=200) returns a real Flow. isinstance() works. New estimators need only declare _CONFIG_SECTIONS and implement setup(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Flow and GaussianFullCov __init__ params are now plain names (max_epochs, net_type, lr, gamma, lr_patience, use_best_models, ...) instead of loop_/network_/optimizer_/inference_ prefixes. deployed_graph.py passes flat YAML dict directly as **kwargs to estimator_cls.__init__; setup() no longer takes a config arg. All example YAML files updated to flat format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
….yml Notebooks are the source of truth — .py mirror files removed. 04_gaussian notebook updated: GaussianFullCov now instantiated with explicit params matching config.yml (max_epochs, lr, gamma, etc.). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #90 +/- ##
========================================
- Coverage 9.72% 9.20% -0.53%
========================================
Files 33 34 +1
Lines 4154 4411 +257
========================================
+ Hits 404 406 +2
- Misses 3750 4005 +255
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Update epoch override paths from loop.max_epochs to max_epochs to match the flattened estimator YAML structure introduced in the previous commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extract _build_optimizer() in GaussianFullCov and LossBasedEstimator to eliminate duplicated optimizer/scheduler setup between _initialize_model and load - Delete plans/COLAB_API_PLAN.md and plans/spikes/cloudpickle_spike.py - Drop "TODO: refactor" prefix from proposal bias correction comment - Remove stale TODO comment in deployed_graph._launch Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Breaking changes
Test plan
🤖 Generated with Claude Code