Skip to content

Add paths.buffer and PathConfig; rename import→imports with list support#87

Merged
cweniger merged 5 commits into
mainfrom
feature/paths-buffer
Jun 6, 2026
Merged

Add paths.buffer and PathConfig; rename import→imports with list support#87
cweniger merged 5 commits into
mainfrom
feature/paths-buffer

Conversation

@cweniger

@cweniger cweniger commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds paths.buffer config key to redirect buffer snapshot storage to a separate volume (e.g. HPC scratch), mirroring the existing paths.samples pattern
  • Introduces PathConfig dataclass to validate cfg.paths against a typed schema, consistent with how BufferConfig validates cfg.buffer
  • Renames paths.importpaths.imports: list[str] — avoids the Python keyword clash and supports multiple search directories prepended to sys.path in Ray workers
  • Renames model_pathimport_dirs throughout DeployedGraph, NodeWrapper, MultiplexNodeWrapper

Motivation

Buffer snapshots can be large (grows with max_samples × field_size × adaptive_rounds) and should land on fast scratch storage, not alongside model checkpoints and configs on persistent storage. paths.buffer gives users independent control over where snapshots land.

PathConfig brings cfg.paths in line with the rest of the config: typed, validated, with unknown-key rejection via OmegaConf structured config merge.

imports as a list allows specifying multiple module search directories, which was not possible with the old scalar import key.

Migration

Before After
paths.import: "." paths.imports: ["."]

Test plan

  • Run falcon launch with no paths.buffer set — snapshots land in {run_dir}/buffer/snapshots/ as before
  • Run with paths.buffer: /scratch/myproject — snapshots redirect correctly
  • Run with paths.imports: [".", "../shared"] — both dirs appear in sys.path in workers
  • Config with unknown key under paths: raises OmegaConf error

🤖 Generated with Claude Code

oleg-savchenko and others added 4 commits May 22, 2026 16:45
Mirrors the existing paths.samples pattern: if paths.buffer is set in the
config (or passed as a Hydra override), buffer snapshots are written to
{paths.buffer}/snapshots/ instead of the default {run_dir}/buffer/snapshots/.

Useful when run_dir is on a fast local filesystem but buffer snapshots
(which can be large) should land on a separate scratch volume.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Keep snapshot_every (renamed from store_fraction in main branch).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ort_dirs

- PathConfig dataclass validates cfg.paths against a typed schema (analogous to BufferConfig for cfg.buffer)
- imports: Optional[List[str]] replaces the bare import: str key; avoids Python keyword clash and supports multiple search dirs
- DeployedGraph/NodeWrapper/MultiplexNodeWrapper accept import_dirs (list) instead of model_path (str) and iterate over all entries when extending sys.path
- _resolve_paths() helper merges cfg.paths against PathConfig and returns a plain dict; used in launch_mode and sample_mode
- PathConfig exported from falcon top-level alongside BufferConfig

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@codecov

codecov Bot commented Jun 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 20.58824% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 9.72%. Comparing base (a20f1c7) to head (1785b5d).

Files with missing lines Patch % Lines
falcon/cli.py 0.00% 18 Missing ⚠️
falcon/core/deployed_graph.py 0.00% 9 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##            main     #87      +/-   ##
========================================
+ Coverage   9.61%   9.72%   +0.11%     
========================================
  Files         33      33              
  Lines       4141    4154      +13     
========================================
+ Hits         398     404       +6     
- Misses      3743    3750       +7     
Flag Coverage Δ
unit 9.72% <20.58%> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@cweniger cweniger merged commit 7974ab3 into main Jun 6, 2026
6 checks passed
@cweniger cweniger deleted the feature/paths-buffer branch June 8, 2026 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants