distributed multi-process and multi-node orchestration

[yai-dev]

Summary

Support distributed multi-process and multi-node orchestration in the current orchestration subsystem.

Current State

OrchestrationManager currently coordinates work within the current process model. Persistence primitives are more mature now, but there is still no remote worker queue / claim / retry lifecycle for distributed execution.

Proposed Design

1. Define a durable orchestration job model
2. Introduce worker processes that can claim and execute jobs from shared persistence/queue state
3. Support retries, dead-letter handling, priority, and worker routing
4. Preserve the existing high-level spawn/wait model where possible

Affected Areas

• packages/capabilities/src/orchestration/
• shared persistence contracts backing orchestration state
• worker process/runtime integration

Acceptance Criteria

• Distributed job lifecycle states are defined
• Remote workers can claim jobs without double execution
• Retry/dead-letter behavior is implemented
• Worker routing and fire-and-forget modes are supported
• Integration tests cover orchestrator + multiple workers against shared state