Commit b14d4fd
committed
Add state controller with continuous drift detection and self-healing
Implements a Kubernetes-style reconciliation controller that:
- Continuously polls Docker + DNS + Keycloak state via handler healthchecks
- Detects drift when actual state diverges from desired spec
- Automatically re-applies diverged components (self-healing)
- Emits events for monitoring/alerting integration
- Persists drift history to RuntimeState for auditability
Key changes:
1. netengine/core/drift_controller.py: Main DriftDetectionController that polls
healthchecks and triggers self-healing on drift detection
2. netengine/core/self_healing.py: SelfHealingStrategy for phase re-execution
3. netengine/core/state.py: Added drift_history, last_drift_check_at,
current_drift_phases fields to RuntimeState
4. netengine/core/orchestrator.py: Added start_drift_detection() method
5. netengine/cli/main.py: Added drift-watch and drift-status CLI commands
6. tests/test_drift_controller.py: Unit tests for drift detection logic
7. tests/integration/test_drift_detection.py: Integration tests
The controller runs as a background consumer via ConsumerSupervisor with:
- Configurable poll interval (default 30s)
- Optional auto-healing (enabled by default, --no-auto-heal to disable)
- Dependency-aware re-execution when upstream phases heal
- Graceful restart on crashes with exponential backoff
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DyX7e2PSJ4zELLGHkQv6F11 parent aa49f40 commit b14d4fd
7 files changed
Lines changed: 1108 additions & 1 deletion
File tree
- netengine
- cli
- core
- tests
- integration
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
410 | 410 | | |
411 | 411 | | |
412 | 412 | | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
413 | 498 | | |
414 | 499 | | |
415 | 500 | | |
| |||
0 commit comments