Refactor observability into a first-class module and replace fragmented persistence paths

[fdidonato]

Summary

Introduce a new moralstack/observability/ module as the single entry point for runtime observability, audit logging, and trace persistence.

This change replaces the current fragmented flow spread across:

• moralstack/persistence/sink.py
• moralstack/persistence/db.py
• moralstack/persistence/context.py
• moralstack/orchestration/persistence_helpers.py
• controller.py persistence fallbacks

The goal is to create a clean, library-safe, frontend-agnostic observability architecture that supports both current admin tooling and future SDK / multi-turn evolution without coupling tracing to the existing persistence implementation.

Why

MoralStack currently works well as a research prototype, but observability is still structurally fragile:

• write paths are distributed across multiple modules with overlapping responsibilities
• persistence/db.py has become a mixed read/write God module
• context tracking (run_id, request_id, cycle) is coupled to persistence concerns
• controller.py and helper layers rely on fragile delegation chains and defensive try/except patterns
• MORALSTACK_PERSIST_MODE mixes transport concerns with storage concerns
• JSONL output is not modeled as a structured audit trail
• current architecture is not a solid base for standalone library usage, future multi-turn/session support, multiple frontends, or consistent export/reporting contracts

This refactor establishes a dedicated observability layer with explicit event envelopes, sink routing, and a single read contract.

Current problems in the codebase

Fragmented write path

Today the flow is effectively:

controller.py / orchestration -> persistence_helpers.py -> persistence/sink.py -> persistence/db.py

This introduces too many delegation layers, scattered exception handling, and unclear ownership.

persistence/db.py is overloaded

persistence/db.py currently mixes:

• SQLite write logic
• read/query logic
• unit-of-work style concerns
• event persistence helpers
• dashboard data retrieval

This makes evolution risky and prevents clear module boundaries.

Context is in the wrong layer

persistence/context.py contains runtime context vars such as:

• get_current_run_id
• get_current_request_id
• get_current_cycle

These belong to observability/runtime context, not to a storage backend.

Storage mode semantics are weak

MORALSTACK_PERSIST_MODE currently controls:

• db_only
• dual
• file_only

But this is not expressed as a true observability routing abstraction, and file output is not defined as a normalized audit event stream.

Proposed architecture

Create a new module:

moralstack/observability/
├── __init__.py
├── events.py
├── context.py
├── config.py
├── service.py
├── router.py
├── read_store.py
└── sinks/
    ├── base.py
    ├── sqlite_sink.py
    └── jsonl_sink.py

Responsibilities

__init__.py

Expose a singleton-style entry point, for example:

• obs.emit(...)
• obs.flush()
• obs.shutdown()

events.py

Define:

• EventEnvelope
• typed event payloads / event kinds
• normalization helpers for consistent audit serialization

context.py

Move runtime context vars here from persistence/context.py, including:

• current run id
• current request id
• current cycle
• future-safe session/turn metadata

config.py

Introduce ObservabilityConfig sourced from env vars and defaults.

This should replace the old MORALSTACK_PERSIST_MODE with a clearer setting such as:

• MORALSTACK_OBSERVABILITY_MODE

service.py

Implement ObservabilityService as the main orchestration layer:

• validate/build event envelopes
• enrich with runtime context
• dispatch to router
• expose lifecycle methods (emit, flush, shutdown)

router.py

Implement EventRouter to dispatch each event to one or more active sinks.

read_store.py

Define the single backend/frontend read contract.

This must become the only supported read interface for:

• admin dashboard
• future user-facing UI
• exports/reports
• benchmark inspection tools

sinks/base.py

Define the sink protocol / interface.

sinks/sqlite_sink.py

Move SQLite write responsibilities out of persistence/db.py.

sinks/jsonl_sink.py

Implement structured JSONL output as a first-class audit sink, ideally with routing by event_type.

Event envelope requirements

The new EventEnvelope should include current and future-safe metadata.

Required baseline fields

At minimum, each event should carry stable identifiers and audit metadata such as:

• event_id
• event_type
• timestamp
• run_id
• request_id
• cycle
• payload

Future-safe fields for multi-turn support

Add these now, even if initially optional:

• session_id: Optional[str]

• turn_number: Optional[int]

• parent_event_id: Optional[str]

• audit_level: str = "turn" where allowed values are:

  • "turn"
  • "session"
  • "export"

These fields make future session grouping possible without another storage schema redesign.

Architectural constraints

1. No FastAPI dependency

The observability layer must work in:

• standalone Python scripts
• tests
• CLI/benchmark runs
• future SDK/library usage

It must not depend on the web app layer.

2. ReadStore is the only read contract

No frontend or admin UI should query low-level persistence helpers directly.

All read access should be mediated by ReadStore.

3. Backward-compatible transition

Legacy modules may remain temporarily, but only as thin deprecated shims.

4. Structured event parity

In dual mode, the same logical event must be written identically to both DB and JSONL outputs.

Migration plan

Step 1 — Introduce the new observability module

Create the new package and establish the service/router/sink abstractions.

Step 2 — Migrate runtime context

Move context vars from:

• persistence/context.py

to:

• observability/context.py

Leave the old module as a deprecated compatibility alias if needed.

Step 3 — Move write logic out of persistence/db.py

Split responsibilities:

• SQLite writes -> observability/sinks/sqlite_sink.py
• read/query logic -> observability/read_store.py

Step 4 — Convert persistence/sink.py into a shim

Keep the public function names temporarily if needed, but make them thin wrappers over:

• obs.emit(...)

Examples:

• persist_llm_call()
• persist_decision_trace()
• persist_debug_event()
• persist_orchestration_event()

Step 5 — Update orchestration helpers

Refactor moralstack/orchestration/persistence_helpers.py so methods like:

• record_llm_call
• record_decision_trace

delegate to obs rather than low-level persistence functions.

Step 6 — Simplify controller integration

Update controller.py to always use the observability service directly and remove defensive persistence import fallbacks such as try/except ImportError branches.

Step 7 — Route the admin UI through ReadStore

Preserve current admin UI behavior, but make it read only from the new read contract.

Files to modify / deprecate

New files

moralstack/observability/__init__.py
moralstack/observability/events.py
moralstack/observability/context.py
moralstack/observability/config.py
moralstack/observability/service.py
moralstack/observability/router.py
moralstack/observability/read_store.py
moralstack/observability/sinks/base.py
moralstack/observability/sinks/sqlite_sink.py
moralstack/observability/sinks/jsonl_sink.py

Existing files to refactor

• moralstack/persistence/sink.py
• moralstack/persistence/db.py
• moralstack/persistence/context.py
• moralstack/orchestration/persistence_helpers.py
• controller.py

Existing files likely affected indirectly

Any module currently emitting or reading:

• LLM call logs
• decision traces
• debug events
• orchestration events
• benchmark persistence data

Acceptance criteria

This issue is complete only if all of the following are true:

Functional criteria

• All existing event types flow through ObservabilityService.emit()
• Current event coverage is preserved for all 10 event types
• file_only mode works without SQLite being available
• dual mode writes semantically identical events to both DB and JSONL
• Admin UI behavior remains unchanged from the user perspective
• Benchmark/reporting flows continue to work

Architectural criteria

• No write path bypasses the observability service
• ReadStore becomes the only supported read contract
• persistence/context.py is migrated or downgraded to a deprecated alias
• persistence/sink.py contains no real business logic anymore
• write logic is removed from persistence/db.py

Regression criteria

• Full benchmark remains functionally invariant
• 84/84 benchmark set remains stable
• overall compliance remains unchanged at current target behavior (~98.8%)
• no change in governance outcomes caused by the observability refactor

Non-goals

This issue should not attempt to solve the following in the same PR unless strictly necessary for migration:

• full multi-turn conversation support
• user-facing frontend redesign
• SDK packaging work
• CI/docs/examples overhaul
• schema redesign for every historical admin view

Those are enabled by this refactor, but they are not the direct scope.

Implementation notes

• Prefer a typed event model over ad-hoc dict assembly
• Keep event normalization deterministic across sinks
• JSONL should be structured for audit export, not treated as a debug dump
• Avoid new direct dependencies between orchestration and SQLite internals
• Preserve compatibility where useful, but make deprecation explicit

Expected outcome

After this change, MoralStack should have:

• one clear observability entry point
• one clear read contract
• backend/frontend decoupling for trace access
• structured audit-ready JSONL output
• a storage-agnostic runtime tracing model
• a future-safe foundation for sessions, turns, SDK usage, and alternative frontends