ferrum-evals

Continual Learning Evaluation Harness for the Ferrum Platform.

Serves forge-agent, agent-runtime, and cl-experiments.

Two distinct jobs:

1. Operational CI gate — Score a single agent run for tool correctness, trajectory quality, and guardrail compliance. Fail a PR if scores fall below thresholds.
2. Continual learning research instrument — Measure whether forge-agent is getting better over time. Compute BWT (Backward Transfer), FWT (Forward Transfer), and Plasticity Index across a curriculum of tasks.

Architecture

ferrum-evals/
├── ferrum_evals/
│   ├── cli.py                          # Typer CLI: run, cl-report, compare, dataset
│   ├── config.py                       # Env var config
│   ├── main.py                         # FastAPI app + lifespan
│   ├── contracts/
│   │   └── eval_models.py              # Pydantic v2 models
│   ├── metrics/
│   │   ├── tool_correctness.py         # Jaccard similarity on tool call pairs
│   │   ├── trajectory.py               # Trajectory scorer
│   │   ├── guardrails.py               # PII/injection rule-based checker
│   │   ├── deepeval_wrappers.py        # AnswerRelevancy + Hallucination wrappers
│   │   └── cl_metrics.py               # BWT, FWT, Plasticity computation
│   ├── storage/
│   │   └── sqlite_store.py             # ScoreStore (aiosqlite + WAL)
│   ├── api/
│   │   ├── eval.py                     # /eval/* routes
│   │   ├── collector.py                # /v1/traces (OTLP span ingestion)
│   │   └── dataset.py                  # /dataset/* routes
│   └── reports/
│       ├── html_report.py              # Jinja2 HTML scorecard
│       ├── json_report.py              # JSON scorecard
│       └── junit_xml.py                # JUnit XML for CI
├── evals/fixtures/
│   ├── golden_dataset.json             # 5 initial eval test cases
│   └── task_curriculum.json            # 20-task CL curriculum
└── tests/
    ├── conftest.py
    ├── unit/
    └── integration/

Quick Start

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -e ".[dev]"

# Run tests
pip install pytest pytest-asyncio pytest-cov
pytest tests/unit/ --cov=ferrum_evals --cov-report=term-missing -v

Configuration

All configuration is via environment variables with sensible defaults:

Variable	Default	Description
FERRUM_EVALS_PORT	8002	Server port
SCORE_DB_PATH	~/.ferrum-evals/scores.db	SQLite database path
FERRUM_MEMORY_URL	http://localhost:8000	ferrum-memory service URL
JUDGE_LLM_URL	http://localhost:8080	LLM judge service URL
JUDGE_LLM_MODEL	claude-haiku-4-5-20251001	Judge model
TOOL_CORRECTNESS_THRESHOLD	0.80	Minimum tool correctness score
TRAJECTORY_THRESHOLD	0.70	Minimum trajectory score
ANSWER_RELEVANCY_THRESHOLD	0.80	Minimum answer relevancy score
HALLUCINATION_THRESHOLD	0.20	Maximum hallucination score
OTEL_EXPORTER_OTLP_ENDPOINT	http://localhost:4317	OTLP collector endpoint

API Endpoints

Operational Eval

Method	Endpoint	Description
POST	/eval/run	Trigger evaluation for a session
GET	/eval/status/{run_id}	Check eval run status
POST	/eval/cl-report	Compute CL report (BWT, FWT, Plasticity)

OTLP Span Ingestion

Method	Endpoint	Description
POST	/v1/traces	Receive OpenTelemetry spans

Dataset

Method	Endpoint	Description
POST	/dataset/upload	Upload golden dataset entries
GET	/dataset/entries	Query dataset entries

System

Method	Endpoint	Description
GET	/health	Health status
GET	/openapi.json	OpenAPI schema

Metrics

Operational Metrics (per-session)

Metric	Type	Threshold	Formula
ToolCorrectnessMetric	Jaccard similarity	≥ 0.80	Jaccard on (tool_name, params) pairs
TrajectoryScore	Custom	≥ 0.70	(necessary/total) × (ordered/necessary)
GuardrailCheck	Rule-based	pass	PII/injection pattern matching
AnswerRelevancyMetric	deepeval	≥ 0.80	LLM-as-judge
HallucinationMetric	deepeval	≤ 0.20	LLM-as-judge

Continual Learning Metrics (over episodes)

Metric	Formula	Interpretation
BWT	(1/N) Σ[R(i,N) - R(i,i)]	Negative = forgetting. Zero = stable.
FWT	(1/N) Σ[R(i,i) - R_zero_shot(i)]	Positive = prior helps new tasks.
Plasticity	R_current / R_baseline	< 0.5 = catastrophic forgetting.

CLI

# Run eval on a specific session
ferrum-evals run --session <session_id>

# Run eval on last N sessions
ferrum-evals run --last 5

# Generate CL report
ferrum-evals cl-report --since 2026-04-01 --output cl_report.html

# Compare two branches
ferrum-evals compare --baseline main --current feature/new-tool-router

# Upload golden dataset
ferrum-evals dataset upload --file golden_dataset.json

Test Coverage

pytest tests/unit/ --cov=ferrum_evals --cov-report=term-missing --cov-fail-under=95 -v

Unit Tests

• test_cl_metrics.py — BWT/FWT/Plasticity with synthetic data (known mathematical outputs)
• test_tool_correctness.py — Jaccard edge cases (empty, partial, parameter mismatch)
• test_trajectory.py — Trajectory scoring edge cases
• test_guardrails.py — PII/injection pattern detection
• test_sqlite_store.py — ScoreStore CRUD + WAL mode

Integration Tests

• test_full_eval_run.py — End-to-end: store → compute → query → verify

Design Principles

1. CL metrics are research-grade — mathematically exact, independently testable with synthetic data
2. Operational and CL metrics are separate — no code conflation between CI gate and research
3. SQLite only — aiosqlite + WAL mode, no external dependencies for storage
4. No ORM — raw SQL throughout; Pydantic models for API serialization only
5. Mirrors ferrum-memory patterns — same coding conventions, lifecycle, and API patterns

License

MIT