added a 100 example test for evaluation

Author: drago-codes-21

.github/workflows/ci.yml

@@ -0,0 +1,23 @@
+name: CI
+
+on:
+  pull_request:
+  push:
+    branches: [ main ]
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install -r dev-requirements.txt
+      - name: Run tests
+        run: pytest -q
+

README.md

@@ -102,6 +102,92 @@ Response example:
 }
 ```
 
+## MLOps Additions (Open Source + Free)
+
+- Reproducible training: `src/train.py` now accepts `--seed` and fixes RNGs. After training it writes `metrics.json` and `run_info.json` into the output model folder with args, data hash, and git commit (if available) for traceability.
+- Health/CORS/metrics: FastAPI exposes `/healthz` and `/readyz`. CORS is enabled for React by default. Optional Prometheus metrics (set `ENABLE_METRICS=1`) if `prometheus-fastapi-instrumentator` is installed.
+- Tests: Added unit and API tests under `tests/`. Dev deps in `dev-requirements.txt`. CI via GitHub Actions runs tests on pushes/PRs in public repos for free.
+
+### Configure CORS for React
+
+- By default all origins are allowed. To restrict:
+
+```bash
+set ALLOW_ORIGINS=http://localhost:5173,http://localhost:3000  # Windows PowerShell: $env:ALLOW_ORIGINS="http://..."
+uvicorn app.main:app --reload --port 8000
+```
+
+### Enable Prometheus metrics (optional)
+
+```bash
+export ENABLE_METRICS=1   # Windows PowerShell: $env:ENABLE_METRICS=1
+uvicorn app.main:app --reload --port 8000
+```
+
+### Run Tests Locally
+
+```bash
+pip install -r requirements.txt -r dev-requirements.txt
+pytest -q
+```
+
+### Demo-Ready Metrics Report
+
+Share visuals and plain-English talking points with stakeholders in one command:
+
+```bash
+pip install -r requirements.txt -r dev-requirements.txt
+python reports/generate_report.py \
+  --model_dir models/distilbert_component_classifier \
+  --train_path data/train.csv \
+  --report_dir reports/latest
+```
+
+You will get:
+
+- `reports/latest/report_summary.md`: non-technical explanation of precision, recall, F1, exact-match accuracy, and loss trends.
+- `reports/latest/report_data.json`: structured metrics and per-label stats ready for slide tables or dashboards.
+- `reports/latest/figures/*.png`: validation F1/loss curves plus a top-component coverage bar chart for quick storytelling.
+
+Run `python reports/generate_report.py -h` to tweak thresholds, validation split, or the destination folder.
+
+### Manual Scenario Audit (Edge Cases)
+
+When you need to show strengths *and* improvement areas, run the curated scenario harness:
+
+```bash
+python reports/run_manual_eval.py \
+  --cases_path reports/manual_eval_cases_100.jsonl \
+  --model_dir models/distilbert_component_classifier \
+  --output_dir reports/manual_eval \
+  --top_k_fallback 3
+```
+
+Artifacts:
+
+- `reports/manual_eval/manual_eval_results.{json,csv}`: per-scenario expectations vs. predictions, true/false positives, misses.
+- `reports/manual_eval/manual_eval_summary.md`: bullet-point narrative calling out gaps for non-technical leads.
+- `reports/manual_eval/figures/*.png`: outcome distribution, per-case recall, and top missed components for slide-ready visuals.
+
+Use `--top_k_fallback` (default 0) to add the best-scoring labels even when the sigmoid score is below the threshold—handy for exploratory edge-case analysis. Edit `reports/manual_eval_cases_100.jsonl` directly or regenerate it with:
+
+```bash
+python reports/build_manual_cases.py \
+  --limit 100 \
+  --output_path reports/manual_eval_cases_100.jsonl
+```
+
+The generator spans authentication, lending, collections, KYC, payments, reporting, disputes, core integration, omni-channel comms, and regulatory scenarios so each component in the taxonomy appears multiple times.
+
+### Lightweight Run Tracking Artifacts
+
+- After training, check your output dir (e.g., `models/distilbert_component_classifier/`) for:
+  - `metrics.json`: evaluation metrics from the Trainer
+  - `run_info.json`: training args, data SHA256, git commit
+  - `label2id.json`: label mapping used at inference
+
+These files are simple, portable, and versionable in git or any storage.
+
 ## Optional: Docker
 
 ```dockerfile
@@ -125,3 +211,22 @@ docker run -p 8000:8000 component-identifier
 
 - Training defaults target CPU-friendly settings (batch size 4, max length 256, 3–5 epochs). Adjust `--num_epochs`, `--learning_rate`, and other CLI flags as needed.
 - The provided synthetic dataset is only for demonstration. Replace it with real, labeled production data for meaningful predictions.
+
+## Frontend Integration (React)
+
+- Point your React app to the FastAPI endpoint:
+
+```ts
+// Example using fetch
+async function predictComponents(text: string, threshold = 0.5) {
+  const resp = await fetch("http://localhost:8000/predict", {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({ text, threshold }),
+  });
+  if (!resp.ok) throw new Error("Prediction failed");
+  return await resp.json();
+}
+```
+
+- Ensure the backend has CORS configured (default is permissive) or set `ALLOW_ORIGINS` accordingly in production.

app/main.py

@@ -1,8 +1,10 @@
 from pathlib import Path
+import os
 import sys
 from typing import List
 
 from fastapi import FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
 from pydantic import BaseModel, Field
 
 PROJECT_ROOT = Path(__file__).resolve().parents[1]
@@ -34,6 +36,16 @@ class PredictResponse(BaseModel):
     summary="Predict impacted components from requirement statements.",
 )
 
+# CORS for React UI
+ALLOW_ORIGINS = os.getenv("ALLOW_ORIGINS", "*").split(",") if os.getenv("ALLOW_ORIGINS") else ["*"]
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=ALLOW_ORIGINS,
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
 MODEL_DIR = PROJECT_ROOT / "models" / "distilbert_component_classifier"
 MODEL = None
 TOKENIZER = None
@@ -43,11 +55,21 @@ class PredictResponse(BaseModel):
 @app.on_event("startup")
 async def _load_model() -> None:
     global MODEL, TOKENIZER, LABELS  # pylint: disable=global-statement
-    if not MODEL_DIR.exists():
-        raise RuntimeError(
-            f"Model directory '{MODEL_DIR}' not found. Train the model before starting the API."
-        )
-    MODEL, TOKENIZER, LABELS = load_assets(str(MODEL_DIR))
+    if MODEL_DIR.exists():
+        MODEL, TOKENIZER, LABELS = load_assets(str(MODEL_DIR))
+    else:
+        # Defer loading; endpoint guard will respond 503 until trained
+        MODEL, TOKENIZER, LABELS = None, None, None
+
+
+@app.get("/healthz")
+async def healthz() -> dict:
+    return {"status": "ok", "version": app.version}
+
+
+@app.get("/readyz")
+async def readyz() -> dict:
+    return {"ready": MODEL is not None}
 
 
 @app.post("/predict", response_model=PredictResponse)
@@ -67,3 +89,13 @@ async def predict_components(payload: PredictRequest) -> PredictResponse:
     predictions.sort(key=lambda item: item["score"], reverse=True)
     scores = [ComponentScore(**item) for item in predictions]
     return PredictResponse(components=scores, threshold=payload.threshold)
+
+
+# Optional Prometheus metrics if installed and enabled
+if os.getenv("ENABLE_METRICS", "0") == "1":
+    try:
+        from prometheus_fastapi_instrumentator import Instrumentator  # type: ignore
+
+        Instrumentator().instrument(app).expose(app)
+    except Exception:  # pragma: no cover - optional dependency
+        pass

dev-requirements.txt

@@ -0,0 +1,4 @@
+pytest>=7.4
+requests>=2.31
+matplotlib>=3.8
+seaborn>=0.13