amicus-eval

A retrieval benchmark for Spanish-language Peruvian legal QA, and the evidence to reproduce it. This is the open, verifiable half of the amicus legal assistant: the gold set, the annotations, the metrics, and the per-query rankings. You do not have to trust our pipeline. You re-score our rankings yourself.

Companion to the research log: research.crafter.ing/research/measuring-legal-retrieval

What this is

amicus does hybrid retrieval (Spanish full-text search + embeddings, fused by RRF, with query expansion and reranking) over the legalize-pe corpus of ~21,000 Peruvian legal norms. This repo evaluates how well it retrieves the right norm, with standard IR metrics over a hand-built gold set.

The retrieval code itself lives in the private amicus repo. What is public here is everything needed to verify the claims: the benchmark and the rankings each configuration produced.

Verify it yourself

bun verify.ts

This reads the published rankings (results/results-n35.json) and the gold set, recomputes every metric with lib/metrics.ts (pure functions, no database, no network), and checks them against our reported numbers. Output:

OK  fts          recomputed MRR=0.0886  reported=0.0886
OK  vec          recomputed MRR=0.5108  reported=0.5108
OK  rrf          recomputed MRR=0.4011  reported=0.4011
OK  rrf+expand   recomputed MRR=0.6050  reported=0.6050
OK  rrf+rerank   recomputed MRR=0.7917  reported=0.7917
OK  best         recomputed MRR=0.7611  reported=0.7611

If you distrust our metric code, reimplement P@k / MRR / nDCG yourself and run them over results/results-n35.json. The rankings are the ground truth of the experiment.

The gold set

data/gold-firme.jsonl: 35 firm query→norm pairs across six strata (colloquial, technical-legal, multi-norm, core-vs-regulation, subnational, out-of-scope). Each norm id maps to a file in the legalize-pe corpus (pe/<id>.md).

It was built with a deliberately auditable method:

1. data/candidates.jsonl: 50 query seeds sampled from the real corpus.
2. Dual blind annotation: two models (Claude Opus + Codex gpt-5.5) annotated the same sheet independently, blind to the seed norm. See data/annotations-claude.jsonl and data/annotations-codex.jsonl.
3. Overlap → gold automatically. Divergence → arbitrated by reading the norm text (data/arbitration-decisions.jsonl, marked arbitrated_by: non-lawyer).
4. results/reconciliation-report.md reports inter-annotator agreement per stratum: 100% on technical-legal, 22% on subnational. That number is a finding, not a flaw: the regional corpus is intrinsically ambiguous.

Honest limitations

• N=35 is small. Confidence intervals are wide. These are signals, not final results. Scaling the gold past 100 is ongoing.
• No lawyer on the team yet. Vigency, repeal, and which-norm-prevails were marked needs_lawyer and excluded from the firm set. Arbitration was done by reading text, by non-lawyers, and is flagged as such.
• The conclusion moved as the gold improved. An earlier N=19 single-annotator run suggested the shipped pipeline was not optimal. That died on scaling. The full story (including the near-miss) is in the research log.

Files

data/
  gold.jsonl                  all reconciled pairs (firm + needs_lawyer)
  gold-firme.jsonl            35 firm Nivel-A pairs (the benchmark)
  candidates.jsonl            50 query seeds
  annotations-claude.jsonl    blind annotation, Claude
  annotations-codex.jsonl     blind annotation, Codex
  arbitration-decisions.jsonl divergences resolved by reading text
results/
  results-n35.json            per-query rankings for all 6 configs + metrics
  results-n35.md              the ablation table, human-readable
  reconciliation-report.md    inter-annotator agreement
lib/
  metrics.ts                  pure P@k / Recall@k / MRR / nDCG (no deps)
verify.ts                     reproduces every reported metric from the rankings

License

The benchmark data is released for research use. Peruvian legal texts are public domain (DLeg 822 Art. 9). The corpus lives at crafter-research/legalize-pe.

Part of Crafter Research.