Smarter relation deduplication: fuzzy pre-filter + LLM semantic validation

[renaudcepre]

Problem

Current deduplication uses fuzz.ratio >= 75 which produces false positives (semantically opposite relations can match) and false negatives (semantically equivalent relations with different wording don't match).

Example: "ally in the quest" + "companion and fellow traveler" → same thing, but fuzzy won't catch it reliably.

Proposed architecture

Three-stage pipeline:

1. fuzz.ratio >= ~85-90  → fast pre-filter (raise threshold from 75)
2. LLM semantic check    → confirm or reject the candidate merge
3. user clarification    → only if LLM returns "unsure"

LLM validation prompt

The LLM should receive full context, not just the relation strings:

Character A: Gimli
Character B: Legolas

Existing relation in DB: "companion met during the Council of Elrond"
New candidate relation: "ally forged through shared battle"

Current profile of Gimli: <arc, background>

Are these describing the same relationship, or distinct aspects worth keeping both?
→ merge / keep_both / unsure

Without context, the LLM judges words. With context, it judges meaning — "ally" and "companion" can be the same for Frodo/Sam but distinct for Aragorn/Boromir.

Notes

• Keep fuzzy as a cheap pre-filter, not the decision maker
• LLM call only triggered when fuzzy score is in ambiguous range (~70-90)
• unsure escalates to user clarification (existing mechanism)
• Raising the fuzzy threshold alone would already reduce noise

[renaudcepre]

Partiellement résolu :

• Ajout d'un seuil bas score < 30 dans _is_relation_duplicate (orchestrator.py) — les relations clairement distinctes (ex: "messagère" vs "Tisserand of the Vermeil Order", score 15) sont court-circuitées sans appel LLM ni clarification humaine.
• Le flow reste : score ≥ 90 → merge auto, score < 30 → keep_both auto, entre les deux → LLM dedup → clarification si "unsure".

Reste à faire : améliorer le prompt du dedup agent pour réduire les "unsure" sur des cas évidents (effort faible, priorité basse).