Skip to content

feat(engine): add per-claim confidence persistence (#465)#812

Merged
manavgup merged 4 commits into
mainfrom
feat/per-claim-confidence-465
May 23, 2026
Merged

feat(engine): add per-claim confidence persistence (#465)#812
manavgup merged 4 commits into
mainfrom
feat/per-claim-confidence-465

Conversation

@manavgup
Copy link
Copy Markdown
Owner

Summary

  • Update the compiler prompt to request subjects and source_ids per claim, enabling per-claim source attribution from LLM output
  • Pass source UUID in user prompt metadata so the LLM can reference it in claim source_ids
  • Add integration tests verifying claim-level confidence aggregates correctly into article-level confidence

Context

PR #800 shipped Phase 1 of issue #465 (article-level confidence with persistence). This PR advances the issue by closing the gap between the LLM prompt and the existing CompiledClaim table schema — the table already had subjects, source_ids, and confidence_score fields, but the compiler prompt never asked the LLM to populate them. Now it does.

What was already in place (from PR #800 and #784):

  • CompiledClaim table with confidence_score, source_ids, last_reinforced_at, subjects
  • compute_claim_confidence() and aggregate_claim_confidence() pure functions
  • _persist_claims() storing claims with per-claim confidence scores
  • _refresh_confidence_score() aggregating claim scores to article level
  • GET /wiki/articles/{id}/claims endpoint and service layer
  • Full DTOs, migrations, and unit tests

What this PR adds:

  • Compiler prompt schema now includes subjects and source_ids fields per claim
  • Prompt rules instruct the LLM to set subjects (1-3 canonical entity names) and source_ids (UUIDs from metadata)
  • _build_user_prompt passes Source ID: <uuid> in metadata block
  • Integration test: article confidence is the mean of its claim scores
  • Integration test: _persist_claims correctly stores subjects and source_ids from DTOs
  • Unit test: _build_user_prompt includes source ID in output

Not included (deferred to future PRs):

  • Concept identity / embedding clustering
  • Cross-article claim deduplication

Test plan

  • All existing unit tests pass (35 compiler tests, 24 claim confidence tests, 30 citation tests)
  • All existing integration tests pass (3 confidence integration tests)
  • New unit test: test_build_user_prompt_includes_source_id verifies source ID in prompt
  • New integration test: test_article_confidence_aggregated_from_claims verifies claim-to-article aggregation
  • New integration test: test_persist_claims_stores_source_ids_and_subjects verifies claim persistence
  • ruff check passes with no errors
  • mypy passes (4 pre-existing errors in unrelated files)

🤖 Generated with Claude Code

@github-actions github-actions Bot force-pushed the feat/per-claim-confidence-465 branch from ada66ce to 09054e5 Compare May 23, 2026 23:00
manavgup and others added 3 commits May 23, 2026 23:15
Update the compiler prompt to request per-claim source attribution
(subjects and source_ids) so claims carry provenance metadata from
the LLM response. Pass source ID in the user prompt metadata so the
LLM can reference it. Add integration tests verifying claim-level
confidence aggregates into article-level confidence and that
_persist_claims stores subjects/source_ids correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot force-pushed the feat/per-claim-confidence-465 branch from 323b1ee to bc25d84 Compare May 23, 2026 23:15
@manavgup manavgup merged commit 2955682 into main May 23, 2026
13 checks passed
@manavgup manavgup deleted the feat/per-claim-confidence-465 branch May 23, 2026 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant