fix: dedupe duplicate observations in retrieval output

[sunwz1115]

Summary

• dedupe exact normalized duplicate observations at representation formatting time
• dedupe reasoning-chain source/premise/child tool output without mutating stored observations
• add regression coverage proving duplicates are removed per observation level while same content across different levels is retained

Safety

This is retrieval-time only. It does not delete, rewrite, or merge raw documents rows, vector-store data, source IDs, premise links, or session metadata. Ranking remains stable by keeping the first document in input order.

Test plan

• python -m pytest tests/utils/test_representation_retrieval_dedupe.py -q

Summary by CodeRabbit

• Bug Fixes

  • Eliminated duplicate observation documents from reasoning chains to prevent repeated content in output.
  • Enhanced document deduplication for LLM representations by intelligently normalizing content comparisons across whitespace and casing variations.

• Tests

  • Added test coverage for document deduplication behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 97419dbe-18ef-4f27-b689-2a4b1b7025a5

📥 Commits

Reviewing files that changed from the base of the PR and between a420264 and c3b3410.

📒 Files selected for processing (3)

• src/utils/agent_tools.py
• src/utils/representation.py
• tests/utils/test_representation_retrieval_dedupe.py

---

Walkthrough

This PR adds deduplication logic to remove duplicate observation documents across document retrieval and tool output rendering. Deduplication is based on normalized content (case-insensitive, whitespace-collapsed) and level matching. The core helpers are applied in Representation.from_documents() and in _handle_get_reasoning_chain reasoning output formatting.

Changes

Document Deduplication Feature

Layer / File(s)	Summary
Deduplication helpers and representation integration src/utils/representation.py	Added private helpers to normalize document content and deduplicate by (level, normalized_content) tuple, preserving first occurrence order. Integrated deduplication into Representation.from_documents(...) before observation construction.
Tool output deduplication for reasoning chain src/utils/agent_tools.py	Added Sequence import, introduced _dedupe_observation_docs_for_tool_output() helper, and applied deduplication to premises, sources, and derived observations in _handle_get_reasoning_chain before response rendering.
Test verification for deduplication behavior tests/utils/test_representation_retrieval_dedupe.py	Added test module with document helper function and test case that verifies deduplication of normalized duplicate content at the same level while preserving observations at different levels.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 In documents deep where duplicates hide,
A deduplication spell opens wide—
Same content, same level, they merge as one,
Normalized content, the duplicate's done!
Observations now clean, no repeats remain,
Clarity flows through the reasoning chain.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: deduplication of duplicate observations in retrieval output, which is the core purpose of all three modified/added files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

• Generate code and open pull requests
• Plan features and break down work
• Investigate incidents and troubleshoot customer tickets together
• Automate recurring tasks and respond to alerts with triggers
• Summarize progress and report instantly

Built for teams:

• Shared memory across your entire org—no repeating context
• Per-thread sandboxes to safely plan and execute work
• Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}