Harden review prompts for consistency and noise reduction

[mariusvniekerk]

Summary

• Define impact-based severity levels — Replace bare high/medium/low labels with concrete definitions tied to real-world impact (data loss, exploitability, blast radius). Gives all agents a shared calibration standard so severity is consistent across reviews.
• Require concrete harm articulation — Every finding must now explain what specifically goes wrong if left unfixed. Eliminates vague "violates best practices" findings by forcing agents to justify each issue with concrete reasoning.
• Add evidence thresholds — Explicit "do not report" instructions suppress the most common false positive categories: hypothetical issues in unseen code, style opinions, unfounded "missing tests" claims, and flagging codebase conventions as issues.
• Add intent-implementation alignment check — Reverse the old "do not review the commit message" instruction. The commit message now serves as the primary lens for evaluating the diff, catching gaps between what the developer intended and what they actually wrote.
• Add self-review quality gate — Before outputting, agents must verify every finding has a specific file/line reference, severity matches described impact, and no findings contradict each other. Drops findings that fail.
• Add evidence thresholds to insights analysis — Tiered confidence thresholds (1-2 = data point, 3-5 = candidate, 6+ = strong recommendation) prevent guideline suggestions from single occurrences and give high confidence to well-evidenced patterns.

🤖 Generated with Claude Code

[roborev-ci]

roborev: Combined Review (3934535)

Summary Verdict: The changes successfully tighten review prompts with severity definitions and evidence thresholds, but introduce regressions by restricting blind-spot-driven guideline generation and creating false-negative paths for
security vulnerabilities outside the immediate diff.

Medium Severity

• Location: internal/prompt/prompt.go:39, 78, 108, 921

  • Problem: The new instructions tell reviewers not to report issues in code not shown in the diff
    and require a "plausible exploit path visible in the diff" for security reviews. A malicious contributor can exploit this by submitting a small change that connects external input to an existing dangerous sink outside the patch. The prompt directs the reviewer to drop the finding if the sink or full taint flow lives in unchanged code, creating a
    predictable false-negative path for real security bugs.
  • Fix: Keep the anti-speculation guardrail, but explicitly allow reviewers to inspect unchanged surrounding code when needed to validate whether the changed path introduces a vulnerability. A safer rule is: "Do not speculate without evidence, but you may use nearby unchanged
    code to confirm whether the diff creates or exposes an exploit path."

• Location: internal/prompt/insights.go:53

  • Problem: Section 5 now allows guideline suggestions only from section 1 or section 3 evidence, but excludes section 2 recurring blind spots. This
    means the insights pass can identify a repeated missing-guideline pattern in section 2 and still be unable to recommend the corresponding guideline text, which is a direct regression in the output's usefulness.
  • Fix: Allow section 2 patterns with the same evidence threshold to feed section 5 guideline suggestions.

---

Synthesized from 3 reviews (agents: codex, gemini | types: default, security)