Skip to content

LLM layer should be able to suppress false positives from rule engine #1

@Ap6pack

Description

@Ap6pack

Problem

The scan pipeline is strictly additive — each layer can only add findings, never suppress or demote findings from earlier layers. This means the LLM analyzer's contextual understanding is wasted when it correctly identifies rule engine findings as false positives.

Example: securityreview skill on ClawHub

malwar crawl scan securityreview

Result: MALICIOUS (100/100) — but the findings are FPs:

Severity Rule What it found
HIGH MALWAR-CRED-001 AKIAIOSFODNN7EXAMPLE (AWS example key)
HIGH MALWAR-CRED-001 wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY (AWS example key)
LOW llm-deception "Prior Detector Flags Are False Positives from Example Code"

The LLM correctly identifies the credentials as AWS's well-known placeholder keys used in documentation, but it has no mechanism to demote or suppress the HIGH findings from the rule engine. The final score remains 100/100 despite the LLM's correct assessment.

Proposed Solution

Add a feedback path from the LLM analyzer back to the risk scoring layer:

  1. When the LLM identifies prior findings as false positives (example code, documentation patterns, test fixtures), it should emit a suppression annotation referencing the original finding IDs
  2. The risk scorer should process suppression annotations by reducing confidence or removing suppressed findings from the score calculation
  3. Suppressed findings should still appear in the output (for transparency) but marked as suppressed with the LLM's reasoning

Impact

Without this, security-focused skills that reference example credentials, document attack patterns, or include test vectors will consistently score as MALICIOUS — reducing trust in scan results and causing alert fatigue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions