Skip to content

feat: Comprehensive Analysis Runner (M121) #265

@hlin99

Description

@hlin99

Problem

Users must run multiple CLI subcommands (summary, tail, jitter, convergence, health-check, outlier-impact, scorecard) individually to get a complete picture of their benchmark data quality and results.

Solution

Add a comprehensive subcommand that orchestrates multiple analysis modules in a single invocation and produces a unified report with aggregated findings.

Implementation

  • ComprehensiveRunner class in comprehensive.py
  • ComprehensiveConfig, AnalysisSection, ComprehensiveReport Pydantic models
  • Default suite: summary, SLA check, tail analysis, jitter, convergence, health check, outlier impact, scorecard
  • Configurable sections via --sections / --skip flags
  • Aggregated findings: warnings, recommendations, key metrics from all analyses
  • Overall health summary with composite grade
  • CLI comprehensive subcommand with table + JSON output
  • Programmatic run_comprehensive() API

Acceptance Criteria

  • comprehensive subcommand produces unified report
  • Sections are independently toggleable
  • JSON output includes all section results
  • ~24 new tests
  • All existing tests pass

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions