[Batch 6] Performance benchmark harness + regression detection

[MichaelFisher1997]

Summary

Create an automated benchmark harness that loads a world, follows a fixed camera path, and records frame time metrics across all presets. Integrate into CI for performance regression detection.

Depends on: #385 (all presets must be defined before benchmarking)

Requirements

Benchmark Tool

• New executable target: zig build benchmark
• Loads world at seed 12345 (deterministic terrain)
• Flies camera along a fixed path (waypoints)
• Records per-frame: GPU time, CPU time, draw calls, vertex count, chunk count
• Runs for N seconds (configurable, default 60s)
• Outputs: JSON results file

Fixed Camera Path

const BENCH_PATH = [_]Waypoint{
    .{ .pos = Vec3.init(8, 100, 8), .look = Vec3.init(1, 0, 0), .duration = 5.0 },
    .{ .pos = Vec3.init(200, 150, 200), .look = Vec3.init(0, -0.3, 1), .duration = 10.0 },
    .{ .pos = Vec3.init(-500, 80, 300), .look = Vec3.init(1, 0, -1), .duration = 10.0 },
    // ... covers multiple biomes, directions, elevations
};

• Interpolate between waypoints (smooth camera movement)
• Covers: plains, mountains, ocean, forest, desert biomes
• Covers: looking up, looking down, spinning

Metrics Collection

{
  "preset": "high",
  "render_distance": 100,
  "frames": 3600,
  "duration_s": 60.0,
  "fps": {
    "min": 42,
    "avg": 67,
    "max": 120,
    "p1": 45,
    "p5": 51,
    "p50": 68,
    "p95": 89,
    "p99": 102
  },
  "gpu_ms": {
    "shadow_avg": 2.1,
    "opaque_avg": 4.3,
    "total_avg": 12.8
  },
  "draw_calls_avg": 2340,
  "vertices_avg": 2100000,
  "chunks_rendered_avg": 890
}

CI Integration

• New workflow: .github/workflows/benchmark.yml
• Triggers on: push to dev, manual dispatch
• Runs benchmark at LOW, MEDIUM, HIGH presets
• Compares against baseline (stored in repo or artifact)
• Posts results as PR comment or commit status
• Regression threshold: 10% FPS drop = warning, 20% = failure

Baseline Management

• Store baseline JSON in docs/benchmarks/baseline.json
• Update baseline when intentional performance changes land
• Compare: scripts/compare_benchmarks.sh baseline.json new.json

Implementation Plan

Step 1: Benchmark executable

• src/benchmark.zig: main entry, world setup, camera path
• Reuse existing -Dsmoke-test infrastructure for headless world loading
• Add waypoint system with smooth interpolation
• Run fixed duration, collect metrics

Step 2: Metrics collector

• Wrap existing GPU timing infrastructure (rhi_timing.zig)
• Accumulate frame times in arrays
• Compute percentiles and averages at end
• Write JSON output

Step 3: Script runner

• scripts/run_benchmark.sh: runs benchmark across presets, collects results
• Configurable: duration, seed, presets to test
• Output: one JSON per preset, summary report

Step 4: Comparison tool

• scripts/compare_benchmarks.sh: diff two JSON files
• Print regression/improvement per metric
• Exit code: 0 if no regression, 1 if regression exceeds threshold

Step 5: CI workflow

• Runs on ubuntu-latest with Vulkan software renderer (or skip GPU tests)
• Alternatively: runs on self-hosted GPU runner
• Posts results as artifact

Files to Create

• src/benchmark.zig — benchmark executable
• scripts/run_benchmark.sh — runner script
• scripts/compare_benchmarks.sh — comparison tool
• docs/benchmarks/baseline.json — initial baseline

Files to Modify

• build.zig — add benchmark build target
• .github/workflows/benchmark.yml — CI workflow (optional, can be follow-up)

Testing

• Benchmark runs to completion at each preset
• JSON output is valid and contains all metrics
• Comparison tool detects regressions correctly
• Results are reproducible (±5% variance across runs)
• CI workflow runs without crashing

Roadmap: docs/PERFORMANCE_ROADMAP.md — Batch 6, Polish

[opencode-agent]

This is a Feature Request for a comprehensive performance benchmark harness. The issue is well-structured and includes detailed requirements.

Scope Assessment: Should Be Broken Down

This is a substantial feature spanning multiple independent deliverables. I recommend splitting this into 4 sub-issues:

1. Sub-Issue A: Benchmark Executable (src/benchmark.zig, build.zig)

   • Camera waypoint system with smooth interpolation
   • Fixed path traversal over deterministic world (seed 12345)
   • JSON metrics output
   • Reuse existing -Dsmoke-test and -Dskip-present infrastructure

2. Sub-Issue B: Runner & Comparison Scripts (scripts/run_benchmark.sh, scripts/compare_benchmarks.sh)

   • Multi-preset benchmark runner
   • Baseline comparison with regression thresholds (10%/20%)

3. Sub-Issue C: CI Workflow (.github/workflows/benchmark.yml)

   • GitHub Actions integration
   • Result reporting (PR comments/commit status)

4. Sub-Issue D: Baseline Management (docs/benchmarks/baseline.json)

   • Initial baseline creation
   • Update procedures

Existing Infrastructure You Can Leverage

GPU Timing is already implemented in:

• src/engine/graphics/vulkan/rhi_timing.zig:40-126 - Per-pass timing (shadow, G-pass, SSAO, LPV, main, bloom, FXAA, post-process)
• timing_results.total_gpu_ms aggregates all passes

Headless Mode exists:

• build.zig:14 - -Dskip-present flag for headless rendering
• build.zig:11 - -Dsmoke-test flag for automated world loading

Presets System is in place:

• src/game/settings/json_presets.zig:43-124 - Dynamic preset loading from assets/config/presets.json
• src/game/settings/data.zig:97-102 - LPV quality presets

Build Pattern for new executables:

• Follow the robust-demo pattern in build.zig:122-137
• Add benchmark target: b.step("benchmark", "Run performance benchmark")

Questions

1. Dependency [Batch 4] Expand render distance presets + LOD tuning #385: What's the current status of preset definitions? This issue is blocked until presets are finalized.

2. CI GPU Access: You mention "self-hosted GPU runner" as an alternative. Do you have access to GPU runners, or should the CI workflow use software Vulkan (Mesa lavapipe)? This affects implementation significantly.

3. Baseline Storage: Will baselines be stored directly in git (docs/benchmarks/baseline.json), or do you plan to use Git LFS or artifacts? Consider baseline size if storing multiple historical results.

4. Reproducibility: The ±5% variance target is ambitious for GPU benchmarks. Have you considered:

   • Running multiple iterations and averaging?
   • Thermal throttling mitigation?
   • Warm-up frames before recording?

Suggested Implementation Order

1. Start with Sub-Issue A (benchmark executable) - this delivers immediate value for manual testing
2. Add Sub-Issue B (scripts) for local regression detection
3. Sub-Issue D (baseline) once you have stable results
4. Sub-Issue C (CI) last, as it depends on runner infrastructure

Would you like me to create these sub-issues, or do you prefer to track this as a single epic issue?

opencode session  |  github run