[Batch 5] Occlusion culling integration with depth pyramid

[MichaelFisher1997]

Summary

Add a second pass to the GPU culling compute shader that tests chunk AABBs against the depth pyramid from #383. Chunks fully occluded by closer terrain are culled, reducing draw calls by 30-50% in hilly/mountainous terrain.

Depends on: #383 (depth pyramid shader)

Current State (after #379)

GPU frustum culling eliminates chunks outside the camera's view frustum. But chunks inside the frustum that are completely hidden behind a hill or mountain are still rendered. In mountainous biomes this can be 30-50% of visible chunks.

How It Works

The depth pyramid (#383) stores hierarchical depth information from the previous frame. For each chunk AABB, we can test whether it's fully behind the closest surface already rendered:

1. Project the AABB's 8 corners to screen space
2. Find the bounding box of those projections in screen space
3. Sample the depth pyramid at the appropriate mip level (matching the AABB's screen size)
4. If the AABB's closest depth is further than the pyramid's maximum depth at that region → occluded

Conservative Testing

• Use the AABB's nearest corner depth for the test (conservative: err on the side of visibility)
• Sample the depth pyramid mip level that matches the AABB's screen-space coverage
• One sample at the appropriate mip level is sufficient (that's the beauty of the hierarchy)

Implementation Plan

Step 1: Add depth pyramid binding to culling shader

Update culling.comp:

• New binding: layout(binding=3) uniform sampler2D depth_pyramid
• New push constant: vec2 screen_size and float previous_frame_valid
• If previous_frame_valid == 0: skip occlusion test (first frame, no previous depth)

Step 2: AABB projection + occlusion test

bool isOccluded(vec3 aabb_min, vec3 aabb_max, mat4 view_proj, 
                sampler2D depth_pyramid, vec2 screen_size) {
    // Project all 8 AABB corners to clip space
    // Find nearest depth and 2D screen bounds
    // Select mip level based on screen-space AABB size
    // Sample depth pyramid at mip level
    // If nearest AABB depth > pyramid max depth → occluded
}

Step 3: Two-pass culling in compute

void main() {
    uint chunk_idx = gl_GlobalInvocationID.x;
    if (chunk_idx >= chunk_count) return;
    
    // Pass 1: Frustum test (already exists)
    if (!frustumTest(chunk_idx)) return;
    
    // Pass 2: Occlusion test (new)
    if (previous_frame_valid > 0 && isOccluded(chunk_idx, ...)) return;
    
    // Visible: write draw command
    writeDrawCommand(chunk_idx);
}

Step 4: Depth pyramid lifecycle

• Depth pyramid generated at end of frame N
• Used for occlusion in frame N+1
• First frame after startup: no occlusion (previous_frame_valid = 0)
• After camera cut/teleport: invalidate pyramid for one frame

Step 5: Debug visualization

• Add toggle to debug overlay showing occlusion culling statistics
• Display: chunks frustum-culled, chunks occlusion-culled, chunks rendered
• Color-coded wireframe: green=visible, red=frustum-culled, blue=occlusion-culled

Files to Modify

• assets/shaders/vulkan/culling.comp — add occlusion test
• src/engine/graphics/vulkan/culling_system.zig — bind depth pyramid, pass screen size
• src/engine/graphics/vulkan/depth_pyramid.zig — expose texture for binding
• src/engine/ui/debug_menu.zig — occlusion culling debug toggle

Testing

• Camera faces wall: all chunks behind wall are culled
• Mountain terrain: chunks behind mountain are culled, visible ones render
• Flat terrain: no chunks incorrectly culled (nothing occludes)
• First frame after startup: no occlusion (graceful skip)
• After teleport: occlusion re-validates within 1 frame
• Statistics show expected cull ratio (30-50% in mountains)
• No false positives (visible chunks never culled)

Roadmap: docs/PERFORMANCE_ROADMAP.md — Batch 5, Issue 3A-2

[opencode-agent]

Looking at this issue, I can provide some analysis on the proposed occlusion culling implementation.

Classification: This is a Feature Request - a well-defined enhancement to the existing GPU culling system.

Analysis:

The approach is technically sound. Using a depth pyramid for hierarchical Z-testing is the standard technique for GPU-driven occlusion culling. The two-pass structure (frustum → occlusion) maintains the existing architecture while adding the new culling stage.

A few clarifying questions:

1. Depth pyramid format: What format is the depth pyramid stored in? For the conservative test you describe (AABB nearest depth vs pyramid max depth), you'll need the pyramid to store maximum depth per tile, not minimum. Most implementations use either:

   • VK_FORMAT_R32_SFLOAT with max-reduction
   • VK_FORMAT_R16_UNORM for memory bandwidth
   • Hardware-supported Hi-Z if available

2. Camera teleport detection: How do you plan to detect when previous_frame_valid should be set to 0? Common approaches:

   • Large position delta between frames (> threshold)
   • Explicit camera cut event from game code
   • Time-based heuristic (first N frames after load)

3. Mip level selection: The screen-space AABB size calculation should account for the pyramid's power-of-2 dimensions. Something like:

   vec2 screen_aabb_size = max(screen_max - screen_min, vec2(1.0));
   float max_size = max(screen_aabb_size.x, screen_aabb_size.y);
   int mip_level = clamp(int(log2(max_size)), 0, max_mip);

Scope Assessment: This is appropriately scoped as a single issue. All changes are tightly coupled to the culling system and depend on #383 being complete first.

The testing checklist is comprehensive. The 30-50% cull rate target for mountainous terrain is realistic for this technique.

Is #383 (depth pyramid shader) already implemented? This issue is blocked until that's available.

opencode session  |  github run