Fix audio lagging behind video (audio latency reduction + AudioSync diagnostics)

[oliversluke]

Problem

Audio plays noticeably behind video during streaming. The video path is deliberately unbuffered — decoded frames display immediately to keep input latency minimal — but the audio path buffered at every stage. Measured on a real device (Apple TV, HDMI): ~160 ms of audio pipeline delay.

Buffering the video to match is not an option (it would add controller-to-photon latency), and the GFN server sends no RTCP sender reports, so WebRTC's built-in lip-sync can never engage on these streams. The only lever is cutting audio latency.

Root causes (measured with the diagnostics added in this PR)

Stage	Measured	Cause
Device playout	100 ms	.moviePlayback session mode engages tvOS's high-latency, quality-optimized output path (80 ms outputLatency + 20 ms IO buffer)
Voice processing	(included above)	Default factory init routes game audio through Apple's Voice-Processing I/O unit — built for calls (AEC/AGC/NS), adds latency and voice-band filtering
NetEQ jitter buffer	~60 ms	Sat ~20 ms above its own 40 ms target without ever draining (acceleration never triggered)

Fixes

1. Bypass voice processing: peer connection factory created with bypassVoiceProcessing: true. Game audio is full-range; the call-oriented processing chain only hurts. Trade-off: no echo cancellation on the (rarely used) microphone path.
2. .default audio session mode instead of .moviePlayback, plus a 10 ms preferred IO buffer on both the playback and mic session paths.
3. Lean jitter buffer: audioJitterBufferFastAccelerate = true (drains excess above target) and audioJitterBufferMaxPackets = 50 (caps worst-case buildup at 500 ms; default was 2 s).

Diagnostics (new)

In Diagnostics stats mode only, a per-second AudioSync os_log line reports: NetEQ actual/target/floor delay, device playout delay, live OS route output latency, NetEQ time-stretch/concealment activity, packet jitter/loss, and the audio−video estimated-playout-timestamp offset (when the server ever sends RTCP SR). Plus a one-shot audio session configuration line (category, mode, sample rate, IO buffer, output latency, route). Zero overhead outside diagnostic mode.

Results

• Movie-mode/VPIO device overhead eliminated; NetEQ held nearer its target.
• Subjectively confirmed on-device: audio sync is clearly improved.
• Remaining structural terms, documented in-code for follow-up: NetEQ's ~40 ms self-computed target (no public API lowers it below the computed value; a follow-up will experiment with preferring plain Opus over RED in the answer SDP) and the HDMI route's ~80 ms reported output latency (tvOS expects apps to delay video to compensate — we deliberately do not, to protect input latency).

Testing

• Real device (Apple TV 4K, HDMI): multiple sessions across different games/zones, before/after AudioSync logs compared.
• Build (tvOS simulator) + SwiftFormat + SwiftLint --strict clean.

🤖 Generated with Claude Code

[aarikmudgal]

Hi @oliversluke
Thanks for the PR, how can I test the changes? not sure if we have audio info in diagnostics menu

[aarikmudgal]

aah just saw you added diagnostics, testing now

[aarikmudgal]

@oliversluke I see higher Jitter and Decode buffer comparing these changes to the main branch. Also, I did not see audio diagnostics in diagnostics menu, just console logs. Maybe its me but I did not notice audio latency differences but did notice the 5-10ms increase in jitter and decode values in diagnostics. Could you please cross-check. Thanks

[oliversluke]

@aarikmudgal Thanks for testing — good questions, and they led to a rework. On your observations:

Jitter / decode buffer +5–10 ms: the two values in the diagnostics menu are video-side metrics (video jitter buffer and decode time). Everything in this PR is strictly in the audio path, and the one WebRTC mechanism that could couple audio to video buffering — lip-sync raising the video playout delay to match audio — cannot engage on GFN, because the server sends no RTCP sender reports (documented in the PR description). In our before/after sessions the video numbers moved by more than 5–10 ms between two runs of the same build: GFN assigns a different rig per session and the video jitter-buffer target adapts to live network conditions, so an unpaired A/B comparison sits inside normal variance. If you still see a consistent delta across several paired sessions (same game/region/codec, a few minutes in), please share — that would be worth digging into.

No audio diagnostics in the menu: fair hit — they were os_log-only. Fixed properly now, see below.

Couldn't hear the difference: expected, honestly — the steady-state gain is ~45 ms (~160 → ~115 ms), right around the detectability threshold. The bigger win of fastAccelerate is recovery: after a network hiccup the buffer drains back down in seconds instead of latency creeping up for the rest of the session.

Closing this PR: consolidated into #62, which now carries the latency fix, 5.1 surround, and a GFN-style Statistics HUD — pause menu → Statistics cycles Off → Compact → Standard, with separate, clearly-labeled Audio and Video jitter-buffer rows on screen, so you can verify all of this quantitatively without a console attached. Would love a re-test over there.