Audio: latency reduction, true stereo/5.1 surround, and a GFN-style statistics HUD#62
Conversation
aarikmudgal
changed the title
oliversluke
force-pushed
the
feature/audio-format
branch
from
596c100 to
a98704a
Compare
|
testing changes |
Audio played noticeably behind video (~160 ms measured). The video path is deliberately unbuffered (frames display immediately on decode to minimize input latency), while every stage of the audio path buffered: WebRTC's NetEQ jitter buffer (~60 ms), Apple's Voice-Processing I/O unit, and a .moviePlayback audio session whose quality-optimized output path measured 100 ms of device playout delay. The GFN server sends no RTCP sender reports, so WebRTC's built-in lip-sync can never engage — trimming audio latency is the only lever. Fixes, all verified with the new diagnostics: - Create the peer connection factory with bypassVoiceProcessing: the VPIO unit is built for calls (AEC/AGC/NS) and adds output latency plus voice-band processing to full-range game audio. Trade-off: no echo cancellation on the rarely-used microphone path. - Use .default audio session mode instead of .moviePlayback and request a 10 ms preferred IO buffer; movie mode engages tvOS's high-latency, quality-optimized output path. - Keep the audio jitter buffer lean: audioJitterBufferFastAccelerate drains excess above target; audioJitterBufferMaxPackets=50 caps worst-case buildup at 500 ms (default was 2 s). Also adds an AudioSync diagnostic log (Diagnostics stats mode only): per- second NetEQ actual/target/floor delay, device playout delay, live OS output latency, time-stretch/concealment activity, and the audio-video estimated- playout-timestamp offset when RTCP SR is available. Remaining structural latency, documented for follow-up: NetEQ's ~40 ms self-computed target (no public API lowers it below the computed value) and the HDMI route's reported 80 ms output latency (tvOS expects apps to delay video to compensate; we deliberately do not, to keep input latency minimal). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The app has always played MONO: WebRTC's built-in Apple audio device module pins playout to 1 channel (audio_device_ios.mm asserts channels == 1 and reports stereo unavailable), so the stereo we negotiate via stereo=1 was silently downmixed before reaching the speaker. This PR replaces the built-in module with a custom AVAudioEngine-based RTCAudioDevice and adds full 5.1 surround support on top. Audio Format setting (Automatic / Stereo / 5.1 Surround, default Automatic): - Automatic requests 5.1 on surround-capable routes and lets tvOS render it for the actual speakers: discrete 5.1 on receivers (device-verified: AVR locks onto multichannel PCM), spatial virtualization on Atmos TVs, graceful 6:2 downmix elsewhere. tvOS exposes no exact sink-capability API (the route port reports only the currently active format), so preferring surround with graceful degradation is the reliable behavior. - Localized in all 34 languages. 5.1 negotiation (probe-verified against the GFN server): - CloudMatch create/resume requests declare surround via surroundAudioInfo (GameStream encoding, 5.1 = (0x3F << 16) | 6), upon which the server offers multiopus/48000/6 instead of stereo opus — the same multichannel Opus the official web client uses. - WebRTC does not advertise multiopus as a receive codec, so createAnswer rejects the audio m-line (port 0, mid dropped from BUNDLE), breaking the bundle transport. SDPMunger.mungeAudioAnswer rebuilds the answer's audio section to accept multiopus with the offer's exact fmtp — the SDP-munging path libwebrtc explicitly supports for this codec. Custom audio device (GFNAudioDevice): - AVAudioSourceNode pulls interleaved Int16 straight from the native ADM at the negotiated channel count: 2ch (finally true stereo) or 6ch with an explicit MPEG_5_1_A layout matching WebRTC's multiopus decode order. - AVAudioSinkNode keeps the GFN microphone path working (Float32 to Int16 conversion); route changes rebuild the graph and re-notify the ADM. Latency: the answer munge also drops RED (redundant audio) from the accepted payloads — NetEQ no longer budgets delay for redundancy the healthy network never needs. Combined with the fast-accelerate drain, NetEQ now settles as low as 35 ms in clean sessions (was pinned at 55-65 ms). Diagnostics: the AudioSync/AudioDevice os_log lines report the negotiated audio codec, requested/granted/port channel counts and route, gated to Diagnostics stats mode. Verified on real hardware across a stereo-TV room and a 5.1 AVR room: negotiation, 6-channel decode and playout, AVR multichannel lock, stereo regression, HDR-switch resilience, and correct channel placement by ear. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
oliversluke
force-pushed
the
feature/audio-format
branch
from
a98704a to
7577f6a
Compare
…evels Add a persistent statistics HUD during gameplay, mirroring the official client's Statistics overlay: Compact shows the vital signs (FPS, ping, bitrate, loss — with sparklines), Standard a full sectioned panel (Network, Video, Video Format, Audio, Audio Format, Session) including server location and, for the first time, the audio pipeline: NetEQ jitter buffer, concealment, time stretching, output latency, negotiated codec/channels and the active output route. The pause menu becomes a left sidebar like the official overlay, with a Statistics button cycling Off → Compact → Standard that takes effect immediately (the HUD stays visible while the menu is open, never takes focus, and never pauses input). B/Menu closes the sidebar like Resume. The old statsMode picker splits into two independent settings: the HUD level (statsMode: off/compact/standard, default off, cycled in-stream) and a Diagnostics toggle in Settings gating the developer extras (video-pipeline tracing, AudioSync logging, RTC event log, Debug HUD section, which also absorbs the pause menu's old diagnostic rows). Persisted settings migrate losslessly: "hud" → off (its stats now always show in the overlay), "diagnostic" → standard + diagnostics on; unknown values fall back to off instead of resetting all settings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
oliversluke
changed the title
|
@aarikmudgal following up from #59: this PR now includes the Statistics HUD — open the pause menu during a stream and press Statistics to cycle Off → Compact → Standard (takes effect immediately, HUD stays up while playing). Standard mode shows separate Audio and Video sections, each with its own jitter buffer (current/target) — the video jitter buffer and decode time you compared on #59 live under Video; the audio path this PR changes is under Audio / Audio Format. Compact is just FPS/ping/bitrate/loss + server location. For an A/B against main: same game + region + codec, give it 2–3 minutes to settle, and ideally a couple of runs each — GFN hands out a different rig per session, so single-session deltas of a few ms are noise. |
|
Nice @oliversluke , Will do the testing in a bit and share my observations. |
|
@oliversluke Once in stream, after opening Diagnostics. There are no further options you mentioned. tested on simulator as well as real apple tv. I also noticed that you have incoming and outgoing commits in the PR. can you please check once and confirm? (Maybe a rebase) Press "Play/Pause" button on Siri remote or hold options button opens the diagnostic menu 
|
|
I will fix the commits later today. I need to rebase them. |
|
Sounds good, @oliversluke Tag me after, now just to see the changes I am trying to directly test on your forked branch. Maybe that works. |
|
@oliversluke Tested on your fork and branch, features work, haven't tested in depth but on high level it works and I found a UI bug when we switch from compact to standard. The UI breaks, seems like the entire stream tries to fit the diagnostics menu from top to bottom and zooms into the stream. Maybe you can fix that as well. Here are the screenshots, tested both on simulator as well as ATV 4K. 
You can see the cropped FH6 logo on left bottom |
…ontrast Review fixes for the statistics HUD: - The HUD's outer padding sat outside its full-screen frame, so panel height + margins exceeded the screen, grew the ZStack, and stretched the video underneath (reported as the stream 'zooming' when switching to Standard). Padding now applies inside the frame; the HUD can no longer influence the video layout. - Standard level condensed from six sections to four with combined rows: jitter/loss, bitrate (max bandwidth inline), drops/freezes, one-line video format (codec · color mode · HW), conceal/stretch, and audio format/output (layout @ rate · route). Full decoder implementation moved to the Debug section. - Show both FPS values like the official client: game engine FPS first, then stream FPS, in Compact and Standard. The game FPS arrives on a new 'stats_channel' data channel (unordered, no retransmits) whose binary messages the server sends for the official overlay: message type 3/4, then little-endian float64 at payload offset 25. - Pause-menu buttons switch from borderedProminent (always-filled, focus barely visible) to bordered, restoring the native tvOS focus contrast: translucent chips with the focused button in a solid bright fill. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@aarikmudgal thanks for the thorough testing — all addressed in 27e27c1:
About the incoming/outgoing commits: nothing is wrong with the PR itself — it's based on current Would appreciate a re-test of the Compact ↔ Standard switch on your ATV! |
surroundAudioInfo only switches the WebRTC transport to multiopus — the session was carried as 6-channel audio, but audioMode stayed at the stereo default, so the cloud rig's audio endpoint remained 2.0 and games rendered stereo: front speakers played, rear channels carried encoded silence. The official client sets audioMode to the selected channel count (audioMode = audioChannelCount); do the same. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 6-channel source fed AVAudioEngine's main mixer, which folds multichannel input into the front channels when the input and output layouts differ: the HDMI link stayed 6-channel (receivers locked onto 5.1) but the rear channels left the mixer silent. Device-verified with per-channel level metering — all six channels arrive from WebRTC's multiopus decode; only the mixer dropped the rears. Surround now connects the source straight to the output node, which maps by channel layout; stereo keeps the mixer path. Also logs the output hardware format/layout once per graph build for diagnostics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Connection candidates raced their state callbacks on a concurrent global queue, so a losing candidate's cancellation could run concurrently with (or right after) its own .ready callback and resume the same checked continuation twice — crashing with SWIFT TASK CONTINUATION MISUSE when both candidates connected near-simultaneously. Each connection now starts on its own serial queue and every resume goes through a lock-guarded resume-once gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Two more device-verified fixes pushed:
|
The badge showed whenever a game supports Reflex, but the session request only enables it at 120+ fps, which tvOS's 60 Hz cap rules out — so it advertised a feature no session ever uses. HDR and RTX badges already gate on actual usability; Reflex now simply never shows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@owenselles @aarikmudgal From our side this PR is now complete and ready for review/merge — no further changes planned unless review or testing turns something up. Final state, 8 commits in reading order:
Device-tested across a stereo room and a 5.1 receiver room: latency, discrete 5.1 with working rears, HUD levels/persistence, and the Compact↔Standard switch. Happy to address anything that comes up. |
|
Thanks @oliversluke for explaining the changes well and in detail. I will pick the testing as soon as possible. If not today then tomorrow for sure. |
|
Testing @oliversluke , quick question, holding option doesn't open the in-stream overlay. Was this explicitly modified, now we have to click the siri remote to open the overlay, we should keep the hold option button as well since with controller you kind of replace the Siri Remote experience of tvOS. |
|
@aarikmudgal - no there was no change for this. it is one of the two buttons (depending on your settings). and a long press opens the HUD which shows on the left side. maybe a side effect of some other PR you are testing? I can also retest later today, but I am quite sure I did not change this. |
|
@aarikmudgal Double-checked the diff: Two things to check on your end: the overlay trigger is a single configurable button (Settings → Controller, default "Start (≡)" = the ☰ Menu button on Xbox / Options on DualSense) — holding the other small button triggers the Steam-overlay chord instead, which shows nothing outside Steam games. And the hold needs to be fairly long (~1.8 s); shorter presses are passed through to the game on purpose. Both behaviors are unchanged since the configurable-trigger commit from April. If it still doesn't open with the right button and a long hold, can you check whether it works on a main build with the same controller? |
|
Absolutely @oliversluke will Test it thoroughly. For now I started testing and noticed few things.
Take this with a grain of salt since it's really fresh testing observations. Would need to test it thoroughly for extensive feedback and review. |
|
@aarikmudgal One more thought: does your test build include #65 (tvOS keyboard input) by any chance? Its default keyboard shortcut is Options + Y, and its shortcut state machine intercepts the Options button on press — it withholds the button while waiting for the rest of the sequence and actively suppresses the overlay/Steam hold gestures while engaged ( |
|
@oliversluke you were right. I took clean build and now the game started and overlays stats etc are working well. 5.1 audio also working well. Will do one by one test of all the listed changes. |
aarikmudgal
left a comment
aarikmudgal
left a comment
There was a problem hiding this comment.
Tested on ATV with FH6 and Witcher 3. All mentioned features are working.
|
@oliversluke Tested, looks good., May I recommend to also update Readme.md for all your PRs, I think last few PRs were also missing the documentation and I completely overlooked that. |
|
@aarikmudgal Good point, thanks. To keep this PR's approval intact I'll do the docs as a dedicated follow-up PR right after this one merges — it will cover the README debt from the recent merged PRs too (0% deadzone from #46, the H.265 Main10/HDR work from #49) plus this PR's statistics HUD and surround audio. The still-open #67 and #68 will get their README updates pushed directly into those PRs so they land documented. |
|
Thank you @oliversluke Sounds like a very good idea. |
…ree (#69) Documents what recent PRs shipped but the README missed: 0-30% controller deadzone (#46), H.265 Main10 SDP handling and the HDR-preserving VideoToolbox decoder (#49), the pause menu sidebar, statistics HUD, and surround audio (#62). Also adds the nine source files missing from the architecture tree. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2 participants
Supersedes and absorbs #59 — this PR is now self-contained (three commits, reviewable independently) and no longer depends on any other PR.
What this PR does
1. Audio latency reduction + AudioSync diagnostics (commit 1)
Audio played ~160 ms behind the (deliberately unbuffered) video. Since the GFN server sends no RTCP sender reports, WebRTC lip-sync can never engage — trimming audio-path latency is the only lever:
.defaultplayback session mode + 10 ms IO buffer (was.moviePlayback, which measured ~100 ms device playout)audioJitterBufferFastAccelerate+ capped max packets, so the buffer drains back down after network hiccups instead of latency creeping upMeasured: ~160 ms → ~115 ms total audio latency (NetEQ settles at ~35 ms in clean sessions; the remaining ~80 ms is fixed HDMI/TV route latency).
2. True stereo & 5.1 surround with an Audio Format setting (commit 2)
The bundled WebRTC audio device plays mono on Apple platforms. This PR replaces it with a custom
RTCAudioDevice(GFNAudioDevice) built on AVAudioEngine:surroundAudioInfofrom CloudMatch, accepts the server's multiopus offer via answer munging (libwebrtc rejects the m-line in its own answer; the LiveKit binary supports multiopus decode once accepted), and plays out 6 channels with an explicit 5.1 channel layoutmaximumOutputNumberOfChannels >= 6) and lets tvOS downmix gracefully otherwise — device-verified in a stereo room (TV with spatial audio) and a 5.1 AVR room (receiver locks discrete 5.1)3. GFN-style Statistics HUD (commit 3)
Replicates the official client's Statistics overlay, and directly addresses the review feedback on #59 (audio diagnostics were console-only, and video vs audio metrics were easy to confuse):
hud/diagnosticvalues map sensibly; unknown values can't reset settings).Testing
🤖 Generated with Claude Code