feat(linux): native video playback with GPU-accelerated rendering#142
feat(linux): native video playback with GPU-accelerated rendering#142skoruppa wants to merge 8 commits into
Conversation
|
@FeelThePoveR, you were active in #58, want to make some testing here as well? ;) |
|
Thank you for modifying the Projectand paying attention to Linux users |
|
Sure I can test it out. This took me way longer than it should, but the package doesn't compile by default due to 2 issues:
After figuring both of those out the package got built. |
|
As for the actual running of the app on my KDE Plasma 6.7 desktop with Wayland session:
Zoom/Fit/Stretch doesn't do anything. Only gpuMode 2 I tested on an actual release build the rest I just did ./gradlew run |
skoruppa
force-pushed
the
linux-gpu-player
branch
2 times, most recently
from
6b8583a to
3f728a7
Compare
|
Thanks for testing! I made some changes and let's hope the gpuMode2 now works for you ;) gpuMode 1 is for x11, so it will fail for you, and gpuMode 0 is the software fallback Would really appreciate if you could re-test on your KDE Plasma 6.7 Wayland setup. Let me know if you still get any errors. |
skoruppa
force-pushed
the
linux-gpu-player
branch
from
3f728a7 to
edcd096
Compare
|
I rebuilt the package and the gpuMode 2 still doesn't work, but now it falls back to gpuMode 0 so playback still starts. For gpuMode 1 it shouldn't really matter that I'm not in X11 session. KWIN still renders Nuvio as an X11 window through XWayland. I looked it up and apparently Kotlin Multiplatform doesn't support native Wayland yet so Wayland native probably isn't possible right now without adding QT/GTK or something like that as the backend. From other issues:
|
That is not true :D Yes, JVM runs through XWayland. But mpv's vo=gpu-next with an XWayland window ID is unreliable. It silently fails on many compositors because mpv's X11 GPU path fights with the Wayland compositor's surface management. That's exactly what you saw (no output, no errors).
Could you share your GPU info? Something like: The GL render context creation is failing on your setup. EGL initializes fine but mpv rejects the GL context. Likely a driver/profile compatibility issue. On my Intel ARL it works with GL 4.6 Core Profile, but your GPU might need Compatibility Profile or a different GL version. What GPU/driver are you on? I left some logs in latest changes maybe we can see what is happening on your side (both x11 and wayland) |
|
Here's the Player bridge logs For the X11 gpuMode 1 path - it doesn't reach any of the new logs, the last log produced is this So still before the player initialization. |
|
Ahh, NVIDIA why I'm not suprised :P I made a new push that should fix the NVIDIA Wayland issue. EGL context is now created and activated on the same render thread For X11 - no |
|
Hahah, yeah the Nvidia GPU for sure is the biggest regret in my current build. For the new build on Wayland session unfortunately it still fails on the same line and falls back to gpuMode 0 X11 side on the other hand decided to produce some logs this time (even without resizing) |
|
New push with a potential fix for NVIDIA Wayland Also pushed a fix for X11. The |
|
Both Nvidia Wayland and X11 still behave in the same way. Wayland logs: X11 logs: |
|
Try again >< maybe this time we will have different results |
|
For the Nvidia Wayland there's no change, it still falls back to gpuMode 0 For the X11 side there's some progress the playback starts, but there's no UI and after the player loses focus it starts being drawn over the cursor |
|
I decided to get rid of gpuMode 1 - gpuMode 2 with software fallback should also work there. Can you test both again? I want to see logs |
|
Nvidia Wayland continues to fallback to gpuMode 0 X11 side on the other hand seems to be working as expected with the gpuMode 0 fallback. |
|
Good. Progress. Can you check again with Wayland Nvidia? |
|
No change there unfortunately |
|
Ok, next idea :D lets test again |
|
Hahah, it's still failing but there's some new info in the log, apparently |
|
Ok, check newest changes. Can you also run those command and give me results? |
|
Result is the same Command output Maybe this will also help - I run eglinfo and it reports the following devices:
|
|
thx for that. This helped. Can you try with new changes? |
|
A little bit of progress with this one now the override config passes, but the eglMakeCurrent still fails. |
|
I also tried one other thing - running the app with the EGL Vendor passed in the environment |
and how it works? Everything smooth? |
Renderer starts in gpuMode 2 successfully using the GLX fallback There's an issue with cached instance - changing sources or backing out of the player and restarting playback blacks out the video playback with that error in the console |
Cache was storing EGL fields but not useGLX flag or eglSurface. On restore, makeGLCurrent tried EGL (useGLX=0) and failed because GLX context was the active backend. Now caches useGLX and eglSurface alongside other GL state.
|
yeah, cache reuse will not work with glx. Check now @FeelThePoveR |
Now it's fine for this case. There's also a different issue where the player refuses to die when backing out of playback, after it hangs the app itself also refuses to close. Logs don't really say anything useful. I though that was an anomaly during the first test, but it happened for a second time. The end of the log output |
Render thread could deadlock during dispose when mpv_command('stop') blocks
while render callback is active. Fix:
- Detach mpv render update callback before stopping playback
- Signal frameCond again after stop to wake render thread from wait
|
@FeelThePoveR check now. Maybe managed to handle it |
This seems to have fixed it, I tried for a couple of minutes and nothing so I think we should be good. Last issue (I think :D) is that when you back out of playback the resume position doesn't sync - it only syncs once you back out of the stream list. |
|
Nevermind spoke too soon, of course I'm going to get that immediately after posting. I'll see if I can repro it reliably |
|
After trying it out for 10 more minutes I got 3 players stuck like this but there's no common pattern for this from what I can see 1 of those failures also showed another random issue with no reliable repro - during the second attempt my cursor started being drawn under the player for a sec until I focused on another app and then back on Nuvio |
…ose deadlock
On NVIDIA/GLX, backing out of playback could hang the player ("won't die") and
then the whole app (won't close). dispose() blocks forever on pthread_join of the
render thread.
Root cause (confirmed by thread dump): dispose() sent mpv `stop` BEFORE joining the
render thread. `stop` makes mpv's core thread tear down the decoder
(vd_lavc_destroy -> vkDestroyDevice) inside the NVIDIA driver, while the render
thread still owns and is using the GLX context. Both threads then block forever
inside libnvidia-glcore (concurrent GL + Vulkan teardown on the same context), so
the render thread never exits and dispose()'s pthread_join never returns.
Fix: join the render thread FIRST. It finishes its current frame, unbinds the GL
context (glXMakeCurrent(None)) and exits, so the context is idle before mpv tears
the decoder down — no concurrent GL/VK teardown, no driver deadlock. Then stop
playback as before. Single-threaded change, GLX path only; EGL/Mesa unaffected.
Tested on Fedora/Wayland (XWayland), RTX 4060 Ti, driver 595.80: repeated
play -> back out -> replay -> back out -> close cycles, all clean
(dispose() -> "GL instance cached for reuse", app closes normally).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@skoruppa Was able to reproduce the stuck-player deadlock on |
I also can't reproduce the resume position issue after this so it's possible the dispose hang was blocking whatever process triggers the watch progress sync? Needs further testing though. |
fix(linux/nvidia): join render thread before mpv stop to fix GLX dispose deadlock
Merged. @FeelThePoveR can you test as well? |
Looks good to me I couldn't get the player to hang and the playback's good. 2 remaining issues:
- Backing out of playback doesn't update the resume time (To reproduce this one you can enter the player seek to a random place, back out and go back in - the position won't align, the UI element also doesn't update with the new resume time untill you leave the list of stream sources)
|
- Align setResizeMode with macOS bridge (keepaspect/panscan) - Fit: letterbox with subs in margins (sub-use-margins=yes) - Zoom: panscan=1.0 crop to fill, subs stay in visible area - Stretch: keepaspect=no, subs on video - Draw full FBO in Compose Canvas (no Kotlin-side cropping) - Remove unused videoInFbo calculation from Canvas drawing
|
Subtitles fixed ;) But this one
I'm not sure if I understand. Tried what you wrote but everything seems fine |
|
Huh on my end it looks like this. This is the initial resume position from previous playbacks I go into the stream and resume is set correctly I seek to a different position And after going back out it didn't update - it's still at the old resume position (this is the issue) And going back into a stream confirms that But if I do the same thing and go all the way back to the home page then the resume time updates I don't know if that's the same case you tried. |
|
Oh, now I get it, but I think this is a global bug, not related to Linux bridge ;) |
|
Ah alright, I'll probably file a separate bug after this is merged (if it's not there that is). So I think that would be it from the issues that I can see sooo I would say it's an ACK from me on NVIDIA :D |
|
Unless you want me to check out the X11 side again that is |
NVIDIA uses GLX fallback anyway, so GLES was only needed for Intel/AMD where Desktop GL works equally well (Mesa supports both). Desktop GL gives mpv access to full GL extensions and better compatibility. EGL path now requests GL 3.3 Core Profile with fallback to default.
|
@skoruppa I think I may have to stop this direction here. The desktop player already has a shared web UI for controls in Linux should follow the same model too: shared So I don’t want to continue with the Compose Canvas / frame-readback approach. Let’s redirect the Linux work toward the same native libmpv + desktop webview architecture later. For now, enjoy your vacation. We can continue this when you’re back. |
|
Tested on AMD + VAAPI + KDE Plasma — the combo listed as "needs testing". 🎉 Everything works, but I hit a build-blocking bug first. 🐛 Build-blocking bug (blocks any clean checkout)
Root cause: the bundled-runtime dir Fix (in the shared abstract class GenerateNativeRuntimeIndexTask : DefaultTask() {
- @get:InputDirectory
+ @get:InputFiles
+ @get:PathSensitive(PathSensitivity.RELATIVE)
abstract val runtimeDir: DirectoryProperty
@get:OutputFile
abstract val indexFile: RegularFileProperty
@TaskAction
fun generate() {
val dir = runtimeDir.get().asFile
+ dir.mkdirs()
val files = dir
.listFiles { file -> file.isFile && file.name != indexFile.get().asFile.name }
.orEmpty()This keeps incremental input tracking for the Windows path (which has a populated runtime dir) while tolerating the empty case on Linux. With this, the build runs clean. ✅ AMD VAAPI test resultHardware: AMD Radeon RX 6650 XT (navi23) · Mesa 26.1.3 · Arch · KDE Plasma 6 (Wayland) Bridge log on playback:
Happy to test further variants if useful. |
|
Followed up on my own report — pulled the branch, built it, and ran it on AMD RX 6650 XT + KDE Plasma 6 (Wayland). After the build fix above, two playback bugs surfaced. Both now fixed and verified working on this hardware. 1. Timeline click doesn't seek (only dragging works)
@@ -43,6 +43,8 @@ import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.getValue
+import androidx.compose.runtime.mutableStateOf
+import androidx.compose.runtime.remember
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.clip
@@ -496,14 +498,21 @@ private fun ProgressControls(
val audioPainter = appIconPainter(AppIconResource.PlayerAudioFilled)
Column(modifier = modifier) {
+ // Track the latest value from onValueChange: on a single track-click the Slider fires
+ // onValueChange then onValueChangeFinished within one frame, before displayedPositionMs
+ // recomposes — so seeking to displayedPositionMs would use the stale (pre-click) position.
+ val pendingScrubMs = remember { mutableStateOf(displayedPositionMs) }
Slider(
modifier = Modifier
.fillMaxWidth()
.height(metrics.sliderTouchHeight)
.graphicsLayer(scaleY = metrics.sliderScaleY),
value = displayedPositionMs.coerceIn(0L, durationMs).toFloat(),
- onValueChange = { value -> onScrubChange(value.toLong()) },
- onValueChangeFinished = { onScrubFinished(displayedPositionMs.coerceIn(0L, durationMs)) },
+ onValueChange = { value ->
+ pendingScrubMs.value = value.toLong()
+ onScrubChange(value.toLong())
+ },
+ onValueChangeFinished = { onScrubFinished(pendingScrubMs.value.coerceIn(0L, durationMs)) },
valueRange = 0f..durationMs.toFloat(),
)
Row(2. Addon (external URL) subtitles never displayEmbedded subtitles worked, but addon subtitles (OpenSubtitles/Wizdom) did nothing. The addon handler calls only @@ -1293,7 +1293,7 @@ JNIEXPORT void JNICALL Java_com_nuvio_app_features_player_desktop_NativePlayerBr
JNIEXPORT void JNICALL Java_com_nuvio_app_features_player_desktop_NativePlayerBridge_addSubtitleUrl(JNIEnv *env, jobject thiz, jlong hdl, jstring url) {
(void)thiz; CreateTask *task = getTask(hdl); if (!task || !task->mpv || !url) return;
const char *u = (*env)->GetStringUTFChars(env, url, NULL);
- if (u) { const char *cmd[] = {"sub-add", u, "auto", NULL}; mpv_command_async(task->mpv, 0, cmd); (*env)->ReleaseStringUTFChars(env, url, u); }
+ if (u) { const char *cmd[] = {"sub-add", u, "select", NULL}; mpv_command_async(task->mpv, 0, cmd); (*env)->ReleaseStringUTFChars(env, url, u); }
}
JNIEXPORT void JNICALL Java_com_nuvio_app_features_player_desktop_NativePlayerBridge_clearExternalSubtitles(JNIEnv *env, jobject thiz, jlong hdl) {All three fixes are on a branch if useful. Verified on AMD: hardware VAAPI playback, click-to-seek, embedded + addon subtitles all working. |
|
Heads-up on scope: of the fixes I found while testing this PR, only two are Linux-player-specific and belong here — the build fix (runtime-index dir) and the addon-subtitle
So this PR can stay scoped to the two Linux items above. |
Alternative approach: X11 direct mode (
|
|
I've been messing around with Webkit + MPV based player prototype (FeelThePoveR#1) and I must say NVIDIA proprietary truly sucks on Linux. So far I think UI overlay rendering basically can't be done on it under XWayland with webkitgtk 6.0 and 4.1 (modversion 2.52.4 on both - that's what I have in Fedora repos. I've seen that 2.40 modversion may work but haven't tested it), the following attempts were made on both webkitgtk versions:
Doing research on the web confirmed multiple NVIDIA proprietary issues exist for XWayland webkitgtk. Not doing PR against main repos as the implementation is messy and is supposed to serve as inspiration/research before the actual implementation will be done. |
7 participants
PR type
Why
Linux users cannot play video at all - the app shows "Desktop in-app playback is not available yet." This adds native libmpv playback on Linux via JNI, matching what macOS/Windows already have.
Desktop scope
Linux only. Single unified rendering path for all Linux sessions (X11 and Wayland):
EGL GBM offscreen + Compose Canvas overlay:
/dev/dri/renderD*(enumerates render nodes for multi-GPU)MPV_RENDER_PARAM_DRM_DISPLAY_V2- enables VAAPI/nvdec zero-copy DMA-BUF interopglReadPixelsRGBA - JNIrenderFrameBytesconverts to BGRA (Skia N32) into JavaByteArrayLinuxPlayerHostwraps into SkiaImage, Compose Canvas draws viadrawImageRectIf EGL/GBM init fails (no GPU, missing render node, old drivers), falls back to mpv SW rendering (
MPV_RENDER_API_TYPE_SW) withhwdec=auto-copy- GPU decode still works, just with a RAM copy for compositing.Key technical decisions in
player_bridge.c:MPV_RENDER_PARAM_DRM_DISPLAY_V2with render node fd lets mpv open VA display for hw decode interoppthread_condsignaled bympv_render_context_set_update_callbackmpv_render_context_free- crashes Gallium shared pipe_screen. Cache instance (glCache static) and reuse on next createShared code touched minimally: NativePlayerController now accepts a
PlayerHostinterface so it can drive both the AWT host (macOS/Windows) and the new LinuxPlayerHost without duplication. macOS/Windows paths unchanged.Issue or approval
Addresses #27. Loosely based on ideas from #58 but rewritten from scratch for minimal diff.
UI / behavior impact
Policy check
CONTRIBUTING.md.Scope boundaries
Testing
Tested personally:
Tested by volunteer (NVIDIA RTX 4070 SUPER, KDE Plasma 6.7, driver 610.43.02):
Needs additional testing:
Build verification:
./gradlew :composeApp:compileKotlinDesktop- passes clean./gradlew :composeApp:buildLinuxPlayerBridge- produces libplayer_bridge.so (~60KB)Build requirements on Linux:
libmpv-dev(pkg-config: mpv)libEGL,libGL,libgbm(Mesa)gccScreenshots / Video
Not a UI change - player surface renders video frames using existing controls overlay.
Breaking changes
None. All new code gated behind
DesktopHostOs.current == LINUXorhost is LinuxPlayerHostchecks. macOS/Windows paths unchanged.Linked issues
Addresses #27 (Linux native playback).
Loosely based on #58 but rewritten from scratch for minimal diff.
@tapframe could you review this? Would appreciate regression testing on macOS and Windows to confirm nothing broke. AMD GPU and KDE Plasma testing also needed.