CUDA backend hard-crashes on RTX 5050 Blackwell during backend load

[leehack]

Context from leehack/llamadart#222:

While testing listGpuDevices(probeBackends: [GpuBackend.cuda]), @elana-voss reported that the CUDA backend hard-crashes on an NVIDIA GeForce RTX 5050 Laptop GPU (Blackwell). Vulkan on the same Windows laptop enumerated both the AMD Radeon 860M iGPU and RTX 5050 dGPU correctly, and Metal enumeration worked on macOS.

This appears separate from the Dart listGpuDevices enumeration change: the crash happens when the native CUDA backend module loads, before device enumeration runs. The relevant path is ggml_backend_load("cuda") initializing the upstream llama.cpp CUDA backend; CUDA runtime calls in that initializer can abort the process via GGML_ABORT.

Could you provide a few more details so we can isolate whether this is a llamadart-native packaging issue, an upstream llama.cpp CUDA issue, or a driver/runtime issue?

• Native stderr/stdout from the crash, especially any CUDA error or GGML_ABORT lines.
• Windows Event Viewer / crash dialog details if available.
• NVIDIA driver version and nvidia-smi output.
• The llamadart-native tag / package version used.
• Whether the same machine can reproduce with upstream llama.cpp directly, for example llama-cli --list-devices or any minimal upstream command that loads the CUDA backend.
• Whether updating the NVIDIA driver changes the behavior.

Until we have that, the expected workaround is to avoid explicit CUDA probing on this hardware and use the Vulkan backend path, which was reported to enumerate correctly.

[elana-voss]

Can't reproduce the CUDA crash anymore. On the current driver (610.62) and recent llama.cpp (b9700), the CUDA backend loads, lists the RTX 5050, and runs a model fully on the GPU with no GGML_ABORT or CUDA error.

For context, the crash I originally filed (PR #22750 and the NVIDIA forum post) was actually the Vulkan cooperative-matrix one, not CUDA.

[leehack]

Thanks, that helps. Could you confirm whether the crash still reproduces through the current llamadart path, for example listGpuDevices(probeBackends: [GpuBackend.cuda]), and which llamadart-native tag/package version is being used?

If CUDA works both in upstream llama.cpp and in the current llamadart-native artifact, we can treat this as resolved or driver/upstream-fixed. If it only works with upstream b9700 but still crashes through llamadart-native, then this is likely a native artifact bump/rebuild issue on our side.

[elana-voss]

Tested through llamadart main with llamadart-native b9694, calling listGpuDevices(probeBackends: [GpuBackend.cuda]) with cuda enabled in the build (llamadart_native_backends: [cpu, vulkan, cuda]).

No crash. The CPU backend loads, then ggml-cuda.dll fails to load (load_backend: failed to load …ggml-cuda.dll:, empty reason) and it returns 0 devices — cudart64_12/cublas64_12/cublasLt64_12 are all deployed alongside it.

Raw upstream llama.cpp CUDA (b9700) runs fine on this RTX 5050, so this looks like the packaged ggml-cuda.dll not loading through llamadart-native — the native-artifact branch you mentioned. I didn't dig into the load failure itself.

[leehack]

Thanks — that narrows it down.

Given that upstream llama.cpp b9700 now loads CUDA on the same RTX 5050, and llamadart-native b9694 includes cudart64_12.dll, cublas64_12.dll, and cublasLt64_12.dll beside the CUDA backend, I think this is no longer the original Blackwell/CUDA crash. It looks like our llamadart Windows dynamic backend-load path is not resolving the CUDA redistributable DLLs when the backend is loaded from the native-asset bundle directory.

I opened a package-side fix here:

• fix: load Windows backend modules with altered search path llamadart#227

Could you test against that PR branch directly?

In your test app/package, temporarily override llamadart like this:

dependency_overrides:
  llamadart:
    git:
      url: https://github.com/leehack/llamadart.git
      ref: watcher/windows-cuda-dll-preload

Then force a fresh native-assets build from the app directory:

rm -rf .dart_tool build
dart pub get
# or, for Flutter:
# flutter pub get

Please keep the same CUDA-enabled native config you used before, e.g. llamadart_native_backends: [cpu, vulkan, cuda], and retry:

await LlamaEngine.listGpuDevices(
  probeBackends: [GpuBackend.cuda],
);

Ideally, test without manually adding the native bundle directory to PATH. If this PR works, CUDA enumeration should find the RTX 5050 through the normal llamadart path without needing a PATH workaround.

If it still returns 0 devices, could you share the updated backend load log from the PR branch? The PR now preloads the bundled CUDA redistributable DLLs by absolute path before loading the CUDA backend module, so the next failure mode should be more specific.

[leehack]

Thanks for the follow-up validation. leehack/llamadart#227 has now been merged.

The confirmed failure mode was package-side Windows backend loading, not the original upstream CUDA crash: ggml-cuda.dll and the CUDA redistributable DLLs were present in the native asset bundle, but the Dart package load path was not resolving the backend-owned transitive DLLs reliably.

The merged fix preloads the Windows backend module with LoadLibraryExW(..., LOAD_WITH_ALTERED_SEARCH_PATH) before handing the same backend path to llama.cpp ggml_backend_load().

Validation on the PR head confirmed the fix on the RTX 5050 with the native bundle directory not on PATH: listGpuDevices(probeBackends: [GpuBackend.cuda]) loaded ggml-cuda.dll from .dart_tool/lib and enumerated the RTX 5050 successfully.

Closing this as fixed by leehack/llamadart#227 / merge commit d8244024. If this still reproduces with the merged package path or the next release, please reopen or comment with the exact llamadart / llamadart-native versions and the backend load log.