CUDA Support

[chrishayuk]

At the moment, i haven't added any CUDA support to the repo.. that would be a very welcome addition.
And i'm very unlikely to do this anytime soon.
I think the larql-compute crate is stable enough just now, so good time for it to be included

[metavacua]

I make no promises, but I'll probably be experimenting with CUDA, OpenCL, Vulkan, and WebGPU as part of trying to get this all to run across Ubuntu, ChromeOS, and AndroidOS.

[chrishayuk]

@metavacua that would be amazing

[plam40]

Hi Chris, you are superstar - not for being super smart ( many smart people ) ... but for opening and releasing this for free to the world !!!! Thank you !
I have no proper coding skills, but I am very tinkery, and I have been pushing the local Qwen3.6 crunching on CUDA part for the past few days. I have no idea how bad the code is, will do a review pass with Claude at the end and will upload it to a repo so a more knowledgeable person can check it if they want to and if good - eventually - you can decide merging.

[metavacua]

@chrishayuk I figured that getting the system to build on smartphones is pretty high priority given your stated goals. The primary appeal of your project to me and my research objectives is in the possibility of having extremely lightweight language models running on extremely lightweight hardware.

WebGPU and WASM seem like they have a particular synergy with Rust.

I've been looking at wgpu, and I think that if wgpu is well supported then specializing into Vulkan, CUDA, OpenCL, and similar hardware acceleration features will be relatively painless as compared to trying to implement CUDA and Metal first then agglomerating up to WebGPU.

It looks like MoltenVK is a possible bridge technology between Metal and general open source *nix GPU acceleration via Vulkan.

OpenCL seems lower priority.

I am contemplating forking your main repo multiple times starting with a per-OS forking structure then creating durable per-feature branches within each of them; this will allow lighter weight and more modular CI/CD.

Organizationally, it makes sense to me to decompose the crate and workspace structure, but I'm only starting to approach thinking about that problem; I have vague thoughts about Gists or per-crate/per-workspace branches. Chances are you'll be a lot faster than me about that.

[chrishayuk]

@metavacua yeah, running on smartphones is a high priority for me as you can guess. i think i see the path forward to getting large models running on tiny hardware. if i'm honest.. i'd ideally like to break the dependence on gpu completely.. i managed to do this with the FFN (which is now cpu based)... i think the same can be achieved with attention. i developed the metal shaders just so people don't shout at me for the tokens per second being slow.. but ideally i'd like rid of it completely one day...

with that being said, my goals aside.. i think having a healthy library of support for gpu (cuda, vulkan, webgpu) as well as wasm, would definitely be appreciated. i think there is a lot of space here for folks to innovate. i'm open to all suggests for refactoring. i've tried to keep everything shader related in larql-compute.. i can easily see that splitting to larql-compute-metal , larql-compute-vulkan would be sensible. my recent changes have introduced a bunch of traits to make that easier. so i'm easy and open with those crates splitting.. and definitely any contributions are welcome.

i'm probably gonna be focused on attention and the distributed grid for a bit.. which is why i'm unlikely to look at CUDA etc. so i'm pretty sure you can have some fun in this area without any intereference from me

[codeninja]

@chrishayuk I fully applaud the push for mobile, however, as a backend and agentic engineer I automate backend processes and I can attest to the need to have powerful models behind your firewall for organizational tasks.

I see value in the project in that I can take an enterprise codebase, transpose it into a semantic graph, and stuff it in a local model's knowledge base without complex retraining. And then, to be able to distribute the KG within the infrastructure.

Tokens per second is the bottleneck and for that we need CUDA and GPU support badly. And without it, it constrains our ability to scale.

My target platform is Ubuntu 24 in AWS & GCP. Just to give you a different extreme to consider.

@metavacua I would be more than happy to contribute to the CUDA efforts if some have already been started.

[chrishayuk]

@codeninja @metavacua yeah, cuda is a definite need... i've just finished the massive refactor that i did last week, which seperates out larql-compute and larql-compute-metal combined with a trait model that keeps the independence. I think that refactor now enables you to do larql-compute-cuda larql-compute-vulkan etc etc etc without intermingling... i needed to get that right to detangle things.. so i think this is the safe base to build upon...

[metavacua]

@codeninja @chrishayuk As I said, I make no promises. Bluntly, CUDA support is moving down the list of my priorities and is not in my work queue for the immediate future; my development machine lacks a dedicated GPU, so my personal value proposition is more towards reducing LARQL to something that doesn't need a GPU and producing synthetic domain-specific language models that can run on 8 GBs or less of RAM.

I suspect--but haven't looked into--we could get a lot of value from identifying where raw outputs need to be buffered and other blocking behaviors that cause the CPU, RAM, GPU, and Storage to have to wait on the monitor microcontroller. This is a common problem for CLI that use unbuffered stdout and stderr, and I would not be surprised if it is part of the per-token latency. This is where my mind starts to go when we make optimization priority over raw effectiveness of expressive power.

Most immediately, a synthetic language model that corresponds to the LARQL code base itself--or a minimum viable subgraph--is an obvious major benefit; this places an adaptor for GraphQL for GitHub and similar repositories as a priority technology for development. As Chris has demonstrated, the knowledge base of the LLM can be hosted remotely, and we can just run attention on a smartphone.

I've been surveying coding agent harnesses that could dovetail with LARQL, and I've tentatively picked out Tabby for this purpose; a major milestone will occur when we have a LARQL language model strapped into an effective coding harness attached to the LARQL codebase with GraphQL support.

I reviewed some of the other forks, and CUDA support is one of the more common features that they have added or are working on. https://github.com/ianblenke/larql has some work on CUDA that I've been looking at.

It seems like the basics for that are something that would be a matter of cloning a branch, doing some adjustments and testing, and making a PR back to this repo.

My focus is on WebGPU support as that is consistent with my focus on getting an in-browser subset of LARQL working via WASM32 support; the process of converting the existing crates to target wasm32-unknown-unknown for building and testing in Node and Firefox reduces the code base down to a much cleaner architecture that separates OS/IO from raw compute and rather critically reduces to something that should more easily parallelize on CPU and GPU. Adding CUDA support and GPU support for all operating systems before doing that refactor is MASSIVE technical debt whereas doing that refactor before the matrix of OS and GPU support makes adding additional OS and GPU support relatively trivial.

For me, CUDA support is behind Vulkan support and WebGPU in large part because I don't have hardware with which to test CUDA.

I'm focused on laying foundations for WebGPU first in my own fork in large part because wgpu is a low-hanging fruit with a Rust project like this and immediately gives us Vulkan support with seemingly mild configuration changes. Also, WebGPU support and WASM total conversion allows us to interact more directly with the Apache 2.0 project, WebLLM, that I've identified as relatively ideal for adaptation to LARQL particularly for a front end, and I suspect that we could get the biggest GPU support bang for effort by making an LARQL to MLC/TVM adapter.

I haven't audited the CUDA licensing situation, but I suspect it is not favorable for an Apache 2.0 project, and my efforts are moving towards a CC-BY-SA v4.0 and AGPLv3 licensing suite for the state-of-the-art work that I'll be doing in my fork after I get the Apache 2.0 licensed work for basic OS/IO done, so I know that NVidia is not my friend when it comes to coding projects.

Rough shape of the problem:

To build a GitHub Actions matrix testing workflow for GPU-accelerated computing across CUDA, OpenCL, Vulkan, and WebGPU, you must balance differing hardware requirements, driver installations, and runtime environments. [1]
The primary challenge in standard cloud CI environments (like standard GitHub-hosted runners) is the lack of a physical GPU, which restricts you to software emulation or specialized runner configurations. [2]

---

Strategy Matrix Overview

API [3, 4, 5, 6, 7]	CI Testing Capability	Implementation Strategy
CUDA	High (Compile-only / CPU Simulation)	Install CUDA Toolkit via GitHub Action; use Ocelot or NVVM for basic software emulation, or use a Self-Hosted Runner with an NVIDIA GPU for true hardware testing.
OpenCL	High (Full Emulation)	Use PoCL (Portable Computing Language) or Mesa Clover for pure CPU execution.
Vulkan	High (Full Emulation)	Use SwiftShader (Google) or Lavapipe (Mesa) as a software Vulkan driver.
WebGPU	High (Headless Testing)	Run headless testing inside a browser (e.g., Playwright/Puppeteer with flags) or use native backends like wgpu-native / Dawn running over SwiftShader.

---

Implementation Best Practices## 🏋️ Real Hardware (For Complete Performance Metrics)

Software emulators verify compute logic, but they do not capture hardware bottlenecks, concurrency, or performance regressions. If you require matrix multiplication benchmarking on physical chips, you must transition to Self-Hosted Runners. [1, 8, 9]

1. Link an on-premise machine containing an NVIDIA GPU to GitHub Actions.
2. Modify your workflow file to target those runners directly

🌐 WebGPU Native Cross-Compiling

If your WebGPU application is built natively (e.g., using Rust's [wgpu](https://github.com/gfx-rs/wgpu) or Google's Dawn), you can bundle WebGPU validation alongside Vulkan. On Linux, wgpu automatically targets Vulkan down-level features if it runs over a CPU driver like Lavapipe. [8, 10, 11, 12, 13]

⚠️ Conformance Test Suite (CTS)

When testing a WebGPU framework or application layout, consider hooking into the official [W3C WebGPU Conformance Test Suite](https://github.com/gpuweb/cts). It contains automated hooks tailored for continuous integration platforms to run headless tests safely inside sandboxed CI environments. [14, 15]

[1] [https://github.com](https://github.com/software-mansion-labs/webgpu-workshop/actions)
[2] [https://medium.com](https://medium.com/codex/build-a-custom-ci-bot-integrate-github-apps-with-lightning-for-fast-scalable-mlops-workflows-9a0f99b8a815)
[3] [https://forums.developer.nvidia.com](https://forums.developer.nvidia.com/t/gpu-semi-diagnostic-tool-s-cuda-memtest-wooo/14770)
[4] [https://skywork.ai](https://skywork.ai/skypage/en/gpu-shader-sandbox-ai/1980565837505355776)
[5] [https://github.com](https://github.com/JuliaGPU/Vulkan.jl)
[6] [https://github.com](https://github.com/google/graphicsfuzz/blob/master/docs/finding-a-vulkan-driver-bug-using-spirv-fuzz.md)
[7] [https://github.com](https://github.com/Liquid4All/cookbook/blob/main/examples/vl-webgpu-demo/README.md)
[8] [https://github.com](pygfx/wgpu-py#47)
[9] [https://www.youtube.com](https://www.youtube.com/watch?v=rVq-SCNyxVc)
[10] [https://github.com](https://github.com/gfx-rs/wgpu)
[11] [https://rustify.rs](https://rustify.rs/articles/rust-gpu-computing-wgpu-2026)
[12] [https://github.com](https://github.com/eliemichel/WebGPU-distribution)
[13] [https://maplibre.org](https://maplibre.org/maplibre-native/docs/book/platforms/android/android-webgpu.html)
[14] [https://github.com](https://github.com/gpuweb/cts)
[15] [https://github.com](https://github.com/gpuweb/gpuweb/wiki/GPU-Web-2024-10-F2F)