Support GGUF

[mliand]

Feature Idea

I have tested the quantization of gguf on SD3 and Flux, and the results are great, smaller memory occupancy and faster speed, hope to support

Existing Solutions

https://github.com/city96/ComfyUI-GGUF

Other

https://github.com/leejet/stable-diffusion.cpp

[JorgeR81]

I hope this lora issue, for GGUF models, in lowvram, is also fixed. 
city96/ComfyUI-GGUF#33

---

EDIT : Fixed !

[arcum42]

Having gguf built in instead of in a custom node would be very useful, as newer models are larger and larger, and needing things like gguf to run on consumer grade hardware more and more. ComfyUI-GGUF is already becoming an essential custom node to install, and anything along that line should generally be folded into the main project.

[polarathene]

GGUF only supports reduced storage and memory when not an LLM model AFAIK, with some issues/workarounds required for applying LoRA due to dequantization upcasting the data type I think?

Those concerns don't apply with some alternative quantization formats IIRC, some of which also quantize the activations which provides faster performance too.

[SRStwo]

GGUF only supports reduced storage and memory when not an LLM model AFAIK, with some issues/workarounds required for applying LoRA due to dequantization upcasting the data type I think?

Those concerns don't apply with some alternative quantization formats IIRC, some of which also quantize the activations which provides faster performance too.

Chroma Q8_0 GGUF runs faster on my secondary system than Chroma FP16. (4070 12 GB VRAM, 32 GB RAM.) The same is true of Flux Dev.

The advantage of Q8_0 is that it is close to FP16 quality while allowing larger models to able to run in lower VRAM cards. Someone claimed the new mixed FP8 is better than Q8_0 but I have never seen any articles with proof of that. What I have seen are articles with data showing Q8_0 is quite close to FP16 in quality.

I doubt I could run Wan 2.2 on this system with FP16 but it runs in Q8_0.

[polarathene]

Chroma Q8_0 GGUF runs faster on my secondary system than Chroma FP16. (4070 12 GB VRAM, 32 GB RAM.) The same is true of Flux Dev.

Are you on Windows? Chroma FP16 isn't going to fit into that VRAM is it?

• You'd notice in the task manager that "Shared" GPU memory would be in use as it spills over into system memory, so that'd naturally make things slower.
• Since the Q8 GGUF would require much less memory, you're not going to get the notable performance degradation when that fits fully into VRAM (or at least the bulk of it by comparison to FP16 weights as there would be less memory transfers slowing it down).

EDIT: If using the city96/ComfyUI-GGUF node, that loads weights into system memory and only transfers to VRAM what is needed on-demand for inference apparently. That may complicate the allocation comparison a bit 😓

On Linux, the proprietary nvidia drivers lack the equivalent support to spill over into system memory (no GTT support like the OSS drivers have), so you'd just get an OOM if unable to satisfy the allocation into VRAM.

NOTE: This difference regarding GPU sharing system memory to "extend" VRAM (like swap/pagefile does for system memory) differs from offloading individual layers of a model to system memory.

Individual layers offloading is more common/straight-forward for LLM models, but gets a bit more complex with projects like ComfyUI that have various tunables for managing memory (the recent default enabled dynamic vram feature is also said to conflict with loaders like GGUF nodes which can potentially make it worse).

---

With offloading concerns aside, GGUF has overhead of dequantizing blocks of tensors into a native type for the GPU to use (such as FP16).

• The GGUF node has this PR which improves on that performance a bit and adds the ability to customize the dtype to dequantize to which could help with runtime performance AFAIK (current default is apparently FP16).
• Alternatives to GGUF that quantize weights into a native dtype for the GPU to use avoid that overhead from dequantizing weights, so those should be able to run faster (assuming no slowdown from offloading involved).

This was something I tried to emphasize earlier regarding GGUF is typically you're only benefiting from the quantized weights (reduced memory usage and reduced disk storage), not quantized activations (GPU dtype used at inference time) that provides additional processing speed. You technically gain speed if the reduced memory requirements better fit into VRAM, but that's conditional as opposed to the inference dtype being a perf win regardless.

This report for example notes BF16 vs Q4 had no notable difference in performance (one user supposedly had sufficient VRAM, and another user with plenty of VRAM observed no difference with the GGUF vs the BF16). Additionally, when you add loras to the mix, this compounds overhead much further slowing down the overall time to get an output.

[SRStwo]

The secondary system is Windows. The primary is Linux. Unfortunately, I am apparently sensitive to infrasound generated by that so I am back to using the very weak 4070 instead of my 5090 in order to reduce sleep issues caused by tinnitus. Bizarrely, I have the 5090 system in my detached garage and it's still not far away enough to not cause problems with my hearing, even when I wear earplugs. The 5090 system seems to make my spouse irritable as well. I'll probably have to sell it and be unhappy with this ultra-weak 4070 system.