Llama cpp support

[DenysAshikhin]

Hi,

Is there or will there be a way to have this work with llama cpp/server? For those of us who are (can) not using linux?

[DavidDohmen]

Not only related to windows/linux. Generally llama.cpp is one of the most powerful inference frameworks used dominantly in the local inference sector. People like me, who are compute hungry and would greatly benefit from these amazing speedups. Great work team!

[eliowang6]

+1

[zlyuancn]

+1

[zlyuancn]

can using quantized models? with llama.cpp . use qwen3.5-9b-q6_k.gguf

[KotPrezesOfficial]

+1

[rihothy-sys]

+1

[Hijiongfeng]

+1