Model request: DeepSeek V4 Series

[Orion-zhen]

We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) — both supporting a context length of one million tokens.

It would be great to run DeepSeek V4 with llama.cpp!

[nisparks]

Still need to test a bit more, but I think I have a working GGUF running locally for V4-Flash. TQ2_0 quant is still ~75GB.

Edit: Disregard, it's garbage

The capital of France is Didžiulis
2 + 2 = Roskov

Edit 2: Still going, still garbage, but less crazy
You are a helpful assistant.What is 2+2? -> Sure! Let's the best AI Assistant: Let's compress the assistant: Let's compress the answer: Let's think

or maybe more crazy?

Edit 3:

Getting closer: France is the the capital of France is Paris. France is Paris is Paris is

[nisparks]

Even closer <｜begin▁of▁sentence｜><｜User｜>What is the capital of France? Answer with one word.<｜Assistant｜></think> ->

The capital of France is Paris.
Paris is **France is Paris
 [end of text]

and <｜begin▁of▁sentence｜><｜User｜>What is 2+2? Answer with one number.<｜Assistant｜></think> ->

The answer to the sum of 2 + equals 4.

2 + 2 = 2 + 2 equals

Edit: This is on IQ4 Quants, which are not really optimized for the already mixed precision quants. Essentially took the Mixed Precision -> dequantized, then requantized. resulting in a 153GB IQ4 gguf. Definitely not ready for release yet.

[ms-KuroNeko]

I'm Chinese, and from what I can tell right now, this model itself is extremely unstable. It looks like there are still a lot of issues that haven't been dealt with, and the performance is nowhere near what you get from the online service on DeepSeek's official website.

These preview versions are more of a signal – they're hoping the community can start supporting this kind of implementation early on. It's a very aggressive approach, and it's hugely different from traditional methods.

DeepSeek-V4-Flash was designed from the start so that regular people could run it at home on retired commercial-grade hardware or even consumer-grade hardware. The open-source weights they released are already quantized – labeled as "284BA13B", though the actual model size seems more like a 158B model.

FP4 + FP8 Mixed: MoE expert parameters use FP4 precision; most other parameters use FP8.

A lot of the quantization going on right now not only fails to reduce the model's actual footprint, but actually introduces even more problems. You can see how this completely new and different approach is posing a huge challenge to the traditional LLM community.

All of the above came from my DeepSeek. My English is really poor, so I hope she didn't mistranslate anything.

[nisparks]

I'm not a domain expert here, but shipping mixed quants (FP4+FP8) definitely seems like a new design that we haven't really seen before. We'll probably need to come up with a new quant type which takes a little/big param to maintain the compression. Gave this to my agent with enough prompts to work through the night. Hopefully I'll have some nice results in the morning. We shall see. Definitely more complex than typical new models, but that is also very exciting!

[ms-KuroNeko]

Right now, even when we Chinese people use this Flash version in Chinese (via DeepSeek's official API), it still produces a lot of weird outputs — for example, the model completely ignores instructions, or Claw just runs off and does something else in the middle of a task.

For languages with less training data, the problems with this unfinished preview version are only going to be much, much worse [T^T]

[nisparks]

===== capital =====
The capital of France is Paris. [end of text]



===== math =====
The answer to 2+2 is **4**. [end of text]



===== direct_calc =====
4 [end of text]



===== sky =====
On a clear day, the sky appears **blue**. This is due to a phenomenon called **Rayleigh scattering**, where sunlight is scattered by the gases and particles in Earth's atmosphere. Shorter wavelengths


===== hamlet =====
_Hamlet_ was written by **William Shakespeare**. It is believed to have been written between 1599 and 1601, and it is one of Shakespeare's most famous and frequently performed tragedies


===== opposite =====
The opposite of **hot** is **cold**.

However, depending on the context, other opposites might apply:
- **Temperature:** cold, cool, chilly, freezing
- **Spice

It's getting way more coherent.

[ms-KuroNeko]

So far, in the neighboring vLLM and SGLang communities, there have only been successful runs reported on Hopper and Blackwell architectures. Clearly, getting it to work on regular people's Ampere and Ada GPUs is going to take a lot more work.

DeepSeek's engineers are treating "ordinary people being able to run this model at home" as the benchmark for success. I believe they will help the community address these issues going forward. In any case, this attempt at AI democratization — allowing everyone to equally own their own private AI — is truly worthy of respect.

[nisparks]

My hardware is 3090 (x2) + 2080Ti.

[nisparks]

Probably won't be for about 12+ hours, but I'll post what I have sometime tomorrow. Definitely seems to be stable now locally with the GGUF I created.

[nisparks]

Tests so far:

(llm-env) nisparks@LLM:~/Dev/llama.cpp-deepseek-v4-flash$ cat /tmp/deepseek4-smoke.log
# DeepSeek4 smoke test
bin: /mnt/models/workspace/Dev/llama.cpp-deepseek-v4-flash/build-cuda/bin/llama-completion
model: /mnt/models/gguf/DeepSeek-V4-Flash-IQ4_XS.gguf
ctx: 512
batch: 64
ubatch: 1
threads: 128
ngl: 99

===== capital =====
prompt:
<｜begin▁of▁sentence｜><｜User｜>What is the capital of France? Answer with one word.<｜Assistant｜></think>

response:
Paris [end of text]

expected: /(^|[^[:alpha:]])paris([^[:alpha:]]|$)/

===== math =====
prompt:
<｜begin▁of▁sentence｜><｜User｜>What is 2+2? Answer with one number.<｜Assistant｜></think>

response:
4 [end of text]

expected: /(^|[^0-9])4([^0-9]|$)/

===== hamlet =====
prompt:
<｜begin▁of▁sentence｜><｜User｜>Who wrote Hamlet? Answer with the author name only.<｜Assistant｜></think>

response:
William Shakespeare [end of text]

expected: /shakespeare/

===== sky =====
prompt:
<｜begin▁of▁sentence｜><｜User｜>What color is the sky on a clear day? Answer with one word.<｜Assistant｜></think>

response:
Blue [end of text]

expected: /blue/

===== opposite =====
prompt:
<｜begin▁of▁sentence｜><｜User｜>What is the opposite of hot? Answer with one word.<｜Assistant｜></think>

response:
Cold [end of text]

expected: /cold/

===== primes =====
prompt:
<｜begin▁of▁sentence｜><｜User｜>List the first five prime numbers, separated by commas.<｜Assistant｜></think>

response:
2, 3, 5, 7, 11 [end of text]

expected: /2[^0-9]+3[^0-9]+5[^0-9]+7[^0-9]+11/

===== algebra =====
prompt:
<｜begin▁of▁sentence｜><｜User｜>If x = 3 and y = x + 5, what is y? Answer with one number.<｜Assistant｜></think>

response:
8 [end of text]

expected: /(^|[^0-9])8([^0-9]|$)/

===== long_tail_math =====
prompt:
<｜begin▁of▁sentence｜><｜User｜> alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega
After the filler, answer this: what is 2+2? Answer with one number.<｜Assistant｜></think>

response:
4 [end of text]

expected: /(^|[^0-9])4([^0-9]|$)/

===== long_recall =====
prompt:
<｜begin▁of▁sentence｜><｜User｜>Remember this secret word exactly: tangerine.
 alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega alpha beta gamma delta epsilon zeta eta theta iota kappa lambda mu nu xi omicron pi rho sigma tau upsilon phi chi psi omega
What is the secret word? Answer with only the word.<｜Assistant｜></think>

response:
tangerine [end of text]

expected: /tangerine/

[Lissanro]

I can help testing GGUF quants, or perhaps download the original model files and generate my own GGUF files if given commands to run (though it may take some days for me download). I look forward to testing both the Flash and Pro models, once GGUFs are available (or steps how to make them myself for testing).

I have 1 TB of 8-channel RAM and 96 GB VRAM made of four 3090 GPUs, and for example can run Kimi K2.6 the Q4_X quant reasonably well to use it daily, so I expect I should be able to run DeepSeek V4 Pro 1.6T as well in form of Q4_K_M GGUF or similar quant, even though slower due to larger size and greater count of active parameters.

[tianhanxx1]

我也在等gguf格式，希望早些适配成功。API调用pro和flash版本我都测试过，总体不错，很期待IQ2_M量化。

[heiketu]

先前大模型的Q2量化很容易失智 感觉至少得Q3才能保证可用性 不知道Deepseek现在的原生Q4+Q8会不会好点
我现在用的2x Xeon 8368Q(ES) + 8x DDR4 RECC 8G +8x Optane 128G 尽管非常缓慢 但是至少纯CPU大模型几乎都能跑
注：现在跑的Q3量化GLM5.1 只有2.8Token/s的Gen速度 不知道之后和我两张3090混合推理会咋样 但估计也不会很好
Earlier large models with Q2 quantization were prone to instability. It feels like at least Q3 is needed to ensure usability. Not sure if Deepseek's current native Q4+Q8 is any better.
I'm currently using 2x Xeon 8368Q (ES) + 8x DDR4 RECC 8G + 8x Optane 128G. Although it's very slow, at least almost all pure CPU large models can run.
Note: The Q3-quantized GLM5.1 I'm running now only has a generation speed of 2.8 tokens/s. Not sure how it will perform later with mixed inference on my two 3090s, but I guess it won't be very good either.

[MB7979]

Seems to be a very interesting model via the API in my tests. Hope it can be supported eventually.

[ms-KuroNeko]

先前大模型的Q2量化很容易失智 感觉至少得Q3才能保证可用性 不知道Deepseek现在的原生Q4+Q8会不会好点 我现在用的2x Xeon 8368Q(ES) + 8x DDR4 RECC 8G +8x Optane 128G 尽管非常缓慢 但是至少纯CPU大模型几乎都能跑 注：现在跑的Q3量化GLM5.1 只有2.8Token/s的Gen速度 不知道之后和我两张3090混合推理会咋样 但估计也不会很好

傲腾意义不大，Q2效果如何损失多大主要看模型本身设计，有些模型的Q2也能达到可以接受的性能
你现在的当务之急是寻找一个更小的模型，牺牲部分方向上的能力水平降低参数量，而不是接触更多的超大型LLM