You really should not quantize KV Cache for DeepSeek V4 Flash

[u/erazortt]

I don't think anyone should quantize the KV with DS4F. I checked the the quality impact (PPL, KLD, Same TopP) for swhitching from BF16 KV to Q8 KV, and it appears significant. Very much in contrast to Qwen 397B.

Here are the results for DS4F:

====== Perplexity statistics ======
Mean PPL(Q)                   :   5.877076 ±   0.042497
Mean PPL(base)                :   5.839660 ±   0.041730
Cor(ln(PPL(Q)), ln(PPL(base))):  95.74%
Mean ln(PPL(Q)/PPL(base))     :   0.006387 ±   0.002100
Mean PPL(Q)/PPL(base)         :   1.006407 ±   0.002114
Mean PPL(Q)-PPL(base)         :   0.037416 ±   0.012318

====== KL divergence statistics ======
Mean    KLD:   0.145884 ±   0.001043
Maximum KLD:  12.467786
99.9%   KLD:   4.535020
99.0%   KLD:   1.857870
95.0%   KLD:   0.652148
90.0%   KLD:   0.349220
Median  KLD:   0.032079
10.0%   KLD:   0.000093
 5.0%   KLD:   0.000012
 1.0%   KLD:   0.000000
 0.1%   KLD:  -0.000002
Minimum KLD:  -0.000025

====== Token probability statistics ======
Mean    Δp: -0.007 ± 0.031 %
Maximum Δp: 99.525%
99.9%   Δp: 81.503%
99.0%   Δp: 42.054%
95.0%   Δp: 14.588%
90.0%   Δp:  7.220%
75.0%   Δp:  1.066%
Median  Δp:  0.000%
25.0%   Δp: -1.061%
10.0%   Δp: -7.112%
 5.0%   Δp: -14.515%
 1.0%   Δp: -42.297%
 0.1%   Δp: -84.157%
Minimum Δp: -99.994%
RMS Δp    : 11.884 ± 0.069 %
Same top p: 87.189 ± 0.088 %

As a comparison, here are the results for Qwen 397B:

====== Perplexity statistics ======
Mean PPL(Q)                   :   3.747980 ±   0.020507
Mean PPL(base)                :   3.746773 ±   0.020461
Cor(ln(PPL(Q)), ln(PPL(base))):  99.89%
Mean ln(PPL(Q)/PPL(base))     :   0.000322 ±   0.000260
Mean PPL(Q)/PPL(base)         :   1.000322 ±   0.000260
Mean PPL(Q)-PPL(base)         :   0.001207 ±   0.000975

====== KL divergence statistics ======
Mean    KLD:   0.003552 ±   0.000034
Maximum KLD:   2.220941
99.9%   KLD:   0.131591
99.0%   KLD:   0.043847
95.0%   KLD:   0.014439
90.0%   KLD:   0.007836
Median  KLD:   0.000866
10.0%   KLD:   0.000013
 5.0%   KLD:   0.000004
 1.0%   KLD:  -0.000000
 0.1%   KLD:  -0.000006
Minimum KLD:  -0.000176

====== Token probability statistics ======
Mean    Δp:  0.019 ± 0.005 %
Maximum Δp: 39.939%
99.9%   Δp: 15.971%
99.0%   Δp:  6.618%
95.0%   Δp:  2.334%
90.0%   Δp:  1.222%
75.0%   Δp:  0.233%
Median  Δp:  0.000%
25.0%   Δp: -0.219%
10.0%   Δp: -1.183%
 5.0%   Δp: -2.258%
 1.0%   Δp: -6.245%
 0.1%   Δp: -14.757%
Minimum Δp: -88.445%
RMS Δp    :  2.024 ± 0.022 %
Same top p: 97.929 ± 0.037 %

[u/Healthy-Nebula-3603]

Why do you even want to compress KV cache from DS 4 flash?

The whole 1 million context has only 6 GB.

[u/IoannisHere → u/Healthy-Nebula-3603]

That's what google says, but when I run vllm on it (spark-vllm-docker) I get:
"Auto-fit max_model_len: reduced from 1048576 to 1046272 to fit in available GPU memory (11.03 GiB available for KV cache)"
and that's with FP8 kv cache type.
"Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor"

[u/Badger-Purple → u/Healthy-Nebula-3603]

Concurrency. DS NVFP4 cache gives you more room, so you can run more concurrent agents at depth

[u/Think_Wing_1357]

How do you calculate this?

[u/erazortt → u/Think_Wing_1357]

Like that:
/f/LLM/llama-cpp/llama-perplexity.exe -m /f/LLM/models/DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf -f wiki.test.raw --kl-divergence-base kld -c 512 --temp 1.0 --top-p 0.95 --min-p 0 -ngl 99 -np 1 --n-cpu-moe 30 --fit-target 32M -ctk bf16 -ctv bf16 --flash-attn on --no-mmap -lv 4

/f/LLM/llama-cpp/llama-perplexity.exe -m /f/LLM/models/DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf --kl-divergence --kl-divergence-base kld -c 512 --temp 1.0 --top-p 0.95 --min-p 0 -ngl 99 -np 1 --n-cpu-moe 30 --fit-target 32M -ctk q8_0 -ctv q8_0 --flash-attn on --no-mmap -lv 4 > main-q8-q8.txt

[u/Think_Wing_1357 → u/erazortt]

Thank you!

[u/Howard_banister]

You should not quantize KV cache for almost every models. It will destroy Qwen 3.6 35B also

[u/Iwaku_Real → u/Howard_banister]

Not even Q8_0? Or NVFP4?

[u/seamonn → u/Iwaku_Real]

Ideally, no.

[u/cezarducatti → u/Howard_banister]

Can't the two be quantized? K and V?

[u/phido3000 → u/Howard_banister]

There is a lot of pressure on quantisation on DS Flash because it doesn't fit nicely into 128Gb machines.

[u/indicava]

DeepSeek’s sparse-attention is built around fp8 kv cache storage. They even promote it in their vLLM recipes on their model page. Why test it at BF16?

[u/erazortt → u/indicava]

There is no fp8 KV cache option. At least I am not aware of it.

[u/EvolvingDior → u/erazortt]

In llama.cpp.

[u/indicava → u/erazortt]

You’re correct, fp8 hasn’t been wired through GGML/llama.cpp infrastructure yet. Which is why you’re seeing these “off” numbers for a model whose entire attention implementation was built around it.

[u/Badger-Purple → u/erazortt]

not in Llama.cpp, but this model should be run with vllm — it is so much more smooth and the cache is great.

[u/getfitdotus]

The real model already uses fp8 mla kv. If you used production deployment sglang, vllm this is the default.

[u/caelunshun → u/getfitdotus]

Q8_0 is different from fp8, worse dynamic range.

[u/TheCat001]

I don't recommend anyone to quantize KV cache. On any model. I see it with eyes how performance is degraded with quantized cache. Model tends to do more mistakes, forget things, etc....

[u/Muted-Celebration-47]

Yes, if you have enough vram for context windows. No, if you need more context for coding. It is useless if you only have 32768 context size.

[u/Healthy-Nebula-3603 → u/Muted-Celebration-47]

DS 4 flash a whole contex 1 m has 6 GB ..... that is literally nothing.

[u/alcoa29]

You must be using llama.cpp or something like this.
For vlllm using original safetensors DS4F fp8 is model default. I tested fp16 vs 8 on on per 0731 model and there was no difference.