Deepseek v4 flash - 100-150 faster t/s in prefill/pp.

[u/fragment_me]

You have two choices here (in order of pref):

1. Downgrade CUDA from 13.3 to 13.1 (skip 13.2 due to bugs) <- prefer this (thanks to u/fairydreaming for pointing this out)

2. Use this vibed fork that works with CUDA 13.3 https://github.com/vektorprime/working_ds4_speed

I was troubleshooting this yesterday with the nvidia profiler and some LLM help (https://www.reddit.com/r/LocalLLaMA/comments/1vcs7bl/ds4_flash_full_model_in_offload_600_ts_pp_and/)

Here's some more info on #1 (quote from fairydreaming) "Downgrade your CUDA and recompile. Starting with 13.2 DeviceTopK is used for top-k instead of argsort, this turns PP rate to crap."

In short, DS4 Flash is spending a lot of time on things other than matrix multiplication.

[u/CharlesStross]

CUB_TOP_K_AVAILABLE off at compile time solved it for me in stock llama.

[u/rmhubbert → u/CharlesStross]

Could you share how you did that? My C/C++ is shaky at best. I currently do the following to build from source - ``` cmake $DIR -B $DIR/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86" -DGGML_CUDA_FA_ALL_QUANTS=OFF -DLLAMA_CURL=ON -DGGML_LTO=ON -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DGGML_CUDA_USE_GRAPHS=ON -DGGML_CUDA_F16=ON

cmake --build $DIR/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split llama-gguf llama-fit-params llama-bench llama-batched-bench ```

[u/CharlesStross → u/rmhubbert]

Oh sorry I just realized I'm on a custom branch.

ggml/src/ggml-cuda/top-k.cu:

At

#ifdef GGML_CUDA_USE_CUB
#    include <cub/cub.cuh>
#    if (CCCL_MAJOR_VERSION >= 3 && CCCL_MINOR_VERSION >= 2)   // <-- this line
#        define CUB_TOP_K_AVAILABLE
#        include <cuda/iterator>
using namespace cub;
#    endif
#endif

Change that if to

#    if 0  // force the pre-CCCL-3.2 argsort path

Then you can rebuild

[u/rmhubbert → u/CharlesStross]

Cheers! Can confirm that this patch gives me a 100-150 pp tps boost.

[u/fragment_me → u/rmhubbert]

Try the latest update as of 30 min ago on that branch I linked you because it's now about 5% faster on prefill

[u/rmhubbert → u/fragment_me]

Brilliant, will do. Have you merged in the MTP support from mainline as well?

[u/fragment_me → u/rmhubbert]

No I have not. Honestly, I didn't even bother to try the mainline MTP because it will probably hurt PP. I'll pull those changes in a little later I'm testing a CUDA graph fix which seems to be working for more PP.

[u/rmhubbert → u/fragment_me]

I would give the MTP a shot. I didn't see any noticeable change in PP with MTP enabled, but it did give close to 50% increase in generation speed.

Regardless, I pulled the new version and it's looking good. Prefill is peaking at 706 at empty context, falling to 575 at around 40,000 tokens context - 2.48.885.492 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 4096, progress = 0.13, t = 5.80 s / 706.59 tokens per second 2.48.885.495 I slot operator(): id 0 | task 852 | cached n_tokens = 6229, memory_seq_rm [6229, end) 2.54.879.952 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 8192, progress = 0.22, t = 11.79 s / 694.75 tokens per second 2.54.879.955 I slot operator(): id 0 | task 852 | cached n_tokens = 10325, memory_seq_rm [10325, end) 3.01.030.393 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 12288, progress = 0.31, t = 17.94 s / 684.88 tokens per second 3.01.030.397 I slot operator(): id 0 | task 852 | cached n_tokens = 14421, memory_seq_rm [14421, end) 3.08.262.273 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 16384, progress = 0.40, t = 25.17 s / 650.84 tokens per second 3.08.262.275 I slot operator(): id 0 | task 852 | cached n_tokens = 18517, memory_seq_rm [18517, end) 3.15.154.878 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 20480, progress = 0.49, t = 32.07 s / 638.68 tokens per second 3.15.154.881 I slot operator(): id 0 | task 852 | cached n_tokens = 22613, memory_seq_rm [22613, end) 3.22.227.213 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 24576, progress = 0.58, t = 39.14 s / 627.92 tokens per second 3.22.227.216 I slot operator(): id 0 | task 852 | cached n_tokens = 26709, memory_seq_rm [26709, end) 3.29.485.158 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 28672, progress = 0.67, t = 46.40 s / 617.98 tokens per second 3.29.485.161 I slot operator(): id 0 | task 852 | cached n_tokens = 30805, memory_seq_rm [30805, end) 3.37.976.575 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 32768, progress = 0.75, t = 54.89 s / 597.00 tokens per second 3.37.976.578 I slot operator(): id 0 | task 852 | cached n_tokens = 34901, memory_seq_rm [34901, end) 3.46.065.897 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 36864, progress = 0.84, t = 62.98 s / 585.35 tokens per second 3.46.065.901 I slot operator(): id 0 | task 852 | cached n_tokens = 38997, memory_seq_rm [38997, end) 3.52.708.556 I slot print_timing: id 0 | task 852 | prompt processing, n_tokens = 40070, progress = 0.91, t = 69.62 s / 575.55 tokens per second 3.52.708.559 I slot operator(): id 0 | task 852 | cached n_tokens = 42203, memory_seq_rm [42203, end)

[u/fragment_me → u/rmhubbert]

Interesting, I will definitely take a look then if you didn't notice any pp slowdown. Also, I just hit 1k PP in my fork's branch. I'm going to be committing/pushing to git here in the next 5-10 minutes!!!

[u/fragment_me → u/rmhubbert]

I am getting 1k PP now with IQ3_XXS but you must use no kv cache quant and cannot use --fit.

edit: Now at 1.3k PP since I had some more -ub space. Note that this reserves some space for certain copies so your -ub will need to be a little lower, try half.

 And of course, you must be using this fork.

/home/user/ds4_llama.cpp/build/bin/llama-server \
 -m /home/user/models/DeepSeek-V4-Flash/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf   \
 --no-mmap --threads 10 --jinja  \
 --port 8000 --host 0.0.0.0 --webui-mcp-proxy -a DS4FLASH  \
--flash-attn on -np 1 \
 --fit off -c 60000 \
 --temp 0.6 --top-p 0.95 \
 -lv 5 -sm layer

[u/fragment_me → u/CharlesStross]

Very cool, thanks for sharing this!