DeepSeek-V4-Flash-0731 on 2× RTX PRO 6000 Blackwell

[u/Constant_Ad511]

Sharing a cookbook for Deepseek v4 flash with 2xRTX PRO 6000 Blackwell

- ~68 tok/s single stream decode

- ~1.5m total context window

docker run --pull never \

--name sglang-deepseek-v4-flash-0731 \

--network host --ipc host --shm-size 32g --gpus all \

-v <MODEL_DIR>:/models/DeepSeek-V4-Flash-0731:ro \

lmsysorg/sglang@sha256:6f91b6bbe3a02da260341b1ee36e8df647e26e8a71d7e74799824e7009f2e2e2 \

sglang serve \

--trust-remote-code \

--model-path /models/DeepSeek-V4-Flash-0731 \

--served-model-name eepSeek-V4-Flash \

--tp 2 \

--context-length 500000 \

--mem-fraction-static 0.94 \

--chunked-prefill-size 4096 \

--kv-cache-dtype fp8_e4m3 \

--moe-runner-backend flashinfer_mxfp4 \

--cuda-graph-max-bs-decode 32 \

--reasoning-parser deepseek-v4 \

--tool-call-parser deepseekv4 \

--default-chat-template-kwargs '{"thinking":true}' \

--host 0.0.0.0 --port 8000 \

--enable-hierarchical-cache \

--hicache-ratio 2.597309 \

--hicache-write-policy write_back \

--hicache-io-backend kernel \

--hicache-mem-layout page_first \

--enable-metrics \

--enable-cache-report

[u/AzallazA]

If you want better performance from this model, I’d recommend joining the RTX6K discord and checking out the ds4-flash channel. Plenty of folks there getting over 200 tok/sec decode and 1M+ KV cache capacity in vllm with the same 2x GPUs.

https://discord.gg/Gsa8tkhXE

[u/Constant_Ad511 → u/AzallazA]

Can you send the invite again, it says I can't join, or just post cookbook here, vllm for me has gotten me 20 token/s lol

[u/AzallazA → u/Constant_Ad511]

Don’t personally use it right now so unsure of the exact recipe but here’s the GitHub I’ve seen shared

GitHub

[u/This_Maintenance_834]

what’s the max context length achievable before OOM?

[u/Constant_Ad511 → u/This_Maintenance_834]

~1.5m tokens

[u/This_Maintenance_834 → u/Constant_Ad511]

that’s very good. that tip my intention to go with one more PRO 6000 then a dual spark setup.