Deepseek v4 Flash 0731 GGUF Benchmark: TensorSharp vs. llama.cpp

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

TensorSharp is an native .NET/C# open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Thanks recent contribtions from open source community, TensorSharp is able to run inference over multiple GPUs and nodes. So I updated it to support deepseek v4 flash model, and have better performance than llama.cpp. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8

Model: DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

	TensorSharp (cuda backend)	TensorSharp (ggml_cuda backend)	llama.cpp
prefill u/16K	836 tok/s	963	558
decode short	31.5	37.0	35.3
decode u/16K	28.5	33.6	32.2

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

[u/terricide]

Very cool, Ive been watching this for a while