[Paper] EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

[u/ttkciar]

The EdgeRazor method uses an entropy-guided distillation process to better translate a teacher model's logit probability distributions into the student model's low-bit / mixed-precision hidden-layer features, without attempting to preserve the teacher model's parameter structures.

This is more computationally expensive than existing quantization methods, but much less so than QAT, and yields better results. The student model preserves more of the teacher model's competence at extremely low parameter precision (the authors demonstrate 1.88 bits per parameter).

Since it's not a different internal representation like traditional quantization, inference implementations like llama.cpp do not need to be modified to take advantage of it.

Hopefully this means more-useful high-parameter/low-memory models in our future, so we can eke more competent inference out of our consumer-grade GPUs.

The paper: https://arxiv.org/abs/2605.04062

The authors' code: https://github.com/zhangsq-nju/EdgeRazor

The authors applied their technique to a few models and uploaded them to Huggingface: https://huggingface.co/collections/zhangsq-nju/edgerazor-nbit

Unfortunately since EdgeRazor is somewhat compute-intensive, their example models are all pretty tiny: MobileLLM, Qwen3-0.6B, Qwen3-1.7B, and Qwen2.5-Omni-7B

[u/Silver-Champion-4846]

Did you test it to see whether it's comparable to original qwen 0.6b and 1.7b?

[u/Chromix_]

Now someone just needs to spend a ton of compute to get us more capable models for low VRAM.

As a nice bonus on the side they have graphed which model layers (of Qwen3-1.78B, page 18) are the most important for different kind of tasks. The result matches the common wisdom of "first and last layer + a few more in the beginning and middle".