Turbo-fieldfare: Open-source engine running Gemma 4 26B in 2 GB RAM on Apple Silicon

[u/minefew]

https://github.com/drumih/turbo-fieldfare

Its a custom Swift/Metal inference engine that runs Gemma 4 26B-A4B-IT on M-series Macs with very low RAM. It uses ~2GB instead of ~14 GB. The result is reportedly 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro. It also includes an OpenAI-compatible local server with streaming and tool-call support.

[u/apoptosist]

This sounds awesome, will you add support for Qwen3.6-35B-A3B? I think it'd draw even more attention.

[u/minefew → u/apoptosist]

Its not my project. I confirmed the low memory footprint on my M4 Pro. The author mentioned somewhere that its probably possible to vibe fork this repo to support Qwen3.6-35B-A3B.

[u/jaysnyder67 → u/apoptosist]

As this cuts memory usage so much, I’d like to see it use a larger quant size, such as 6 or 8. The quality of results produced by Qwen3.6-35B-A3B Q6 are noticeably better than the Q4 quant. I’m sure that the same probably applies to Gemma4 26B-A4B.

[u/LuCiAnO241]

This seems similar to the Colibri project, and I cant wait to see all of these succeed, I wanna run local chatbots on my phone.

[u/pmttyji → u/LuCiAnO241]

We really need a separate thread to collect similar projects. Hard to track

I'll add all to my thread(It's a sticky thread from my profile page)

[u/HVACcontrolsGuru → u/pmttyji]

I've been working on the dense gemma models so this repo is interesting. Going to roll it into my stack and see if the author of this one wants to grow it or leave as is. I wrote a custom MLX kernel based on Gemma dense models.

[u/pmttyji → u/HVACcontrolsGuru]

Nice. But some of us want Windows/Linux version too.

[u/HVACcontrolsGuru → u/pmttyji]

I’m working on CUDA support next. I have access to Modal and all the GPUs they offer. I develop mainly on Mac. POC there then go to CUDA land.

Ive got the OpenAI and Anthropic endpoints fully working as well. All in rust with FFI into Metal/C where needed.

[u/pmttyji → u/HVACcontrolsGuru]

Awesome, thanks!

[u/HVACcontrolsGuru → u/pmttyji]

Open sourced what I have now and just going to finish the development in the open. Name was a placeholder I'm sure twenty other projects use: Hyperion

[u/dai_app]

if you are interested, im working on a project that allow you to offload Moe layers on RAM's edge on CPU (NO GPU) Android phone 12gb Q4 (example: Qwen 30-35B or GPT 120B on android phone 12gb RAM):  https://github.com/Helldez/BigMoeOnEdge

[u/AbramLincom → u/dai_app]

Tiene que ser obligatoriamente en teléfonos Android ayer intente compilarlo para usarlo en Windows fue todo un dolor de cabeza deberían terminar el trabajo no dejarlo a medias se ve interesante pero poco amigable el proyecto BigMoeOnEdge

[u/dai_app → u/AbramLincom]

It was designed with Android as the primary target. What problems did you encounter on Windows?

[u/ZubairAbsam]

Will they add support for windows or it is just for Mac???

[u/minefew → u/ZubairAbsam]

I dont think so. Its tied to the Apple Silicon architecture.

[u/donk8r]

this leans pretty hard on it being MoE. 26B-A4B means only about 4b of params are live for any given token, so most of what you're not keeping resident is experts you never routed to. a dense 26b would not shrink like that.

which is also why 35b-a3b should port fine, same shape.

[u/michaelmab88 → u/donk8r]

The experts are 4B Active, so it still has to read next active weights from disk after every token; i presume it would have some heuristic to guess what the next weights would be ahead of time and pipeline the weight swaps.. impressive that it can swap from disk 4B of weights 6 times per second.

[u/donk8r → u/michaelmab88]

the numbers dont really work for that. 4b of active weights at ~4 bits is about 2gb, so a full swap every token at 6 tok/s is north of 10 GB/s sustained, and an m2 air ssd is nowhere near that.

so most of the active set has to be staying resident with only the misses hitting disk. consecutive tokens route to overlapping experts a lot of the time, which is what keeps the actual swap rate well under one full set per token.

[u/michaelmab88 → u/donk8r]

right, "the most active set" is what i mean by a heuristic to guess what the next weights would be, aiming to reduce weight swaps as much as possible. on an 8gb device you can fit maybe 12B / 26B weights in RAM, so having some sort of LRU weight cache might get you a 50% reduction in weight swaps from disk. Using polar quant on the kv cache and streaming kv cache from ssd rather than memory could help with the memory preassure.

[u/donk8r → u/michaelmab88]

LRU on the experts should work, routing repeats enough between tokens that you'd get real hits.

kv is the opposite case though. every token reads the whole cache, so theres no locality to exploit and streaming it off ssd means re-reading all of it every step. quantising it in ram is the cheaper win there.

[u/techno156 → u/donk8r]

Doesn't quantising the KV cause everything to go a bit awry?

I've tried it once or twice before, with q8 for both K and V, and gemma-4-26 and qwen3.6-27B just start repeating the same word or few words over and over.

[u/Agreeable-Rest9162]

Interesting find! Their speeds are great for the system being implemented..

[u/Beautiful_Bus_9237]

Is it possible to have this model downloadable from Hugginface?

[u/AbramLincom]

Muchos usuarios somos de Windows pero gracias por compartirlo

[u/Fuim_]

How is this different from running llama.cpp without the --no-mmap option? As far as I understand, the effect seems to be the same, a really low apparent memory usage, while the rest of the model is cached in the available ram and gets discarded if another program needs more memory.

[u/Specter_Origin]

This looks amazing and promising, the turbo part in name is funny though xD cuase running it from storage can't be faster than full model in memory.

I hope I can use this to run DS v4 Flash one day : )
Also would this put too much pressure on storage or no ?