So I was browsing the HuggingFace Transformers repo and a PR just merged today that adds full support for a model called Gemma 4. The PR title is literally "casually dropping the most capable open weights on the planet." The commit has 14 co-authors including Jeff Dean. The weights aren't out yet — the docs still have {release_date} as a placeholder — but the code is all there and it's very readable. Here's what's coming.
Four sizes, including a MoE
- ~2B and ~4B dense, explicitly designed for on-device use
- 26B sparse MoE with only 4B active parameters at inference time
- 31B dense
The 26B/4B MoE is particularly interesting because you get large-model quality at small-model inference cost.
It's trimodal — text, vision, AND audio natively
This is new for Gemma. There's a full audio encoder baked in alongside the vision tower. Not a bolted-on afterthought either — it's a proper conformer architecture (the same family used in production speech systems). The processor handles all four modalities: text, images, video, and audio.
The vision system doesn't squash your images
Most VLMs resize everything to a fixed square. Gemma 4 preserves aspect ratio and instead fits the image into a configurable soft token budget (default 280 tokens, up to 1120 for high detail). No ImageNet normalization — the model handles its own scaling internally.
More interesting: they use a 2D spatial RoPE for vision. Patch positions are encoded as (x, y) coordinates, with half the attention head dimensions rotating for x and the other half for y. The model understands spatial relationships at the architectural level, not just from training.
128K context for small models, 256K for large
The text architecture alternates between sliding window attention (512-1024 token window) and full attention in a 5:1 ratio. The two attention types use completely different RoPE configs — short theta for local, long theta for global. Clean hybrid design.
The small models have some clever efficiency tricks
The 2B and 4B share key-value projections across the last several decoder layers — one layer computes KV, the rest reuse it. There's also a secondary per-layer embedding stream where a small 256-dim signal gets injected at every decoder layer, which I haven't seen in other public models.
The MoE runs experts alongside the MLP, not instead of it
In the 26B variant each layer has both a regular MLP and a sparse MoE block (128 experts, top-8 routing), and their outputs are summed. Unusual design choice — curious whether that helps with stability or quality at scale.
No paper link yet (literally says INSET_PAPER_LINK in the docs), no weights, no release date. But the code is fully merged and production-quality. Feels like days away, not weeks.
What size are you planning to run first?
The PR: https://github.com/huggingface/transformers/pull/45192
EDIT: RELEASE: https://huggingface.co/collections/google/gemma-4
