Gemma 4: how MoE and QAT let a 26B-parameter model run in 15 GB of RAM

For the past decade, a language model's power was measured in parameters, and running it was measured in the price of the GPU. Gemma 4, Google's open-weights model family, breaks that correlation with two engineering techniques that move the conversation from compute to RAM. The result has direct consequences for developers, small businesses and startups that want private AI without unpredictable API bills.

The problem: the model doesn't fit, but the cloud is expensive

A large language model needs to load all of its weights into memory to generate text. Historically that meant data-center GPUs with tens of gigabytes of VRAM, or paying per token to a cloud API. For a small company, both routes have friction: the first because of hardware cost, the second because of variable cost and the need to send sensitive data to a third party.

The Gemma 4 family attacks that problem from the design stage. Its variants range from tiny models for phones (E2B, ~2.3B effective; E4B, ~4.5B) to a dense 31B model for maximum quality, by way of the most interesting piece for professional use: the 26B-A4B, a Mixture of Experts model. What makes running these models on accessible hardware viable is not a single trick, but the combination of two.

First technique: Mixture of Experts (MoE)

The Mixture of Experts architecture splits the network into many specialised sub-models —the "experts"— and uses a router that decides, token by token, which ones to activate. In the case of Gemma 4 26B-A4B, the "A4B" stands for active 4 billion: of its 26 billion total parameters, only around 3.8 billion are activated for each token processed.

Concretely, the model organises its knowledge into 128 small experts, of which only 8 fire for any given token, according to the published technical breakdown of the model [2] [3]. The consequence is counterintuitive but powerful: the model thinks like a 26B one, but costs like one under 4B.

Quality without paying for the full compute

Google's 2026 benchmarks show that this efficiency does not sacrifice capability. The 26B-A4B reaches 88.3% on the AIME 2026 mathematics exam, 77.1% on competitive programming (LiveCodeBench v6) and 82.3% on graduate-level science (GPQA Diamond) [1] [4]. In the 14–16 GB VRAM hardware bracket, the model holds a substantial quality advantage on reasoning tasks —maths, code and multi-step logic— over comparable dense alternatives.

The difference against a dense model is key for the inference budget: in a dense model of equivalent size, every parameter activates on every token. MoE keeps the memory occupied by the full model, but drastically cuts compute per token, which translates into faster responses and lower energy consumption.

Second technique: Quantization-Aware Training (QAT)

MoE solves compute, but the model still has to fit in memory. That is where the second technique comes in. On 5 June 2026, Google DeepMind released Gemma 4 checkpoints optimised with Quantization-Aware Training (QAT), a technique that, instead of compressing the model after training it, simulates 4-bit arithmetic during training so the model learns to tolerate that reduced precision [6].

The effect on memory is decisive: QAT cuts RAM usage by roughly 72% while preserving quality close to the original. In practice, the 26B-A4B QAT checkpoint loads at around 15 GB in 4-bit, which means a 16 GB unified-memory Mac or a 16 GB GPU (such as an RTX 4060) can run it [5] [6].

The table that matters: RAM per variant

With 4-bit quantization (QAT), approximate total memory requirements land like this [5] [7]:

Tools such as Ollama download the INT4 version (Q4_K_M) by default, reducing memory usage by between 55% and 60%, and automatically manage GPU offloading based on available hardware. Running on CPU without a GPU is even possible, though between 5 and 10 times slower.

The mental rule that follows for choosing hardware: 8 GB of RAM is enough for 4B to 12B models; 16 GB unlocks the 26B-A4B MoE —the quality/cost "sweet spot"—; and 32 GB or more leaves room for the dense 31B.

What this means for the business

The MoE + QAT combination shifts the barrier to entry for private AI in a measurable way. For a development team or a small business, there are three implications.

Privacy and compliance: tasks such as document classification, data extraction, summarisation or code drafts can run locally, without the information ever leaving the machine. Cost predictability: the variable spend of an API is replaced by the fixed cost of hardware the company, in many cases, already owns. Accessibility: 16 GB of unified RAM —present in mid-range laptops— is enough for a model with competitive reasoning capability.

This does not eliminate the role of cloud APIs or frontier models; for maximum-complexity tasks, a cloud orchestrator still makes sense. But it does change the calculation: a growing share of routine AI work can migrate to local, and the hardware purchasing criterion becomes, above all, a question of gigabytes of memory.

Conclusion

Gemma 4 condenses a broader 2026 trend: architectural efficiency is democratising access to powerful models. MoE reduces compute by activating only a fraction of the parameters; QAT reduces memory by training the model to live in 4 bits. Together, they let a model with elite reasoning quality fit on a 16 GB laptop. The message for anyone planning AI infrastructure is direct: stop measuring models in parameters and start measuring them in the gigabytes of RAM you have available. That is, today, the metric that decides which AI you can run —and how much it costs you.

Did this breakdown help? At No BS AI we publish technical analysis without the smoke on AI applied to development and business. Subscribe to the newsletter and tell us which tool you want us to take apart in the next episode.

Sources {#sources}

1. Google AI for Developers. "Gemma 4 model overview". 2026. https://ai.google.dev/gemma/docs/core

2. Google (Hugging Face). "google/gemma-4-26B-A4B". 2026. https://huggingface.co/google/gemma-4-26B-A4B

3. Govrani, Pulkit. "Gemma 4 26B A4B: What 'Mixture of Experts' Actually Means for Your Inference Budget". DEV Community. 2026. https://dev.to/pulkitgovrani/gemma-4-26b-a4b-what-mixture-of-experts-actually-means-for-your-inference-budget-13hj

4. Tang, James. "Benchmarking Gemma-4–26B (A4B) on the DGX Spark". Medium. May 2026. https://medium.com/@james-tang/benchmarking-gemma-4-26b-a4b-on-the-dgx-spark-dc8245292095

5. Oflight Inc. "Gemma 4 System Requirements — 5–62GB VRAM, RTX 3060 to H100 by Variant (E2B/E4B/26B/31B) [2026 Guide]". 2026. https://www.oflight.co.jp/en/columns/gemma4-hardware-requirements-local-ai-spec-2026

6. Lushbinary. "Gemma 4 QAT Self-Hosting Guide: Ollama, vLLM". 2026. https://lushbinary.com/blog/gemma-4-qat-self-hosting-guide-ollama-llama-cpp-vllm/

7. Aurigait. "Gemma 4 by Google: Specs, Benchmarks, Model Sizes, and How to Run It Locally (2026 Guide)". 2026. https://aurigait.com/blog/gemma-4-features-benchmarks-guide/

8. Labellerr. "Google Gemma 4: A Technical Overview". 2026. https://www.labellerr.com/blog/gemma-4-open-weight-ai-model-overview/