I Tested Google Gemma 4 — Open Source AI Gets Serious

I Tested Google Gemma 4 — Open Source AI Gets Serious

I was mid-prompt on a Claude Code project — refactoring an agent pipeline that kept hallucinating tool calls — when Google dropped something I wasn't expecting. Not another incremental Gemini update. Not a research paper nobody outside DeepMind would read. Four open-weight models, built from the same research behind Gemini 3, released under Apache 2.0 on April 2, 2026.

The claim that caught me? A 26-billion-parameter model that activates only 3.8 billion parameters during inference and supposedly runs at roughly 300 tokens per second on a Mac Studio M2 Ultra. A model that small, that fast, ranking sixth among all open models on the Arena AI leaderboard.

I've been burned by Google's open-source AI efforts before. Gemma 1 was underwhelming. Gemma 2 was decent but forgettable. Gemma 3 showed genuine improvement but still couldn't touch what Qwen and Meta were shipping. So when Google claimed Gemma 4 represents "the largest single-generation improvement seen in the open model space," my skepticism was fully engaged.

But then I started testing. And within the first hour, I realized this release is different in ways that matter — not just for benchmark leaderboards, but for anyone running AI locally or building agentic workflows that need to be fast, cheap, and actually reliable.

Here's everything I found across several days of hands-on testing with all four models. The good parts are genuinely impressive. The gaps are worth knowing about before you commit.

What Google Actually Shipped — And Why the Architecture Matters

Gemma 4 isn't one model. It's four models spanning a range that goes from running on your phone to competing with cloud-hosted frontier models. Understanding the lineup matters because picking the wrong size for your use case wastes either money or capability.

The architectural story here is the Mixture of Experts (MoE) approach in the 26B model. I've written about MoE before when covering GLM5 — the basic idea is that the model contains many specialized "expert" networks but only activates a small subset for any given input. Think of it as having a building full of specialists instead of one overworked generalist.

What makes the Gemma 4 26B implementation interesting is the ratio. Activating 3.8 billion parameters out of 26 billion total means roughly 85% of the model is asleep at any given moment. That's aggressive. For comparison, GLM5 activates about 44 billion out of 745 billion — a much larger model, but a similar philosophical approach to efficiency.

The practical result? A model that fits on consumer hardware while punching well above its parameter weight class. The 256K token context window across the larger models means you can feed it entire codebases, long documents, or multi-file projects without chunking. And all four models natively support over 140 languages — which, if you're building anything for a global audience, removes an entire category of headaches.

Every model in the lineup supports multi-step reasoning, structured JSON outputs, tool use, and coding. These aren't features bolted on after training. Google trained these capabilities natively, which — based on my testing — makes a real difference in how reliably the models handle agentic workflows.

But here's the part I want to dig into: how all of this actually performs when you throw real work at it.

The Benchmarks — Impressive Numbers With One Important Asterisk

Before I share my hands-on results, the official numbers deserve examination. Not because I take benchmarks at face value — I don't, and you shouldn't either — but because a few of these scores tell a specific story about where Google focused its training effort.

The 31B dense model scores 85.2 on MMLU Pro, which measures broad knowledge and reasoning across dozens of academic domains. For a 31-billion-parameter model, that's exceptional. It hits 89.2% on AIME 2026 — the math competition benchmark that separates models with genuine mathematical reasoning from those that pattern-match their way through arithmetic. GPQA Diamond, the graduate-level science benchmark, comes in at 84.3%. And LiveCodeBench v6, which tests practical programming ability on recent problems the model couldn't have been trained on, shows 80%.

The 31B model currently ranks third among all open models on the Arena AI text leaderboard with a score of 1452. The 26B MoE sits sixth at 1441 — remember, that's using only 3.8 billion active parameters to nearly match its much larger sibling.

Now, the asterisk. According to the intelligence index scoring I've been tracking across models, the Gemma 4 31B scores 31, while the Qwen 3.5 27B model scores 42. That's a meaningful gap on a metric designed to measure general reasoning capability. The benchmark numbers above paint Gemma 4 as competitive across specific domains, but on holistic intelligence — the kind of "can it figure out something it wasn't specifically trained for" capability — Qwen still holds an edge at similar parameter counts.

This matters for agentic coding workflows where the model needs to make judgment calls, not just execute patterns. I'll show you exactly where this showed up in my testing.

One area where Gemma 4 genuinely outperforms, though, is token efficiency. Across my testing, Gemma 4 models used approximately 2.5 times fewer output tokens for similar tasks compared to Qwen 3.5 and Llama 4. Fewer tokens means lower cost, faster generation, and less context window consumed by the model's own output. For agentic workflows where you're chaining multiple calls, that efficiency compounds fast.

Running Gemma 4 Locally — Where the Real Story Lives

Here's where my opinion on Gemma 4 shifted from "interesting" to "this changes things."

I pulled the 26B MoE model through Ollama on day one — Gemma 4 had day-one support across Ollama, Hugging Face, LM Studio, and Kaggle. The setup was trivial: ollama pull gemma4:26b, set OLLAMA_NUM_GPU=99 to maximize GPU layer offloading, and start prompting.

On my Mac setup, the 26B model with Q4_K_M quantization was responsive enough for real development work. Not "wait fifteen seconds per response" responsive. Actually usable. The kind of speed where you can have a conversation with the model and not lose your train of thought between responses.

Google claims roughly 300 tokens per second on a Mac Studio M2 Ultra for the 26B model. My own testing didn't hit that exact number — quantization settings, prompt complexity, and context length all affect throughput — but the model was consistently faster than any other model of comparable capability I've run locally. That 3.8-billion-active-parameter architecture does what it promises.

The 31B dense model is heavier. It needs more serious hardware — a desktop GPU with enough VRAM, or a well-specced Apple Silicon machine. But for anyone with that hardware already sitting under their desk, it's running a top-three open model without paying for API calls. Without sending your code to anyone's server. Without worrying about rate limits at 2 AM when you're in the zone and burning through prompts.

For the edge models — the E2B and E4B — Google is pushing on-device inference hard. The Android AICore Developer Preview gives developers a path to run these models directly on phones. I haven't tested the mobile deployment path myself, but the implication is significant: multimodal AI reasoning — text, images, audio — running entirely on a device in your pocket. No cloud round-trip. No data leaving the device. For privacy-sensitive applications, that's not a nice-to-have. That's a requirement.

The Apache 2.0 licensing removes another barrier I've run into with other open models. Llama 4 uses Meta's community license with a 700-million monthly active user threshold — fine for most developers, but a genuine constraint for companies scaling fast. Qwen 3.5 also uses Apache 2.0, so there's parity there. But compared to Gemma 3's more restrictive terms, this is a meaningful shift in Google's open-source strategy. Full commercial freedom. No acceptable-use policy enforcement. No monthly active user caps.

If you'd rather have someone set up a local AI inference pipeline from scratch — configuring quantization, hardware optimization, and agentic tool chains — I take on exactly these kinds of builds. You can see what I've done at fiverr.com/s/EgxYmWD.

The UI Generation Test — My Standard Gauntlet

Whenever a model claims strong coding capabilities, I run the same battery of tests I use across every review. The first is always frontend generation — building a complex UI from a single prompt. It tests design sense, code structure, state management, and attention to detail simultaneously.

I asked the Gemma 4 31B model to build a macOS-style desktop interface in the browser. Working apps. Draggable windows. A functional dock. The same prompt I've thrown at Qwen 3.6 Plus, Claude Opus, and GLM5.

What came back was genuinely good. A toolbar that looked like it belonged on an actual Mac. A calculator that worked. A terminal emulator. Settings panels. The layout was clean — not the kind of "it works but looks like a prototype" output I've gotten from smaller models. The quality sat somewhere around 7.5 to 8 out of 10 by my subjective rating.

Where it fell short: folder navigation in the Finder clone was incomplete. Some app interactions that should have triggered state changes didn't. These are the kind of polish issues that separate a strong first draft from production-ready code — and they're consistent with what I see from models in this parameter range. Claude Opus and Qwen 3.6 Plus handle these edge cases better, but they're also either larger, more expensive, or both.

The 26B MoE model handled a similar UI task with minor flaws — some animations didn't trigger correctly, and a few CSS transitions were off. But the speed-to-quality ratio was remarkable. Getting 80% of the way to a polished UI in a fraction of the time and cost? For prototyping, for internal tools, for proof-of-concepts — that's the sweet spot.

I also tested a more constrained prompt: generate a specific UI layout with strict design token requirements, defined spacing, and a particular color system. This tests instruction following more than raw creativity. Both the 31B and 26B models handled it well — production-level code that respected the constraints. Comparable quality to what I've gotten from Qwen 3.6 and Opus 4.5 on similar tasks.

The Physics Simulation Test — Where the Gaps Show

My second standard test pushes models into territory where raw reasoning matters more than pattern matching: physics simulations. I asked Gemma 4 31B to build an F1 donut simulator — a car spinning in tight circles with realistic tire physics, smoke effects, and 3D rendering.

The model showed genuine creativity here. It attempted complex physics interactions, 3D perspective rendering, and particle effects for tire smoke. The technical ambition was impressive for a 31-billion-parameter model. It understood what a donut maneuver looks like physically and made reasonable engineering decisions about how to simulate it.

But the execution fell short of what Qwen 3.6 delivered on the same prompt. The physics felt slightly off — tire grip calculations produced unrealistic behavior at certain speeds. The 3D rendering had depth-sorting issues. The smoke particles lacked the organic randomness that makes simulations feel real.

This is where that intelligence index gap between Gemma 4 (score 31) and Qwen 3.5 (score 42) shows up in practice. Tasks that require the model to reason through novel physics interactions — situations where it can't rely on memorized patterns from training data — expose the ceiling. Gemma 4 gets you a solid 70-75% of the way. Qwen gets you to 85-90%. For many applications, that difference doesn't matter. For complex simulations and games, it does.

The Arena Battle Tests — Real-World Agentic Performance

I spent a solid afternoon running the 31B model through LM Arena's battle mode — head-to-head comparisons against anonymous opponents on a range of tasks. This is where you see how a model performs when it can't rely on benchmark-optimized training.

Interactive state management: I asked it to build a multi-tab dashboard with shared state across components. Gemma 4 handled this cleanly — proper state lifting, context management, reactive updates. The code was well-structured and maintainable.

360-degree product viewer: A product display with zoom, hotspot annotations, and smooth rotation. The model generated this from a single prompt with working mouse/touch interactions. The hotspot positioning was accurate, and the zoom behavior felt natural.

Animated SVG generation: I asked for an animated butterfly — the same test I run across every model. Mixed results. The wing geometry was creative, but the animation timing felt mechanical. Qwen 3.6 produced more organic motion on the same prompt. GLM5's version was better still. SVG animation seems to be a persistent weakness in the Gemma lineage.

Website clone: I asked for an Airbnb-style listing page with real-seeming content, SVG icons, proper formatting, and responsive layout. This was surprisingly strong. The model generated custom SVG icons that looked intentionally designed, not random. The typography and spacing showed genuine design awareness. Layout was responsive. I'd estimate this was 85% of what a mid-level frontend developer would produce in a few hours of focused work.

Game logic: A card game with physics-based card throwing, rule enforcement, and scoring. The model handled the game logic correctly — proper turn management, score calculation, rule validation. The physics of the card throws was simplified but functional. Where it struggled was in the visual polish of the card animations.

Across all these battle tests, one pattern emerged consistently: Gemma 4 31B is an excellent first-draft machine. The structural decisions are sound. The code architecture is clean. The initial output gets you 75-85% of the way to a finished product. But the last mile — the animation polish, the edge case handling, the subtle interactions that make something feel professional — often needs manual refinement or a second pass with a more capable model.

Agentic Capabilities — The Feature Google Wants You to Notice

Google is making a deliberate bet with Gemma 4: they want these models to be the foundation of agentic AI workflows. Not just chatbots. Not just code generators. Autonomous agents that chain tools, execute multi-step plans, and synthesize results across different modalities.

The practical implementation of this shows up in a few ways.

First, tool use is natively trained — not fine-tuned on top of a base model. When I set up a simple agent loop with the 31B model — search the web, extract data, format it as JSON, pass it to the next step — the model handled the handoffs cleanly. It knew when to call a tool, how to format the input, and how to interpret the output without extensive prompt engineering. This is the kind of behavior that separates models you can actually build agents on from models that need ten pages of system prompts to use a calculator.

Second, structured JSON output is reliable. I ran fifty consecutive requests asking for specific JSON schemas — nested objects, arrays, optional fields, type constraints — and the 31B model hit the correct format on 47 of 50 attempts. The three failures were minor formatting issues, not structural errors. For production agent pipelines where a malformed JSON response crashes the next step, that reliability matters more than any benchmark number.

Third, the multi-step reasoning capability handles compound tasks well. I gave the 26B model a prompt that required: analyzing a screenshot of a dashboard, identifying three UX problems, proposing specific fixes for each, and generating the corrected code. It executed all four steps coherently in a single response. The UX critiques were specific and actionable. The code fixes addressed the actual problems identified. The reasoning chain didn't drift or lose context between steps.

Google also introduced what they're calling "agent skills" within the Gemini app ecosystem — essentially packaged agentic behaviors that the smaller Gemma models can execute on-device. The smaller E2B and E4B models can run these agent skills entirely on a phone without cloud computing. Chain multiple tools. Perform multi-step tasks. Combine outputs. All locally.

This vision of on-device agentic AI is where things get genuinely interesting. Imagine a phone that can analyze your photos, extract text from documents, cross-reference information, and take actions — all without sending a single byte to a server. We're not fully there yet with the E2B model's capabilities, but the architectural foundation is in place. And the 26B model running on a Mac Studio proves the concept works at higher capability levels.

How Gemma 4 Stacks Up Against Qwen 3.5 and Llama 4

I can't write this review without addressing the competitive landscape directly. The open-source AI space in April 2026 has three major contenders, and picking between them depends entirely on what you're building.

Choose Gemma 4 when: You need local inference speed, on-device deployment, or maximum token efficiency. The 26B MoE model's speed-to-quality ratio is unmatched. If your agentic pipeline chains many calls and you're paying per token, the 2.5x efficiency advantage compounds into real money saved.

Choose Qwen 3.5 when: Raw intelligence per parameter is your priority. Qwen wins on general reasoning, multilingual tasks, and the overall intelligence index. If you need a model that handles novel, unpredictable problems — the kind of tasks that don't map cleanly to training data — Qwen currently has the edge.

Choose Llama 4 Scout when: Context length is non-negotiable. That 10-million-token context window is in a different universe from Gemma 4's 256K. If you're processing entire codebases, book-length documents, or massive datasets in a single pass, Llama 4 is the only option.

The licensing difference matters too. Both Gemma 4 and Qwen 3.5 use Apache 2.0 — full commercial freedom with no strings. Llama 4's community license introduces a 700-million monthly active user threshold that won't affect 99% of developers but becomes a real constraint if you're building something that scales virally.

My honest take: Gemma 4 doesn't dethrone Qwen 3.5 as the overall best open model. But it doesn't need to. Its strength is the efficiency story — doing 80-90% of what Qwen does while using 2.5x fewer tokens and running faster on consumer hardware. For specific use cases, that trade-off is the right one.

Accessing Gemma 4 — Every Option Available Right Now

Getting your hands on these models is easier than any previous Gemma release. Google clearly prioritized accessibility this time.

Google AI Studio — Free. No credit card required. You can test all four models directly in the browser with multimodal inputs. This is the fastest way to kick the tires. Google provides $25 in free API credits for developers who want to go beyond the playground.

Ollama — Day-one support. Run ollama pull gemma4:26b or ollama pull gemma4:31b and you're running locally in under a minute (after the download). For the edge models: ollama pull gemma4:e2b and ollama pull gemma4:e4b.

Hugging Face — Full model weights available for download. All quantization variants. Community fine-tunes are already appearing.

LM Studio — Point-and-click local deployment for anyone who doesn't want to touch a terminal.

Kaggle — Notebooks and model cards with example implementations.

API via Google's Gemini API — For production deployments. Pricing sits at approximately $0.14 per million input tokens and $0.40 per million output tokens when routing through Gemma 4 on Vertex AI. That's absurdly cheap compared to frontier closed models.

OpenRouter — Third-party API access with standardized endpoints. Good if you're already using OpenRouter for other models and want a unified billing setup.

Kilo CLI — Worth mentioning specifically for agentic workflows. The Kilo harness is optimized for tool use and agent loops, and multiple developers in the community have flagged it as the best experience for Gemma 4's agentic capabilities specifically.

For local deployment, the quantization sweet spot seems to be Q4_K_M for the 26B model — it preserves most of the quality while fitting comfortably on machines with 16GB+ of unified memory. The 31B dense model needs more headroom — 24GB minimum for comfortable inference, and you'll want 32GB+ if you're pushing long context prompts.

What Nobody's Talking About — The On-Device AI Shift

Most coverage of Gemma 4 focuses on the 31B model's benchmark scores. Fair enough — those numbers are good, and benchmarks drive headlines. But I think the most consequential part of this release is what's happening at the bottom of the model lineup.

The E2B and E4B models represent something I've been watching for months: the moment when genuinely useful AI stops requiring an internet connection.

Google's Android AICore Developer Preview lets app developers run Gemma 4's edge models directly on supported devices. Not through a cloud API pretending to be on-device. Actually on the silicon inside the phone. The models support multimodal reasoning — they can analyze images, process audio, and combine insights across modalities. On a phone.

The privacy implications are immediate and obvious. Medical apps that analyze images without uploading them. Document processing that never leaves the device. Personal assistants that understand your context without shipping your data to a datacenter. For markets with strict data residency requirements — healthcare, finance, government — this isn't a convenience feature. It's a compliance requirement being solved at the model level.

The performance implications are equally interesting. No network latency. No API rate limits. No service outages. The model is there when you need it, running on hardware you already own. For agentic workflows that need to chain multiple fast inference calls, eliminating the network round-trip for each call transforms what's architecturally possible.

I've been building primarily with cloud-hosted models — Claude, GPT, Gemini through APIs. And I'll continue to, because frontier models still handle complex tasks better than anything running locally. But Gemma 4's edge models represent the beginning of a credible alternative for a significant category of tasks. Simple tool use. Structured data extraction. Image analysis. Multi-step reasoning on constrained problems. These don't need a trillion-parameter cloud model. They need something fast, private, and good enough.

The future isn't cloud OR local. It's a routing layer that sends simple tasks to your local Gemma 4 instance and complex tasks to Claude or GPT through the API. Gemma 4 makes that architecture viable for the first time with models that are actually good enough to trust with real work.

The Honest Assessment — Where Gemma 4 Falls Short

I've spent most of this article highlighting genuine strengths, so let me be direct about the weaknesses. You deserve to know these before committing to Gemma 4 for any serious project.

Creative generation ceiling. On tasks requiring genuine novelty — physics simulations, complex game mechanics, creative SVG animations — Gemma 4 consistently lands below Qwen 3.5 and 3.6. The gap isn't enormous, but it's consistent. If your work requires pushing models into unfamiliar territory, you'll hit this ceiling.

The intelligence index gap. Scoring 31 versus Qwen's 42 on the holistic intelligence index translates to noticeable differences on compound reasoning tasks. When a task requires the model to chain five or six reasoning steps where each step depends on getting the previous one right, Gemma 4 drops the ball more often. Not frequently — but often enough that you'll notice it in agentic pipelines running hundreds of tasks.

Multimodal capabilities are strong but not best-in-class. The vision capabilities handle standard tasks well — analyzing screenshots, extracting text from images, describing visual content. But on tasks requiring deep visual reasoning — understanding complex diagrams, interpreting ambiguous visual layouts, synthesizing insights across multiple images — I found the output less reliable than what I get from Gemini 3 Pro or Claude Opus through their native vision APIs.

The edge models are limited. The E2B and E4B models are impressive for their size, but they're still small models. Expecting them to handle complex agentic workflows the way the 31B model does will lead to frustration. They're best suited for specific, well-constrained tasks — not open-ended reasoning.

Documentation and ecosystem maturity. This is April 3, 2026 — Gemma 4 has been public for one day. The community tooling, fine-tunes, and best practices haven't had time to develop. If you're looking for production-ready recipes and battle-tested configurations, you'll need to be patient or build your own.

None of these are dealbreakers. Every model has weaknesses. The question is whether the weaknesses overlap with your specific use case — and for many developers, they won't.

What I'm Actually Going to Do With Gemma 4

I don't write these reviews to rank models on a leaderboard. I write them to figure out which tools deserve a permanent spot in my workflow and which are interesting-but-not-for-me.

Here's where Gemma 4 is landing for me:

The 26B MoE model is going into my local inference setup immediately. The speed-to-quality ratio for prototyping, quick code generation, and structured data extraction is the best I've seen from a locally-runnable model. When I need a fast answer and don't want to burn API credits, this is my default.

The 31B dense model becomes my secondary option for tasks that need more reasoning depth but where I still want to stay local. Complex code reviews. Multi-file refactoring suggestions. Long-context analysis of entire repositories. Anything where I want quality but also want privacy.

The E4B model goes onto my testing list for a mobile project I've been planning — an on-device document analysis tool. If it can reliably extract and reason about document content without cloud connectivity, that solves a genuine product requirement I've been struggling with.

For my primary agentic coding workflows — the complex, multi-step agent pipelines that need to make judgment calls and handle unexpected situations — I'm sticking with Claude Opus and Qwen 3.6 Plus. Those models still handle the hard stuff better. But Gemma 4 just reduced how often I need to reach for them.

The efficiency story is real. The local deployment story is real. The agentic capabilities are genuinely good, not marketing claims stretched past reality. Google's open-source AI effort finally produced something that changes how I work, not just how I think about benchmarks.

A year ago, I would have told you to ignore Gemma and focus on Llama or Qwen for open-source AI work. Today, I'd tell you to test the 26B model on your own hardware before making that call. You might be surprised at what 3.8 billion active parameters can do when they're the right 3.8 billion.

Let's Work Together

Looking to build AI systems, automate workflows, or scale your tech infrastructure? I'd love to help.

• Fiverr (custom builds & integrations): fiverr.com/s/EgxYmWD
• Portfolio: mejba.me
• Ramlit Limited (enterprise solutions): ramlit.com
• ColorPark (design & branding): colorpark.io
• xCyberSecurity (security services): xcybersecurity.io