<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qwtitqYOqDQ17Vu69OlmJA.png"><figcaption>The speed got my attention. The stability fixes made me stay.</figcaption></figure><h4>BeeLlama • Gemma 4 • DFlash</h4>
<h4>A week later, it’s still running beside my main llama.cpp setup.</h4>
Two crashes in two days. Then 177 tok/s. So I opened the repo again.
A month ago I wrote BeeLlama off for serious agentic coding.
Then v0.2.0 landed with a claimed 177.8 tok/s on <a href="https://ai.google.dev/gemma">Gemma 4 31B</a>. I wasn’t planning to revisit it. A week later, I was using it daily.
<figure><img alt="Two crashes in two days. Then 177 tok/s. So I opened the repo again | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*4VMHRONQUdnx8RajKnEowA.png"><figcaption>Two crashes in two days. Then 177 tok/s. So I opened the repo again.</figcaption></figure>A few weeks ago, in my article “<a href="https://medium.com/ai-in-plain-english/best-local-llm-for-8gb-vram-coding-i-tried-everything-51a9528dd77e">Best Local LLM for 8GB VRAM Coding? I Tried Everything</a>”, I mentioned BeeLlama almost in passing.
My verdict wasn’t exactly positive.
<blockquote>Fast when it works. Too risky for long unattended sessions.</blockquote>
At the time, that felt fair.
I’d hit two failures in two days. One ended with a segfault. Another silently stopped making progress during an overnight agent run.
After that, I went back to mainline llama.cpp and stopped thinking about it.
Then <a href="https://github.com/Anbeeld/beellama.cpp">BeeLlama v0.2.0</a> showed up.
The headline benchmark was hard to ignore:
177.8 tok/s on Gemma 4 31B using a single RTX 3090.
Compared to roughly 36 tok/s on llama.cpp, that’s not a small improvement. That’s the kind of number that makes you wonder whether you’re leaving performance on the table.
So I installed it again.
<h3>What BeeLlama Actually Is</h3>
BeeLlama is a CUDA-focused fork of <a href="https://github.com/ggml-org/llama.cpp">llama.cpp</a>.
It’s built around a custom attention implementation called DFlash.
The simple explanation is that DFlash reduces how much time the GPU spends moving data during token generation. Instead of changing the model itself, it changes how decoding is performed.
Same model.
Same output.
Much faster generation.
The obvious question is why llama.cpp doesn’t just adopt it.
The answer is that BeeLlama gets to optimize aggressively for CUDA. Mainline llama.cpp has to support CUDA, Metal, Vulkan, ROCm, CPUs, and everything in between.
A CUDA-specific optimization can be excellent and still not fit comfortably into a project designed to run everywhere.
That’s why forks like BeeLlama often become the ecosystem’s R&amp;D lab. Ideas get tested here first before influencing mainstream projects later.
<h3>What Changed in v0.2.0</h3>
The benchmark numbers grabbed attention.
The changelog is what got mine.
Most of my issues with earlier releases happened around tool calls, reasoning traces, and speculative decoding fallbacks, the places where systems stop taking the happy path.
v0.2.0 introduced:
<ul>
<li>Safer verifier fallback logic</li>
<li>Better handling around reasoning and tool-call boundaries</li>
<li>Cleaner prefill-to-decode transitions</li>
<li>Stricter draft-target validation</li>
</ul>
None of those improvements are exciting on social media.
They’re also exactly the kind of fixes that make software usable.
The funny thing is that the speed numbers weren’t what changed my mind.
If BeeLlama had stayed unstable, 177 tok/s wouldn’t have mattered.
Fast software that crashes is still slow software.
What got my attention wasn’t the benchmark graph.
It was the changelog.
<h3>Is It Actually Stable Now?</h3>
This is the question that matters.
And the answer is surprisingly simple:
Yes. BeeLlama v0.2.0 is meaningfully more stable than previous versions.
Is it as reliable as mainline llama.cpp?
No.
Would I trust it with production inference serving real users?
Also no.
Would I use it for daily agentic coding?
Absolutely.
Over the past week, I ran multiple long coding sessions involving large repositories, tool calls, test runs, file edits, and debugging loops.
No crashes.
That matters more than ever because local agents are increasingly handling multi-step workflows.
In a previous experiment, I built a Gemma 4 coding agent that kept looping until I fixed a subtle reasoning issue:
<a href="https://ai.plainenglish.io/i-built-a-gemma-4-ai-agent-it-kept-looping-until-i-fixed-this-1ef53f50158c">I Built a Gemma 4 AI Agent. It Kept Looping Until I Fixed This.</a>
One session briefly paused before continuing, which looked like BeeLlama dropping out of its fast path and recovering cleanly.
Earlier releases probably would have failed there.
This one didn’t.
That’s obviously not a scientific benchmark, but the specific failure modes that pushed me away a month ago have stopped reproducing.
For me, that’s enough to keep using it.
<h3>The Speed Numbers</h3>
The headline benchmark remains impressive.
<h3><a href="https://ai.google.dev/gemma">Gemma 4 31B</a></h3>
<ul>
<li>BeeLlama: 177.8 tok/s
</li>
<li>llama.cpp: ~36 tok/s
</li>
<li>Improvement: ~4.9x
</li>
</ul>
<h3><a href="https://qwen.ai/research">Qwen 3.6 27B</a></h3>
<ul>
<li>BeeLlama: ~164 tok/s
</li>
<li>llama.cpp: ~37 tok/s
</li>
<li>Improvement: ~4.4x
</li>
</ul>
<figure><img alt="36 tok/s vs 177 tok/s. Same GPU. Same model. Different inference engine | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*BJxN2PBn4LVEFykg-5gmtw.png"><figcaption>36 tok/s vs 177 tok/s. Same GPU. Same model. Different inference engine.</figcaption></figure>What’s important is where those gains appear.
DFlash primarily accelerates decoding, not prompt processing.
That’s especially relevant if you’re already running larger local models such as Qwen 3.6 in Thinking Mode, where long generations quickly become the dominant cost.
<a href="https://ai.plainenglish.io/i-ran-qwen3-6-locally-the-thinking-mode-changed-how-i-use-it-5e5db6a14c78">I Ran Qwen3.6 Locally. The Thinking Mode Changed How I Use It</a>
For coding agents, that’s exactly where most of the time is spent.
A single completion finishing two seconds faster isn’t particularly exciting.
An agent making hundreds of tool calls throughout a workday is different.
That’s where the speedup becomes noticeable.
After a week with BeeLlama, switching back to vanilla llama.cpp felt slower than I expected.
That’s usually when an optimization becomes real, when you miss it after it’s gone.
<h3>DFlash vs MTP</h3>
Anyone who read my recent article:
<a href="https://ai.plainenglish.io/gemma-4-mtp-local-inference-benchmarks-6711c8589d2f">I Spent 3 Nights Testing Gemma 4 (MTP)</a>
will probably ask the same question.
<h4>How does DFlash compare to MTP?</h4>
Both are trying to solve slow token generation.
They just take different approaches.
<h4>MTP (Multi-Token Prediction)</h4>
MTP generates future tokens and asks the larger model to verify them. When outputs are predictable, it performs extremely well.
DFlash DFlash accelerates the underlying attention computation itself.
It doesn’t care whether the model is generating boilerplate code or reasoning through a bug.
For long-context agentic coding workloads, DFlash currently feels more consistent.
MTP still shines on shorter generations with high acceptance rates.
The interesting possibility is that BeeLlama can combine DFlash with speculative decoding techniques, meaning these approaches aren’t necessarily competing — they can be complementary.
<figure><img alt="MTP bets on predictability. DFlash doesn’t need to | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*owCfyKSk6riiSoEY-1jCsQ.png"><figcaption>MTP bets on predictability. DFlash doesn’t need to.</figcaption></figure><blockquote>MTP bets on predictability. DFlash doesn’t need to.</blockquote>
If you’re evaluating raw decoding speed versus speculative decoding, my Gemma 4 MTP testing is worth reading alongside this BeeLlama benchmark because the two approaches solve the same bottleneck in very different ways.
<h3>Who Should Use It?</h3>
<h4>Switch now if:</h4>
<ul>
<li>You have an RTX 3090, 4090, or similar CUDA GPU</li>
<li>You primarily use Gemma 4 31B</li>
<li>Your primarily use Qwen 3.6 27B</li>
<li>Your workload is agentic coding</li>
<li>You’re comfortable using a fast-moving fork</li>
</ul>
<h4>Wait if:</h4>
<ul>
<li>You need maximum stability</li>
<li>You’re serving production workloads</li>
<li>You’re on AMD</li>
<li>You’re Apple Silicon</li>
<li>You’re primarily focused on low-VRAM setups</li>
</ul>
The practical solution is simple:
Run BeeLlama for active coding sessions and keep a llama.cpp server as a fallback.
That’s currently what I’m doing.
<h3>The Bigger Question</h3>
Some people are understandably hesitant to depend on a fork maintained by a single developer.
That’s a real risk.
If development stops, the project stops.
At the same time, many of the most impactful optimizations in local AI started life in forks before becoming mainstream.
Whether DFlash eventually influences llama.cpp remains to be seen.
But after using v0.2.0, it feels less like an experiment and more like an early glimpse of where local inference performance is heading.
<h3>Final Verdict</h3>
A month ago, BeeLlama lived in my “Interesting benchmark project” folder.
Today, it’s running alongside my primary inference stack.
I’m still keeping a llama.cpp server as a fallback.
I’m still not ready to call BeeLlama boringly reliable.
But for the first time, I’m recommending it for real agentic coding workloads.
Benchmarks made me reinstall it.
Stability is what made me keep using it.
And that’s ultimately the conclusion of this BeeLlama v0.2.0 DFlash review.
<figure><img alt="171 tok/s. Not a benchmark suite. Just the server log | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*pAT3lLTWFvsKeGQJJmnSXQ.png"><figcaption>171 tok/s. Not a benchmark suite. Just the server log.</figcaption></figure><h4>Continue Reading</h4>
<h4>If you’re building a local AI stack, these articles connect directly with the ideas explored here:</h4>
<ul>
<li><a href="https://ai.plainenglish.io/best-local-llm-for-8gb-vram-coding-i-tried-everything-51a9528dd77e">Best Local LLM for 8GB VRAM Coding? I Tried Everything.</a></li>
<li><a href="https://ai.plainenglish.io/gemma-4-mtp-local-inference-benchmarks-6711c8589d2f">I Spent 3 Nights Testing Gemma 4 (MTP)</a></li>
<li><a href="https://ai.plainenglish.io/i-ran-qwen3-6-locally-the-thinking-mode-changed-how-i-use-it-5e5db6a14c78">I Ran Qwen3.6 Locally. The Thinking Mode Changed How I Use It</a></li>
</ul>
<h3>QUESTION I HAD BEFORE TRYING IT OUT MYSELF</h3>
<h3>Does BeeLlama work with Ollama or LM Studio?</h3>
Not directly. BeeLlama runs as its own llama.cpp-compatible server and exposes an OpenAI-compatible API.
<h3>Is DFlash the same as <a href="https://github.com/Dao-AILab/flash-attention">Flash Attention</a>?</h3>
No.
Flash Attention: <a href="https://github.com/Dao-AILab/flash-attention">https://github.com/Dao-AILab/flash-attention</a> DFlash specifically targets decoding performance and memory-bandwidth bottlenecks, while Flash Attention is a broader attention optimization.
<h3>What happened to TurboQuant?</h3>
TurboQuant is still available in BeeLlama. The v0.2.0 release primarily focused on DFlash and stability improvements.
<h3>Can DFlash be combined with speculative decoding?</h3>
Yes. The maintainer has confirmed support for combining DFlash with techniques like ngram speculation and copyspec.
<h3>Will BeeLlama eventually merge into llama.cpp?</h3>
Probably not directly, but successful ideas from forks often influence future upstream implementations.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=46d88b046e07" width="1" height="1" alt=""><hr>
<a href="https://ai.plainenglish.io/i-uninstalled-beellama-then-v0-2-0-changed-my-mind-46d88b046e07">I Uninstalled BeeLlama. Then v0.2.0 Changed My Mind.</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

I Uninstalled BeeLlama. Then v0.2.0 Changed My Mind.

<figure><img alt="Best Local LLM for 8GB VRAM Coding? I Tried Everything | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*_847RIGhAW1RJVPq9q9fKw.png"><figcaption>Best Local LLM for 8GB VRAM Coding? I Tried Everything.</figcaption></figure><h4>LOCAL AI / BUDGET HARDWARE</h4>
<h4>Gemma 4 killed my KV cache. Dense Qwen crawled at 3 tok/s. After weeks of testing local LLMs on a GTX 1070, one setup finally stopped fighting me.</h4>
After weeks of testing Gemma 4, dense Qwen, BeeLlama.cpp, and every “8GB-friendly” setup Reddit recommended, only one actually felt usable for real coding work.
If you just want the answer:
<blockquote>
The best local LLM for 8GB VRAM coding right now is probably Qwen3.6–35B-A3B running through llama.cpp with--n-cpu-moe enabled.
</blockquote>
<figure><img alt="Still loud. Still refusing to die. Turns out still useful. | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*kmnqgMw4xKkBxpqNxZcdsQ.png"><figcaption>Still loud. Still refusing to die. Turns out still useful.</figcaption></figure>Every local LLM thread eventually turns into someone benchmarking a 70B model on a 4090 and calling it “surprisingly accessible.”
Meanwhile my GTX 1070 sounds like it’s preparing for takeoff every time <a href="https://github.com/ggml-org/llama.cpp">llama.cpp</a> starts loading weights.
So I went looking for the best local LLM for 8GB VRAM coding because honestly, I got tired of reading advice written by people whose GPU costs more than my entire desktop.
And after a few weeks of testing different setups, I think most “8GB compatible” recommendations are technically true in the same way sleeping in your car is technically a housing solution.
Yes, the models load.
That’s not the same thing as usable.
There’s a huge difference between:
<blockquote>the model generates tokens</blockquote>
and
<blockquote>you can leave it running on a coding task without the whole setup collapsing into a 3 tok/s swamp.</blockquote>
That gap matters way more than benchmarks do.
Especially for agentic coding.
<h3>Why Most Local LLM Setups on 8GB VRAM Fall Apart</h3>
The weird part is that loading the model usually isn’t the problem anymore.
Quantization got good enough that almost anything can be bullied into memory if you compress it hard enough and throw enough system RAM at <a href="https://github.com/ggml-org/llama.cpp">llama.cpp</a>.
The problems show up later.
Usually around the point where your coding agent has:
<ul>
<li>Opened twelve files</li>
<li>Dumped stack traces into context</li>
<li>Retried the same broken test four times</li>
<li>and quietly accumulated 40k tokens of garbage memory</li>
</ul>
That’s when the “works great on my machine” setups start dying.
Two things consistently killed performance for me:
1️⃣ KV Cache Growth
This is the real VRAM killer on long coding sessions.
Fresh context benchmarks are misleading because almost every model feels fast at 4k context. The collapse happens later, when the KV cache starts eating memory alongside the model itself.
Gemma 4 was especially bad here.
Around 30–40k context, I could literally watch VRAM disappear in nvtop while generation slowed to a crawl.
And coding agents hit long context fast. Faster than most people expect.
A single debugging session can accumulate:
<ul>
<li>Tool calls</li>
<li>Logs</li>
<li>Diffs</li>
<li>Retries</li>
<li>Terminal output</li>
<li>File contents</li>
<li>Reasoning traces</li>
</ul>
Suddenly your “small task” is dragging around half a novel worth of context.
2️⃣ PCIe Bottlenecks on Dense Models
Dense models on 8GB feel terrible once offloading starts.
Technically the setup works. Practically llama.cpp starts hauling weights back and forth between VRAM and RAM every forward pass like it’s carrying groceries up apartment stairs.
I tested Dense <a href="https://github.com/QwenLM/Qwen">Qwen3.6–27B</a> this way.
Around 3 tok/s.
Which sounds survivable until the agent makes a mistake and spends the next several minutes slowly explaining the mistake it slowly made.
That was the point where I realized most people defining “usable” have a much higher tolerance for suffering than I do.
<h3>The Candidates I Actually Tested (And What Broke)</h3>
<h4>Gemma 4 26B MoE</h4>
This one genuinely tricked me at first.
On paper it looks perfect for budget hardware:
<ul>
<li>MoE architecture</li>
<li>Smaller active parameter count</li>
<li>Reasonable quant sizes</li>
<li>Decent coding performance</li>
</ul>
Short sessions felt great.
Then context grew and the KV cache started eating the machine alive.
At around 32k tokens, performance became inconsistent enough that I stopped trusting unattended runs entirely.
That became the pattern with almost every “8GB-ready” setup:
fine at the beginning, terrible once the session became realistic.
Single-turn coding help? Sure.
Long agentic sessions? I gave up on it pretty quickly.
I actually spent a full week trying to make Gemma 4 behave locally because on paper it looked like the perfect budget setup. The longer I tested it though, the stranger the KV cache behavior became. I’ll probably write that up separately because some of the degradation patterns were genuinely weird.
<a href="https://ai.plainenglish.io/i-ran-gemma-4-locally-heres-what-nobody-s-telling-you-806dcd485925">I Ran Gemma 4 Locally. Here’s What Nobody’s Telling You.</a>
<figure><img alt="Three candidates. Two failures. One setup that held up | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*umMyTr7cY8RZbRfYKGbAqQ.png"><figcaption>Three candidates. Two failures. One setup that held up.</figcaption></figure><h3>Dense Qwen3.6–27B</h3>
I understand why people love this model.
I also understand why most of those people own more VRAM than I do.
With 32GB system RAM, I got it running through layer offloading in llama.cpp.
The problem was speed.
Or more specifically: accumulated waiting.
I gave it a normal coding task:
<ul>
<li>Refactor a module</li>
<li>Write tests</li>
<li>Fix failures</li>
</ul>
The full loop took around 40 minutes.
Not because the model was dumb. Mostly because inference felt like wading through wet cement.
The psychological effect surprised me more than the benchmark numbers did.
You stop experimenting when every mistake costs another ten minutes.
That part surprised me more than the benchmarks did, honestly.
Slow inference changes how you think. You become weirdly conservative. You stop trying ideas because every failed attempt turns into another coffee break.
<h3>BeeLlama.cpp</h3>
This one felt like driving a heavily modified project car.
Sometimes incredible.
Sometimes concerning.
The throughput gains are real. Short-context speed noticeably improved on my GTX 1070, even compared to standard llama.cpp.
But I also hit:
<ul>
<li>One hard segfault</li>
<li>One silent inference hang</li>
<li>And one session where my coding agent waited forever because the server quietly died mid-task</li>
</ul>
That matters more for coding agents than regular chat.
If I’m sitting there babysitting inference, fine.
If I leave for coffee and come back to a dead process after an hour-long task, the performance gains stop feeling very important.
I still think <a href="https://github.com/beeai/bee-agent-framework">BeeLlama.cpp</a> is worth watching though. It just feels early.
<h3>The One Setup That Actually Worked</h3>
The weird answer ended up being the model I originally ignored:
<blockquote>Qwen3.6–35B-A3B.</blockquote>
I ended up using a GGUF quant through <a href="https://huggingface.co/models?library=gguf">Hugging Face</a>, mostly because the llama.cpp workflow gave me more control than Ollama did.
The name alone made me assume there was absolutely no way this thing would behave on 8GB VRAM.
But because it’s MoE and only around 3B parameters stay active during inference, the real footprint is much smaller than the headline number suggests.
That changes everything about how the system behaves under load.
More importantly: The KV cache behavior stayed stable.
That was the real breakthrough.
Not peak benchmark speed. Not first-token latency. Just consistency during ugly, realistic, long-running coding sessions.
Fresh context on my GTX 1070 sits around:
<ul>
<li>35–40 tok/s early</li>
<li>20-ish tok/s during longer sessions</li>
</ul>
And honestly? Once the speed stays predictable, your brain stops thinking about the hardware constantly.
That’s the first time local coding on 8GB stopped feeling like a science experiment to me.
<h3>The llama.cpp Flag That Made the Biggest Difference</h3>
This was the command that finally stopped fighting me:
<pre>./llama-server \ -m Qwen3.6-35B-A3B-Q4_K_M.gguf \ --n-gpu-layers 99 \ --n-cpu-moe \ --ctx-size 65536 \ --flash-attn \ -ctk q4_0 \ -ctv q4_0 \ -t 8</pre>
The important flag is:
<pre>--n-cpu-moe</pre>
That one matters way more than most people realize.
Without it, inactive MoE experts still create enough VRAM pressure that the KV cache starts competing for memory during long sessions.
With it, memory behavior becomes predictable enough that sessions stop randomly collapsing near the 30k-token mark.
That stability mattered more than raw speed.
<figure><img alt="One flag. The difference between running out of VRAM at 30k tokens and running clean past 60k | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*dqIclKq2JjXrX2w472b9MA.png"><figcaption>One flag. The difference between running out of VRAM at 30k tokens and running clean past 60k.</figcaption></figure><h3>Is Local Agentic Coding on 8GB VRAM Actually Viable?</h3>
Honestly?
Yeah. More than I expected.
Not “replace a cloud 4090 cluster” viable. Not “run giant autonomous repo agents for 14 hours” viable.
But absolutely viable for:
<ul>
<li>Focused coding sessions</li>
<li>Feature implementation</li>
<li>Debugging</li>
<li>Test iteration</li>
<li>Medium-sized refactors</li>
</ul>
Especially if you pair:
<ul>
<li>8GB VRAM</li>
<li>32GB RAM</li>
<li>Qwen3.6–35B-A3B</li>
<li>llama.cpp</li>
<li>KV cache quantization</li>
</ul>
The funniest part is that the setup people kept dismissing as “too weak” was the first one that actually let me stop thinking about hardware and just code.
That’s probably the real benchmark.
<figure><img alt="38 tok/s at 12k context. Not a 4090. Good enough | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*Z-2TGnhOMhqb3ATgrxKEgw.png"><figcaption>38 tok/s at 12k context. Not a 4090. Good enough.</figcaption></figure>I’ve also been experimenting with replacing cloud coding assistants entirely with local setups. That’s probably the next rabbit hole because the tradeoffs get weird fast once you start comparing local models against tools like Copilot or Claude Code directly.
<a href="https://ai.plainenglish.io/replaced-github-copilot-with-local-ai-gemma-ollama-2d2dbd7f47e2">I Replaced GitHub Copilot With a Local Setup. Here’s What Nobody Tells You</a>
If you’ve been waiting to try local agentic coding until you could “afford a proper GPU” — you don’t have to wait.
The setup exists.
It’s just buried underneath a lot of benchmark screenshots from people running hardware most normal developers don’t own.
Honestly, that was probably the most useful thing I learned from all of this.
<h3>A message from our Founder</h3>
Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.
If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>. And before you go, don’t forget to clap and follow the writer️!
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=51a9528dd77e" width="1" height="1" alt=""><hr>
<a href="https://ai.plainenglish.io/best-local-llm-for-8gb-vram-coding-i-tried-everything-51a9528dd77e">Best Local LLM for 8GB VRAM Coding? I Tried Everything.</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Best Local LLM for 8GB VRAM Coding? I Tried Everything.

<figure><img alt="Google Gemma 4 MTP Speculative Decoding Benchmarks | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*YPyq7iDyn-tBeoLE-8LdGg.png"><figcaption>Google Gemma 4 MTP Speculative Decoding Benchmarks</figcaption></figure><h4>Nobody mentions the annoying parts.</h4>
Google’s new Multi-Token Prediction update promises 2x faster inference with zero quality loss.
So I set it up, tested that claim, and found a few things worth knowing before you do the same.
<figure><img alt="Three evenings, a lot of llama.cpp flags, and one Reddit comment from somebody named catgpu92 | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*m1XYeXH_c4Ttx_WVDwvrBQ.png"><figcaption>Three evenings, a lot of llama.cpp flags, and one Reddit comment from somebody named catgpu92.</figcaption></figure>I’ve been running <a href="https://deepmind.google/models/gemma/">Gemma 4</a> locally for a while now. I meant to spend maybe an hour testing Gemma 4 MTP.
Pull the models. Run a few prompts. Check the throughput numbers. Write a quick post. Done.
Instead, I ended up spending three evenings poking at <a href="https://github.com/ggml-org/llama.cpp">llama.cpp</a> flags, reading half-finished GitHub threads, and trying to figure out why one setup felt weirdly slower despite the benchmarks saying otherwise.
The speedups are real. That’s the important part.
But the release notes smooth over a bunch of details that matter once you actually try running this locally.
And honestly, that’s been the pattern with local LLM tooling lately. The demos look frictionless. The real setup usually involves one Reddit comment from somebody named catgpu92 explaining the thing the docs forgot to mention.
If you’ve been experimenting with local AI workflows recently, especially around quantized coding models or VRAM-constrained setups, you’ve probably seen similar behavior already.
<h3>So What Actually Is MTP, and Why Should You Care?</h3>
Multi-Token Prediction (MTP) is Google’s answer to a fundamental bottleneck in local LLM inference.
Normally, a model generates tokens one at a time:
<ol>
<li>Generate token</li>
<li>Wait</li>
<li>Generate next token</li>
<li>Repeat forever until your GPU heats the room</li>
</ol>
Gemma 4 MTP changes that by attaching a small draft model alongside the main model.
The draft model guesses several tokens ahead. The main model then verifies those guesses in a single pass.
When the guesses are correct — which happens surprisingly often for code and predictable text — generation speeds up a lot.
Google claims up to 2x decoding speedup with zero degradation in output quality.
That last part zero degradation is actually the interesting claim.
<a href="https://arxiv.org/abs/2211.17192">Speculative decoding</a> in general has been around for a while. What makes Gemma 4 MTP worth paying attention to is that Google trained these draft models specifically on Gemma 4’s distribution, rather than bolting on a generic small model.
In theory, that means higher acceptance rates.
In practice, I wanted to see what that looked like on real hardware.
<h3>The Release Is Slightly More Confusing Than It Looks</h3>
At first I thought Google had released entirely new models.
They didn’t.
The <a href="https://huggingface.co/google">Hugging Face Gemma Repos</a> are draft assistants meant to pair with the base Gemma 4 models:
<ul>
<li>
gemma-4-E2B-it-assistant → draft for the 2B model (78M draft head)</li>
<li>
gemma-4-E4B-it-assistant → draft for the 4B model (156M draft head)</li>
<li>
gemma-4–26B-A4B-it-assistant → draft for the 26B MoE model (156M draft head)</li>
<li>
gemma-4–31B-it-assistant → draft for the 31B dense model (930M draft head)</li>
</ul>
The naming is a little confusing at first.
These aren’t standalone models.
So you need two things running together:
<ul>
<li>The actual Gemma 4 model</li>
<li>The matching draft assistant</li>
</ul>
This was obvious after twenty minutes.
It was not obvious during the first five minutes where I was staring at the repo names wondering if Google had invented the world’s worst naming convention on purpose.
The part people immediately noticed was the 31B assistant only being around ~930MB.
That’s small enough that the overhead feels reasonable even on tighter local inference setups.
<h3>How I Set This Up (and Where It Gets Annoying)</h3>
I’m running on a 24GB VRAM card.
For the 31B dense model pushing it a little too hard, so I went with the 26B MoE variant. Active parameters stay relatively low during inference, which makes it a much more comfortable fit for local deployment.
The setup that actually worked:
<pre># Pull the base model ollama pull gemma4:26b # Run speculative decoding through llama.cpp ./llama-server \ -m gemma-4-26B-Q4_K_M.gguf \ --draft-model gemma-4-26B-A4B-Q8_0.gguf \ --draft-max 8 \ -ngl 99 \ --ctx-size 32768</pre>
The part that slowed me down wasn’t llama.cpp itself.
Here’s the thing nobody quite spelled out in the initial release thread: Ollama doesn’t support MTP natively yet as of the time I’m writing this.
You can run Gemma 4 in Ollama just fine.
You just can’t do the speculative decoding part there yet, at least not cleanly.
I saw similar story with <a href="https://lmstudio.ai/">LM Studio</a> when I tested it.
Maybe that’ll change by the time you’re reading this. Right now it’s mostly llama.cpp territory.
Directly, or a server that exposes the--draft-model flag.
For llama.cpp, the command looks like this:
<pre># Run speculative decoding through llama.cpp ./llama-server \ -m gemma-4-26B-Q4_K_M.gguf \ --draft-model gemma-4-26B-A4B-Q8_0.gguf \ --draft-max 8 \ -ngl 99 \ --ctx-size 32768</pre>
Also:
--draft-max 8 seemed like the practical ceiling for me.
I tried pushing it higher because bigger number obviously means faster, right?
Didn’t really help.
At some point, the draft model starts missing too many predictions, verification overhead increases, and throughput actually dips.
For coding workloads, 6–8 felt consistently stable.
Creative writing was noticeably messier.
Which honestly makes sense.
<h3>The Speed Gains Are Real, Just Not Uniform</h3>
The benchmark headline is
<blockquote>Up to 2x faster inference.</blockquote>
That’s technically true.
I saw it.
Mostly on boring code.
<a href="https://medium.com/media/1f118956ff321dd4e8b970e07a3782c6/href">https://medium.com/media/1f118956ff321dd4e8b970e07a3782c6/href</a>For code completion and short generations (under ~200 tokens), the speedup was closer to 1.7–1.9x in practice.
Not quite the 2x ceiling, but noticeable.
Watching tokens fly out at that speed after weeks of standard inference felt a bit surreal, honestly.
For long-form generation behaved differently.
Explanations, documentation, and multi-step chains generally landed closer to 1.3–1.5x improvements.
The reason is fairly intuitive once you think about it: The less predictable the output becomes, the more often the draft assistant misses predictions.
Rejected drafts still cost compute.
For repetitive boilerplate code though — imports, CRUD scaffolding, unit tests — Gemma 4 MTP absolutely flies.
That’s where speculative decoding shines.
One thing I still find a bit counterintuitive: the output quality is genuinely identical to running without MTP.
I was half-expecting some subtle degradation like the model cutting corners somewhere. But speculative decoding is mathematically equivalent to standard decoding when the main model is doing the verification. The math holds up.
<figure><img alt="Not all tasks benefit equally. Boilerplate code is where MTP earns its headline number | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*yrwMRfYnYb1aSOJob5q3SA.png"><figcaption>Not all tasks benefit equally. Boilerplate code is where MTP earns its headline number.</figcaption></figure><h3>
The Weird Thing Is Gemma 4 MTP Doesn’t Reduce Output Quality</h3>
This part genuinely surprised me more than the raw speed gains.
I kept expecting subtle degradation.
<ul>
<li>Shortcuts.</li>
<li>Slightly sloppier reasoning.</li>
<li>Sloppier outputs.</li>
<li>Subtle degradations.</li>
<li>Something.</li>
</ul>
Didn’t happen.
The outputs matched exactly in my testing.
Which feels counterintuitive until you remember how speculative decoding actually works.
The main model still verifies every generated token before outputting anything.
So mathematically, the output remains equivalent to standard decoding.
That part impressed me more than the benchmark numbers themselves.
<h3>What’s the Catch With Gemma 4 MTP?</h3>
There Is One.
Actually several.
<h4>VRAM overhead is real</h4>
A few things the benchmarks won’t tell you:
The draft assistant still sits in VRAM alongside the main model.
For the 31B setup, you’re looking at roughly an additional ~1GB overhead.
If your GPU is already running near 95% utilization, MTP can absolutely push you over the edge.
Check your VRAM headroom before assuming it’ll slot in cleanly.
<h4>Prompt processing does not get faster</h4>
This only speeds up decoding.
If you throw a giant 64K context prompt into the model, prefill time still costs exactly what it used to cost.
A few people seemed disappointed by this in early threads, probably because “2x faster inference” sounds broader than what’s actually happening.
<h4>
The draft model acceptance rates vary heavily by workload</h4>
This is the biggest thing benchmark charts usually hide.
Your effective speedup isn’t fixed — it depends on how predictable your output is.
<ul>
<li>Coding = high acceptance rate = big gains.</li>
<li>Open-ended creative tasks = lower acceptance rate = more modest gains.</li>
</ul>
Keep that in mind when you see benchmark numbers measured on code benchmarks exclusively.
<h4>The llama.cpp PR situation is still a little messy</h4>
There’s an experimental llama.cpp MTP PR floating around that some people are using for even more aggressive throughput gains.
The maintainers themselves have described parts of it as unstable.
Personally, I stopped there.
I’ve reached the stage where:
<blockquote>“slightly faster inference” no longer outweighs “random segfault at 1AM.”</blockquote>
<figure><img alt="Your VRAM headroom matters more than the benchmark sheet suggests | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*jOhjeRpeBdAF6-GnobKhvQ.png"><figcaption>Your VRAM headroom matters more than the benchmark sheet suggests.</figcaption></figure><h3>Which Gemma 4 MTP Model Should You Actually Use?</h3>
If you have 16GB VRAM: The 4B + E4B assistant is your realistic option. Gains are smaller in absolute terms, but the model is fast enough already that the UX improvement is still noticeable.
If you have 24GB VRAM: The 26B MoE + its assistant is the sweet spot. Active parameters are around 4B during inference despite the larger model size, so it fits comfortably and the quality jump over 4B is meaningful.
If you have 48GB+: The 31B dense + its 930MB assistant is where MTP really shines. Dense models have higher draft acceptance rates than MoE models for most tasks. The 31B is also competitive with cloud models on coding benchmarks — which is the whole point of running this locally.
<h3>Is Gemma 4 MTP Worth Using Right Now?</h3>
For local coding workflows? Honestly, yes!
After a couple days I stopped turning MTP off for coding tasks entirely.
The speed difference is large enough that going back feels sluggish pretty quickly.
For long-form reasoning or creative work, I’m less certain the setup friction is worth it yet. The gains are still there, just noticeably smaller.
The main thing is this:
Gemma 4 MTP works better than I expected once it’s configured correctly.
The annoying part is getting to that point.
Setting everything up took me maybe 45 minutes total, including the time spent reading model cards, digging through Reddit comments, and realizing a few instructions were already outdated by the time I found them.
Would’ve been faster if somebody had written this article first.
<h3>How Does Gemma 4 MTP Compare to Qwen?</h3>
Conceptually, the workflows are very similar.
In my limited testing, throughput gains were in roughly the same range for coding tasks.
Gemma 4 felt a little more coherent during longer reasoning chains, but that’s one setup, one GPU, and not enough testing to turn into a grand conclusion.
I’ll probably do a deeper Qwen speculative decoding comparison later because the acceptance-rate behavior between dense and MoE models gets interesting pretty quickly.
<figure><img alt="Once it’s configured, you stop noticing it’s fast. You just notice that the old speed feels broken | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*JCH4U3HgN_pj1sD1VWEhvg.png"><figcaption>Once it’s configured, you stop noticing it’s fast. You just notice that the old speed feels broken.</figcaption></figure>If you’re already comfortable running local LLMs, this is worth an afternoon.
If you’re brand new to local inference, I’d still start with a normal Gemma setup first before layering speculative decoding on top.
MTP adds just enough extra moving pieces that debugging becomes a lot easier once you already know what “normal” looks like.
Related Reading
If you’re interested in local AI infrastructure and inference optimization, you’ll probably also enjoy:
<ul>
<li><a href="https://ai.plainenglish.io/i-ran-gemma-4-locally-heres-what-nobody-s-telling-you-806dcd485925">I Ran Gemma 4 Locally. Here’s What Nobody’s Telling You.</a></li>
<li><a href="https://ai.plainenglish.io/gemma-4-vs-gpt-vs-claude-real-world-test-ae676ddf0f19">I Tried Gemma 4 Against GPT and Claude for a Week</a></li>
<li><a href="https://ai.plainenglish.io/i-ran-qwen3-6-locally-the-thinking-mode-changed-how-i-use-it-5e5db6a14c78">I Ran Qwen3.6 Locally. The Thinking Mode Changed How I Use It</a></li>
</ul>
<h3>A message from our Founder</h3>
Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.
If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>. And before you go, don’t forget to clap and follow the writer️!
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=6711c8589d2f" width="1" height="1" alt=""><hr>
<a href="https://ai.plainenglish.io/gemma-4-mtp-local-inference-benchmarks-6711c8589d2f">I Spent 3 Nights Testing Gemma 4 (MTP)</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

I Spent 3 Nights Testing Gemma 4 (MTP)

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2f77aVgNtRuKom3ka-70YA.png"><figcaption>A 3-day test that changed how I use local AI</figcaption></figure><h4>This Isn’t What the Leaderboards Show</h4>
<h4>I tested Qwen3.6–27B locally using Ollama, what the benchmarks skip, why thinking mode matters, and whether it’s worth the setup cost.</h4>
I almost ignored it.
<figure><img alt="I almost didn’t bother. Then one number pulled me back | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*MsS1FsDokyq7CGMx1KQ3Vg.png"><figcaption>I almost didn’t bother. Then one number pulled me back.</figcaption></figure>Every week, there’s a new model that supposedly changes everything.
At this point, that phrase means almost nothing.
So when Qwen3.6–27B dropped, I skimmed the benchmarks… and moved on.
Then one number pulled me back.
Then I noticed something weird: A 27B model apparently outperforming a 397B one on coding tasks.
That didn’t make sense.
So I did what I usually do: I ran it locally.
So I went back, pulled it locally, and ran it the same way I test everything else no API, no wrapper UI, just Ollama and actual work I had to finish.
And after using it for a few days, I realized:
<blockquote>The benchmarks aren’t the interesting part.</blockquote>
<h3>Why I Paid Attention This Time</h3>
I’ve been testing local models for a while now.
<ul>
<li><a href="https://ai.plainenglish.io/i-ran-gemma-4-locally-heres-what-nobody-s-telling-you-806dcd485925?source=user_profile_page---------0-------------845264e55c6f----------------------">I Ran Gemma 4 Locally. Here's What Nobody's Telling You.</a></li>
<li><a href="https://ai.plainenglish.io/replaced-github-copilot-with-local-ai-gemma-ollama-2d2dbd7f47e2">I Replaced GitHub Copilot With a Local Setup. Here’s What Nobody Tells You</a></li>
</ul>
After running Gemma 4 locally (which performed surprisingly well in controlled tasks), I didn’t expect another model to shift my workflow this quickly.
But Qwen3.6 did not because it’s bigger or faster.
Because of how it thinks.
<h3>Setting Up Qwen3.6 Locally Is Almost Too Easy</h3>
This used to be the annoying part.
This was not always true. A year ago, running a capable model locally meant dealing with dependency hell, quantisation format confusion, and RAM calculations done on the back of a napkin.
With Qwen3.6–27B, the whole thing took about six minutes.
Now it’s… kind of boring:
<pre>ollama pull qwen3.6:27b ollama run qwen3.6:27b</pre>
That’s it.
No setup rabbit hole. No weird configs.
The Q4_K_M version is around 17GB, which fits comfortably if you’ve got 24GB VRAM. If not, or runs mostly in RAM if you’re on a machine without a beefy GPU.
On my machine, it booted in under half a minute.
I’m not going to pretend the output speed was indistinguishable. It’s not cloud fast…
You notice it at first. Especially when you turn on thinking mode. But after a bit, your brain adjusts and stops caring.
<h3>The 27B vs 397B Thing Isn’t What It Sounds Like</h3>
The Benchmark Confusion (27B vs 397B)
This part gets misrepresented everywhere.
Before getting to thinking mode, it’s worth addressing the thing that made me pay attention in the first place.
Qwen3.6–27B is a dense model.
Qwen3.5–397B is Mixture-of-Experts.
Which means… the 397B number isn’t really what’s active during inference.
<a href="https://ai.plainenglish.io/gemma-4-vs-gpt-vs-claude-real-world-test-ae676ddf0f19">I Tried Gemma 4 Against GPT and Claude for a Week</a>
That headline number? Misleading.
Roughly ~17B parameters are actually being used per token in that MoE model.
So the comparison is closer to:
<blockquote>~17B active vs 27B dense</blockquote>
That’s a very different story.
So what’s actually happening is:
It’s that Qwen’s team trained a dense 27B model efficiently enough that it can compete at that level at all.
That’s the actual story. Most of the posts I read just screenshot the leaderboard without explaining why.
I still find this a bit counterintuitive, honestly my brain keeps wanting to say “but 397 is bigger than 27” but once you understand active parameters, the confusion mostly dissolves.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9IsA_4-yw_xXzKtpc9p_vw.png"><figcaption>The ‘27B beats 397B’ headline is technically true but deeply misleading. Here’s why.</figcaption></figure><h3>Thinking Mode Is the Only Feature That Actually Changed How I Used It</h3>
This is the part I want to spend time on, because I’ve seen it mentioned in nearly every Qwen3.6 post and almost never explained in practical terms.
So here’s the version after actually using it for a few days:
Qwen3.6 ships with two modes:
Standard generation mode and a “thinking mode” that uses extended chain-of-thought reasoning before giving you an answer.
You can toggle between them mid-conversation. In Ollama, you can activate it directly in the system prompt, or just ask the model to “think step by step” on harder tasks it recognises the intent.
Qwen3.6 introduces a “thinking mode” a reasoning-heavy generation mode that takes longer but processes tasks more deeply.
Sounds familiar. But the practical difference is where it matters.
Here’s the honest breakdown of what that actually means in practice:
When Thinking Mode Actually Helps
Anything involving logic, debugging, multi-step reasoning, or tasks where the first plausible answer is usually wrong.
I threw a particularly ugly async bug at it, the kind where the error message is pointing at the wrong line, and the thinking-mode response traced through the execution order and caught the real issue.
Standard mode gave me a confident but incorrect fix. That gap was significant.
<h3>When thinking mode doesn’t help:</h3>
Short, factual lookups. Writing tasks.
Anything where the answer is relatively direct.
In these cases, the thinking mode just adds latency without meaningfully improving the output. I ran a batch of simple questions through both modes, and the results were nearly identical, but the thinking mode was consistently 2–3× slower.
<h3>The practical takeaway</h3>
Treat thinking mode like a debugger, not a default.
Activate it for the 20% of tasks where you actually need the model to reason carefully. Leave it off for everything else.
Once I settled into that pattern, Qwen3.6 became noticeably more useful than anything I’d run locally before.
There’s also a third approach, and I say this somewhat reluctantly because it involves a prompt engineering pattern I used to roll my eyes at where you explicitly tell the model when to switch modes as part of your workflow. Something like a system prompt that says “use thinking mode for any task involving code review or debugging, standard mode otherwise.” It sounds fussy. In practice, it actually works really well.
<h3>Using It for Actual Work (Not Benchmarks)</h3>
I used Qwen3.6–27B as my primary coding assistant for three days before writing this.
Not for toy examples, for actual tasks from my Shopify development work.
<h4>Here’s what I found:</h4>
Code generation: Solid. Not Claude Sonnet-level, but genuinely useful for scaffolding, boilerplate, and explaining unfamiliar library APIs. On straightforward tasks, it produced clean, runnable code more often than not.
Debugging: This is where thinking mode earns its keep. Given a stack trace and relevant code context, it was more methodical than most cloud models I’ve used at identifying the actual root cause rather than the surface symptom.
Long context tasks: I fed it a 4,000 word codebase context and asked questions about architecture. It handled this reasonably well, better than I expected for a local model, though I noticed it started losing track of earlier context on very long exchanges. This is still an area where cloud models have an edge.
Tool calls / MCP: Mixed. Standard prompt-based tasks are fine. If you’re building agents, you’ll hit some of the same tool-call reliability issues that affect most local models at this size. It’s better than Gemma 4 was at launch for this, but I wouldn’t stake a production pipeline on it without thorough testing.
The honest version: Qwen3.6–27B is the first local model where I didn’t feel like I was making a compromise the moment I needed it for something real. Previous “best local models” had a way of suddenly reminding you they were local models. This one does too, occasionally, but the reminders are less frequent.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fPgs4W3fJpSwHkvQcfby7g.jpeg"><figcaption>Bigger models don’t guarantee better outcomes, what matters is how efficiently they use what they have.</figcaption></figure><h3>The Part Nobody Talks About: What You’re Actually Trading</h3>
Every “run X locally” article I’ve read eventually turns into an ad for local models. This one isn’t going to do that, because the tradeoffs are real and worth naming.
What you gain: Privacy. Zero per-token cost. No rate limits. The ability to run offline. And — this matters more than people admit — the feeling of actually understanding the tool you’re using rather than renting access to a black box.
What you give up: Raw output speed (cloud models are still faster). The latest models, the moment they drop (cloud APIs deploy updates faster than Ollama catches up). And most practically, the confidence that comes from using a model that’s been tested by millions of people in production.
I don’t know if any of this is true for setups different from mine. On my machine, Qwen3.6–27B made local AI feel genuinely practical for the first time. Whether that applies to a machine with less VRAM, a different workflow, or different tasks — that’s something you’d need to test yourself.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QgFkVltELox3yrOg3sEtWw.png"><figcaption>Every local model setup comes with tradeoffs only discover after you’ve committed.</figcaption></figure><h3>So… Is It Worth Running?</h3>
Depends where you’re starting.
If you already use local models, yes immediately!
Qwen3.6–27B is the clearest upgrade to the local model ecosystem in 2026 so far, and the thinking mode genuinely adds capability rather than just being a marketing feature.
If you’re coming from cloud APIs and wondering whether local is worth the setup cost.
<blockquote>The honest answer is that it depends entirely on your use case.</blockquote>
If you’re doing a lot of reasoning-heavy work and you have the hardware, a machine with 24GB VRAM or more, the economics make sense within a few months. If you’re primarily doing quick lookups or short generation tasks, the latency difference will frustrate you more than the cost savings will delight you.
If you’ve never run a local model before, start with the Qwen3.6 7B first.
Get comfortable with Ollama.
Understand how quantisation affects the tradeoffs.
Then graduate to 27B when you know what you’re actually looking for.
The thinking mode alone made this worth an afternoon of my time. Whether it’s worth yours depends on what you’re building.
If you found this useful, I write about local AI setups, developer tools, and the honest side of building with AI. Follow for more.
Or if you are feeling pumped &amp; excited after reading this, you’ll feel the same after reading these:
<a href="https://ai.plainenglish.io/i-built-a-gemma-4-ai-agent-it-kept-looping-until-i-fixed-this-1ef53f50158c">I Built a Gemma 4 AI Agent. It Kept Looping Until I Fixed This.</a>
<h3>A message from our Founder</h3>
Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.
If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>. And before you go, don’t forget to clap and follow the writer️!
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=5e5db6a14c78" width="1" height="1" alt=""><hr>
<a href="https://ai.plainenglish.io/i-ran-qwen3-6-locally-the-thinking-mode-changed-how-i-use-it-5e5db6a14c78">I Ran Qwen3.6 Locally. The Thinking Mode Changed How I Use It</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

I Ran Qwen3.6 Locally. The Thinking Mode Changed How I Use It

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tmxBX4PceNLf_qMBpApEBg.png"><figcaption>The Gemma Specialization Supernova</figcaption></figure><h4>Beyond Gemini</h4>
<h4>An ecosystem of focused AI models is replacing the “one model does all” approach.</h4>
From cancer detection to dolphin communication, Google’s open-weight <a href="https://ai.google.dev/gemma">Gemma models</a> are evolving into the most diverse specialized AI ecosystem developers can actually use.
Most people think Google’s AI strategy is Google Gemini.
The big model.
The flagship.
The one that tries to do everything.
But while everyone is watching Gemini…
<figure><img alt="Google’s Gemma has quietly expanded into one of the most domain-diverse open ecosystems in AI, and most developers haven’t noticed | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*2j7RTZDTx1SC42XUwjYkQw.png"><figcaption>Google’s Gemma has quietly expanded into one of the most domain-diverse open ecosystems in AI, and most developers haven’t noticed.</figcaption></figure>Google is quietly building something far more dangerous:
<blockquote>AI that doesn’t try to do everything — it tries to do one thing perfectly.</blockquote>
And almost nobody is talking about it.
We are witnessing the rise of Google Gemma specialized AI models. An ecosystem where developers can run models locally, fine-tune them, and avoid API lock-in entirely.
<h4>The Gemmaverse Is Bigger Than You Think</h4>
<a href="https://medium.com/ai.google.dev/gemma">Google Gemma</a> started in early 2024 as a lightweight alternative to Gemini.
A set of modest, open-weight AI models that developers could actually download and run locally. Two sizes. Modest ambitions.
Then something unexpected happened.
<ul>
<li>400M+ downloads</li>
<li>
100K+ community variants (explore ecosystem: <a href="https://huggingface.co/models?search=gemma">HuggingFace</a>)</li>
<li>Massive developer experimentation</li>
</ul>
And Google started listening very carefully to what people were doing with them.
And the signal was clear:
<blockquote>Developers didn’t want bigger models. They wanted sharper ones.</blockquote>
So Google changed direction.
Not toward scale. Toward specialization.
<h4>One Family. Dozens of Domains.</h4>
Here’s what most people don’t realize:
<blockquote>Gemma is no longer a model. It’s becoming an operating system for domain-specific AI.</blockquote>
Instead of one general intelligence…
Google is building purpose-trained models, optimized for specific tasks:
<ul>
<li>Medicine</li>
<li>Code</li>
<li>Translation</li>
<li>Safety</li>
<li>Search</li>
<li>Edge computing</li>
</ul>
This is a shift from intelligence → infrastructure.
Let’s break down the most fascinating branches of this ecosystem.
<figure><img alt="The Gemmaverse: 9 specialized models, each purpose-built for a distinct domain. 100,000+ community variants. 400 million downloads | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*jxFgkux9Zm591Qf-f1rcjg.png"><figcaption>The Gemmaverse: 9 specialized models, each purpose-built for a distinct domain. 100,000+ community variants. 400 million downloads.</figcaption></figure><h4>MedGemma: AI That Understands Medicine</h4>
<a href="https://deepmind.google/models/gemma/medgemma/">MedGemma</a> is built on advanced Gemma 3 and fine-tuned specifically for medical data, text, imaging, and clinical reasoning.
This isn’t generic AI trying to “understand healthcare.” (see <a href="https://health.google/">Google Health</a>)
This is an AI trained to think in a medical context.
It’s designed for tasks like reading medical imaging, parsing clinical notes, and supporting diagnostic reasoning, the kind of work that typically requires either a trained clinician or an extremely expensive proprietary model.
Real-world example:
A team in Gurgaon used it to build a diabetes management tool running entirely on their own infrastructure.
<ul>
<li>No API dependency.</li>
<li>No data is leaving their system.</li>
</ul>
That’s a big deal.
Because in healthcare, <a href="https://www.who.int/publications/i/item/9789240029200">privacy isn’t optional</a> — it’s existential.
<blockquote>⚠️ Important: MedGemma is not clinical-grade yet.</blockquote>
It shouldn’t replace doctors. But that’s not the point.
It replaces the busywork around doctors.
<ul>
<li>Intake summaries</li>
<li>Documentation</li>
<li>Symptom structuring</li>
</ul>
That alone is massive.
<h4>DolphinGemma: The Model That Breaks Your Mental Model</h4>
DolphinGemma is where things get weird and important.
<a href="https://deepmind.google/models/gemma/dolphingemma/">DolphinGemma</a> is roughly ~400M parameter model built with Google DeepMind, Georgia Tech, and the <a href="https://www.wilddolphinproject.org/">Wild Dolphin Project</a>.
Its purpose?
Understanding dolphin communication.
<ul>
<li>Not text.</li>
<li>Not images.</li>
</ul>
Dolphin vocalizations.
Let that sink in.
<blockquote>This isn’t an experiment. This is proof that LLM architectures can escape human language entirely.</blockquote>
And it forces a question:
And it raises a profound question worth sitting with: if you can train a specialized language model to parse the high-frequency click-and-whistle sound patterns of dolphins, what can’t you build a domain-specific model for?
DolphinGemma serves as a proof of concept in the most literal sense. It is irrefutable proof that the Gemma architecture is flexible enough to work far beyond the standard text-and-image domains that people usually associate with LLMs.
<figure><img alt="DolphinGemma is trained on dolphin vocalizations. It treats their sounds the way a language model treats human text. That is not a metaphor | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*TXo9jmLVbaRp7Hh9XtN9nA.png"><figcaption>DolphinGemma is trained on dolphin vocalizations. It treats their sounds the way a language model treats human text. That is not a metaphor.</figcaption></figure><h4>
<a href="https://deepmind.google/models/gemma/codegemma/">CodeGemma</a> — The End of Cloud Subscriptions?
</h4>
<a href="https://huggingface.co/google/codegemma-7b-it">CodeGemma</a> comes in highly optimized 2B and 7B-parameter sizes, trained explicitly for code completion, debugging, and general software engineering tasks.
Not scraped knowledge.
<blockquote>Not AI that also codes.</blockquote>
It supports:
<ul>
<li>Python</li>
<li>JavaScript</li>
<li>TypeScript</li>
<li>C++</li>
<li>Java</li>
</ul>
It integrates naturally into the broader open-source ecosystem (<a href="https://github.com/">GitHub</a>).
And here’s the real advantage:
<ul>
<li>A smaller, specialized model fine-tuned on your codebase</li>
<li>Can outperform a massive general model.</li>
</ul>
That flips the usual assumption:
Bigger ≠ better. Closer to the problem = better.
<h4>The Others You Probably Haven’t Heard Of</h4>
The Gemmaverse doesn’t stop at the three above. It keeps going:
<h4>TranslateGemma</h4>
<a href="https://huggingface.co/collections/google/translategemma">TranslateGemma</a> is available in 4B, 12B, and 27B sizes and is built to help people communicate across 55 languages. Not a general translation dedicated translation architecture.
<blockquote>55-language communication layer</blockquote>
<h4>ShieldGemma 2</h4>
<a href="https://deepmind.google/models/gemma/shieldgemma-2/">ShieldGemma 2</a> — A robust 4B image safety classifier. It outputs real-time safety labels across dangerous content, sexually explicit material, and violence categories. It acts as a ready-made content moderation layer that any backend developer can instantly drop into their ingestion pipeline.
<blockquote>Built-in safety classification</blockquote>
<h4>FunctionGemma</h4>
<a href="https://huggingface.co/google/functiongemma-270m-it">FunctionGemma</a> — A staggeringly small 270M parameter model built specifically for robust function calling at the edge.
It is tiny.
It is fast.
It is designed precisely to work on constrained hardware where sending every structured JSON request to a cloud API isn’t practical, stable, or affordable.
<blockquote>Ultra-light edge function calling</blockquote>
<h4>EmbeddingGemma</h4>
<a href="https://deepmind.google/models/gemma/embeddinggemma/">EmbeddingGemma</a> — Sitting at 308M parameters, this is purpose-built for generating embeddings — the core numeric representations that power semantic search, dense RAG (Retrieval-Augmented Generation) pipelines, and modern recommendation systems.
semantic search + RAG backbone
<h4>PaliGemma 2</h4>
<a href="https://deepmind.google/models/gemma/paligemma-2/">PaliGemma 2</a> — A dynamic vision-language model available in 3B, 10B, and 28B sizes, designed for multi-modal tasks that require a deep understanding of both complex images and text simultaneously.
<blockquote>Vision + Language reasoning</blockquote>
<h4><a href="https://deepmind.google/models/gemma/gemma-3n/">Gemma 3n</a></h4>
The “n” stands for on-device.
Optimized specifically for mobile and edge hardware. E2B and E4B variants that run efficiently on Android devices and laptops.
Each of these is a focused tool — not a general-purpose assistant trying to do everything.
<blockquote>Optimized for on-device deployment</blockquote>
Each model is a tool, not a general assistant.
<h4>Here’s the Shift Most People Are Missing</h4>
Everyone is focused on:
<ul>
<li>Benchmarks</li>
<li>
<a href="https://arena.ai/leaderboard/text">Arena rankings</a> (see Chatbot Arena)
</li>
<li>“Which model is #1?”</li>
</ul>
They debate which model beats which at MMLU or HumanEval.
That’s the wrong game.
That’s not the game Google is playing with Gemma.
The future of AI isn’t one model dominating everything.
It’s many models dominating something.
This is the “surface area strategy.”
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nFzl8vn0NCO7fvWcVg4LVg.png"><figcaption>400 million downloads. 100,000+ community variants. The Gemmaverse is not an experiment, it is already infrastructure.</figcaption></figure>Gemma 4 just launched in April 2026 under an Apache 2.0 license — fully open for commercial use. That’s not a small detail. It means a startup can build a clinical documentation tool on MedGemma, a content moderation system on ShieldGemma, and a multilingual customer support tool on TranslateGemma, all without a usage agreement, a legal review, or a per-token bill.
Cover enough use cases…
And you don’t need to win the benchmark war.
You win the ecosystem.
<h4>What Google Gemma Specialized AI Models Mean for Developers Right Now</h4>
This isn’t just product design.
It’s positioning.
If you’re building any software in 2026 that touches a specialized domain, the first question worth asking yourself isn’t “which frontier API should I call?”
It is: Is there a Gemma variant for this?
Now a startup can build:
<ul>
<li>Healthcare tools → MedGemma</li>
<li>Dev tools → CodeGemma</li>
<li>Moderation → ShieldGemma</li>
<li>Search → EmbeddingGemma</li>
</ul>
Without asking permission.
That’s not a model advantage.
That’s an ecosystem advantage.
With Gemma 4 (Apache 2.0 licensed), Google removed:
<ul>
<li>API lock-in</li>
<li>Usage restrictions</li>
<li>Cost barriers</li>
</ul>
The only thing left is to start.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjC0aSP_0p8_3RgpPmuG2A.png"><figcaption>The tools are open. The license is permissive. The community is already building. The only thing left is to start.</figcaption></figure><h4>Here are some Questions which might be bothering you, I’ll clarify right now:</h4>
<h4>What are all the Google Gemma model variants?</h4>
Here’s a quick breakdown of the most interesting ones:
Gemma 4: The core general-purpose foundation model (available in 4 sizes). MedGemma: Optimized for medical text parsing and clinical imaging.
CodeGemma: Purpose-built for code completion and offline development.
TranslateGemma: Dedicated translation architecture spanning 55 languages.
ShieldGemma 2: Image safety and content moderation classification.
FunctionGemma: Edge-optimized, ultra-lightweight function calling.
EmbeddingGemma: Built specifically for semantic search and RAG embeddings.
PaliGemma 2: Multi-modal vision-language understanding.
Gemma 3n: Highly optimized for on-device and mobile hardware.
DolphinGemma: Experimental model for marine biology and cetacean acoustic analysis.
<h4>Is MedGemma safe to use in clinical settings?</h4>
Not yet for final, unassisted diagnosis — Google explicitly states MedGemma isn’t clinical grade. But it is a remarkably strong foundation for building healthcare-adjacent tools: documentation workflows, triage support, intake summarization, and medical imaging assistance, where a human clinician ultimately reviews the output. Developers should treat it as powerful infrastructure, not a replacement for human clinical judgment.
<h4>Can I use Gemma models commercially?</h4>
Yes. Gemma 4 is released under the Apache 2.0 license, making it fully usable in commercial applications.
<h4>What is the difference between Gemma and Gemini?</h4>
Gemini is closed and API-based.
Gemma is open-weight and locally deployable.
<h4>How does DolphinGemma work?</h4>
It applies transformer-based modeling to dolphin vocalizations, treating sound patterns similarly to language sequences.
<h3>Considering all of the facts</h3>
Google’s biggest AI bet isn’t building the smartest model.
It’s building the most useful collection of models.
And that’s a completely different game.
Everyone is watching the frontier race.
Bigger models.
Better benchmarks.
Flashier demos.
Meanwhile…
The landscape is shifting under our feet. Through Google Gemma specialized AI models, the future of development isn’t just getting smarter.
It’s getting hyper-specific, incredibly efficient, and radically open.
<blockquote>And developers are downloading it 400 million times.</blockquote>
<h3>A message from our Founder</h3>
Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.
If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>. And before you go, don’t forget to clap and follow the writer️!
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=dbceb5ebdb2d" width="1" height="1" alt=""><hr>
<a href="https://ai.plainenglish.io/google-is-quietly-building-an-ai-for-everything-and-no-one-is-noticing-dbceb5ebdb2d">Google Is Quietly Building an AI for Everything (And No One Is Noticing)</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google Is Quietly Building an AI for Everything (And No One Is Noticing)

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WWP1nx2WyKo3p5iSXgaPsA.png"><figcaption>The code wasn’t the problem. The pause was.</figcaption></figure><h4>Beyond Productivity</h4>
<h4>It wasn’t the output. That part got better. It was something else, slower, quieter.</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PD0ustDZAeqWbe4ukBxfFQ.png"><figcaption>Around day 15, I noticed something. I was waiting for Claude to say something. That was the tell.</figcaption></figure>Around day 15, I noticed something.
I got stuck on something that shouldn’t have slowed me down.
I was debugging a <a href="https://nodejs.org/en/docs/guides/debugging-getting-started/">Node.js</a> issue, a fairly routine <a href="https://redis.io/docs/">Redis</a> connection timeout.
The kind of thing I’d fixed probably thirty times in the past two years, that it usually doesn’t take long to at least guess what’s wrong.
But this time?
But I just sat there…
Not stuck exactly. Just… waiting a bit. Like I’d forgotten the first move.
And then it clicked:
I wasn’t trying to figure it out. I was waiting for Claude to say something first.
That part bothered me more than the bug.
That was the moment I realised something had shifted.
Not in Claude. In me.
<h3>This isn’t really about “AI makes you worse”</h3>
Agentic coding skills don’t disappear overnight. They drift.
You delegate a task. <a href="https://www.anthropic.com/claude">Claude</a> handles it well. You delegate another.
Over 30 days of using Claude Code heavily, tool calls, multi-step tasks, full feature builds, I started noticing specific things getting slower.
Not my output. My instincts.
There’s a loud debate happening right now:
<ul>
<li>This makes you 10x faster</li>
<li>Or this will replace developers
</li>
</ul>
The honest answer to “Does letting AI code for you makes you worse?”
<blockquote>You don’t get worse at shipping. You get weaker at the things you’d need if the AI disappeared tomorrow.</blockquote>
That distinction matters more than most people are willing to say out loud.
<h4>What I Actually Tracked</h4>
I want to be clear about what I mean when I say I “tracked” this, because I’m not reporting a controlled experiment.
I kept notes.
I paid attention to how long certain tasks took me when I did them manually, and compared that to how they felt a month earlier.
A few patterns kept repeating:
1️⃣ Debugging instinct slowed down.
Earlier, if something broke, I’d start forming a guess almost immediately.
Now, because my first instinct was now “let me ask Claude” rather than “let me think”.
<blockquote>“Let me just paste this and see what Claude says”</blockquote>
If you’ve ever pasted an error into AI before thinking, you’ve probably felt this.
It sounds harmless.
And sometimes it is.
But something subtle changes:
You stop initiating the diagnosis.
2️⃣ I was “reading” code without really reading it.
When Claude generates code, you review it.
But you don’t read it the way you’d read code you wrote yourself.
I caught myself approving chunks that I understood at a high level but couldn’t fully explain if you stopped me mid-way.
That’s new. I didn’t love that.
And honestly, a bit uncomfortable.
<figure><img alt="Skills you don’t exercise don’t stay sharp. The brain stops treating delegated tasks as something worth retaining | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*j7JgiDB_5DTjXXPH2N0Vaw.png"><figcaption>Skills you don’t exercise don’t stay sharp. The brain stops treating delegated tasks as something worth retaining.</figcaption></figure>3️⃣ My mental model had gaps
This one’s the hardest to explain.
When you write code yourself, even messy code, you kind of own the structure.
When AI writes it:
You inherit the result, not the reasoning.
I started noticing moments where I knew what the system did… but couldn’t easily explain why it was shaped that way.
Not a great feeling during debugging.
<h4>I don’t think this is a “problem” with Claude</h4>
This is a problem with how humans learn.
It feels more like a side effect of how brains work.
There’s research from cognitive science on how skill retention works under assisted performance.
<blockquote>
The core finding: when a tool reliably handles a task, the brain stops treating it as something worth retaining.</blockquote>
There’s a concept in cognitive science called <a href="https://en.wikipedia.org/wiki/Cognitive_offloading">cognitive offloading</a>.
Same mechanism behind why GPS navigation genuinely degrades spatial memory with extended use. Same reason surgeons who rely heavily on robotic tools maintain strong outcomes but slower manual fallback times.
Same reason muscle memory fades if you stop using it.
Nothing new. Just happening faster because the handoff is frictionless.
You don’t decide to delegate. You just… do.
This is actually the same pattern I wrote about in the context drift problem — models losing track of instructions mid-conversation, developers losing track of their own reasoning mid-workflow.
Both are attention management problems.
One happens in the model. One happens in you.
If you would like to understand why your AI Agent forgets instruction and context mid conversation, check out my article on context drift here:
<a href="https://ai.plainenglish.io/your-ai-agent-isnt-dumb-it-has-adhd-4686585bc5f2">Your AI Agent Isn’t Dumb. It Has ADHD</a>
<h4>What I changed (and yeah, none of this is dramatic)</h4>
I didn’t stop using AI.
I’m not going to tell you to stop using Claude Code. That’s not the lesson and it’s not what I did.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VAK4Sph53FqMixHjE6lUrA.png"><figcaption>The fix isn’t using AI less. It’s staying in the loop during the parts that build instinct.</figcaption></figure>Here’s what actually changed how I work:
1️⃣ I force myself to have an opinion first
Before I hand a problem to Claude, I spend ~5 minutes on it myself.
Not solving it.
Just:
<blockquote>“I think this smells like a race condition in Redis”</blockquote>
Half the time I’m wrong.
Still useful.
Still valuable.
Because:
Forming a hypothesis keeps your diagnostic instinct alive.
AI can still do the heavy lifting.
But you stay involved in the part that actually builds skill.
2️⃣ If I can’t explain it, I don’t ship it
Clean code is dangerous.
Especially when you didn’t write it.
If I can’t explain a block of generated code in plain terms:
I don’t commit it yet.
Yes, it slows things down slightly. Some sessions added ten minutes.
Some sessions it exposed things I genuinely didn’t understand and would have shipped without noticing. I think the second case happened more than I’d like to admit.
3️⃣ I do one manual session per day, no AI
No philosophy here.
Just… Something done completely manually.
Bug fix, small function, whatever.
Just enough to keep the baseline from drifting.
I still find this slightly annoying, which is probably the point.
I still catch myself wanting to open Claude out of habit. Which probably says enough.
<h4>The Part I’m Still Working Out</h4>
I don’t know if what I noticed is actual skill decay or just adaptation.
After a few weeks of these habits, things feel sharper again.
I’ve been running these three habits for about three weeks and things feel sharper, but I genuinely can’t tell if that’s real recovery or just familiarity with a new workflow.
Also not sure how much any of this matters long term.
This still confuses me a little, honestly.
Also not sure how much this matters long-term.
If tools keep improving, maybe the baseline just shifts and this whole concern becomes irrelevant. Or maybe debugging becomes the one skill that really matters, and this gets more important.
Honestly, I’m guessing a bit here. I don’t have a clean take on that yet.
<h4>If there’s one thing I’d keep in mind</h4>
It’s not:
<blockquote>“Use AI less”</blockquote>
That framing feels wrong.
It’s more like:
<blockquote>Stay involved in the part where you’re forming explanations, not just accepting them.</blockquote>
Because the risk isn’t that AI writes bad code.
It’s that it writes good code that you only half-understand.
Because the weird moment for me wasn’t bad output.
It was sitting in front of a problem I understood… and hesitating.
Waiting for something external to kick things off.
<h3>And yeah, that moment stuck with me, Back to that Redis bug</h3>
I eventually fixed it.
It wasn’t even complicated.
Instead, I paused.
Looked at the screen.
And waited for Claude to say something.
That’s the part I’m paying attention to now.
Not the speed gains. Not the productivity graphs.
<figure><img alt="The tools aren’t going anywhere. Make sure you aren’t either | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*ki170zTbUIUNmkrB7mG9Qw.png"><figcaption>The tools aren’t going anywhere. Make sure you aren’t either.</figcaption></figure>Just that small hesitation.
That was the tell.
The tools aren’t going anywhere. Make sure you aren’t either.
<h3>A message from our Founder</h3>
Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.
If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>. And before you go, don’t forget to clap and follow the writer️!
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=14eb8c39e6a3" width="1" height="1" alt=""><hr>
<a href="https://ai.plainenglish.io/ai-coding-skill-drift-30-days-14eb8c39e6a3">I Let Claude Code for Me for 30 Days. Here’s What Nobody’s Telling You.</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

I Let Claude Code for Me for 30 Days. Here’s What Nobody’s Telling You.

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JhI_FsrjipnXgqOAte7MvA.png"><figcaption>I genuinely feel the change in perception on using AI.</figcaption></figure><h4>Beyond Benchmarks</h4>
<h4>Real developer tasks — coding, debugging, writing — and what actually matters in practice.</h4>
<figure><img alt="A week of actual work. Not screenshots | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*58PvaAsRy3uradbih4IOZg.png"><figcaption>A week of actual work. Not screenshots.</figcaption></figure>The numbers didn’t feel real at first. The benchmarks looked suspicious.
89.2% on <a href="https://artofproblemsolving.com/wiki/index.php/AIME">AIME</a>. 80% on <a href="https://livecodebench.github.io/">LiveCodeBench</a> v6. A Codeforces ELO that jumped from 110 to 2,150 in a single generation that reads like someone forgot to sanity check the slide before publishing it.
<a href="https://deepmind.google/">Google DeepMind</a> released Gemma 4 on April 2, 2026. The numbers read more like it had been optimized for headlines.
So instead of arguing about benchmarks (which I’ve done before, and it goes nowhere), I decided to stop reading and start testing.
I spent a week running Gemma 4 against GPT and Claude on the kinds of tasks I actually do: debugging, writing component logic, explaining unfamiliar APIs, and drafting docs.
<blockquote>This is Gemma 4 vs GPT vs Claude from someone who was genuinely skeptical going in.</blockquote>
<h4>What Gemma 4 Actually Is (Before We Compare Anything)</h4>
Gemma 4 is Google DeepMind’s latest family of open-weight models. Four sizes: E2B, E4B, 26B MoE, and 31B Dense. Apache 2.0 license, which means you can use it commercially, self-host it, fine-tune it, and ship it inside a product without negotiating with anyone.
<blockquote>The 31B Dense is the flagship. That’s what I tested most heavily.</blockquote>
The MoE variant 26B total parameters, but only 3.8B active per inference, is the one I’d actually use in production. It matches roughly 97% of the 31B’s quality at a fraction of the compute cost. I’ll come back to that.
<h4>What I actually did (nothing fancy)</h4>
I didn’t run synthetic benchmarks. I ran the things that were already on my to-do list:
<ul>
<li>A custom React hook with some messy state management</li>
<li>An async race condition in a Node.js service that had been bugging me for two days</li>
<li>A piece of the Next.js App Router docs I kept avoiding because I didn’t fully understand it.</li>
</ul>
Also: writing test cases for a utility function, a short technical spec, some changelogs, and PR descriptions. The unglamorous stuff that takes more time than it should.
I rotated between Gemma 4, GPT, and Claude without overthinking it. Same prompts where possible. Sometimes not. I wasn’t trying to be fair, just honest.
The same kind of work I described when I wrote about automating 80% of my workflow with AI, not glamorous, but time-consuming.
<a href="https://generativeai.pub/i-automated-80-of-my-workflow-with-ai-bb8115b600bd">I Automated 80% of My Workflow With AI</a>
<h4>Where Gemma 4 Genuinely Surprised Me</h4>
The coding tasks.
That’s where I least expected it to hold up, and it did.
On the React hook, Gemma 4 31B produced clean, idiomatic code on the first pass. No hallucinated APIs. No incorrect hook dependency arrays. It even flagged a stale closure issue I hadn’t explicitly mentioned.
That part felt… slightly uncomfortable, honestly. Like it inferred more than I intended.
The async bug was more interesting. I pasted ~80 lines, described the symptoms, and Gemma 4 correctly identified that the problem was in how I was handling <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled">Promise.allSettled</a> results, specifically, that I wasn’t guarding against a rejected promise before accessing its value property. GPT got there too, eventually, but asked three clarifying questions first. Claude nailed it in one shot and explained it more clearly. Gemma 4 nailed it in one shot with less explanation.
<figure><img alt="Test suite. All green. First pass. | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*YXA_LusLQFU4wMaTFIXcCA.png"><figcaption>Test suite. All green. First pass.</figcaption></figure><blockquote>I’m not sure if “less explanation” is a feature or a bug. Depends on the day.</blockquote>
For generating test cases, Gemma 4 was genuinely fast and solid. Edge cases included, structure was sensible, no unnecessary padding. That’s probably where I’d trust it the most without thinking twice.
<h4>Where GPT and Claude Still Win</h4>
Here’s the part that got buried in every benchmark post I read.
When things get a bit ambiguous, Gemma 4 starts to feel like it’s skipping steps.
I asked all three something architectural, where to put shared validation logic in a slightly messy service setup.
Claude leaned into the tradeoffs. It kind of walked around the problem a bit before landing somewhere.
GPT did a version of that too.
Gemma 4 gave a reasonable answer, but it felt like it jumped straight to the conclusion. Less “thinking with me” more “here’s what you probably want”.
It’s not wrong. Just… lighter.
And yeah, writing is another gap.
Changelogs, PR descriptions, Claude still sounds like someone who’s done this before. There’s a rhythm to it. GPT is close.
Gemma 4 gets the job done, but you can feel the difference if you read it twice.
Also: long prompts.
I had one messy ~600-word spec with constraints buried everywhere. Claude followed all of it. GPT missed a couple small things. Gemma 4 missed more than that, not dramatically, but enough that I had to go back and fix it.
That surprised me more than the reasoning gap, actually.
<h4>The benchmark question (the annoying part)</h4>
Let me be direct about something.
Gemma 4 31B scores 84.3% on <a href="https://arxiv.org/abs/2311.12022">GPQA</a> Diamond, graduate-level scientific reasoning. Claude’s top models and GPT sit above that on the same benchmark. And yes, in practice, the gap does show up. Not on simple tasks. On the tasks where you need a model to reason carefully through something genuinely ambiguous, the frontier proprietary models aren’t playing the same game yet.
But here’s the thing I kept coming back to: for maybe 70–75% of my actual daily developer work, Gemma 4 was good enough. And “good enough, running locally, at zero per-token cost” is a different value proposition than “slightly better, $X per million tokens, data leaving my machine.”
I can’t speak to every use case. In the tasks I ran, the proprietary models were ahead on the complex reasoning end, and the gap was real, just not as wide as I’d assumed.
For teams handling sensitive code, or for anyone watching API costs quietly double over the last year, that narrowed gap changes the calculation. The math looks different when “a bit worse” also means “free, local, and yours.”
<figure><img alt="Not either/or. Just routing | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*sdcF2b2kqJtZQqM9uuiwMQ.png"><figcaption>Not either/or. Just routing.</figcaption></figure><h4>The Hybrid Workflow That Actually Makes Sense</h4>
I didn’t replace anything.
I just… split the work differently.
Run Gemma 4 locally (the 26B MoE is my choice — 40+ tokens/sec on a decent GPU, manageable VRAM footprint) for routine tasks: test generation, boilerplate, quick explanations, PR descriptions. Switch to Claude or GPT for anything where the reasoning depth actually matters — complex architectural decisions, subtle debugging, long-document understanding.
Both Claude Code and similar tooling now support fast model switching. The workflow is mature. It’s not an either/or choice.
Honestly, I think this is how most developers will end up using open models — not as replacements for the cloud APIs, but as a cheaper first pass that handles the 70% and routes the harder stuff up. The “open models will kill proprietary AI” framing misses this. They’ll coexist, and the tools will just get better at switching between them automatically.
This wasn’t a planned system. It just sort of emerged after a few days.
And now it feels obvious.
<h4>One thing that might matter more than people are saying</h4>
There’s an article to be written about how much the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache 2.0 license</a> actually matters.
Gemma 3 had restrictions. Llama still has usage thresholds for large commercial deployments. Gemma 4 is clean. You can slot it into a product, fine-tune it on your codebase, and deploy it internally — without reading the fine print twice.
That’s not exciting to talk about, but if you’ve ever had to deal with compliance or enterprise constraints, it’s the difference between “interesting” and “usable”.
<figure><img alt="Still not cancelling the Claude subscription | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*WNLzM8QIIdZkYvhWW8o4hg.png"><figcaption>Still not cancelling the Claude subscription.</figcaption></figure><h4>So… is there a winner?</h4>
Not really.
<a href="https://deepmind.google/models/gemma/gemma-4/">Gemma 4</a> vs <a href="https://openai.com/index/chatgpt/">GPT</a> vs <a href="https://claude.com/product/overview">Claude</a> isn’t a single winner situation. It’s a question of what you need it for.
If you’re building something where the AI model runs locally, costs matter, or data privacy is non-negotiable, Gemma 4 is now a serious option in a way that previous open models weren’t. The benchmark numbers, for once, roughly held up in my experience.
Gemma 4 is good enough for a lot of real work. More than I expected.
GPT and Claude are still better when things get complicated. Also, more consistent.
Both of those things can be true at the same time.
<blockquote>My honest take: Gemma 4 made me rethink how I split work between local and cloud models. It didn’t make me cancel my Claude subscription.</blockquote>
Both of those things can be true.
<h4>Small afterthought</h4>
I went into this expecting to dismiss Gemma 4.
Didn’t happen.
Didn’t switch entirely either.
It just… earned a place in the workflow.
That’s a quieter outcome than the benchmarks suggest, but probably a more useful one.
<h4>Questions I had to Google before writing this:</h4>
Is Gemma 4 better than GPT for coding?
For straightforward to medium-complexity coding tasks, Gemma 4 31B is competitive with GPT and sometimes faster. For complex multi-file refactoring or deep architectural reasoning, GPT and Claude still have an edge in my experience.
Can Gemma 4 replace Claude for developers?
Not entirely, at least not yet. For 70–75% of routine developer tasks, Gemma 4 handles things well — especially if you’re running it locally. But for nuanced reasoning, writing quality, and complex instruction following, Claude is still ahead.
How does Gemma 4 31B perform on real tasks?
Better than I expected going in. Coding and test generation were strong. Complex reasoning and long-prompt instruction following were where it showed more limitations compared to proprietary frontier models.
What is the best way to run Gemma 4 locally?
The 26B MoE variant is the practical choice for most developers — it achieves roughly 97% of the 31B’s quality with significantly lower memory requirements, and runs at 40+ tokens/sec on a single modern GPU.
If you loved reading this, you’ll love this too:
<ul>
<li><a href="https://ai.plainenglish.io/i-ran-gemma-4-locally-heres-what-nobody-s-telling-you-806dcd485925">I Ran Gemma 4 Locally. Here’s What Nobody’s Telling You.</a></li>
<li><a href="https://ai.plainenglish.io/replaced-github-copilot-with-local-ai-gemma-ollama-2d2dbd7f47e2">I Replaced GitHub Copilot With a Local Setup. Here’s What Nobody Tells You</a></li>
</ul>
<h3>A message from our Founder</h3>
Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.
If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>. And before you go, don’t forget to clap and follow the writer️!
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=ae676ddf0f19" width="1" height="1" alt=""><hr>
<a href="https://ai.plainenglish.io/gemma-4-vs-gpt-vs-claude-real-world-test-ae676ddf0f19">I Tried Gemma 4 Against GPT and Claude for a Week</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

I Tried Gemma 4 Against GPT and Claude for a Week

<h4>What Google Actually Shipped</h4>
<h4>I built a local AI agent, and it broke in ways nobody explains (here’s the fix)</h4>
<figure><img alt="The agent kept calling the same tool. Over and over. 15,000 tokens later, it still hadn’t moved | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*ZSXDQo9FsqR40ahjVcP0Yw.png"><figcaption>The agent kept calling the same tool. Over and over. 15,000 tokens later, it still hadn’t moved.</figcaption></figure>I spent the last two weeks building a Gemma 4 local AI agent.
It worked… until it didn’t.
Not benchmarking it. Not running a single “Hello World” prompt and calling it a day.
Actually building:
<ul>
<li>Tool calls</li>
<li>File reads</li>
<li>Multi-step reasoning</li>
<li>Real tasks</li>
</ul>
And then it broke.
My Gemma 4 local AI agent kept calling the same tool over and over again.
It wouldn’t stop.
If you’ve read my earlier breakdown. You already know I don’t care about hype. 👇🏻
<a href="https://ai.plainenglish.io/i-ran-gemma-4-locally-heres-what-nobody-s-telling-you-806dcd485925">I Ran Gemma 4 Locally. Here’s What Nobody’s Telling You.</a>
That article was about performance.
This one is about what happens when you try to build something real.
Some things worked better than anything I’ve run locally before.
A lot of things didn’t.
And the fixes were almost never obvious.
Here’s everything I found: Including the exact config that finally made Gemma 4 local AI agent stable.
<h4>What “Agentic” Actually Means for Gemma 4</h4>
To understand why things break, you need to understand what you’re actually running.
Gemma 4 is the first model in <a href="https://deepmind.google/models/gemma/gemma-4/">Google</a>’s open-weight family with native function calling built in.
Not hacked through prompts. Not bolted on through wrappers. Trained into the model.
That changes everything.
Earlier local models (<a href="https://ollama.com/library/mistral">Mistral</a>, LLaMA variants) could use tools, but only if you forced them.
Gemma 4 understands the loop:
<ul>
<li>Call tool</li>
<li>Receive result</li>
<li>Reason</li>
<li>Decide on the next step</li>
</ul>
The <a href="https://huggingface.co/google/gemma-4-26B-A4B">26B MoE</a> variant is the one most developers are running locally.
<ul>
<li>~4B active params at inference</li>
<li>~80–110 tokens/sec on RTX 3090</li>
</ul>
Fast enough for real agent workflows
<blockquote>So the hardware story is real. The agent's story is where it gets complicated.</blockquote>
<h4>The First Bug That Kills Your Agent: Infinite Tool Loops</h4>
Day 1: Everything broke.
I gave the agent a simple task:
<ul>
<li>Read file</li>
<li>Summarize</li>
<li>Write report</li>
</ul>
What happened:
<ul>
<li>read_file</li>
<li>read_file</li>
<li>read_file</li>
<li>forever</li>
</ul>
The agent called read_file. Got the result. Then called `read_file` again. And again. It never moved forward.
This is the most common failure mode right now, widely discussed in communities like <a href="https://www.reddit.com/r/LocalLLaMA/">r/LocalLLaMA</a>.
<h4>Root cause</h4>
The model never actually receives the tool result properly.
So it keeps retrying the same call.
<figure><img alt="Without — jinja, the tool result never reaches the model in the format it expects. The loop is structural, not random | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*VgV8dasEy_B1IRpki5x2BA.png"><figcaption>Without --jinja, the tool result never reaches the model in the format it expects. The loop is structural, not random.</figcaption></figure><h4>The Fix (Most People Miss This)</h4>
The fix is in how you format the tool result message back to the model. Most frameworks append the result as a plain text string.
Gemma 4 expects tool responses in a specific chat template format.
Not plain text.
If you’re using the llama.cpp server:
👉 You NEED the -- jinja flag
<pre>llama-server \ -m ./gemma-4-26B-A4B-it-Q3_K_M.gguf \ --port 1234 \ -ngl 99 \ -c 32768 \ --jinja \ -ctk q8_0 \ -ctv q8_0</pre>
Without it:
<ul>
<li>Tool responses are injected incorrectly</li>
<li>The model loses state</li>
<li>Loops forever</li>
</ul>
This single flag fixes most “Gemma 4 agent looping” issues!
That -- jinja The flag is not optional for agentic work. It enables the Jinja2 chat template parser, which is what correctly formats tool results. I missed it for the first three days and couldn’t understand why every agent run turned into a loop.
If this feels familiar, it’s the same pattern I saw earlier — models appearing correct until you actually push them into real workflows:
→ <a href="https://ai.plainenglish.io/i-ran-gemma-4-locally-heres-what-nobody-s-telling-you-806dcd485925">I Ran Gemma 4 Locally. Here’s What Nobody’s Telling You</a>
<h4>Bug #2: The Thinking Loop That Never Closes</h4>
The model never stops thinking
Gemma 4 has a built-in reasoning mode. Gemma 4 uses &lt;think&gt; tags internally for reasoning.
During agentic tasks, this sometimes breaks.
The model opens a &lt;think&gt; tag and simply never closes it.
I sat and watched a single agent step burn through 15,000 tokens doing nothing useful.
<h4>The Counterintuitive Fix</h4>
Set your temperature to exactly `1.0` and `top_k` to `40`.
You’d expect lower temperature = stability.
Wrong.
For Gemma 4:
<blockquote>Temperature must be EXACTLY 1.0</blockquote>
<pre>params = { "temperature": 1.0, "top_k": 40, "top_p": 0.95, "repeat_penalty": 1.1 }</pre>
Why?
At temperature: 1.0It has just enough variance to break out of the loop.
<ul>
<li>Low temperature reinforces bad loops</li>
<li>
1.0 , giving just enough randomness to escape</li>
</ul>
<blockquote>This is one of those fixes nobody tells you.</blockquote>
<h4>Bug #3: Context Collapse (Even with 256K)</h4>
The Context Window Is Real, But Context Management Isn’t Free
Yes, Gemma 4 supports massive context.
<blockquote>Gemma 4’s 26B model supports up to 256K context.</blockquote>
Yes, it works.
But…
Practically, it changes how you think about agent design, but not in the way the marketing suggests.
<blockquote>Long context ≠ reliable agent memory</blockquote>
Running an agent with a long context is not the same as running one with a well-managed context.
Here’s what happens at around the 80K token mark in a multi-step agent run:
<ul>
<li>Tool calls repeat</li>
<li>Results get hallucinated</li>
<li>Decisions contradict earlier steps</li>
</ul>
<h4>The Real Fix: External State</h4>
Stop relying on the model to remember everything.
Instead:
I now inject a structured, fresh system prompt at every single turn
<pre>system_prompt = f""" You are a task-completing agent. ORIGINAL TASK: {task} TOOLS AVAILABLE: {tool_descriptions} COMPLETED STEPS SO FAR: {completed_steps_log} Your job: decide the next action. """</pre>
This is the biggest reliability upgrade you can make.
<blockquote>The model doesn’t have to remember its history. It just has to look at the immediate payload and decide the next action.</blockquote>
<h4>Hidden Problem: The llama.cpp Build Problem Nobody Mentions</h4>
This one is specific to a window of llama.cpp builds from late March 2025.
If you are seeing random character insertions mid-generation
<blockquote>e.g., “The file contaiens the folloiwng data…”</blockquote>
You do not have a model quality problem. You have a build problem.
A window of <a href="https://github.com/ggerganov/llama.cpp">llama.cpp</a> builds from late March 2026 introduced an inference bug that causes random typos.
Check your build version:
<pre>llama-server - version</pre>
<h4>Fix:</h4>
<ul>
<li>Use b3000+ (recommended)</li>
<li>
b3447+ (with flash attention)</li>
</ul>
If you are in the b28xx range, update immediately. You want b3000+ for general inference, or b3447+ for full flash attention support.
<h4>The Config That Actually Works For A Gemma 4 Local AI Agent</h4>
After two weeks of testing, here is the definitive, stable setup for running a Gemma 4 local AI agent.
If you want this to work without burning hours of your life, follow these rules exactly:
<h4>Model</h4>
<ul><li>
Unsloth Q3_K_M quant gemma-4–26B-A4B-it-Q3_K_M.gguf (from <a href="https://huggingface.co/unsloth">Hugging Face ecosystem</a>)
</li></ul>
<h4>Inference Server</h4>
<pre>llama-server \ -m ./gemma-4-26B-A4B-it-Q3_K_M.gguf \ --port 1234 \ -ngl 99 \ -c 32768 \ -np 1 \ --jinja \ -ctk q8_0 \ -ctv q8_0</pre>
<h4>Inference params</h4>
<pre>{ "temperature": 1.0, "top_k": 40, "top_p": 0.95, "repeat_penalty": 1.1, "max_tokens": 2048 }</pre>
<h4>Why these specific values</h4>
<ul>
<li>
-np 1 avoids memory overflow</li>
<li>
-ctk q8_0 -ctv q8_0 quantizes the KV cache, cutting it from ~940MB to ~499MB, that headroom matters.</li>
<li>--jinja is non-negotiable for tool calling. Everything breaks without it.</li>
<li>
Q3_K_M from Unsloth, not just any Q3_K_M. Unsloth’s quants are calibrated differently and perform noticeably better on reasoning tasks.</li>
<li>
temperature: 1.0 exactly. Not 0.9. Not 0.7. 1.0.</li>
</ul>
<figure><img alt="Three tools. Three clean completions. No loop. This is what the working config actually looks like at runtime | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*IfNpI66XUrqAsZEROR_skg.png"><figcaption>Three tools. Three clean completions. No loop. This is what the working config actually looks like at runtime.</figcaption></figure>If you’re trying to push this into real-world usage, this connects directly with how I built production workflows:
<a href="https://generativeai.pub/i-automated-80-of-my-workflow-with-ai-bb8115b600bd">I Automated 80% of My Workflow With AI</a>
<h4>What Actually Works Well</h4>
This isn’t a takedown post.
Gemma 4 is the first local model where agents feel real.
What works:
<ul>
<li>Single-step tasks =&gt; very reliable</li>
<li>Code generation =&gt; surprisingly strong</li>
<li>Long context reasoning =&gt; legit</li>
</ul>
I built a workflow where the agent reads a buggy file, writes a fix, and runs a validator.
<blockquote>It completed the loop successfully 80% of the time on the first try. For an entirely offline model, that is staggering.</blockquote>
For any developer who wants to run a private, offline, zero-API-cost coding agent: Gemma 4 is the first local model I’ve used where that feels genuinely viable rather than just technically possible.
<blockquote>The model is capable. The configuration is what determines whether your agent works or loops forever.</blockquote>
<figure><img alt="The model is capable. The configuration is what determines whether your agent works — or loops forever | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*wpHMinAKVh4oqKgerHRSIA.png"><figcaption>The model is capable. The configuration is what determines whether your agent works — or loops forever.</figcaption></figure>Here are some Questions which were concerning to me before I started building. I’ll clear them up now for you:
<h4>Does Gemma 4 support tool calling locally?</h4>
Yes. Fully offline. No API needed.
Using llama.cpp or Ollama, tool calling works, but requires proper formatting -- jinja.
<h4>Why does my Gemma 4 agent loop forever?</h4>
Because tool responses aren’t formatted correctly.
Fix: Use -- jinja So the model receives structured tool outputs.
<h4>Best quantization for Gemma 4 local agent?</h4>
Unsloth Q3_K_M (<a href="https://huggingface.co/models">via Hugging Face ecosystem</a>)
Best balance of:
<ul>
<li>Performance</li>
<li>VRAM</li>
<li>Reasoning stability</li>
</ul>
<h4>Summing it up</h4>
Gemma 4 is the first local model that made me stop thinking of on-device agents as a demo and start thinking of them as a real workflow.
Not because it works.
But it almost works reliably.
And that gap?
<blockquote>The model isn’t the problem. Your configuration is.</blockquote>
I’m documenting everything I learn about local AI systems, LLM agents, and real-world setups.
If you want:
<ul>
<li>Working configs</li>
<li>Real failures</li>
<li>No hype</li>
</ul>
Follow along ❤️
<h3>A message from our Founder</h3>
Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.
If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>. And before you go, don’t forget to clap and follow the writer️!
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=1ef53f50158c" width="1" height="1" alt=""><hr>
<a href="https://ai.plainenglish.io/i-built-a-gemma-4-ai-agent-it-kept-looping-until-i-fixed-this-1ef53f50158c">I Built a Gemma 4 AI Agent. It Kept Looping Until I Fixed This.</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

I Built a Gemma 4 AI Agent. It Kept Looping Until I Fixed This.

I Deleted $500/Month in SaaS and Replaced My Entire Stack

<figure><img alt="You build on someone else’s platform. They decide when the floor moves | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*HkoAyn-NWED1pOZU3lJcrA.png"><figcaption>You build on someone else’s platform. They decide when the floor moves.</figcaption></figure><h4>Artificial Intelligence</h4>
<h4>Most developers focused on the wrong update. This one will break their systems.</h4>
You probably heard GPT-4o got retired.
You likely saw the drama unfold. The #Keep4o hashtag trending. Users are mourning the loss of their “warm, conversational” AI companion.
While everyone was looking at the model and the internet was busy being nostalgic, OpenAI changed the entire developer platform underneath it, and most developers didn’t notice.
OpenAI didn’t just remove a model.
They removed the era it belonged to.
And if you’re still building the old way, this matters more than any upgrade.
<blockquote>The OpenAI Assistants API deprecated timeline is already in motion , and it will break real production systems.</blockquote>
<h4>The Retirement Everyone Talked About — And Misread</h4>
On April 3, 2026, <a href="https://developers.openai.com/api/docs/models">GPT-4o was fully</a> retired across ChatGPT.
The API version? Gone even earlier February 16.
Most people treated it like a product change:
<ul>
<li>“We lost a good model.”</li>
<li>“GPT-5 feels different.”</li>
<li>“Why remove something users liked?”</li>
</ul>
People treated it like a nostalgia story.
<blockquote>OpenAI’s own blog noted that only 0.1% of daily users were still choosing it. The “vast majority” had already moved to GPT-5.2</blockquote>
But that reaction missed the point.
OpenAI didn’t just remove a model.
They removed the era that the model belonged to.
<blockquote>The headline that mattered more was buried six scrolls down in OpenAI’s deprecation docs.</blockquote>
<h4>The Assistants API Is Being Killed. Quietly.</h4>
On August 26, 2025, OpenAI announced something most developers still haven’t internalized: The OpenAI Assistants API is deprecated and will shut down on August 26, 2026.
No keynote.
No major blog post.
Just a dry notice in the documentation and a single thread in the developer forum.
<figure><img alt="The Assistants API wasn’t just a feature. It was the load-bearing wall | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*yts3v2aG1aS8ci4VDBF_oA.png"><figcaption>The Assistants API wasn’t just a feature. It was the load-bearing wall.</figcaption></figure>And that’s a problem, because the Assistants API wasn’t a niche feature. It was the foundation on which many developers built their AI products.
<ul>
<li>Persistent threads (memory)</li>
<li>Built-in tools</li>
<li>Managed agent behavior</li>
<li>Server-side orchestration</li>
</ul>
Now?
It’s all going away.
The replacement is called the <a href="https://developers.openai.com/api/reference/resources/responses">Responses API</a>.
And it works completely differently.
<h4>What Actually Changed Under the Hood</h4>
Most developers think this is an API migration.
It’s not.
It’s a complete reversal in how OpenAI expects you to build.
In plain English: They built it when they didn’t really know what agents needed to be. Now they do. And they’re ripping out the foundation.
Old Way: Assistants API
<ul>
<li>OpenAI manages the agent</li>
<li>State lives on its servers</li>
<li>Tools are abstracted</li>
<li>You orchestrate less</li>
</ul>
<pre># Create an assistant (persistent object) assistant = openai.beta.assistants.create( name="MyAgent", instructions="You are a helpful coding assistant.", model="gpt-5.2", tools=[{"type": "code_interpreter"}], ) # Create a thread (server-side memory) thread = openai.beta.threads.create() # Add a message and run run = openai.beta.threads.runs.create(thread_id=thread.id, assistant_id=assistant.id)</pre>
<h4>New Way: Responses API</h4>
<ul>
<li>You manage orchestration</li>
<li>You control state</li>
<li>You handle tool logic</li>
<li>You own reliability</li>
</ul>
<pre># No persistent assistant object. No threads. # Just send input, get output. response = openai.responses.create( model="gpt-5.2", input=[{"role": "user", "content": "Help me debug this code."}], conversation="conv_abc123", # optional persistent context )</pre>
<blockquote>The surface got simpler. The responsibility moved to you.</blockquote>
<figure><img alt="Three months’ notice on something you’ve been building for a year | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*Dw3cOZTOSdsxs8yDuXxHew.png"><figcaption>Three months’ notice on something you’ve been building for a year.</figcaption></figure><h4>The Model Naming Is Also a Mess Nobody’s Documenting</h4>
While all of this was happening, OpenAI was also quietly overhauling its model lineup in ways that are genuinely confusing.
<ul>
<li><a href="https://developers.openai.com/api/docs/models">GPT-5.1 → GPT-5.2 → GPT-5.3</a></li>
<li>Codex variants removed from UI</li>
<li>Hidden fallback models introduced</li>
</ul>
First, GPT-5.1 stepped in for GPT-4o. Then, almost immediately, GPT-5.2 replaced 5.1. Recently, GPT-5.3-Codex dropped as their first combined training stack. Even stranger, GPT-5.3 Instant Mini is currently acting as the silent fallback model when users hit rate limits, despite not even existing in the developer dashboard’s model picker.
This isn’t accidental.
It’s what rapid infrastructure consolidation looks like.
And it reinforces the real takeaway:
You can’t treat this platform as stable. You have to treat it as evolving infrastructure.
<h4>The Part Most Developers Still Haven’t Realized</h4>
The Assistants API was OpenAI’s early abstraction layer.
It was built before:
<ul>
<li>Reasoning models matured</li>
<li>Tool use stabilized</li>
<li>Agent patterns were understood</li>
</ul>
Now OpenAI is doing something rare:
<blockquote>They’re deprecating their own abstraction to expose lower-level primitives.</blockquote>
This means:
<ul>
<li>More flexibility</li>
<li>Better performance</li>
<li>Fewer hidden constraints</li>
</ul>
But also:
<ul>
<li>More engineering responsibility</li>
<li>More room for mistakes</li>
<li>More complexity in production</li>
</ul>
<h4>What This Actually Means for You</h4>
If you are running production code today, write this date on a sticky note and put it on your monitor: August 26, 2026.
On that day, calls to /v1/assistants and /v1/threads won’t just slow down or return warnings. They will hard-fail. Period.
Three things to do now:
1. Audit your codebase immediately: Run a global search for openai.beta.assistants and openai.beta.threads. If those strings exist anywhere in your active repos, your clock is officially ticking.
2. Read the migration guide.
<ul>
<li>Simple apps → manageable</li>
<li>Agent workflows → significant rewrite</li>
<li>Multi-tool systems → architectural rethink</li>
</ul>
It’s at <a href="https://platform.openai.com/docs/assistants/migration">platform.openai.com/docs/assistants/migration</a>.
The Responses API is genuinely better in some ways, you get MCP support, computer use, deep research tools, and better performance. But the migration is non-trivial for complex workflows.
3. Do not procrastinate.
The <a href="https://community.openai.com/">developer forums</a> are currently flooded with engineers who “almost finished” building an app on the old API right when the notice dropped. Don’t be the dev panicking in late July.
Because when August hits, it’s not degradation.
It’s shutdown.
<h4>The Pattern Most Developers Miss</h4>
This isn’t the first time OpenAI has done this:
<ul>
<li>GPT-4o → retired</li>
<li>Assistants API → deprecated</li>
<li>Models → reshuffled rapidly</li>
</ul>
GPT-4o: retired quietly with minimal notice. Backlash forced a reversal in August 2025, then final retirement in February 2026.
Assistants API: announced deprecation buried in changelog notes, one year horizon.
This isn’t chaos.
It’s velocity.
And the developers who adapt aren’t the ones reading announcements.
They’re the ones:
<ul>
<li>Watching deprecations</li>
<li>Tracking changelogs</li>
<li>Designing for instability</li>
</ul>
<figure><img alt="August 26, 2026. It’s closer than it sounds | Kuldeepsinh Jadeja" src="https://cdn-images-1.medium.com/max/1024/1*Hu17EQtIVdMyLzkv-9TF_Q.png"><figcaption>August 26, 2026. It’s closer than it sounds.</figcaption></figure><h4>Is the Assistants API still working right now?</h4>
Yes. As of April 2026, the OpenAI Assistants API deprecated status is active but still functional. It will shut down completely on August 26, 2026. After that, endpoints like /v1/assistants and /v1/threads will stop working entirely.
<h4>What replaces the Assistants API?</h4>
The Responses API and Conversations API replace it.
<ul>
<li>Responses API → handles generation + tool use</li>
<li>Conversations API → manages persistent context</li>
<li>Prompt objects → replace assistant definitions</li>
</ul>
This new architecture gives more control, but requires more implementation effort.
<h4>Did GPT-4o retirement affect the API too?</h4>
Yes. The chatgpt-4o-latest API endpoint was retired on February 16, 2026.
However, some variants like GPT-4o mini and audio-related models still exist, for now.
<blockquote>The GPT-4o retirement was visible. The OpenAI Assistants API deprecated shift is what will break real systems.</blockquote>
This is not a simple upgrade.
It’s a transition from:
Managed abstractions → To developer-owned architecture
And if your production system still depends on /v1/assistants
You’re already behind the platform.
OpenAI didn’t announce this loudly.
They never do for the things that matter most to developers.
They feel like something quietly disappearing, until your system depends on it.
That’s the real change you needed to notice.
<blockquote>Keep an eye on your codebase, read the docs, and don’t let August sneak up on you.</blockquote>
<h3>A message from our Founder</h3>
Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.
If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>. And before you go, don’t forget to clap and follow the writer️!
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=b10fe7bff661" width="1" height="1" alt=""><hr>
<a href="https://ai.plainenglish.io/openai-assistants-api-deprecated-responses-api-migration-b10fe7bff661">OpenAI Quietly Broke the Way You Build AI Apps</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.