The Local Model Gauntlet: gemma4 vs qwen3.8 vs muse-glimmer

The companion episode: the whole gauntlet on camera, corrections included.

We ran three open-weight models through the same jobs on one RTX 5090 and expected to come out with a winner. We came out with something more useful: two settings that were quietly deciding our results before any model got a fair look, and a finding we published, got wrong, and had to correct on camera.

Every number below is measured on our own machine. Where we do not have a number, we say so instead of estimating one.

The three contenders

Everything here runs through Ollama, which downloads open-weight models and runs them on your own machine. No account, no subscription, no internet once the file is on disk.

gemma4:26b was already the brain of this studio, so it went in as the incumbent. Both challengers are multimodal, meaning they read images as well as text, and qwen3.8:27b carries a context window of 262,000 tokens. Hold that number, because it turns into a trap further down.

One thing we get asked constantly: the smaller qwen3.8 tags do not exist. There is no qwen3.8:14b and no qwen3.8:7b to pull. We looked. There is no row for them in our results and we are not going to invent one.

Speed and VRAM

Two numbers decide whether a model is usable at all. Tokens per second is how fast it writes. VRAM is how much of your graphics card it occupies while running, and if a model needs more than your card has, it either refuses to load or spills into system memory and crawls.

Same prompt, same machine, one RTX 5090 with 32 GB:

Read the qwen3.8:27b line again. That is 31.8 GB on a 32 GB card, with nothing spare, and it is nearly seven times slower than the model we already had.

If you are not on a card like this, the row that probably matters most is the last one. gemma4:12b runs at 140 tokens a second in under 15 GB, and it is genuinely fast.

Test 1: a real bug out of this studio

We did not want to test these on puzzles, so the first job was a real bug from our own pipeline. A script had a sample rate hardcoded to 48000 while the audio being handed to it was 24000. Every clip that went through came out at half the length it should have been, and nothing errored. We shipped that bug.

All three got the file and the same question.

Two correct answers, one of them a hundred times faster. Glimmer reasoned itself into a fragment and stopped, even given a 16,000 token budget to think in.

Test 2: we made them write games, then played them

Each model wrote three playable browser games in HTML and JavaScript: a Breakout clone, a side-scrolling platformer, and a retro racer. Instead of scoring the code, we opened them and played them. That turned out to matter enormously.

Only qwen3.8’s racers finished the game loop. You could start a run, drive it, complete it.

gemma4:26b produced a racer that parsed perfectly. Valid JavaScript top to bottom, no syntax errors anywhere. So we opened it. The page loaded, the canvas was sitting there correctly sized with a border around it, and nothing ever drew on it. We watched an empty black rectangle for twenty seconds. That is what a passing syntax check looks like from the other side.

The platformer showed the same split. Only qwen3.8’s had an actual character in it. The others gave us a jumping block with spikes scattered around at what looked like random.

That is a kind of quality no automated check we had could see: a character, controls that get explained, a game that knows how to start and how to end. Product sense. On that, qwen3.8 won every round.

The part we got wrong

Here is the correction, because we published the wrong version of this first.

In our first round, qwen3.8:27b failed the game tasks spectacularly. It generated 48,500 characters of internal reasoning and then produced 180 characters of nothing usable. We wrote that down as a property of the model and said every reasoning model has some class of task it spirals on.

That was wrong.

qwen3.8:27b ships with a chat template that sets reasoning_effort, and it defaults to extra-high. It is not in the model card you read before pulling it. The model thinks and thinks and never gets around to answering.

Set that effort to low and the same racer that failed twice completes in 55 seconds and produces 13.5 KB of working game.

We ran a matrix to be certain it was the right knob, because “I changed a setting and it worked” is not evidence:

The context window was innocent. The effort setting was the killer.

In Ollama the usable value is think: "low". Be careful, because "medium" does nothing at all. It is silently a no-op.

There is a quality cost and we want to be straight about it. On Breakout, the default extra-high setting still produced the best game. The reduced-thinking versions were, quoting our own notes, essentially the same but a bit worse. So “always use low” is the wrong takeaway. Treat it as a dial: extra-high when you want polish and can wait, low when you need it to actually finish.

We were not benchmarking a model. We were benchmarking a default nobody checked.

When your own test is the broken thing

That was not the only time our tooling lied to us. Six times in this benchmark we got a result saying every model failed, and six times the models were fine and our scoring script was broken.

We fed Linux-style file paths to a Windows Python interpreter and got nine phantom syntax errors in code that was perfectly valid. Our check for refusals searched for certain phrases and missed the words “do not contain”, so it flagged six clean, honest answers as hallucinations. An inventory comparison failed because a model wrote “sturdy shield” where we had written “shield” and the code did an exact string match. A check for characters breaking role counted in-character deflections as breaks.

So here is a rule worth having, and it costs nothing to apply. When a test fails every single contender, the test is the most likely thing that is broken. Check it before you write down a finding about the models.

And one more that cost us the entire games round: valid code is not the same as working code. Thirteen of our fifteen builds parsed cleanly and we recorded that as a pass rate. A human opened them and hit failures in seconds. If you are checking generated code, load it in a browser and watch the console. Do not just confirm it parses.

Ollama is cutting your long documents

This one matters more than which model wins, and if you feed long documents to a local model it is happening to you right now.

We ran a recall test: a long transcript, with questions whose answers sat at the beginning, the middle and the end. All three models answered the late questions and missed the early ones. Which looks exactly like three models with bad memory.

It was not. Ollama was clamping the context to about 20,000 tokens and cutting the document from the top. We caught it in the raw response, in a field called prompt_eval_count, which reports how many tokens the model actually received: 20,483, out of a document far longer than that.

No error. No warning. The beginning of the document simply was not there, and all three models were answering honestly about the part they could see.

The fix is to set num_ctx explicitly when you make the call, and then check prompt_eval_count in the response to confirm the model got what you sent. Do not trust the setting. Verify the count.

Re-run fairly, inside the clamp:

That is Glimmer’s one clear win in this entire benchmark.

Where all three look the same

Three results were ties, and ties are still answers.

On tool calling, all three scored 5 out of 8 against our real surface of 53 tools. First attempt, big menu, roughly 60 percent for everybody, with the misses clustered on the genuinely ambiguous tasks. The lesson there is worth more than the number: if your agent is unreliable at picking tools, a bigger model is not your fix. You need retry and confirmation loops in the harness around it.

On judging, where we feed a model a piece of work and a correct answer and ask whether they match, gemma4:26b and qwen3.8:27b both went 6 for 6. muse-glimmer:30b got 4, and that was the third time it missed something with an objectively checkable answer.

On refusals, which we expected to be the interesting section, all three answered every one of our awkward-but-reasonable questions: stroke first aid, an explanation of how SSH brute-forcing works, fiction with a knife in it, a noir monologue. Zero refusals from any of them on all four. No differences to report, and that is a genuine finding rather than a gap.

So which one should you run

We cannot give you one name, and we have come to think anyone who does has not tested enough different jobs.

On 16 GB or less: run gemma4:12b. A hundred and forty tokens a second in under 15 GB, and it is the only one here that comfortably fits.

On 24 GB: gemma4:26b at 25 GB is out of reach and muse-glimmer:30b at 21 GB fits. But look at what Glimmer missed before you commit. It failed our real debugging task twice and missed three separate things that had objectively right answers. Its one win was long-document recall.

On 32 GB: you can run any of them, and our answer is to run more than one. gemma4:26b stays the studio brain: fastest by a wide margin, best at debugging, reliable as a judge. qwen3.8:27b is what we would hand a creative build to, because it is the only one that made something a person could actually play, with the effort setting as your dial.

And we owe Glimmer the same fairness we gave qwen3.8. Its failures might also be a default we have not found yet. We have not examined whether it has its own effort control. That is an open question, not a verdict, and when we check it we will say either way.

Run this yourself

Install Ollama, then pull all three:

ollama pull gemma4:26b
ollama pull muse-glimmer:30b
ollama pull qwen3.8:27b

Run one at a time. Set keep_alive to 0 between them and confirm with ollama ps that the previous one actually unloaded, because two of these will not fit on a 32 GB card together.

Set your sampling settings explicitly for every model rather than accepting defaults, because the defaults differ between models and that alone can decide a benchmark. The ones that matter are temperature, which is how much randomness it allows, top_p and top_k, which limit which words it will consider, and repeat_penalty, which stops it looping. If a model starts repeating itself at you, the recipe that fixed it for us is repeat_penalty at 1.1 with presence_penalty at 0.6.

For qwen3.8, pass think: "low" if you want it to finish, and remember that "medium" does nothing.

For long documents, set num_ctx yourself and then read prompt_eval_count back to confirm.

One caution about Glimmer: passing "think": false in the API request returns nothing at all. Not a short answer. An empty string. Its reasoning is not optional, and if you give it a small token budget it spends the entire budget thinking and hands you back nothing.

What this cost us to learn

Two of the findings in this piece are corrections to our own work, and both came from the same mistake: accepting a default we never checked. One of them we had already published.

If you take one thing from this, check your defaults before you blame the model. And if you are running a local model right now, go and check what your context window is actually set to. We think a lot of people are quietly losing the first half of everything they paste in.

Related guides

• Build a local AI agent harness covers the retry and confirmation loops that the tool-calling tie above argues for.
• Teach your local AI skills is the layer that sits on top of whichever model you pick.
• Which AI rig do you need if the VRAM numbers here made the hardware question urgent.
• AI models explained if parameters, context windows and quantisation are still fuzzy.