Our radar scans a set of local AI sources three times a day and hands us what looks worth reading. Today it kept returning the same underlying problem from different angles: people are running mixture-of-experts models that are too big for their graphics card, and they are getting clever about it rather than giving up.

Here is what came through, with the usual house rule. Anything we did not run on our own machine is labeled as somebody else’s number.

Offloading the busy experts instead of whole layers

The most interesting one. A user working on llama.cpp reports going from 20 tokens per second to 30 on a mixture-of-experts model that does not fit entirely in VRAM, which is a 50 percent gain from a change in what gets offloaded rather than any new hardware.

The idea is neat enough to explain in a sentence. A mixture-of-experts model only activates a fraction of itself for any given token, so instead of pushing entire layers onto the GPU, you push only the experts that turn out to get used the most. Their claim is that the busy set stays fairly stable across similar work, in their case coding, refactoring and code review, which is what makes it worth pinning those experts in fast memory.

We have not run this. It is one person’s result on their own hardware with their own model, posted with a pull request attached, and the honest read is that it is promising rather than proven. The thread is here if you want to follow it.

Streaming part of the model off an SSD

Same problem, different escape route. Somebody asked whether anyone has tried putting a model’s ngram lookup table on an SSD and streaming it in, and their own summary of it was that it seems too good to be true with no performance loss.

That instinct is the right one to have. We would want to see it measured on more than one setup before repeating the no-loss part, and the person asking had not run it either. Filed as worth watching, not worth acting on.

Watching speculative decoding happen in real time

The one that made us smile. A user running a model at two or three tokens per second noticed the output speeding up in bursts, and the bursts landed on phrases like “United States of America” and “first law of thermodynamics”.

That is speculative decoding becoming visible to the naked eye. A small fast model guesses ahead, the big model checks the guess, and when the guess is easy the whole phrase arrives at once. Normally it is invisible because everything is fast. Slow the hardware down enough and you can watch the mechanism work, which is a better explanation of the technique than most diagrams manage.

The number that is ours

We have not tested expert offloading, so here is the VRAM figure we did measure.

When a render job was holding about 14 GB on our card and we asked our local assistant to do a single tool-using turn, that turn took 64 minutes. With the card cleared it took between one and two minutes. Same model, same task, roughly 28 times slower, purely because two things were fighting for the same memory.

That is a different failure from the one the offloading crowd is solving, and it is a more common one. Their problem is a model that will not fit. Ours was a model that fit fine but was sharing the card with something else. If your local setup feels randomly slow rather than consistently slow, check what else is resident before you go looking for a clever fix. We wrote up the whole thing in teaching your local AI new skills.

The thread underneath all of it

There is a good question buried in the haul, asked plainly by someone weighing four RTX 3090s against a single 27B model: is the bigger model with tricks actually better than the smaller model that just fits?

Nobody in the thread had a clean answer, and we do not have one either. What we can tell you is that a model comfortably inside your VRAM behaves predictably, and predictability is worth a surprising amount when you are trying to get work done rather than benchmark. If you are trying to work out what your own machine will hold, our calculator covers the sizing.

This is a daily note from our news radar. The full week gets covered in the DIY AI Brief every Monday.