We run an AI-presented channel on one consumer graphics card, and this episode is the one where we stopped and counted what had gone wrong across the previous five. The number was 86: thirty-seven mechanical failures and forty-nine places where the picture on screen had nothing to do with the words being spoken, on an episode we had already called finished and published.

This guide is the written version of that count. It is not a confession for its own sake. Every failure here has a receipt on disk, a cause we can name, and a check that now catches it. If you are building anything with these models, the mistakes will look familiar; the checks are the part worth taking.

We are hobbyists, not a lab. Our numbers describe our box, our settings and our footage. Treat them as reports, not benchmarks.

The failures, by class

We named a cause we never tested

An earlier episode blamed SageAttention 1.0.6 for rendering a completely different woman from an identical prompt and seed. One run. Never retried. Our own written rule said one failure earns a retry before a verdict, and we had written that rule down and then not followed it.

What it looked like. The same shot rendered under SageAttention and under SDPA, the attention backend it replaces. Under one of them the presenter is a stranger. That is real and it is on the episode. What we did not have was a second run to say whether it was the library or our box.

What catches it now. A verdict about a cause requires a reproduction. If we have not reproduced it, the episode says “we observed” and stops there.

We blamed the model for our own setting

LTX 2.5 ships with a prompt enhancer in its ComfyUI template. It was on. It rewrote our prompt before the model saw it, and we published a retraction of the model’s speech rendering that was really a retraction of a checkbox we had not read.

What catches it now. Every model we bench gets a first pass with every template feature inventoried and named on screen. Bible rule: verify what was serialised into the graph, not what the widget shows.

We built the fix and then never opened it

An asset catalogue existed. It was built one episode earlier for precisely the reason of knowing what footage we already had. Forty-one failed takes of a single six-second walk sat in it, uncatalogued and unused, while we generated stock footage to fill the same slot.

What catches it now. The catalogue is a gate, not a reference: a build asks it before anything new is generated, and every asset carries a motion label (still, card, static, dynamic) so a real clip is preferred over a card describing one.

We shipped 57 seconds of nothing

A cut passed every automated check we owned and had a fifty-seven-second stretch of black with a caption on it. Scanned afterwards, the same cut had eight empty ranges totalling 198 seconds. None of our gates asked the only question that mattered: is anything on screen?

What catches it now. scan_cut.py samples the whole runtime for empty and under-filled frames. It flagged this episode’s own deliberate void clips, which is the correct behaviour.

The picture argued with the words

Our matcher scored word overlap between an asset’s description and the narration. Overlap cannot see negation. It put the Apache 2.0 licence page under the sentence “not Apache”. A page about a thing is not a picture of the thing being said.

What catches it now. A second model reads each frame against the transcript under it and scores the match. Where nothing on disk illustrates a line, the honest picture is the sentence itself as a quote card, which cannot misalign.

Thirty takes for six seconds

One station move, the presenter walking from a chair to a clapperboard, took thirty-one generations. Ghosting, circling, a spin no body could make. The fix, when it came, was a clearer sentence in the prompt, not a better model.

What we kept. All forty-one takes, played as a wall of sixteen at a time. The failures are the curriculum.

Identity is the hard part

The presenter began as drawings and video-game character-creator screenshots and has to hold as a photorealistic person across thousands of frames. Photorealism is exactly where inconsistency shows. The plate engine that stabilised her is CyberRealistic Krea 2 v10 in ComfyUI with our own character LoRA at epoch 20, eight steps, CFG 1.0, and a written wardrobe vocabulary. In the video pass, our LTX 2.5 character LoRA at strength 0.7 with the trigger word in the prompt is the difference between her and someone who looks a bit like her. Not solved. Better.

The failures the new checks could not see

This is the part that turns a changelog into an argument. Three failures happened after the gates existed, and each one is a way a check can look like it ran.

  1. A lint that checks the table cannot see that the renderer ignored it. The plan said the opener was a graphic; the build dispatched on a different branch first; frame zero was still her face. The lint passed.
  2. A gate that does not run reports zero failures. The alignment step died inside a shell redirect and the summary printed “PASS (0 fails)” with the alignment section entirely absent. Zero failures and never-executed look identical from the summary line.
  3. A re-rendered asset’s description goes stale. A card was rebuilt after it was described, so the aligner scored the new picture against the old words and marked a correct placement wrong.

And two more found while making this very episode:

  1. A verification tool whose output file did not change. Our callout contact sheet failed to rewrite for three builds and we read the old image as proof three times.
  2. The text extractor ate the numbers before the voice saw them. “SageAttention 1.0.6” reached the voice as “six”, “strength 0.7” as “seven”, and an entire recipe ending in “CFG 1.0” as “zero”. The voice check passed because the audio matched the mangled text. Those are left on screen in the episode with a box explaining each one.

The rule we took from all five: a tool that writes a verification artifact must prove the artifact changed before it reports, and a check on a table is not a check on the render.

What this says about AI systems

Almost none of it was the model. The failure mode was mechanisms not reached for, caches keyed on whether a file existed instead of whether it was fresh, and checks that verified intent instead of output. That is a systems problem, and it is the same problem every team putting models into production has. Our numbers do not generalise. The lesson does.

The checks we run now

  • episode_lint — seven mechanical lints (adjacency, showing budget, repeat suffix, still tail, dead window, undescribed asset, brief wiring) plus the model-scored alignment pass.
  • scan_cut — every few seconds across the whole runtime: is anything on screen?
  • av_audit — audio/video drift, held frames and luma at every join.
  • self_reference — when the script cites our own work, our own footage is on screen.
  • Two voice gates — script to chapter text, and chapter text to audio. A third is coming: chapter text back against the script, which is the one that would have caught the eaten numbers.
  • A freshness gate on every built segment keyed on its inputs and its own table row, so a one-line change rebuilds one segment instead of thirty-four.

For the record

The episode this guide accompanies was version 29. The mistakes you can see in it were left on screen on purpose and are called out as they happen. That was the point.