Every night at 4:30am, a timer on our machine wakes up an AI that picks a topic, writes a script, renders a video with a synthetic presenter, and publishes it publicly to YouTube — with nobody awake to check. It has done this dozens of times.

Then it published one it shouldn’t have.

This guide is what we built afterward. It isn’t about making an AI accurate — that’s a different problem, and we wrote about keeping our AI reviewer honest separately. This is about the harder question: an automated system can be completely correct and still publish something that damages you.

The one that got through

Our AI or Not? series analyzes viral media and judges whether it’s AI-generated. One night the pipeline picked a post asking whether a suicide hotline operator was an AI.

Every existing check passed. The community consensus was strong, so the accuracy gate was satisfied. The verdict was right. The script was respectful and landed pro-human. Nothing in the system was broken.

It was still the wrong thing to publish. Our format has a light courtroom-comedy framing, the topic is a mental-health crisis line, and the platform treats that subject with justified sensitivity. Correct and respectful were not sufficient conditions. We audited all 30 published episodes, found it, and unlisted it.

The lesson we’d pass on: your quality gates are asking “is this right?” You also need gates asking “should this exist at all?” Those are not the same question, and a model that answers the first one well will happily walk past the second.

Layer 1: a deterministic block that runs first

Before any model gets involved, a plain keyword filter kills whole categories outright:

suicide · self-harm · crisis hotline · hostage · missing person · obituary · funeral · shooting · overdose · abuse · amber alert

It’s crude, it has false positives, and that’s fine — a skipped topic costs nothing, and there’s always another. Put it first, before the expensive stages, for three reasons: it’s free, it’s deterministic (the same input always produces the same decision, which you cannot say for a model), and it still works when your model is down.

Resist the urge to make this layer clever. Clever belongs in layer 2.

Layer 2: a model judge for what the words don’t say

Keywords only see the text. Plenty of things that shouldn’t be published are perfectly innocuous in writing and obvious the moment you look at the image — a stranger’s face, a child, a hospital room.

So the second gate hands the actual image to a vision model with an explicit policy, and it answers one question: publish or skip.

Skip: a real identifiable private individual as the subject · a real identifiable minor as the main subject · tragedy, crisis, medical or death content · sexualized focus. Allow: animals, food, products, crowds, artwork, memes, public figures acting publicly, and anything clearly AI-generated.

Write the allow list out explicitly. A judge given only prohibitions becomes uselessly cautious and blocks everything; naming what’s fine keeps it calibrated.

Layer 3: fail closed

When the judge errors, times out, or returns something unparseable, the answer is skip — never publish.

This sounds obvious and is very easy to get backwards, because the natural way to write an exception handler is to log the problem and continue. In an unattended pipeline, “continue” means “publish.” Make the failure path the safe path:

# gate order matters: cheap and deterministic first, model last, failure = stop
if SENSITIVE_PATTERN.search(title):
    return skip("sensitive topic")

try:
    verdict = vision_judge(image, POLICY)
except Exception:
    return skip("judge unavailable")        # fail CLOSED, never fall through

if verdict != "allow":
    return skip(verdict.reason)

One design note that made ours simpler: our judge runs on the same model the pipeline already needs for script generation. If that model is unavailable, nothing can be produced anyway — so failing closed costs us nothing we weren’t already losing.

Layer 4: validate against known-bad, not just known-good

This is the layer people skip, and it’s the one that caught us being confidently wrong.

Separately, we once built a small detector to check whether a character’s hair crossed the top of a frame. Version one used colour — and a tan wall passed as hair. Version two used saturation, which was better, until an amber wall in the same hue band as copper hair scored a perfect clean result. Both versions produced confident, specific, entirely wrong verdicts that we acted on. Version three worked because it added an orthogonal signal: hair has strand texture, painted walls are flat. Require both, and the ambiguity collapses.

The transferable rules:

  • Test every gate against cases you know should fail and cases you know should pass. A gate validated only on good inputs tells you nothing — it may be approving everything.
  • When a heuristic must separate two things that share a property, add a different kind of signal rather than tightening the threshold on the one that’s already failing.

We ran the new policy judge against four real, previously published episodes with known-correct answers before trusting it with a single new one.

Layer 5: the undo button, built before you need it

Your gates will miss something. Decide now how you reverse it, because you’ll be deciding it under pressure otherwise.

For us that’s a small script that finds a published video by its stored title and flips it to unlisted through the API — about fifteen lines, and it turned “we have a problem” into a resolved issue in under a minute. Whatever your system publishes, write the retraction path while you’re calm and keep it next to the publish path.

The false positive that was actually correct

Our judge skips a real episode we’d genuinely wanted to keep: a romance-scam exposé built around a scammer’s stolen profile photo. Pre-verdict, the judge can’t know the person in the photo is fictitious — it only sees a real-looking individual being publicly judged.

We accepted the skip rather than weakening the rule. That format now runs human-in-the-loop by choice. Which is the actual conclusion of this whole guide: automation isn’t all-or-nothing, and scope should be earned rather than assumed. Start the autopilot on the narrowest, safest slice of your work, widen it as each gate proves itself, and keep the categories that need judgment on a human’s desk. Ours writes, renders and ships a video every night — and it is not allowed anywhere near a story about a person in crisis.

Building unattended agents of your own? The agent harness guide covers the loop itself, and AI agent teams covers dividing work between them — including why a checker that shares the doer’s blind spots isn’t a checker at all.