How to Make a Music Video with Claude Code (and a Free Local AI)

People are making music videos with Claude where no video model is involved at all. Claude writes a program, and the program draws every frame. Ask it for second 12.4 and it paints second 12.4. A flipbook made of code.

We wanted to know two things. Can our free local agent do this too? And why do so many of these videos flash?

Presenter shots were generated with MiniMax H3, by MiniMax, used under the MiniMax H3 licence. Nothing generated by H3 was used to train anything. The voice in the video is ElevenLabs, in the cloud. Every music video below was drawn by code and rendered on our PC.

Every number below is ours, measured on one RTX 5090 (32 GB), unless it says “reported”.

0. Before you render: five questions

We learned each of these by getting it wrong.

1. Does Chrome get your GPU when no screen is attached? An agent runs as a background job with no display. Render one still the way it will (env -u DISPLAY -u WAYLAND_DISPLAY) and read which GPU it names. If you see Error creating webgl context, stop.
2. Is the tempo right? Beat trackers guess. Ours said 143.55 bpm for a song at 140.29. Check the grid against the song before a single cut is timed to it.
3. Is the set inside still()? Walls, sky, floor and furniture get painted once. Run check_still.sh on a locked shot: colour on the heat map should sit only on the characters and the lyric word.
4. How many designed transitions are there? Three to five in a song, never two in a row, never the same one twice running. Every other cut is a straight cut on a bar line.
5. Is the plan on screen? Open the frames at the times the storyboard promises something, and look. Don’t take the agent’s word for it. Ours wrote “looks good overall” about a payoff that wasn’t there.

And one for after the render: if you change any input of a finished video, check the whole video again.

What you need

Rendering is cheap next to everything else: about 0.25 s a frame on the 5090. The kit’s own 11-second demo rendered in 67 seconds.

Download the kit: ep56-code-music-video-kit.zip. Our stage.js, the render, lint and flicker scripts, the beat and lyric timers, the eight rules, Hermes’ skill and six stage prompts, and the writers’ room. MIT. It does not include the ClaudeAnimationBase kit (clone it from its author) or the producer’s song.

How it works

The kit’s README has an example prompt, and that’s where we started: “make a 15-second video of Clawd trying to catch a butterfly.” The agent reads the kit’s rules, writes a storyboard, then builds the video shot by shot in JavaScript. Every frame is a pure function of time, so the renderer can ask for any frame in any order. After each shot the agent renders a contact sheet, looks at the picture, and fixes what it sees. ffmpeg stitches the frames and the song into an MP4.

The look comes from p5.brush: watercolour washes and ink lines with a hand-made wobble. The kit re-rolls that wobble 12 times a second so the drawing feels alive. Hold on to that detail, it comes back.

Setup, and the run that made 0 frames

Install Node.js, ffmpeg and Python, clone the kit, and copy our files in. The kit’s README and ours have the exact commands.

Then the flag. On Linux, the kit starts Chrome with --use-gl=angle. Our first Hermes run went out as a background service, with no display attached, and that flag got no WebGL context at all. Hermes did the reasonable thing and switched to software rendering. Every render then hit a 180-second timeout. We stopped it at 21 minutes with 0 frames on disk.

That was our setup’s fault, not Hermes’. The fix is one flag: --use-angle=gl-egl reached the GPU with or without a display. Our new_project.sh patches it into every new project. The kit’s renderer can also take it per run as --gpu-angle=gl-egl.

Same prompt, two directors

Same prompt to both, unattended, 110 bpm.

Hermes’ own storyboard said the butterfly lands on Clawd’s head. It lands at his feet. Four times faster, a step below on taste, and a self-review that passed something a viewer catches at once.

The two kinds of flashing

The producer’s note after watching both: both of you had flashing backgrounds. It turned out to be two different problems.

The shimmer. Remember the wobble re-rolled 12 times a second. On an ink line that looks alive. On a sky painted as one big watercolour wash, the whole sky repaints with a new texture 12 times a second, and you see it shimmer. We wrote flicker_shots.py to measure it: in each shot, the share of the picture that should be still and changes anyway, frame to frame. Under 0.5% looks still, and over 2% reads as flashing. Claude’s butterfly scored 4.73%, with every one of its 5 shots over 2%.

The big flash. Hermes’ butterfly scored 1.56%, and on the numbers it looked like the free one flashed less. The producer watched it and saw the whole sky go green, four times (at 4.54, 5.42, 13.5 and 14.71 seconds). Those were Hermes’ brush-wipe transitions covering the entire picture. Our meter treats any frame where more than a quarter of the picture changes as a cut, and skips it. So it skipped all four flashes and measured only the calm stretches between them, less than half of the video’s 15 seconds.

Measure the shimmer. Watch for the big flash yourself. The kit’s copy of flicker_shots.py now prints every “cut” it skipped and how many seconds it really measured, so a wipe shows up as a cut you didn’t plan.

The fix for the shimmer: still()

A theatre trick: paint the set once and leave it alone, and only the actors move. still() is a few lines of JavaScript in our stage.js. Anything drawn inside it gets the same random seed in every frame, so it comes out identical:

function still(key, fn) {
  const b = BOILN; BOILN = 'still'; boilSeed(key);
  try { fn(); } finally { BOILN = b; boilSeed(key + '|after'); }
}

// in a shot: the room is the set, the character is an actor
still('desk', () => room(t));
clawd(x, y, 24, { ...feel('happy', t) });

A prop is an actor only while it moves. A mug on the desk is set; the mug being lifted is an actor. Light can still change, slowly and on purpose, computed from the time instead of from random numbers.

The fix for the big flash: Rule 7

A transition is punctuation. Use one to mark a big change, like a new section or the turn of the song: about 3 to 5 in a 66-second song, never two in a row, never the same one every time. Everything else is a straight cut on a bar line.

Every number in this table has the same blind spot: a full-frame wipe never gets measured. That’s why the rule exists, and why someone watches the whole video.

Rules 7 and 8

The kit’s MUSIC_VIDEO_GUIDE.md has eight rules. One to six we wrote before the producer’s notes on Take 48: split every frame into the set and the actors, one set seen from many angles, a locked camera, a fixed frame, lyrics on screen in time, follow the song’s sections. Seven and eight came from what the producer said about Take 48:

“The frequent use of the transition effect was bothersome and the world went from fully bland to fully colored almost instantly losing that nuance.”

Rule 7. Transitions are punctuation. Above. Our concept_check.py now counts them from the storyboard: each shot that enters on a designed transition says transition_in: <kind>, and the check fails on more than five, on two neighbouring shots, or on the same kind twice running.

Rule 8. Change in steps the viewer can follow. When the picture evolves across a song (colour, light, how finished the drawing is), show the in-between. Plan the steps as a table with at least one partial state between “before” and “after”. Then render a sheet of the first frame of every shot, in order. Neighbouring shots should differ by a step, not a jump.

A real song, and teaching a local model in stages

Next both directors got a real song: Core Loop, 66 seconds, made by the producer in Suno. Same brief, same frame, same karaoke bar, and the timed lyrics from lyrics.py (Whisper pins the lyric sheet to the sung words; 117 of 120 words matched).

Claude did it in one sitting, about 35 minutes: a little yellow robot called Bit, a desk that goes from night to sunrise, and a wire world inside the monitor for the choruses.

Hermes tried the same way and ran out of room. A local model holds only so much conversation at once, and every file it reads and every image it checks takes up space. Around step 76 it filled its 64k context while still fixing the set file, sent back an empty reply, and stopped. 0 frames.

So we split the job into six short sessions, like shift workers. Each one:

• starts fresh, with an empty context;
• reads only what its prompt lists: a handover note (HANDOFF.md) left by the session before, a one-page API sheet (API_QUICK.md) instead of the kit’s long guide, and its own part of the storyboard;
• writes one part of the song, running lint.sh after every single edit (it names the file and line of any syntax error and test-loads one frame, in a few seconds);
• renders sheets, looks at them, fixes, looks again;
• appends what it built and what it left broken to the handover note, then quits.

Stage 1 builds the set and the star, stages 2 to 5 take one part of the song each, and stage 6 reviews the whole video and renders it. Our sixth stage crashed inside Hermes’ own verbose logger (we had passed -v), so a short finishing session rendered it. All told: about 1 hour 45 minutes, three times Claude’s time. But it finished. The six prompts are in the kit with the sung lines blanked out.

The producer’s verdict: “The claude video is objectively better because the hermes video is really just jumping around in front of the monitor on the same desk.”

Too literal: the writers’ room

The producer had a note for Claude’s take too: it drew the lyrics as literally as it could. When the song sings about putting on a pixel scarf, Bit puts on a pixel scarf. (Hermes’ plan said its robot would too. We checked the frames at 10 and 12 seconds: no scarf.) The viral videos tell a second story the lyrics never sing.

So we built a writers’ room, the concept room. It’s in the kit:

1. A brief: the song, its timed lines, what the engine can and can’t draw.
2. A literal list: the obvious picture for every line. Those pictures are banned as main images, with at most two allowed as a wink.
3. Blind pitches: Claude and three local models (qwen3.8 27B, gemma4 26B, and Qwen 3.6, Hermes’ brain), three ideas each, each idea through a different lens (an object that changes state, a ritual, a real UI, a reversal…). Nobody saw anyone else’s.
4. A board: all 12 ideas side by side, shuffled, under letters A to L.
5. A pick, scored on a rubric out of 30: would a stranger get it, is there a second story, does the motif change state, is there a point, is it worth a rewatch, can our engine draw it.

The scorer was Claude, and it knew which three ideas were its own: the Claude pitcher’s reply had shown it the titles. So the scores come with that bias. Claude’s pitches scored 27 to 29. The best local model reached 24. Hermes’ brain scored 18.

The winner was one of Claude’s: a video that paints itself while it plays, racing the song to finish itself. Called Take 48, because in the video it’s on its forty-eighth try. Same model, same engine, about the same time (45 minutes, plus about 5 minutes for the room). All five planted details paid off on screen. The idea changed what the video was about.

And the producer ranked it below Claude’s first, literal take. Better idea, worse craft: too many transitions, and a world that jumped from grey to full colour almost at once. That’s where Rules 7 and 8 came from.

The committee

The room picked one idea and threw the rest away, and it knew nothing about the craft lessons we already had. So it grew into a committee (COMMITTEE.md in the kit):

• each member gets a role: Showrunner, Comedian, Rival analyst, Toolsmith, with the chair as the Archivist;
• every member proposes for every part of the job, blind;
• in the debate, proposals sit behind letters, and each member names the best part of someone else’s, the biggest weakness of each, and a graft (“take B’s ending to fix D’s middle”);
• the chair writes a synthesis with every graft credited;
• ten standing questions get answered against the plan, with evidence. Number 8 is “Can this be funnier?”;
• one human decides.

Our cast was the same four models as the room, chaired by a Claude Code session. The chair and one member are both Claude, so the same bias question applies. The committee planned this episode and the brief for Take 49: keep the room’s idea and the craft of Claude’s first take, no more than five transitions, change in steps.

Time and cost

“$0” means no fees. We didn’t meter the electricity. Every render ran on the 5090 either way.

Failures log

The producer’s pick

Four takes of one song. The producer’s ranking:

1. Take 49, the committee’s take. “I enjoyed take 49 … the high resolution filter drag was cool.”
2. Claude’s first Core Loop, the literal one. It has more going on than Take 48.
3. Take 48, the writers’ room. Pretty good, but not better than Claude’s first one.
4. Hermes’ Core Loop.

Run one yourself

1. Download the kit, clone ClaudeAnimationBase, and follow the kit’s README.
2. Run the no-display smoke test. Fix the GPU before anything else.
3. For a song: beats.py, then phase_check.py, then lyrics.py with your lyric sheet, then new_project.sh with the tempo, the downbeat and the lyrics.
4. Give your agent MUSIC_VIDEO_GUIDE.md. For a local model, give it API_QUICK.md and split the song into stages like ours.
5. Before you render, the five questions at the top. After, flicker_shots.py, then watch the whole thing.

Credits

• ClaudeAnimationBase by John Heibel, MIT. The drawing engine, the renderer and Clawd’s animation code are its; our stage.js sits on top.
• Clawd is Anthropic’s mascot and the kit’s demo character.
• The rival videos we studied, by their creators: “LGTM” (747,398 views when we checked on 12 October), “WHY AM I SLOP?”, whose description says “This took 10 hours and 30% of my weekly quota”, and Pavel Kasík’s song about agents, made with ClaudeAnimationBase.
• Core Loop: made by the producer in Suno. Not in the kit.

Older guides

Our first music video, made with video models on one GPU, is Make a Full AI Music Video on One GPU, and the kids’ version with their own toys is Make a Kids’ Music Video With AI. The last time an agent drew our graphics in code was Let Your AI Agent Make the Motion Graphics. Hermes learned its first skills in Teach Your Local AI New Skills, and the gates around it are in Build a Local AI Agent Harness.

What should we teach Hermes next? Tell me in the comments on the video, or in r/aillex.

The next music video used real footage instead of code: How to Put an AI Character Into Real Stock Footage on One RTX 5090.