This is the written companion to EP39. It walks the whole build in the order we did it, on one RTX 5090, this week. Every number is ours and every failure is one we hit. If you only want the short version of the word-sync trick, the earlier guide on agent-made motion graphics covers that in ten minutes. This one is the long road.

Why not the paid tools

CapCut is a good free editor and Flourish makes good animated charts. Both put a screen full of buttons between your data and the frame, and both live in a cloud with a subscription tier above the free one. For a person editing by hand that is fine. For a pipeline run by an AI agent it is a wall: a tool that needs a person clicking cannot sit inside an automated build, and a tool that lives in somebody else’s cloud cannot be part of a local one.

The requirement we actually had was narrow. Real data in, true transparency out so a chart can sit over the presenter’s shoulder, runs on our own machine, and driven entirely by writing files. Remotion does exactly that. We had it installed for months and were using a fraction of it.

Install in three commands

You need Node and ffmpeg. Then, in a project folder:

npm i --save-exact remotion@4.0.520 @remotion/cli@4.0.520
npx remotion add @remotion/transitions @remotion/motion-blur @remotion/paths @remotion/shapes @remotion/noise @remotion/layout-utils @remotion/captions @remotion/media @remotion/rough-notation
npx remotion skills add

The first line pins the core to one exact version so an update cannot move the floor under a working pipeline. The second line is the one most tutorials skip: the core gives you interpolate and spring, and the packages that make motion look designed are separate. Transitions join scenes without a hard cut, motion blur trails anything moving fast, paths let a line chart draw itself, rough notation draws hand-drawn underlines, captions groups word timestamps into pages. Add them with npx remotion add so they land on the same version as the core. The third line installs Remotion’s own agent skills into the project as markdown rules an AI coding agent reads.

Give the agent ground truth, not memory

Remotion is thin in the data these models were trained on. Every AI session that touched our graphics reached for the two functions it half remembered and stopped there. We had eighteen animated cards, all built that way, on a colour and type system that was genuinely good. When the engineer audited the folder, the core package was pinned 259 releases behind the version already installed, the audio-reactive package had never been imported, and the packages from the second install line were not installed at all.

The fix was a small Python script that walks the installed packages, reads their .d.ts type definition files, and writes one markdown page listing every exported symbol with one line on what it buys us. 404 symbols across 17 packages. The rule since: if it is not in that file, it is not installed, and you look it up before you write it.

Check the generator. Our first version missed spring, Easing and AbsoluteFill because the package re-exports them across several lines and the parser read one line at a time. The count went from 282 to 404 when that was fixed, and the missing 122 included the functions every graphic in the episode is built on.

Lock the motion before the first animation

A design system that stops at colour and type is half a design system. Ours had a locked palette and three fonts and no rule for how anything moved, so every card invented its own bounce. Now there is one short file of motion tokens:

  • one easing curve for things entering, a bezier that starts fast and settles long
  • one for things leaving
  • one for data reaching its value, an ease-out cubic with no overshoot, ever, because a bar that overshoots lies about its number for six frames
  • one for a chart morphing into the next
  • timing in seconds: 0.36 to enter, 0.9 for data to land, a hold of at least 2.5 seconds so a number can be read, 0.14 between staggered siblings
  • a layer order, backdrop to captions, so nothing fights for the same pixels
  • a safe zone: our karaoke captions own the bottom band, so every chart label lives above 900 pixels

Every component imports from that file. Nobody writes damping: 14 inline any more.

Sync to the voice

When we record a line, Whisper writes down every word with a start and an end time. That file was already making the karaoke captions. Now the same file drives the graphics: every component accepts a word as its trigger. Say sixty-two percent and the bar for 62 starts growing on that syllable. The match is by prefix and ignores punctuation, a phrase must be consecutive words, and a word that is not in the audio falls back to a fixed second and the pipeline tells you it did. Both text layers, the caption band and the in-frame line, read one file, so they agree about when a word was said.

The six moves

Each is one JSON file to a producer.

  • Stat hold. One figure counting to its value on the word, held long enough to read. Decimals allowed.
  • Chart reveal. Bars grow, or a line draws along its own path with a dot on the tip. Per-point trigger words. One accent, no gridlines.
  • Metric morph. The same bars re-height and re-order between two states. Before becomes after and the movement is the meaning.
  • Highlight line. The exact sentence in frame, words lighting as spoken, a hand-drawn underline or circle on the phrase that matters.
  • Concept flow. Icon nodes and tokens travelling curved paths between them with a motion-blur trail.
  • Annotate. A framed capture, a punch-in to the region that matters, a hand-drawn box and a connector that draws itself to a label.

And transitions between scenes: fade, wipe, slide, clock wipe, plus a light-leak overlay for the copper look. One piece of arithmetic to get right there, below.

Cams and graphics trade places

The old edit moved on a clock, every twelve seconds or so, and the picture drifted ahead of the sentence. Now a beat starts where a sentence starts, or where a new idea starts inside a long one, read straight from the Whisper file. The presenter’s camera is a video layer whose rectangle glides between full frame, a side column and a corner on each beat, and the beat’s graphic mounts in the space she leaves, on the same frame. Cams span several beats. Graphics attach to beats.

We proved it on a Short first: 75 seconds of narration became 12 beats, three presenter passes through our hybrid camera chain took 16, 10 and 15 minutes of GPU for 25, 20 and 31 seconds of footage, and the graphics and captions rendered in one composition, 1925 frames in one pass on the CPU. EP39 does the same thing at landscape length: one composition renders each segment, ffmpeg only muxes the voice and joins the segments.

Two things moved into Remotion on the back of the Short. The captions are a separate transparent layer rendered from the same words file, so moving a chart never moves a caption. And the warm grade on the camera is applied inside Remotion to the camera layer only, so graphics stay clean.

Let the agent write it

The kit is a markdown file: which component fits which moment, eight rules, the render recipes. The taste lives in the rules. If the point is a sentence it is a highlighted line, not a chart. A comparison is a chart reveal. Before and after is a morph. One number is a stat hold.

We asked four local models to do the producer’s job: read the kit, read the real property interfaces from the source, read one sentence of narration with its word times, and write the graphic’s props as JSON. We scored it mechanically. Does it parse. Are the keys real. Is every trigger a word she actually said. Does a frame render.

The first run measured nothing. Three models returned empty replies in an identical seven seconds because they spent the whole token budget thinking. We fixed the test rig before believing anything it said. On the real run, Gemma 4 at 26B with thinking off wrote a valid, renderable graphic for six of six sentences in about half a second each, and was the only model to choose a bar chart for the sentence with numbers in it and get the numbers right. Its one miss was a paraphrased phrase the word gate refused. Qwen 3.6 at 27B wrote six valid answers in about three seconds each and chose a highlighted line for the numbers sentence. Valid, and the wrong card.

Our routing now: one resident model for prose and graphics, with the word gate and a rendered still between it and the timeline. Every render is versioned and never overwritten, so a worse attempt sits next to a better one and the comparison is free.

What went wrong, so you can skip it

  1. Adding remotion.config.ts makes Remotion require a tsconfig.json. Add one first.
  2. A transition shortens the timeline by its own length because it overlaps both scenes. If the next scene must start on a specific second for the audio, pad the scene before it by the transition length.
  3. Whisper wrote “Remotion” as two tokens. We merge split terms once, at staging, so captions and graphics both get the fix.
  4. GSAP through the official adapter works and is shorter for text choreography, and it fails silently: a ref on the wrong element produced four identical stills and no error. Optional, text only.
  5. The rough-notation package takes padding as an object, not a number, and the number version rendered nothing.
  6. Ours, not Remotion’s: our script splitter cut sentences on periods, and a number with a dot inside it looked like a sentence end, so “SageAttention 1.0.6” reached the voice as “six” in the previous episode. The new splitter cannot lose a character, and a gate refuses any chapter where a number does not survive. Building the episode’s tools from scratch is what found it.

And one more from this build: ElevenLabs appended invented words to the end of four of eighteen long reads. A word-count check waved them through. The voice tool now transcribes every read, checks the first and last five words against the script, retries, and trims an appended tail off a complete read before anything reaches the cams.

Every defect above was caught by rendering one frame, or checking one line, before rendering the movie. A still costs seconds.

What it costs, and who gets it free

Zero dollars a month. Remotion’s licence, in their wording, is free for an individual, a for-profit organisation with up to three employees, and any non-profit; past that a company buys a licence. Read the page against your own situation before you build on it. Rendering is CPU only, which on a one-card machine means the graphics render beside a GPU that is busy for sixteen minutes at a time rendering the presenter. Measured on this episode: one 35-second segment took about a minute of CPU for its graphics and captions (six browser tabs, the GPU busy the whole time) against roughly seventeen minutes of GPU for the presenter cam of the same segment. Your CPU will land somewhere else; the shape of the ratio is the point.

Everything described here is in our public tools folder: the components, the reference generator, the motion tokens, the word-sync helper, the beats script. Plain TypeScript and Python.