Every graphic in the short above knows the exact frame Aillex says the word. Not roughly. The frame. This guide is how that works, what we installed to get there, and what an AI agent can now do with it on one desktop card. Everything below was run here, on one RTX 5090, this week. Nothing is quoted from a vendor page.

The idea in one paragraph

When we record a line of narration, a small speech model called Whisper writes down every word with a start time and an end time. That file was already making our karaoke captions. We now feed the same file to the graphics engine. When she says a number, the bar for that number starts growing on that syllable. When she says a phrase, an underline draws under it in the frame. The camera and the chart trade places on the sentence, not on a timer. Nobody keyframes anything.

The engine: Remotion

Remotion turns React components into video frames through a headless browser. You write the graphic as code, the code reads data, and every frame is a pure function of the frame number. That last part is the whole reason it suits an AI agent: there is no timeline to click in, only files to write.

It is free for individuals and for teams of up to three people, and it runs locally. It takes real numbers as props. It renders true transparency, so a chart can sit over a camera shot. The cloud chart and caption tools we looked at (CapCut, Flourish) put a subscription and a GUI between you and the frame. We wanted neither.

Install

You need Node and ffmpeg. Then, in a folder:

npm i --save-exact remotion@4.0.520 @remotion/cli@4.0.520
npx remotion add @remotion/transitions @remotion/motion-blur @remotion/paths @remotion/shapes @remotion/noise @remotion/layout-utils @remotion/captions @remotion/media @remotion/rough-notation
npx remotion skills add

The second line is the part most tutorials skip. The core package gives you interpolate and spring, and that is where most projects stop. The packages that make motion look designed are separate: paths draws a line chart along its own path, transitions joins scenes without a hard cut, motion-blur puts a trail behind anything moving fast, rough-notation draws hand-drawn underlines and circles, captions groups word timestamps into pages. Add them with npx remotion add so they land on the same version as the core. Mixed versions fail in confusing ways.

The third line installs Remotion’s own agent skills into your project. They are markdown rules that teach an AI coding agent the current API. Keep reading for why that matters more than it sounds.

Give the agent ground truth, not memory

Remotion is thin in model training data. Every AI session we watched reached for the two functions it half-remembered and stopped there. Our own library of eighteen animated cards was built that way, on a good design system, animated with the beginner slice of the API.

The fix was mechanical. We wrote a small script that walks the installed packages, reads their TypeScript declaration files, and writes one markdown page listing every exported function and component with a one-line note on what it buys us. Four hundred symbols. The rule for every session since: if it is not in that file, it is not installed, and you look it up before you write it. The training-data problem went away the day the reference came from the disk instead of the model’s memory.

Two failures taught us to be careful with the generator itself. The first version missed spring, Easing and AbsoluteFill because the package re-exports them across multiple lines and with export *, and our parser only read one line at a time. A generated reference is only as honest as its parser. Check that the symbols you know exist are in it.

Lock the motion before the first animation

A design system that stops at colour and type is half a design system. Ours had a locked palette and three fonts and no rule for how anything moved, so every card invented its own bounce. We added a motion file: one easing for things entering, one for things leaving, one for data reaching its value (no overshoot, ever, because a bar that overshoots lies about its number for six frames), one for a chart morphing into the next. Timing constants for enter, exit, data, hold. A stagger interval. A layer order so cards, cameras and captions never fight. A safe zone, because captions own the bottom band and a chart label at the bottom is read through a karaoke word.

Every component imports from that file. Nobody writes damping: 14 inline any more.

Sync to the voice

This is the part no editing app can do for you. Whisper gives you, for every word, a start and an end in seconds. We stage that file next to the composition and every component accepts a word as its trigger:

{"kind": "chart", "props": {
  "kind": "bar",
  "points": [{"label": "Mechanical fails", "value": 37, "word": "37"},
             {"label": "Picture vs words", "value": 49, "word": "49", "hot": true}]}}

The bar for 37 grows on the frame she says thirty-seven. The one for 49 waits for forty-nine. A highlighted line takes the words spoken in a window and brightens each one on its own timestamp, then draws an underline under a phrase when she reaches it. The matching is by prefix and ignores punctuation, so “sixty” finds “sixty-two,”. A phrase must be consecutive words. A trigger that is not in the words falls back to a fixed second, and the pipeline tells you it did.

We keep captions on the same file. They render in the same composition as their own band, so moving a graphic never moves them, and both text layers agree about when a word was said.

The six moves

These are the components we built on the new packages. Each one is one JSON file to a producer.

  • Stat hold. One figure, counting to its value on the word, held long enough to read. Decimals allowed, which our older counter could not do. It rendered $7.5B as $8B once.
  • Chart reveal. Bars grow. A line draws along its own path with a dot riding the tip. Per-point word triggers. One accent colour, no gridlines.
  • Metric morph. The same bars re-height and re-order between states instead of cutting. Before becomes after; the transition carries the meaning.
  • Highlight line. Her exact sentence in the frame, words lighting as spoken, a hand-drawn underline, circle or box landing on the phrase that matters.
  • Concept flow. Icon nodes, and tokens that travel curved paths between them with a motion-blur trail. A message going from chat to brain to tool and back is one file, not a diagram of boxes.
  • Annotate. A framed capture, a punch-in toward the region that matters, a hand-drawn box, a connector line that draws itself to a label.

Cams and graphics trading places

The short above adds one more layer. The presenter is a video layer whose rectangle glides between full frame, a side column and a corner on each beat. The graphic for that beat mounts in the space she leaves, on the same frame. Beats are not a clock: a small script reads the word file and starts a beat at every sentence end and at pauses where a new idea begins. Cams span several beats. The picture stops drifting ahead of the sentence, which was our most common defect before this.

One thing bit us here that will bite you. A landscape-designed card placed inside a portrait composition lays itself out for the wrong frame, because it reads its size from the composition. Wrap it in a Sequence with an explicit width and height and scale it with CSS. The first still we rendered had words two hundred pixels tall wrapping one per line.

Let the agent write it

The kit is a markdown file: which component fits which moment, the eight rules, the render recipes. We asked four local models to do the producer’s actual job: read the kit, read the real prop interfaces from the source files, read one sentence of narration with its word times, and write the graphic’s props as JSON. We scored it mechanically: does it parse, are the keys real, is every trigger a word she actually said, does a frame render.

The first run measured nothing. Three models returned empty replies in identical seven seconds, because they spent the whole token budget thinking and never wrote an answer. A gate that does not run reports zero. We fixed the harness before believing anything.

On the real run, gemma4 26B with thinking off wrote a valid, renderable graphic for six of six sentences in about half a second each, and it was the only model that chose a bar chart for the sentence with numbers in it and got the numbers right. Its one miss was a phrase it paraphrased, which the word gate refused. qwen3.6 27B wrote six valid answers in about three seconds each and chose a highlighted line for the numbers sentence: valid, and the wrong card. Valid props are not the right card. Taste is not something our gates score yet.

Our routing now: one resident model writes the prose and the graphics props, with the word gate and a rendered still standing between it and the timeline.

What went wrong, so you can skip it

  • Adding a remotion.config.ts made Remotion require a tsconfig.json. Add one.
  • A transition between scenes shortens the timeline by its own length. If the next scene must start on a specific second for the audio, pad the preceding scene by the transition length.
  • Whisper wrote “Remotion” as two tokens, “re” and “-motion”, because the voice pronounced it as Ree-motion. We merge split terms once, at staging, so captions and graphics both get the fix.
  • GSAP through @remotion/gsap works and is genuinely shorter for text choreography, and it fails silently: a ref on the wrong element produced four identical stills and no error. We keep it optional, for text sequences only.
  • Render a single frame and look at it before rendering the movie. Every defect above was caught in a still.

The numbers, and what they are worth

Ours, on one RTX 5090, this week: a 20-second reel of three scenes rendered in one pass on the CPU while the GPU was busy elsewhere; three 20-to-30-second presenter passes took ten to sixteen minutes each through our avatar chain; the whole 77-second short rendered with its audio in one command. Your machine, your models and your numbers will differ. Measure your own.

All of the code lives in our public tools folder alongside the pipeline that made the short. The components, the reference generator, the motion tokens and the word-sync helper are plain TypeScript and Python. Take them.

The full-length build (EP39): Remotion Mastery: free motion graphics from a local AI agent, synced to every word. The three install commands, the reference trick, the six moves, cams and graphics trading places, which local model writes the graphics, and the six things that went wrong.