Companion episode: where LipDub sits among the local talking-avatar methods.
What LipDub is
LipDub is LTX-2.3’s native audio-driven dub: you give it a base video and an audio track, and it re-syncs the mouth to the new audio, video-to-video, entirely local. Its big advantage over mouth-only inpainting is resolution, output up to 1088×1920, sharper than most local lip-sync.
The trap that eats a day
LTX ships with two modes that look almost identical and behave completely differently:
- LipDub (generative): the audio latent is empty, the model generates its own speech-like sound and matching lips. Overlay your real voice on that and the lips never track your words: a foreign-film-dub mismatch, or worse, a “made-up language.”
- LipSync / “Just Dub It” (V2V): the audio latent comes from the encoded input audio, so the model follows your words. This is the one you want.
The fix is a single wire: feed the concat step’s audio latent from the audio-reference tokens (your encoded voice), not from an empty audio latent. Get that right and it truly dubs; get it wrong and it babbles.
Production notes
- Re-mux your clean audio. The workflow round-trips voice through an audio VAE, which blurs consonants (“couldn’t understand her”). Generate the video, then mux the original clean TTS audio back over it, the lips were synced to it, so they still match.
- It’s slower. Two-stage plus upscaling barely fits a 32GB card. You trade speed for resolution.
Where it fits
The higher-res option in the lip-sync tier. When it lands and you’ve wired the true dub path, it’s genuinely nice. It’s finicky and slow enough that for fast daily work a mouth-only method wins, but for a hero shot where resolution matters, LipDub earns its slot.
Honest verdict
Powerful and sharp, with a real footgun: make sure you’re in the audio-driven mode, not the generative one. Test with a sentence you know, if the lips don’t match your actual words, you’re in the wrong mode.