Companion episode — where LipDub sits among the local talking-avatar methods.
What LipDub is
LipDub is LTX-2.3’s native audio-driven dub: you give it a base video and an audio track, and it re-syncs the mouth to the new audio — video-to-video, entirely local. Its big advantage over mouth-only inpainting is resolution — output up to 1088×1920, sharper than most local lip-sync.
The trap that eats a day
LTX ships with two modes that look almost identical and behave completely differently:
- LipDub (generative): the audio latent is empty — the model generates its own speech-like sound and matching lips. Overlay your real voice on that and the lips never track your words — a foreign-film-dub mismatch, or worse, a “made-up language.”
- LipSync / “Just Dub It” (V2V): the audio latent comes from the encoded input audio — so the model follows your words. This is the one you want.
The fix is a single wire: feed the concat step’s audio latent from the audio-reference tokens (your encoded voice), not from an empty audio latent. Get that right and it truly dubs; get it wrong and it babbles.
Production notes
- Re-mux your clean audio. The workflow round-trips voice through an audio VAE, which blurs consonants (“couldn’t understand her”). Generate the video, then mux the original clean TTS audio back over it — the lips were synced to it, so they still match.
- It’s slower. Two-stage plus upscaling barely fits a 32GB card. You trade speed for resolution.
Where it fits
The higher-res option in the lip-sync tier. When it lands and you’ve wired the true dub path, it’s genuinely nice. It’s finicky and slow enough that for fast daily work a mouth-only method wins — but for a hero shot where resolution matters, LipDub earns its slot.
Honest verdict
Powerful and sharp, with a real footgun: make sure you’re in the audio-driven mode, not the generative one. Test with a sentence you know — if the lips don’t match your actual words, you’re in the wrong mode.