Skip to content

Support joint AV latents in LTXVTiledFusionSampler - #592

Open
avtc wants to merge 1 commit into
Lightricks:masterfrom
avtc:feat/tiledfusion-av-latents
Open

avtc wants to merge 1 commit into
Lightricks:masterfrom
avtc:feat/tiledfusion-av-latents

Conversation

@avtc

@avtc avtc commented Nov 14, 2026

Copy link
Copy Markdown

LTXVTiledFusionSampler crashed on a joint AV latent (NestedTensor from LTXVConcatAVLatent). Dropping the audio to get through the tiled path works, but the refine then re-noises faces without audio context and visibly degrades lip sync. Related to #472 (AV support for the Looping/Extend samplers).

Change

  • The video stream is tiled exactly as before.
  • On every tile pass the audio latent rides along via model_function_wrapper: video + audio are packed with comfy.utils.pack_latents + latent_shapes (same as the regular AV sampling path), audio is passed clean and shares the video timestep (audio at timestep 0 lost lip sync in low-sigma refine tests).
  • Audio is never sampled and is returned unchanged with its original noisent` works downstream. The input audio noise mask is not used for sampling: audio is always kept as is.
  • Video-only latents are unaffected.

Usage note: the sampler already drops the appended guide frames, so don't run LTXVCropGuides again after separating its output.

Tests

  • LTX-2.5 lipsync takes, 158f and 277f at 1344×768, RTX 3090 24 GB: 12.6 / 24.0 min; lip sync kept on medium and far shots where the video-only refine broke articulation.
  • Output audio latent is bit-identical to the input.
  • cfg > 1 smoke run (cond/uncond batched).
  • Video-only latent: output unchanged vs master.

🤖 Generated with GLM-5.3 reviewed by Opus 5.5

The sampler crashed on a joint AV latent (NestedTensor). The video stream
is tiled as before; the full audio latent is packed next to every tile
through model_function_wrapper (pack_latents + latent_shapes) and only the
video output goes back to the sampler. Audio is never sampled and is
returned unchanged with its noise mask.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant