Skip to content

LTX-2 condition pipeline #12926

Closed
Closed

Description

@sayakpaul

#12915 added support for LTX-2.

What's supported?

  • Single stage T2V and I2V
  • Upsampling

Similar to the LTX condition pipeline, it would be nice to have support for something similar for LTX-2 as well.

Both the single and double stage pipelines support passing conditions.

Activity

  1. surajyadav-research commented on Jan 8, 2026

    @surajyadav-research

    Hi @sayakpaul,
    I’m interested in contributing to this feature and would love to work on it. Could you please assign it to me?

  2. sayakpaul commented on Jan 8, 2026

    @sayakpaul
    MemberAuthor

    Thanks for your interest!

    Could you share your plan as to how you want to approach the PR?

  3. surajyadav-research commented on Jan 8, 2026

    @surajyadav-research

    I’m looking into the LTX pipeline and will follow up with a plan ASAP.

  4. anjaneya-sharma commented on Jan 8, 2026

    @anjaneya-sharma

    @sayakpaul I'm also very much interested in this, and would love to collaborate with @surajyadav-research , if thats alright.

  5. surajyadav-research commented on Jan 8, 2026

    @surajyadav-research

    Hi @anjaneya-sharma
    I’m looking into LTX pipeline to come up with a plan for the PR. I’ll put together a plan and share it shortly, and then we can discuss it. Happy to collaborate on this.

  6. surajyadav-research commented on Jan 8, 2026

    @surajyadav-research

    Hi @sayakpaul
    I’ve come up with the following plans.

    1) New shared module

    Contains all conditioning logic so pipelines only call helpers.

    In this file

    • LTX2VideoCondition
      • if image (I2V) given → add condition at frame_index=0, strength=1.0
    • encode_conditions_to_latents(conditions, vae, height, width, num_frames, device, dtype) -> encoded_conditions
      • preprocess/resize → VAE encode → map pixel frame_index → latent_frame_index
    • apply_conditions_to_latents(latents, encoded_conditions) -> latents

    2) Stage-1 pipelines (denoising): add conditions=None

    A) T2V

    Modify: pipelines/ltx2/pipeline_ltx2.py
    Add conditions=None to LTX2Pipeline.__call__:

    • encoded = encode_conditions_to_latents(conds, ...)
    • optional pre-loop: latents = apply_conditions_to_latents(latents, encoded)
    • inside denoise loop (after scheduler step): latents = apply_conditions_post_step(latents, encoded)

    B) I2V

    Modify: pipelines/ltx2/pipeline_ltx2_image2video.py
    Add conditions=None to LTX2ImageToVideoPipeline.__call__:

    • auto frame-0 condition
    • same encode + per-step enforcement as T2V

    3) Stage-2 Upsampling: distribution-matched conditioning

    Modify: pipelines/ltx2/pipeline_ltx2_latent_upsample.py
    Add conditions=None to LTX2LatentUpsamplePipeline.__call__:

    • encode conditions at input latent scale: encoded_in = encode_conditions_to_latents(...)
    • upsample main latents: latents_up = upsampler(latents_in)
    • upsample each condition latent with same upsampler - encoded_up
    • inject: latents_up = apply_conditions_to_latents(latents_up, encoded_up)
    • decode
  7. sayakpaul commented on Jan 9, 2026

    @sayakpaul
    MemberAuthor

    Thanks for the plan.

    Could you elaborate a bit about 1?

    Also, I think we can focus only on stage 1 for now.

  8. surajyadav-research commented on Jan 9, 2026

    @surajyadav-research

    Hi @sayakpaul
    thanks for reviewing the plan.
    This is the single-stage plan I came up with.

    1. Create new helper module: pipelines/ltx2/pipeline_ltx2_condition.py

      • Add LTX2VideoCondition dataclass
      • Implement helpers:
        • ensure_conditions_list()
        • encode_conditions_to_latents() -> returns (cond_latents_packed, cond_mask_packed)
        • apply_conditions_to_latents() -> blends/injects into latents
        • expand_mask_for_cfg() -> duplicates mask for CFG batch
        • prepare_noise_latents_packed() (used by I2V to start from noise)
    2. Modify LTX2Pipeline.__call__ (T2V)

      • Add arg: conditions=None
      • After prepare_latents(): encode conditions + inject once
      • In denoise loop:
        • Use per-token timestep: video_timestep = t * (1 - cond_mask_cfg)
        • After scheduler step: re-inject latents = apply_conditions_to_latents(...)
    3. Modify LTX2ImageToVideoPipeline.__call__ (I2V)

      • Add arg: conditions=None
      • Replace special “frame0-only” I2V logic:
        • Start from noise latents (prepare_noise_latents_packed)
        • Add implicit condition: input image as frame_index=0 (unless user provided it)
        • Encode+inject conditions same as T2V
      • In denoise loop:
        • Same masked timestep + re-inject after scheduler step
  9. sayakpaul commented on Jan 9, 2026

    @sayakpaul
    MemberAuthor

    Plan looks good but let's keep LTX2ImageToVideoPipeline untouched for now and instead, do LTX2ConditionPipeline. We could also follow the LTXConditionPipeline and pack the conditions into a dataclass like so:

    That is a good design and it would be nice to stick to it.

  10. surajyadav-research commented on Jan 9, 2026

    @surajyadav-research

    Hi @sayakpaul
    Thanks for reviewing the plan. I’ll come up with an updated plan.

  11. surajyadav-research commented on Jan 9, 2026

    @surajyadav-research

    Hi @sayakpaul
    thanks for reviewing the plan, I updated plane now its similar to LTXConditionPipeline

    1) New file

    Create: diffusers/pipelines/ltx2/pipeline_ltx2_condition.py

    Add:

    • @dataclass LTX2VideoCondition(image=None, video=None, frame_index=0, strength=1.0)
    • class LTX2ConditionPipeline(...same modules as LTX2Pipeline...)
    • def __call__

    2) Parse + preprocess (inside __call__)

    Add helpers:

    • _normalize_condition_inputs
    • _trim_conditioning_sequence
    • _preprocess_conditioning_tensors

    3) Encode conditions into LTX-2 video latents

    Add helpers (copied/adapted from LTX2Pipeline):

    • _retrieve_latents(...)
    • _normalize_latents(...)
    • _pack_latents(...) / _unpack_latents(...)
    • _denormalize_latents(...)

    Core:

    • prepare_latents_with_conditions

    Rules:

    • frame_index == 0: encode→normalize, blend into base latent frames via torch.lerp(...), set mask for those tokens
    • frame_index > 0: encode→normalize→pack as extra tokens, prepend to latents, mask = strength, track extra_tokens

    4) Timesteps + denoise_strength (LTX-2 schedule)

    Reuse LTX-2:

    • calculate_shift(...), retrieve_timesteps(...), audio_scheduler = copy.deepcopy(self.scheduler)
      Add:
    • get_timesteps(..., denoise_strength)

    If denoise_strength < 1: compute latent_sigma and mix provided latents with noise like LTX-1 (edit mode).

    5) Denoising loop (freeze conditioned tokens)

    Add:

    • add_noise_to_image_conditioning_latents(...)

    Loop:

    • run transformer exactly like LTX2Pipeline (video+audio, CFG)
    • scheduler step video/audio
    • freeze tokens:
      • tokens_to_denoise = (t_norm < (1.0 - conditioning_mask)).unsqueeze(-1)
      • latents = torch.where(tokens_to_denoise, denoised_latents, latents)

    6) Cleanup + decode

    • latents = latents[:, extra_tokens:]
    • _unpack_latents → _denormalize_latents → vae.decode
    • audio path unchanged (same audio_vae.decode + vocoder as LTX2Pipeline)

    7) Other files to change

    • diffusers/pipelines/ltx2/__init__.py: export LTX2ConditionPipeline, LTX2VideoCondition
  12. sayakpaul commented on Jan 10, 2026

    @sayakpaul
    MemberAuthor

    Okay plan looks reasonable to me. Thanks for iterating!

  13. surajyadav-research commented on Jan 10, 2026

    @surajyadav-research

    Hi @sayakpaul
    I’ve started working on this and will raise a PR ASAP.

  14. devanshi00 commented on Jan 28, 2026

    @devanshi00

    Hi @surajyadav-research,
    Are you still working on this issue? I’m really interested in contributing and would love to help with the implementation if possible. I’d be glad to begin working on a PR following the plan at the earliest opportunity.

  15. dg845 commented on Jan 30, 2026

    @dg845
    Contributor

    Hi all, we have decided that supporting a LTX-2 condition pipeline is a high priority for us and will thus take over work on it.

    Thanks to everyone who offered to work on this! We appreciate your interest and look forward to collaborating with you in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions