You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi @anjaneya-sharma
I’m looking into LTX pipeline to come up with a plan for the PR. I’ll put together a plan and share it shortly, and then we can discuss it. Happy to collaborate on this.
Plan looks good but let's keep LTX2ImageToVideoPipeline untouched for now and instead, do LTX2ConditionPipeline. We could also follow the LTXConditionPipeline and pack the conditions into a dataclass like so:
Hi @surajyadav-research,
Are you still working on this issue? I’m really interested in contributing and would love to help with the implementation if possible. I’d be glad to begin working on a PR following the plan at the earliest opportunity.
Activity
surajyadav-research commentedon Jan 8, 2026
Hi @sayakpaul,
I’m interested in contributing to this feature and would love to work on it. Could you please assign it to me?
sayakpaul commentedon Jan 8, 2026
Thanks for your interest!
Could you share your plan as to how you want to approach the PR?
surajyadav-research commentedon Jan 8, 2026
I’m looking into the LTX pipeline and will follow up with a plan ASAP.
anjaneya-sharma commentedon Jan 8, 2026
@sayakpaul I'm also very much interested in this, and would love to collaborate with @surajyadav-research , if thats alright.
surajyadav-research commentedon Jan 8, 2026
Hi @anjaneya-sharma
I’m looking into LTX pipeline to come up with a plan for the PR. I’ll put together a plan and share it shortly, and then we can discuss it. Happy to collaborate on this.
surajyadav-research commentedon Jan 8, 2026
Hi @sayakpaul
I’ve come up with the following plans.
1) New shared module
Contains all conditioning logic so pipelines only call helpers.
In this file
LTX2VideoConditionencode_conditions_to_latents(conditions, vae, height, width, num_frames, device, dtype) -> encoded_conditionsframe_index→latent_frame_indexapply_conditions_to_latents(latents, encoded_conditions) -> latents2) Stage-1 pipelines (denoising): add
conditions=NoneA) T2V
Modify:
pipelines/ltx2/pipeline_ltx2.pyAdd
conditions=NonetoLTX2Pipeline.__call__:encoded = encode_conditions_to_latents(conds, ...)latents = apply_conditions_to_latents(latents, encoded)latents = apply_conditions_post_step(latents, encoded)B) I2V
Modify:
pipelines/ltx2/pipeline_ltx2_image2video.pyAdd
conditions=NonetoLTX2ImageToVideoPipeline.__call__:3) Stage-2 Upsampling: distribution-matched conditioning
Modify:
pipelines/ltx2/pipeline_ltx2_latent_upsample.pyAdd
conditions=NonetoLTX2LatentUpsamplePipeline.__call__:encoded_in = encode_conditions_to_latents(...)latents_up = upsampler(latents_in)encoded_uplatents_up = apply_conditions_to_latents(latents_up, encoded_up)sayakpaul commentedon Jan 9, 2026
Thanks for the plan.
Could you elaborate a bit about 1?
Also, I think we can focus only on stage 1 for now.
surajyadav-research commentedon Jan 9, 2026
Hi @sayakpaul
thanks for reviewing the plan.
This is the single-stage plan I came up with.
Create new helper module:
pipelines/ltx2/pipeline_ltx2_condition.pyLTX2VideoConditiondataclassensure_conditions_list()encode_conditions_to_latents()-> returns(cond_latents_packed, cond_mask_packed)apply_conditions_to_latents()-> blends/injects intolatentsexpand_mask_for_cfg()-> duplicates mask for CFG batchprepare_noise_latents_packed()(used by I2V to start from noise)Modify
LTX2Pipeline.__call__(T2V)conditions=Noneprepare_latents(): encode conditions + inject oncevideo_timestep = t * (1 - cond_mask_cfg)latents = apply_conditions_to_latents(...)Modify
LTX2ImageToVideoPipeline.__call__(I2V)conditions=Noneprepare_noise_latents_packed)imageasframe_index=0(unless user provided it)sayakpaul commentedon Jan 9, 2026
Plan looks good but let's keep
LTX2ImageToVideoPipelineuntouched for now and instead, doLTX2ConditionPipeline. We could also follow theLTXConditionPipelineand pack the conditions into a dataclass like so:diffusers/src/diffusers/pipelines/ltx/pipeline_ltx_condition.py
Line 96 in 3981c95
That is a good design and it would be nice to stick to it.
surajyadav-research commentedon Jan 9, 2026
Hi @sayakpaul
Thanks for reviewing the plan. I’ll come up with an updated plan.
surajyadav-research commentedon Jan 9, 2026
Hi @sayakpaul
thanks for reviewing the plan, I updated plane now its similar to
LTXConditionPipeline1) New file
Create:
diffusers/pipelines/ltx2/pipeline_ltx2_condition.pyAdd:
@dataclass LTX2VideoCondition(image=None, video=None, frame_index=0, strength=1.0)class LTX2ConditionPipeline(...same modules as LTX2Pipeline...)def __call__2) Parse + preprocess (inside
__call__)Add helpers:
_normalize_condition_inputs_trim_conditioning_sequence_preprocess_conditioning_tensors3) Encode conditions into LTX-2 video latents
Add helpers (copied/adapted from LTX2Pipeline):
_retrieve_latents(...)_normalize_latents(...)_pack_latents(...)/_unpack_latents(...)_denormalize_latents(...)Core:
prepare_latents_with_conditionsRules:
frame_index == 0: encode→normalize, blend into base latent frames viatorch.lerp(...), set mask for those tokensframe_index > 0: encode→normalize→pack as extra tokens, prepend to latents, mask =strength, trackextra_tokens4) Timesteps + denoise_strength (LTX-2 schedule)
Reuse LTX-2:
calculate_shift(...),retrieve_timesteps(...),audio_scheduler = copy.deepcopy(self.scheduler)Add:
get_timesteps(..., denoise_strength)If
denoise_strength < 1: computelatent_sigmaand mix providedlatentswith noise like LTX-1 (edit mode).5) Denoising loop (freeze conditioned tokens)
Add:
add_noise_to_image_conditioning_latents(...)Loop:
tokens_to_denoise = (t_norm < (1.0 - conditioning_mask)).unsqueeze(-1)latents = torch.where(tokens_to_denoise, denoised_latents, latents)6) Cleanup + decode
latents = latents[:, extra_tokens:]_unpack_latents→_denormalize_latents→vae.decodeaudio_vae.decode+vocoderas LTX2Pipeline)7) Other files to change
diffusers/pipelines/ltx2/__init__.py: exportLTX2ConditionPipeline, LTX2VideoConditionsayakpaul commentedon Jan 10, 2026
Okay plan looks reasonable to me. Thanks for iterating!
surajyadav-research commentedon Jan 10, 2026
Hi @sayakpaul
I’ve started working on this and will raise a PR ASAP.
devanshi00 commentedon Jan 28, 2026
Hi @surajyadav-research,
Are you still working on this issue? I’m really interested in contributing and would love to help with the implementation if possible. I’d be glad to begin working on a PR following the plan at the earliest opportunity.
dg845 commentedon Jan 30, 2026
Hi all, we have decided that supporting a LTX-2 condition pipeline is a high priority for us and will thus take over work on it.
Thanks to everyone who offered to work on this! We appreciate your interest and look forward to collaborating with you in the future.