Repository navigation
fix(wan): trim vace_video to num_frames - #1721
Merged
Merged
Conversation
When vace_video (or vace_video_mask) has more frames than num_frames, the VACE context has more tokens than the video latents, and padding it in VaceWanModel fails with "Trying to create tensor with negative dimension". Only use the first num_frames frames, as other Wan units do for their conditioning videos. Shorter inputs keep the current behavior. Fixes modelscope#1149
Collaborator
|
@fusheng-ji Thank you for your suggestion. We will go ahead and merge this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1149
When
vace_videohas more frames thannum_frames(for example a whole mp4 passed viaVideoData),WanVideoUnit_VACEencodes all of them. The VACE context then has more tokens than the video latents, and the zero padding inVaceWanModel.forwardgets a negative size:With the issue's 3840×1600 resolution, one latent frame is 24000 tokens, so -96000 means 4 extra latent frames: a 97-frame video with the default
num_frames=81.This PR uses only the first
num_framesframes ofvace_videoandvace_video_mask, like other Wan units already do for their conditioning videos (e.g.camera_controlandanimate2_reference_video). Frames are taken by index becauseVideoDatadoesn't support slicing. Inputs withnum_framesframes or fewer behave exactly as before; shorter ones are still zero-padded.Checking
I ran the real
WanVideoUnit_VACEwith a stub VAE, followed by a small random-initVaceWanModel, at 64×64 withnum_frames=81:vace_videoVideoDataof a 97-frame mp4RuntimeError: Trying to create tensor with negative dimension -64End-to-end with
WanVideoPipeline. Setup:Wan-AI/Wan2.1-VACE-1.3B, the depth video fromexamples/wanvideo/model_inference/Wan2.1-VACE-1.3B.py(81 frames, loaded withVideoData), 480×832, 30 steps, seed 1,tiled=True, torch 2.11.0+cu130, 1× B200:vace_videonum_framesVideoData(81 frames)RuntimeError: Trying to create tensor with negative dimension -12480: [1, -12480, 1536]VideoData(81 frames)num_frames.Figure.
Wan-AI/Wan2.1-VACE-1.3B, 480×832, seed 1. The 81-frame depth video from the VACE example is passed asVideoDatawithnum_frames=49. Onmainthis fails with a negative-dimension error; with this PR the control video is trimmed tonum_frames, and the output is bit-identical to passing a pre-trimmed 49-frame list.