Skip to content

fix(wan): trim vace_video to num_frames - #1721

Merged
Artiprocher merged 1 commit into
modelscope:mainfrom
fusheng-ji:fix/wan-vace-frames
Oct 13, 2026
Merged

Artiprocher merged 1 commit into
modelscope:mainfrom
fusheng-ji:fix/wan-vace-frames

Conversation

@fusheng-ji

@fusheng-ji fusheng-ji commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1149

When vace_video has more frames than num_frames (for example a whole mp4 passed via VideoData), WanVideoUnit_VACE encodes all of them. The VACE context then has more tokens than the video latents, and the zero padding in VaceWanModel.forward gets a negative size:

RuntimeError: Trying to create tensor with negative dimension -96000: [1, -96000, 5120]

With the issue's 3840×1600 resolution, one latent frame is 24000 tokens, so -96000 means 4 extra latent frames: a 97-frame video with the default num_frames=81.

This PR uses only the first num_frames frames of vace_video and vace_video_mask, like other Wan units already do for their conditioning videos (e.g. camera_control and animate2_reference_video). Frames are taken by index because VideoData doesn't support slicing. Inputs with num_frames frames or fewer behave exactly as before; shorter ones are still zero-padded.

Checking

I ran the real WanVideoUnit_VACE with a stub VAE, followed by a small random-init VaceWanModel, at 64×64 with num_frames=81:

vace_video before after
VideoData of a 97-frame mp4 RuntimeError: Trying to create tensor with negative dimension -64 OK, context with 21 latent frames
list of 97 frames same error OK, 21 latent frames
list of 81 frames OK, 21 latent frames same
list of 49 frames OK, 13 latent frames same

End-to-end with WanVideoPipeline. Setup: Wan-AI/Wan2.1-VACE-1.3B, the depth video from examples/wanvideo/model_inference/Wan2.1-VACE-1.3B.py (81 frames, loaded with VideoData), 480×832, 30 steps, seed 1, tiled=True, torch 2.11.0+cu130, 1× B200:

vace_video num_frames main this PR
VideoData (81 frames) 49 RuntimeError: Trying to create tensor with negative dimension -12480: [1, -12480, 1536] OK, 49-frame video
list of the first 49 frames 49 OK OK, bit-identical to the row above and to main
VideoData (81 frames) 81 OK OK, bit-identical to main
  • -12480 = 8 extra latent frames (21 − 13) × 1560 tokens per latent frame, the same failure as in the issue.
  • With this PR, passing the whole video gives exactly the same result as passing it already trimmed to num_frames.
Wan2.1-VACE-1.3B with an 81-frame control video and num_frames=49: main fails with a negative-dimension error, this PR trims the video and matches a pre-trimmed input

Figure. Wan-AI/Wan2.1-VACE-1.3B, 480×832, seed 1. The 81-frame depth video from the VACE example is passed as VideoData with num_frames=49. On main this fails with a negative-dimension error; with this PR the control video is trimmed to num_frames, and the output is bit-identical to passing a pre-trimmed 49-frame list.

When vace_video (or vace_video_mask) has more frames than num_frames,
the VACE context has more tokens than the video latents, and padding it
in VaceWanModel fails with "Trying to create tensor with negative
dimension". Only use the first num_frames frames, as other Wan units do
for their conditioning videos. Shorter inputs keep the current behavior.

Fixes modelscope#1149
@Artiprocher

Copy link
Copy Markdown
Collaborator

@fusheng-ji Thank you for your suggestion. We will go ahead and merge this PR.

@Artiprocher
Artiprocher merged commit de122d0 into modelscope:main Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trying to create tensor with negative dimension

2 participants