You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text #7731
Text-to-video diffusion models enable the generation of high-quality videos given text prompts, making it easy to create diverse and individual content. However, existing approaches mostly focus on short video generation (typically 16 or 24 frames), requiring hard cuts when naively extended to the case of long video synthesis. StreamingT2V, enables autoregressive generation of long videos of 80, 240, 600, 1200 or more frames with smooth transitions. The key components are:
A ControlNet-like module which conditions the current generation on frames extracted from the previous chunk, using a cross-attention mechanism to integrate its features into the UNet's skip residual features.
An IP-Adapter-like module which extracts high-level scene and object features from a fixed anchor frame in the first video chunk and is mixed into the prompt embedding features before executing spatial cross-attention.
A SDEdit-based video refinement stage with randomized chunk sampling of overlapped frames per denoising timestep.
Open source status
The model implementation is available.
The model weights are available (Only relevant if addition is not a scheduler).
Hi @dg845! We did indeed plan to support StreamingT2V but other things took priority. Would love to have this if you find time to PR - thanks! I'm familiar with the codebase so would love to be of help in any way
Activity
DN6 commentedon Apr 22, 2024
Sounds like a nice addition. I think we can open it up to the community to work on. Or would you like to work on it @clarencechen?
dg845 commentedon Sep 15, 2024
Hi, I'd like to try working on this if the maintainers still think this is a good addition to the library :).
yiyixuxu commentedon Sep 17, 2024
hi @dg845
it's been a while! Welcome back!
this one I think it'd go into the community folder, to begin with
a-r-r-o-w commentedon Nov 20, 2024
Hi @dg845! We did indeed plan to support StreamingT2V but other things took priority. Would love to have this if you find time to PR - thanks! I'm familiar with the codebase so would love to be of help in any way