Skip to content

StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text #7731

Open

Description

@clarencechen

Model/Pipeline/Scheduler description

Text-to-video diffusion models enable the generation of high-quality videos given text prompts, making it easy to create diverse and individual content. However, existing approaches mostly focus on short video generation (typically 16 or 24 frames), requiring hard cuts when naively extended to the case of long video synthesis. StreamingT2V, enables autoregressive generation of long videos of 80, 240, 600, 1200 or more frames with smooth transitions. The key components are:

  • A ControlNet-like module which conditions the current generation on frames extracted from the previous chunk, using a cross-attention mechanism to integrate its features into the UNet's skip residual features.
  • An IP-Adapter-like module which extracts high-level scene and object features from a fixed anchor frame in the first video chunk and is mixed into the prompt embedding features before executing spatial cross-attention.
  • A SDEdit-based video refinement stage with randomized chunk sampling of overlapped frames per denoising timestep.

Open source status

  • The model implementation is available.
  • The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Activity

  1. DN6 commented on Apr 22, 2024

    @DN6
    Collaborator

    Sounds like a nice addition. I think we can open it up to the community to work on. Or would you like to work on it @clarencechen?

  2. dg845 commented on Sep 15, 2024

    @dg845
    Contributor

    Hi, I'd like to try working on this if the maintainers still think this is a good addition to the library :).

  3. yiyixuxu commented on Sep 17, 2024

    @yiyixuxu
    Collaborator

    hi @dg845
    it's been a while! Welcome back!
    this one I think it'd go into the community folder, to begin with

  4. a-r-r-o-w commented on Nov 20, 2024

    @a-r-r-o-w
    Contributor

    Hi @dg845! We did indeed plan to support StreamingT2V but other things took priority. Would love to have this if you find time to PR - thanks! I'm familiar with the codebase so would love to be of help in any way

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions