Skip to content

Support flash-attn kernel support for non-Hopper GPUs #12308

Closed

Description

@ParagEkbote

Is your feature request related to a problem? Please describe.
In #12236 , flash-attn-3 support was introduced for diffusers using the kernel implementation. I believe that adding the kernels-community/flash-attn can help users who don't have access to Hopper GPUs. If needed, I could help in the integration of the kernel.

cc: @sayakpaul

Describe the solution you'd like.

Integrate kernels-community/flash-attn support into diffusers.

Activity

  1. sayakpaul commented on Sep 10, 2025

    @sayakpaul
    Member
    1. Installing flash-attn on non-hopper architecture GPUs isn't time-consuming.
    2. One can simply enable pipe.transformer.set_attn_backend("flash") to enable flash attention on non-Hopper GPUs once flash-attn is installed.
  2. a-r-r-o-w commented on Sep 10, 2025

    @a-r-r-o-w
    Contributor

    Installing flash-attn on non-hopper architecture GPUs isn't time-consuming.

    but this is incorrect 🤔 First flash-attn install always takes a tonne of time and has high RAM requirements regardless of device. It only seems fast if you already have the binaries cached

  3. sayakpaul commented on Sep 11, 2025

    @sayakpaul
    Member

    Anyway, I will open it up for the community in case someone wants to pick it up.

    The installation time never seemed like an issue to me w.r.t FA3, TBH. But I might be wrong.

  4. ParagEkbote commented on Sep 11, 2025

    @ParagEkbote
    ContributorAuthor

    What steps could be required for the integration of kernels-community/flash-attn. Could you please let me know?

    cc: @a-r-r-o-w

  5. sayakpaul commented on Sep 11, 2025

    @sayakpaul
    Member

    You can follow #12236 to get an overview of what's needed. Feel free to start a PR if you'd like. We'd be happy to help.

  6. a-r-r-o-w commented on Sep 11, 2025

    @a-r-r-o-w
    Contributor

    You might be right @sayakpaul but generally I wouldn't assume users would know how to setup a C++ environment correctly for the compilation to work in the first place 😬 New hub attention backends are probably just going to be more convenient though, and something I'd personally default to eventually

    @ParagEkbote You can follow this PR: #12236

  7. sayakpaul commented on Oct 13, 2025

    @sayakpaul
    Member

    #12387 should close this.

  8. ParagEkbote commented on Dec 4, 2025

    @ParagEkbote
    ContributorAuthor

    Thank you for adding FA2 support through kernels for compatibility with older GPUs in #12439 👍

    cc: @sayakpaul

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions