You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support flash-attn kernel support for non-Hopper GPUs #12308
Is your feature request related to a problem? Please describe.
In #12236 , flash-attn-3 support was introduced for diffusers using the kernel implementation. I believe that adding the kernels-community/flash-attn can help users who don't have access to Hopper GPUs. If needed, I could help in the integration of the kernel.
Installing flash-attn on non-hopper architecture GPUs isn't time-consuming.
but this is incorrect 🤔 First flash-attn install always takes a tonne of time and has high RAM requirements regardless of device. It only seems fast if you already have the binaries cached
You might be right @sayakpaul but generally I wouldn't assume users would know how to setup a C++ environment correctly for the compilation to work in the first place 😬 New hub attention backends are probably just going to be more convenient though, and something I'd personally default to eventually
Activity
sayakpaul commentedon Sep 10, 2025
flash-attnon non-hopper architecture GPUs isn't time-consuming.pipe.transformer.set_attn_backend("flash")to enable flash attention on non-Hopper GPUs onceflash-attnis installed.a-r-r-o-w commentedon Sep 10, 2025
but this is incorrect 🤔 First flash-attn install always takes a tonne of time and has high RAM requirements regardless of device. It only seems fast if you already have the binaries cached
sayakpaul commentedon Sep 11, 2025
Anyway, I will open it up for the community in case someone wants to pick it up.
The installation time never seemed like an issue to me w.r.t FA3, TBH. But I might be wrong.
ParagEkbote commentedon Sep 11, 2025
What steps could be required for the integration of kernels-community/flash-attn. Could you please let me know?
cc: @a-r-r-o-w
sayakpaul commentedon Sep 11, 2025
You can follow #12236 to get an overview of what's needed. Feel free to start a PR if you'd like. We'd be happy to help.
a-r-r-o-w commentedon Sep 11, 2025
You might be right @sayakpaul but generally I wouldn't assume users would know how to setup a C++ environment correctly for the compilation to work in the first place 😬 New hub attention backends are probably just going to be more convenient though, and something I'd personally default to eventually
@ParagEkbote You can follow this PR: #12236
sayakpaul commentedon Oct 13, 2025
#12387 should close this.
ParagEkbote commentedon Dec 4, 2025
Thank you for adding FA2 support through
kernelsfor compatibility with older GPUs in #12439 👍cc: @sayakpaul