Skip to content

[Community Pipeline] Speech to Image  #871

Closed

Description

@osanseviero

Intro

Community Pipelines are introduced in diffusers==0.4.0 with the idea of allowing the community to quickly add, integrate, and share their custom pipelines on top of diffusers.

You can find a guide about Community Pipelines here. You can also find all the community examples under examples/community/. If you have questions about the Community Pipelines feature, please head to the parent issue.

Idea: Speech to Image

You can use a transformer automatic-speech-recognition such as OpenAI whisper to transcribe the text, and pass that to Stable Diffusion. Together, this would create a nice speech-to-image pipeline.

Resources

Activity

  1. MikailINTech commented on Oct 17, 2022

    @MikailINTech
    Contributor

    Hi, would be glad to work on it

  2. osanseviero commented on Oct 17, 2022

    @osanseviero
    ContributorAuthor

    Awesome! Feel free to open a PR and if you have any questions just let us know :) (ideally in the parent issue unless it's very specific to this 🙏 )

  3. akashAD98 commented on Oct 18, 2022

    @akashAD98

    I’m very much interested for this,I’ll try this feature

  4. MikailINTech commented on Oct 19, 2022

    @MikailINTech
    Contributor

    Hi @osanseviero, I opened a PR for this issue. For the moment, I have tested my pipeline for whisper and it works pretty well. I'm waiting for your suggestions (maybe making the pipeline more generic so that we can use other speech-recognition models ?)

  5. patrickvonplaten commented on Oct 20, 2022

    @patrickvonplaten
    Contributor

    It works very well ! Well done @MikailINTech :-)

  6. fffiloni commented on Oct 25, 2022

    @fffiloni

    📌 gradio demo is available here: https://huggingface.co/spaces/fffiloni/speech-to-image

  7. osanseviero commented on Oct 25, 2022

    @osanseviero
    ContributorAuthor

    This is amazing @fffiloni! Great work!

  8. osanseviero commented on Oct 25, 2022

    @osanseviero
    ContributorAuthor

    Closing this issue as this is now done 🔥 amazing work everyone!

  9. darwinharianto commented on Oct 26, 2022

    @darwinharianto

    I tried the sample from https://github.com/huggingface/diffusers/tree/main/examples/community#speech-to-image
    it gives me,
    aa

    Is there anything wrong with the code?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions