Skip to content

datafusion-ray is not published to pypi.org #59

Open

Description

@sairamkrish

Hi

We are trying to use datafusion-ray. To get started we tried to pip install datafusion-ray. However datafusion-ray is not published.

I tried installing directly from github like this :

poetry add git+https://github.com/apache/datafusion-ray.git

Updating dependencies
Resolving dependencies... (0.3s)

Package operations: 1 install, 0 updates, 0 removals

  - Installing datafusion-ray (0.1.0 2783613): Failed

Failure reason is not displayed even in --verbose mode.
If someone wants to do datafusion + ray, should we continue using raysql till then ?

I would like to contribute here if you are open for it.

Activity

  1. robtandy commented on Feb 4, 2025

    @robtandy
    Contributor

    👋 Hi @sairamkrish !

    I've been working on a revised version of datafusion ray for the past month or so. I submitted a PR today in fact.
    It replaces the disk based shuffle with ArrowFlight streaming shuffle, and is able to scale beyond one node.

    It should be in a place where it can build and work for you. You can read the PR here

    #60

    And access the branch from here: https://github.com/robtandy/datafusion-ray/tree/main/src

    If you have any trouble installing or using, you can open an issue here and I'll try to help.

  2. sairamkrish commented on Feb 5, 2025

    @sairamkrish
    Author

    @robtandy Thanks for the response. Since datafusion-ray is not published to pypi.org, finding it difficult to install in a python project and experiment with it. Without a datafusion-ray package in pypi, building it from source and packaging it as a dependency on another project etc., becomes too complex.

    Do you have any suggestion or work around ?

  3. andygrove commented on Feb 15, 2025

    @andygrove
    Member

    Hopefully, we can release version 0.1.0 to PyPi pretty soon now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions