You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've been working on a revised version of datafusion ray for the past month or so. I submitted a PR today in fact.
It replaces the disk based shuffle with ArrowFlight streaming shuffle, and is able to scale beyond one node.
It should be in a place where it can build and work for you. You can read the PR here
@robtandy Thanks for the response. Since datafusion-ray is not published to pypi.org, finding it difficult to install in a python project and experiment with it. Without a datafusion-ray package in pypi, building it from source and packaging it as a dependency on another project etc., becomes too complex.
Activity
robtandy commentedon Feb 4, 2025
👋 Hi @sairamkrish !
I've been working on a revised version of datafusion ray for the past month or so. I submitted a PR today in fact.
It replaces the disk based shuffle with ArrowFlight streaming shuffle, and is able to scale beyond one node.
It should be in a place where it can build and work for you. You can read the PR here
#60
And access the branch from here: https://github.com/robtandy/datafusion-ray/tree/main/src
If you have any trouble installing or using, you can open an issue here and I'll try to help.
sairamkrish commentedon Feb 5, 2025
@robtandy Thanks for the response. Since datafusion-ray is not published to pypi.org, finding it difficult to install in a python project and experiment with it. Without a datafusion-ray package in pypi, building it from source and packaging it as a dependency on another project etc., becomes too complex.
Do you have any suggestion or work around ?
andygrove commentedon Feb 15, 2025
Hopefully, we can release version 0.1.0 to PyPi pretty soon now.