You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Proposal to donate Ray SQL to the DataFusion Project (not into the Python subproject) #872
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In light of the complexities associated with maintaining and debugging Ballista, the community would like to propose exploring the adoption of Ray SQL within the DataFusion Python project.
Ray SQL, with only 1.7k lines of Rust, is significantly simpler compared to Ballista’s 27k lines. Despite its smaller codebase, Ray SQL has demonstrated the ability to run all TPC-H queries, showcasing its robustness and simplicity. This reduction in complexity could ease maintenance, lower the learning curve for contributors, and provide a functional distributed SQL solution that is much simpler than Ballista because Ray provides so much – there is no need for us to build scheduler and executor processes – we can simply execute Python tasks in the Ray cluster.
If there is enough interest and support, we could start working on bringing the Ray SQL code into the DataFusion Python project.
Describe the solution you'd like
Following options are suggested by @andygrove , all feedbacks are welcome!
Bringing the Ray SQL prototype into the DataFusion Python project
Describe alternatives you've considered
Building a new version inspired by the Ray SQL code
Additional context
We might need to go through the Apache IP clearance process for importing the external Ray SQL codebase in datafusion-contrib which is not part of Apache.
Reacted by Andy Grove, Jordan Fox, Edmondo Porcu, Ion Koutsouris, Deepyaman Datta, Vakaris Baškirov, Michael J Ward, dsgibbons, Viktor Kessler, Matthew Aylward and 1 more
Is the complexity of Ballista coming from cluster management or query planning?
If Ray SQL can achieve similar or better performance than Ballista with only 1.7k loc by delegating cluster management to Ray and only keeping the distributed query planning responsibility, that's pretty amazing !
I am curious about the right boundary here. If we go through with this, should the code become a part of datafusion-python, or should the former stay as a single node thing focusing on excellent bindings to core, and we should have a "datafusion-ray" that uses datafusion-python as a library?
Thanks @ozankabak I'm curious on the boundary, too.
One of the goals in Ray SQL is "Drive requirements for DataFusion's Python bindings". So my guess is that if we wanna use datafusion-python as a library for separate datafusion-ray, we might need to ensure APIs of datafusion-ray is a subset of datafusion-python APIs. Otherwise it'll not be able to easily expose complete bindings to python side via PyO3.
100% yes. If Ray SQL can handle all TPC-H queries with just 1.7k lines of code, it’s sounds like a no-brainer. This actually makes a production-ready distributed DataFusion sound achievable and it should attract more contributors.
Can't think of a single reason not to go this route.
One question I have is who would maintain this new code? Ballista I think suffers from a slow decline due to lack of active maintenance and community. We should try to avoid the same thing happening to Ray
Docs/Readme.md probably needs a quick update for anyone joining the catchup. Distributed shuffle using rays object store was implemented in #33.
Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Andrew Lamb ***@***.***>
Sent: Monday, September 16, 2024 6:32:22 PM
To: apache/datafusion-python ***@***.***>
Cc: Jordan Fox ***@***.***>; Comment ***@***.***>
Subject: Re: [apache/datafusion-python] Proposal to Introduce Ray SQL into DataFusion Python (Issue #872)
Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
One question I have is who would maintain this new code? Ballista I think suffers from a slow decline due to lack of active maintenance and community. We should try to avoid the same thing happening to Ray
—
Reply to this email directly, view it on GitHub<#872 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHXVCFKTRUINKJ527GAANVDZW52BNAVCNFSM6AAAAABOIBWHDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJUGI3DMMJUGI>.
You are receiving this because you commented.Message ID: ***@***.***>
It does sound like we have some concensus. The next steps are to start the IP clearance process. I will make a start on that this weekend. It basically involves filling out a form and starting a vote. Once the vote passes, I can create the new datafusion-ray repo and we can push the code there and archive the original repo.
@austin362667 Before we can start the IP clearance process, we need to update the existing source files to use the ASF header. Is that something you would be able to take on?
changed the title [-]Proposal to Introduce Ray SQL into DataFusion Python[/-][+]Proposal to Introduce Ray SQL into DataFusion ~Python~ Project[/+]on Sep 19, 2024
changed the title [-]Proposal to Introduce Ray SQL into DataFusion ~Python~ Project[/-][+]Proposal to Introduce Ray SQL into DataFusion Project (not into the Python subproject)[/+]on Sep 19, 2024
changed the title [-]Proposal to Introduce Ray SQL into DataFusion Project (not into the Python subproject)[/-][+]Proposal to donate Ray SQL to the DataFusion Project (not into the Python subproject)[/+]on Sep 19, 2024
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
@austin362667 Before we can start the IP clearance process, we need to update the existing source files to use the ASF header. Is that something you would be able to take on?
I created an issue in the new repo to track the donation process and for the work that I think needs to happen after that so that we can get to a 0.1.0 release.
As a formality, as part of the ASF IP clearance process, I must remind active committers that they are responsible for ensuring that a Corporate CLA is recorded if such is required to authorize their contributions under their individual CLA.
Activity
andygrove commentedon Sep 15, 2024
Ray SQL project: https://github.com/datafusion-contrib/ray-sql
andygrove commentedon Sep 15, 2024
@franklsf95 fyi
jordandakota commentedon Sep 15, 2024
Things I'm thinking of that should be showcased or discussed:
edmondop commentedon Sep 15, 2024
Is the complexity of Ballista coming from cluster management or query planning?
If Ray SQL can achieve similar or better performance than Ballista with only 1.7k loc by delegating cluster management to Ray and only keeping the distributed query planning responsibility, that's pretty amazing !
andygrove commentedon Sep 15, 2024
Ballista and Ray SQL use the same query planning code. The complexity comes from cluster management/communication.
andygrove commentedon Sep 15, 2024
Since upgrading to a more recent DataFusion version, Ballista is no longer capable of running the TPC-H suite. It works fine in the 0.12.0 release.
ozankabak commentedon Sep 16, 2024
I am curious about the right boundary here. If we go through with this, should the code become a part of datafusion-python, or should the former stay as a single node thing focusing on excellent bindings to core, and we should have a "datafusion-ray" that uses datafusion-python as a library?
austin362667 commentedon Sep 16, 2024
Thanks @ozankabak I'm curious on the boundary, too.
One of the goals in Ray SQL is "Drive requirements for DataFusion's Python bindings". So my guess is that if we wanna use
datafusion-pythonas a library for separatedatafusion-ray, we might need to ensure APIs ofdatafusion-rayis a subset ofdatafusion-pythonAPIs. Otherwise it'll not be able to easily expose complete bindings to python side via PyO3.ozankabak commentedon Sep 16, 2024
Indeed. Do you have an idea of how the subset would look like?
andygrove commentedon Sep 16, 2024
To expand on this, there are several options for adopting the Ray SQL code as part of the DataFusion project.
datafusion-ray-sql(possibly renamed todatafusion-rayor something else)vakarisbk commentedon Sep 16, 2024
100% yes. If Ray SQL can handle all TPC-H queries with just 1.7k lines of code, it’s sounds like a no-brainer. This actually makes a production-ready distributed DataFusion sound achievable and it should attract more contributors.
Can't think of a single reason not to go this route.
alamb commentedon Sep 17, 2024
One question I have is who would maintain this new code? Ballista I think suffers from a slow decline due to lack of active maintenance and community. We should try to avoid the same thing happening to Ray
jordandakota commentedon Sep 17, 2024
andygrove commentedon Sep 17, 2024
Here is the PR in ray-sql that added support for distributed shuffle using the Ray object store:
datafusion-contrib/ray-sql#33
7 remaining items
andygrove commentedon Sep 19, 2024
It does sound like we have some concensus. The next steps are to start the IP clearance process. I will make a start on that this weekend. It basically involves filling out a form and starting a vote. Once the vote passes, I can create the new
datafusion-rayrepo and we can push the code there and archive the original repo.https://incubator.apache.org/ip-clearance/
andygrove commentedon Sep 19, 2024
@franklsf95 Would you be able to file an ICLA (unless you already have one)? The instructions are at https://www.apache.org/licenses/contributor-agreements.html
@austin362667 Before we can start the IP clearance process, we need to update the existing source files to use the ASF header. Is that something you would be able to take on?
[-]Proposal to Introduce Ray SQL into DataFusion Python[/-][+]Proposal to Introduce Ray SQL into DataFusion ~Python~ Project[/+][-]Proposal to Introduce Ray SQL into DataFusion ~Python~ Project[/-][+]Proposal to Introduce Ray SQL into DataFusion Project (not into the Python subproject)[/+][-]Proposal to Introduce Ray SQL into DataFusion Project (not into the Python subproject)[/-][+]Proposal to donate Ray SQL to the DataFusion Project (not into the Python subproject)[/+]austin362667 commentedon Sep 19, 2024
@andygrove Thanks for updating the title and yes, I can help.
I'll open PR to https://github.com/datafusion-contrib/ray-sql that adds:
austin362667 commentedon Sep 19, 2024
The PR Add ASF license header
frank-lsf commentedon Sep 19, 2024
Just submitted one to secretary@apache.org.
andygrove commentedon Sep 19, 2024
@austin362667 I created the new repository: https://github.com/apache/datafusion-ray
Could you open a draft PR against this repository to add the code including ASF headers?
Once this is done, the next steps will be:
Thanks.
andygrove commentedon Sep 20, 2024
I ran some benchmarks from @timsaucer's repo where he has upgraded to DataFusion 41.
Performance is looking good.
austin362667 commentedon Sep 20, 2024
Thanks, @andygrove The result looks super promising!
And here is the draft PR apache/datafusion-ray#1 for the Datafusion Ray donation.
andygrove commentedon Sep 21, 2024
I created an issue in the new repo to track the donation process and for the work that I think needs to happen after that so that we can get to a 0.1.0 release.
apache/datafusion-ray#2
andygrove commentedon Sep 26, 2024
As a formality, as part of the ASF IP clearance process, I must remind active committers that they are responsible for ensuring that a Corporate CLA is recorded if such is required to authorize their contributions under their individual CLA.
cc @franklsf95
andygrove commentedon Sep 30, 2024
The donation has now been accepted, so I will close this issue. Thank you to everyone who participated in this discussion.