Skip to content

Proposal to donate Ray SQL to the DataFusion Project (not into the Python subproject) #872

Closed

Description

@austin362667

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In light of the complexities associated with maintaining and debugging Ballista, the community would like to propose exploring the adoption of Ray SQL within the DataFusion Python project.

Ray SQL, with only 1.7k lines of Rust, is significantly simpler compared to Ballista’s 27k lines. Despite its smaller codebase, Ray SQL has demonstrated the ability to run all TPC-H queries, showcasing its robustness and simplicity. This reduction in complexity could ease maintenance, lower the learning curve for contributors, and provide a functional distributed SQL solution that is much simpler than Ballista because Ray provides so much – there is no need for us to build scheduler and executor processes – we can simply execute Python tasks in the Ray cluster.

If there is enough interest and support, we could start working on bringing the Ray SQL code into the DataFusion Python project.

Describe the solution you'd like

Following options are suggested by @andygrove , all feedbacks are welcome!

Bringing the Ray SQL prototype into the DataFusion Python project

Describe alternatives you've considered

Building a new version inspired by the Ray SQL code

Additional context

We might need to go through the Apache IP clearance process for importing the external Ray SQL codebase in datafusion-contrib which is not part of Apache.

Activity

  1. andygrove commented on Sep 15, 2024

    @andygrove
    Member
  2. andygrove commented on Sep 15, 2024

    @andygrove
    Member

    @franklsf95 fyi

  3. jordandakota commented on Sep 15, 2024

    @jordandakota

    Things I'm thinking of that should be showcased or discussed:

    • Showcase complexity/simplicity
      • Setup and testing of ballista on tpc-h
      • Setup and testing of raysql on tpc-h
    • Benchmark differences ballista/raysql
    • What do we gain/lose by moving towards adopting raysql
    • What does raysql provide that Ballista currently does, what would need to be added/maintained by the integration
      • Need to assess covers all Ballista functionalities and if not document gaps
      • Identify effort requires to close any gaps
    • Discuss learning curve differences
      • At present it's quite complex setting up and running a Ballista cluster while you can start a Ray cluster in a few lines of code reliably
      • Potential to attract contributors due to ease of use
  4. edmondop commented on Sep 15, 2024

    @edmondop

    Is the complexity of Ballista coming from cluster management or query planning?
    If Ray SQL can achieve similar or better performance than Ballista with only 1.7k loc by delegating cluster management to Ray and only keeping the distributed query planning responsibility, that's pretty amazing !

  5. andygrove commented on Sep 15, 2024

    @andygrove
    Member

    Is the complexity of Ballista coming from cluster management or query planning?

    Ballista and Ray SQL use the same query planning code. The complexity comes from cluster management/communication.

  6. andygrove commented on Sep 15, 2024

    @andygrove
    Member
    • Setup and testing of ballista on tpc-h

    Since upgrading to a more recent DataFusion version, Ballista is no longer capable of running the TPC-H suite. It works fine in the 0.12.0 release.

  7. ozankabak commented on Sep 16, 2024

    @ozankabak

    I am curious about the right boundary here. If we go through with this, should the code become a part of datafusion-python, or should the former stay as a single node thing focusing on excellent bindings to core, and we should have a "datafusion-ray" that uses datafusion-python as a library?

  8. austin362667 commented on Sep 16, 2024

    @austin362667
    Author

    Thanks @ozankabak I'm curious on the boundary, too.
    One of the goals in Ray SQL is "Drive requirements for DataFusion's Python bindings". So my guess is that if we wanna use datafusion-python as a library for separate datafusion-ray, we might need to ensure APIs of datafusion-ray is a subset of datafusion-python APIs. Otherwise it'll not be able to easily expose complete bindings to python side via PyO3.

  9. ozankabak commented on Sep 16, 2024

    @ozankabak

    Indeed. Do you have an idea of how the subset would look like?

  10. andygrove commented on Sep 16, 2024

    @andygrove
    Member

    To expand on this, there are several options for adopting the Ray SQL code as part of the DataFusion project.

    1. Create new repo datafusion-ray-sql (possibly renamed to datafusion-ray or something else)
    2. Replace the current Ballista code with Ray SQL (Ballista is already known as the distributed version of DataFusion)
    3. Add as an optional module of the DataFusion Python bindings
  11. vakarisbk commented on Sep 16, 2024

    @vakarisbk

    100% yes. If Ray SQL can handle all TPC-H queries with just 1.7k lines of code, it’s sounds like a no-brainer. This actually makes a production-ready distributed DataFusion sound achievable and it should attract more contributors.
    Can't think of a single reason not to go this route.

  12. alamb commented on Sep 17, 2024

    @alamb
    Contributor

    One question I have is who would maintain this new code? Ballista I think suffers from a slow decline due to lack of active maintenance and community. We should try to avoid the same thing happening to Ray

  13. jordandakota commented on Sep 17, 2024

    @jordandakota
  14. andygrove commented on Sep 17, 2024

    @andygrove
    Member

    Here is the PR in ray-sql that added support for distributed shuffle using the Ray object store:

    datafusion-contrib/ray-sql#33

  15. 7 remaining items

  16. andygrove commented on Sep 19, 2024

    @andygrove
    Member

    It does sound like we have some concensus. The next steps are to start the IP clearance process. I will make a start on that this weekend. It basically involves filling out a form and starting a vote. Once the vote passes, I can create the new datafusion-ray repo and we can push the code there and archive the original repo.

    https://incubator.apache.org/ip-clearance/

  17. andygrove commented on Sep 19, 2024

    @andygrove
    Member

    @franklsf95 Would you be able to file an ICLA (unless you already have one)? The instructions are at https://www.apache.org/licenses/contributor-agreements.html

    @austin362667 Before we can start the IP clearance process, we need to update the existing source files to use the ASF header. Is that something you would be able to take on?

  18. changed the title [-]Proposal to Introduce Ray SQL into DataFusion Python[/-] [+]Proposal to Introduce Ray SQL into DataFusion ~Python~ Project[/+] on Sep 19, 2024
  19. changed the title [-]Proposal to Introduce Ray SQL into DataFusion ~Python~ Project[/-] [+]Proposal to Introduce Ray SQL into DataFusion Project (not into the Python subproject)[/+] on Sep 19, 2024
  20. changed the title [-]Proposal to Introduce Ray SQL into DataFusion Project (not into the Python subproject)[/-] [+]Proposal to donate Ray SQL to the DataFusion Project (not into the Python subproject)[/+] on Sep 19, 2024
  21. austin362667 commented on Sep 19, 2024

    @austin362667
    Author

    @andygrove Thanks for updating the title and yes, I can help.
    I'll open PR to https://github.com/datafusion-contrib/ray-sql that adds:

    # Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    # http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied.  See the License for the
    # specific language governing permissions and limitations
    # under the License.
    
  22. austin362667 commented on Sep 19, 2024

    @austin362667
    Author
  23. frank-lsf commented on Sep 19, 2024

    @frank-lsf

    @franklsf95 Would you be able to file an ICLA (unless you already have one)? The instructions are at https://www.apache.org/licenses/contributor-agreements.html

    @austin362667 Before we can start the IP clearance process, we need to update the existing source files to use the ASF header. Is that something you would be able to take on?

    Just submitted one to secretary@apache.org.

  24. andygrove commented on Sep 19, 2024

    @andygrove
    Member

    @austin362667 I created the new repository: https://github.com/apache/datafusion-ray

    Could you open a draft PR against this repository to add the code including ASF headers?

    Once this is done, the next steps will be:

    1. Hold a vote on the DataFusion mailing list for accepting the donation based on this PR
    2. Once that vote passes, start the IP clearance vote
    3. Once that vote passes, merge the PR

    Thanks.

  25. andygrove commented on Sep 20, 2024

    @andygrove
    Member

    I ran some benchmarks from @timsaucer's repo where he has upgraded to DataFusion 41.

    Performance is looking good.

    Screenshot from 2024-09-19 19-55-55

  26. austin362667 commented on Sep 20, 2024

    @austin362667
    Author

    Thanks, @andygrove The result looks super promising!

    And here is the draft PR apache/datafusion-ray#1 for the Datafusion Ray donation.

  27. andygrove commented on Sep 21, 2024

    @andygrove
    Member

    I created an issue in the new repo to track the donation process and for the work that I think needs to happen after that so that we can get to a 0.1.0 release.

    apache/datafusion-ray#2

  28. andygrove commented on Sep 26, 2024

    @andygrove
    Member

    As a formality, as part of the ASF IP clearance process, I must remind active committers that they are responsible for ensuring that a Corporate CLA is recorded if such is required to authorize their contributions under their individual CLA.

    cc @franklsf95

  29. andygrove commented on Sep 30, 2024

    @andygrove
    Member

    The donation has now been accepted, so I will close this issue. Thank you to everyone who participated in this discussion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions