Skip to content

Datafusion Query Planning tests in CI #9

Closed

Description

@edmondop

Quoting @andygrove on Discord:

we have rust tests for query planning, but they require some tpc-h data to be available
ideally we should be running the tpc-h queries against a small data set (sf=1) in CI and checking that the results are correct
I expect we can re-use some of the CI setup from core DataFusion

Activity

  1. edmondop commented on Oct 2, 2024

    @edmondop
    ContributorAuthor
  2. andygrove commented on Oct 2, 2024

    @andygrove
    Member

    At the moment, we already run some query planning tests, see https://github.com/edmondop/datafusion-ray/actions/runs/11133252895/job/30938921925

    These tests do not actually do anything currently, unless they run on my computer 😞

    async fn do_test(n: u8) -> Result<()> {
        let data_path = "/mnt/bigdata/tpch/sf10-parquet";
        if !Path::new(&data_path).exists() {
            return Ok(());
        }
  3. andygrove commented on Oct 2, 2024

    @andygrove
    Member

    We have a Python script at https://github.com/apache/datafusion-benchmarks/tree/main/tpch for generating TPC-H data:

    https://github.com/apache/datafusion-benchmarks/tree/main/tpch

    We could call this in CI and then figure out how to update theses tests to use that data

  4. edmondop commented on Oct 2, 2024

    @edmondop
    ContributorAuthor

    Excellent, I'll do this as a first iteration, thank for the hint. Then we can decide how to do the comparison between datafusion results and ray results, what do you think?

  5. andygrove commented on Oct 2, 2024

    @andygrove
    Member

    Excellent, I'll do this as a first iteration, thank for the hint. Then we can decide how to do the comparison between datafusion results and ray results, what do you think?

    Sounds good.

    We could run each query first with DF Python and then with DF Ray and compare the results. This could be a good general approach to testing.

  6. changed the title [-]Datafusion Ray TPCH tests in CI[/-] [+]Datafusion Query Planning tests in CI[/+] on Oct 2, 2024
  7. edmondop commented on Oct 4, 2024

    @edmondop
    ContributorAuthor

    Closed with #12 #16 #17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions