You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
we have rust tests for query planning, but they require some tpc-h data to be available
ideally we should be running the tpc-h queries against a small data set (sf=1) in CI and checking that the results are correct
I expect we can re-use some of the CI setup from core DataFusion
Excellent, I'll do this as a first iteration, thank for the hint. Then we can decide how to do the comparison between datafusion results and ray results, what do you think?
Excellent, I'll do this as a first iteration, thank for the hint. Then we can decide how to do the comparison between datafusion results and ray results, what do you think?
Sounds good.
We could run each query first with DF Python and then with DF Ray and compare the results. This could be a good general approach to testing.
Activity
edmondop commentedon Oct 2, 2024
At the moment, we already run some query planning tests, see https://github.com/edmondop/datafusion-ray/actions/runs/11133252895/job/30938921925
andygrove commentedon Oct 2, 2024
These tests do not actually do anything currently, unless they run on my computer 😞
andygrove commentedon Oct 2, 2024
We have a Python script at https://github.com/apache/datafusion-benchmarks/tree/main/tpch for generating TPC-H data:
https://github.com/apache/datafusion-benchmarks/tree/main/tpch
We could call this in CI and then figure out how to update theses tests to use that data
edmondop commentedon Oct 2, 2024
Excellent, I'll do this as a first iteration, thank for the hint. Then we can decide how to do the comparison between datafusion results and ray results, what do you think?
andygrove commentedon Oct 2, 2024
Sounds good.
We could run each query first with DF Python and then with DF Ray and compare the results. This could be a good general approach to testing.
[-]Datafusion Ray TPCH tests in CI[/-][+]Datafusion Query Planning tests in CI[/+]edmondop commentedon Oct 4, 2024
Closed with #12 #16 #17