You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Set execution options in a way more consistent with DataFusion and DataFusion Python #72
While we cannot accept a SessionConfig from datafusion-python due to lack of ABI stability in rust, we should still try to adhere to the DataFusion Python API as much as possible.
So, we allow setting datafusion configuration options via DFRayContext.set.
Setting options in two places is confusion. I think it will be more clear if we set the options in the init via DFRayContext.set instead.
We can call them
datafusion.ray.execution.batch_size
datafusion.ray.execution.partitions_per_processor (use updated name processor instead of worker here)
datafusion.ray.execution.prefetch_buffer_size
datafusion.ray.execution.processor_pool_min (use updated name processor)
Activity
jsai28 commentedon Mar 17, 2025
Mind if I take a stab at this?