ORC Predicate Pushdown #48986

Activity

1. cbb330

   added

   Type: enhancement

   on Jan 26, 2026
2. github-actions

   added

   Component: Python

   on Jan 26, 2026
3. github-actions

   assigned

   cbb330

   on Jan 26, 2026
4. cbb330 commented on Feb 22, 2026

   

   cbb330

   on Feb 22, 2026

   Last edited by cbb330

   Author

   More actions

   Implementation Plan

   Following @raulcd's and @wgtmac's suggestions, here is the detailed plan for ORC predicate pushdown. I will post a consolidated PoC PR (draft) for design review, then work through sub-issues per this plan.

   Architecture

   The implementation follows Parquet's proven pattern, adapted for ORC semantics:

   Query: dataset.to_table(filter=ds.field("id") > 1000)
   
   1. Adapter layer -> boundary between liborc and Arrow via OrcColumnStatistics struct with attributes min/max as Arrow Scalars, plus null and validity info
   2. Expression layer -> DeriveFieldGuarantee() builds (field >= min AND field <= max) expressions
   3. Filter layer -> FilterStripes() calls SimplifyWithGuarantee() per stripe, drops those with literal(false)
   4. Scan layer -> ScanBatchesAsync() passes surviving stripe indices to the reader effectively skipping I/O
   5. Post-filter -> Scanner's Acero "filter" node applies the predicate row-by-row on kept stripes
   

   Key design decisions

   Encapsulate liborc in the adapter layer

   ORC statistics come from liborc::ColumnStatistics which is an external library with its own type hierarchy. We introduce an OrcColumnStatistics struct that converts liborc types into Arrow Scalar values once at the adapter boundary. Then, all downstream filtering code (expression building, stripe pruning) works only with this Arrow-native struct and never depends on liborc.

   Lazy Statistics Extraction

   The lazy approach only extracts statistics for columns that actually appear in the predicate, and caches the results so subsequent queries against the same fragment don't redo the work. This is the same pattern Parquet uses for row group filtering.

   Conservative filtering

   When statistics are missing, corrupted, or for unsupported types, include the stripe so that the logic never produces false negatives.

   Sub-issues

   Issues are in order to the layering noted above:

   1. [C++][ORC] Add stripe statistics API to ORCFileReader #49360 — [C++][ORC] Add stripe statistics API to ORCFileReader
   2. [C++][Dataset] Add OrcFileFragment with stripe filtering and predicate pushdown #49361 — [C++][Dataset] Add OrcFileFragment with stripe filtering and predicate pushdown
   3. [C++][Dataset] ORC predicate pushdown: full operator and type coverage #49362 — [C++][Dataset] Full operator and type coverage
   4. [Python][Dataset] Add filters parameter to orc.read_table() for predicate pushdown #49363 — [Python][Dataset] Add filters parameter to orc.read_table()

   Differences from Parquet

   Aspect	Parquet	ORC	Impact
   Unit of filtering	Row Group	Stripe	Naming only
   Column indexing	Schema-ordered, leaf-only	Depth-first pre-order, col 0 = root struct	Must map Arrow field index to ORC column ID
   Null counting	null_count field	hasNull() + getNumberOfValues() (no direct null count)	Must derive from stripe_rows - num_values; adapter reports -1 (unknown)
   Statistics API	parquet::RowGroupMetaData	liborc ColumnStatistics subclasses	Different API, same information
   Schema manifest	parquet::arrow::SchemaManifest	ORC type tree via Reader::getType()	Must build field to column mapping

   Type support roadmap

   Phase	Types	Notes
   Initial	INT32, INT64	Core integer types, overflow protection for INT32
   Follow-up	DOUBLE, FLOAT	NaN handling required
   Follow-up	STRING	Potential truncation in ORC stats
   Follow-up	DATE, TIMESTAMP	Unit conversion (days/millis/nanos)
   Future	DECIMAL	Scale/precision matching
   Future	Bloom filters	Equality predicates

   Test plan

   • Stripe filtering with selective predicates (skip/keep verification)
   • All comparison and logical operators
   • NULL handling (IS NULL, IS NOT NULL, all-null stripes)
   • INT32 overflow protection
   • Missing/corrupted statistics → conservative include
   • Multi-stripe files with known per-stripe statistics
   • Python integration: Expression format, DNF tuple format, column projection

   I'll post the consolidated PoC PR shortly for design review. Looking forward to feedback on the approach, particularly around the adapter API surface and the sub-issue/PR granularity.

   cc @wgtmac @raulcd
5. wgtmac commented on Feb 26, 2026

   

   wgtmac

   on Feb 26, 2026

   Member

   More actions

   Thanks for working on this and splitting them into several PRs. I will try to review them when I find time. (I'm now suffering from a flood of DoS from reviewing PRs across multiple projects)