Repository navigation
Native Parquet scan does not prune row groups or pages with predicates on struct fields #7037
Description
What is the problem the feature request solves?
The native Parquet scan never prunes row groups or pages with a predicate on a struct field, such as WHERE s.k = 31337, so a selective nested filter reads the whole file. Spark pushes these predicates into Parquet by default (spark.sql.optimizer.nestedPredicatePushdownSupportedFileSources includes parquet), and parquet-mr prunes row groups using the leaf's statistics.
Measured on main at 66c63bd (Spark 4.1.3) over one Spark-written file of 40,000 rows in 37 row groups, with id and s.k both increasing. The numbers are the native scan's metrics:
| Filter | row_groups_pruned_statistics |
page_index_pages_matched |
bytes_scanned |
|---|---|---|---|
id = 31337 |
36 of 37 | 1 | 4,450 |
s.k = 31337 |
0 of 37 | 0 | 322,805 |
s.k IS NULL |
0 of 37 | 0 | 322,805 |
spark.comet.parquet.rowFilterPushdown.enabled=true gives the same numbers.
DataFusion 55's pruning predicate does not resolve a struct field access to its leaf statistics (apache/datafusion#20871, with apache/datafusion#25013 open). Comet also sends struct field access as its own GetStructField, evaluated by ordinal (native/core/src/execution/planner.rs lines 1112-1115), rather than DataFusion's get_field, so the data filters would also need translating before DataFusion could prune with them.
Describe the potential solution
Once DataFusion supports it, emit struct field predicates in the data filters in a form DataFusion's pruning recognizes, and check the pruning metrics against top-level predicates.
Additional context
Found in a nested-type audit of the native scans. #5553 and #5739 depend on this for their TIMESTAMP_MILLIS overflow parity.
Activity