Skip to content

[Bug] Qdp ParquetStreamingReader drops leading all-null rows in List columns #1574

Open

Description

@Howardisme

What

ParquetStreamingReader::read_chunk needs a sample size before it can
zero-fill a null row, and it learns that size from the first non-null row in
the batch (readers/parquet.rs:677-694):

  • some row in the batch is non-null → sample_size is taken from its list
    length, and null rows in that batch are zero-filled as FillZero promises
  • every row is null, but sample_size was already established by an earlier
    batch → the remembered size is reused, and the rows are zero-filled
  • every row is null and sample_size is still None → under
    NullHandling::Reject the reader errors out, but under
    NullHandling::FillZero it continues, and those rows are dropped.

The third case is the problem. The reader cannot know how wide a zero sample
should be yet, so it skips the batch entirely, and the rows are not recoverable
afterwards: by the time a non-null row establishes the sample size, the dropped
rows are already behind the cursor. The caller gets fewer samples than the file
declares, with no error and no warning.

This only affects a List<T> column whose leading rows are all null, enough
of them to fill a whole batch. A null row anywhere after the first non-null row
is handled correctly.

Reproduction

Write a Parquet file whose first row group contains only null List<Float64>
rows, followed by a row group with non-null rows, and read it with
NullHandling::FillZero:

let mut reader = ParquetStreamingReader::new(path, NullHandling::FillZero)?;
let declared = reader.get_num_samples();   // e.g. Some(6)
let mut produced = 0;
while let Some(chunk) = reader.read_chunk(...)? {
    produced += chunk.len() / reader.get_sample_size().unwrap();
}
assert_eq!(produced, declared.unwrap());   // fails: the leading nulls are gone

Expected Output

The reader defers the leading null rows until the sample size is
established from the first non-null row, then emits zero-filled samples for
them, so FillZero holds for every row in the file.

Environment

  • OS: 24.04.4 LTS
  • Python version: 3.12.3
  • Qumat version: 0.7.0.dev0 (qdp-core 0.3.0-dev, current main)

Before #1569 this surfaced downstream as a state vector with an uninitialised
tail, because GpuStateVector::new_batch allocates without zeroing. #1569 adds
a post-loop check that rejects the file instead, with an error hinting at this
case — safe, but still not what FillZero promises.

I'll give it a quick fix later.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions