Skip to content

Make subchunk produce exactly the requested number of sub-chunks - #593

Open
bhaskargurram-ai wants to merge 1 commit into
fsspec:mainfrom
bhaskargurram-ai:fix/subchunk-exact-factor
Open

bhaskargurram-ai wants to merge 1 commit into
fsspec:mainfrom
bhaskargurram-ai:fix/subchunk-exact-factor

Conversation

@bhaskargurram-ai

Copy link
Copy Markdown

kerchunk.utils.subchunk returns fewer sub-chunks than asked for whenever factor is larger than the first non-unit chunk axis. The planning loop works out that the split has to continue on the next axis, but then returns subchunk(store, variable, multi), which splits by the first axis only and drops the rest of the factor. So chunks (2, 12) with factor=4 come back as (1, 12) (x2 instead of x4) and (4, 6) with factor=12 as (1, 6) (x4). The references that are produced are consistent, so the data reads back correctly and nothing warns; the caller just does not get the chunking they asked for. A factor larger than the whole chunk, for example 40 on (2, 10), is silently reduced in the same way, and any factor on a chunk of all ones (chunks [1]) rewrites the references into byte ranges that zarr cannot read.

This change plans the whole split first, consuming the leading axes until the factor is accounted for, and raises ValueError with a message that says which factors are possible when it is not. Each original chunk is then written out as its factor sub-chunks in one pass, enumerated in C order, which is also their byte order within the original chunk because every axis before the last split one ends up with chunk length 1. The byte offsets are the same offset + k * size // factor as before, only over more than one axis. Factors that already worked (those dividing the leading axis) produce exactly the same references as before.

The docstring now says that the split is along the leading (slowest-varying) axes, which is what "largest" meant as explained in #467, and that a factor which only divides a later axis cannot be used because the sub-chunks would not be contiguous. That case raised before and still does; only the message changed.

test_subchunk_exact is parametrised over factors that span one and two axes, test_subchunk_multiple_axes covers a 3-d chunk, and test_subchunk_impossible covers the factors that must raise. Every successful case checks the new chunk shape, the number of references, and that the data read through the sub-chunked references equals the original.

Fixes #592

subchunk planned a split across more than one leading axis when the factor
was larger than the first non-unit chunk axis, but then returned the result
of splitting by that first axis only, so chunks (2, 12) with factor=4 gave
(1, 12) instead of (1, 6), and a factor larger than the whole chunk, or any
factor on a chunk of all ones, silently produced a smaller split or
unreadable references.

Plan the whole split first, consuming leading axes until the factor is
accounted for, and raise ValueError when that is not possible. Then write
each original chunk's sub-chunks in C order, which is also their byte order
because every axis before the last split one has a new chunk length of 1.
Document that the split is along the leading (slowest-varying) axes, which
is what "largest" meant, and why a factor that only divides a later axis
cannot be used.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

utils.subchunk silently produces fewer sub-chunks than requested when the factor spans more than one axis

1 participant