Repository navigation
Make subchunk produce exactly the requested number of sub-chunks - #593
Open
bhaskargurram-ai wants to merge 1 commit into
Open
bhaskargurram-ai wants to merge 1 commit into
bhaskargurram-ai wants to merge 1 commit into
Conversation
subchunk planned a split across more than one leading axis when the factor was larger than the first non-unit chunk axis, but then returned the result of splitting by that first axis only, so chunks (2, 12) with factor=4 gave (1, 12) instead of (1, 6), and a factor larger than the whole chunk, or any factor on a chunk of all ones, silently produced a smaller split or unreadable references. Plan the whole split first, consuming leading axes until the factor is accounted for, and raise ValueError when that is not possible. Then write each original chunk's sub-chunks in C order, which is also their byte order because every axis before the last split one has a new chunk length of 1. Document that the split is along the leading (slowest-varying) axes, which is what "largest" meant, and why a factor that only divides a later axis cannot be used.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kerchunk.utils.subchunkreturns fewer sub-chunks than asked for wheneverfactoris larger than the first non-unit chunk axis. The planning loop works out that the split has to continue on the next axis, but then returnssubchunk(store, variable, multi), which splits by the first axis only and drops the rest of the factor. So chunks(2, 12)withfactor=4come back as(1, 12)(x2 instead of x4) and(4, 6)withfactor=12as(1, 6)(x4). The references that are produced are consistent, so the data reads back correctly and nothing warns; the caller just does not get the chunking they asked for. A factor larger than the whole chunk, for example 40 on(2, 10), is silently reduced in the same way, and any factor on a chunk of all ones (chunks[1]) rewrites the references into byte ranges that zarr cannot read.This change plans the whole split first, consuming the leading axes until the factor is accounted for, and raises
ValueErrorwith a message that says which factors are possible when it is not. Each original chunk is then written out as itsfactorsub-chunks in one pass, enumerated in C order, which is also their byte order within the original chunk because every axis before the last split one ends up with chunk length 1. The byte offsets are the sameoffset + k * size // factoras before, only over more than one axis. Factors that already worked (those dividing the leading axis) produce exactly the same references as before.The docstring now says that the split is along the leading (slowest-varying) axes, which is what "largest" meant as explained in #467, and that a factor which only divides a later axis cannot be used because the sub-chunks would not be contiguous. That case raised before and still does; only the message changed.
test_subchunk_exactis parametrised over factors that span one and two axes,test_subchunk_multiple_axescovers a 3-d chunk, andtest_subchunk_impossiblecovers the factors that must raise. Every successful case checks the new chunk shape, the number of references, and that the data read through the sub-chunked references equals the original.Fixes #592