Skip to content

Pass target_options when MultiZarrToZarr reads reference files from paths - #591

Open
bhaskargurram-ai wants to merge 1 commit into
fsspec:mainfrom
bhaskargurram-ai:combine-target-options
Open

bhaskargurram-ai wants to merge 1 commit into
fsspec:mainfrom
bhaskargurram-ai:combine-target-options

Conversation

@bhaskargurram-ai

Copy link
Copy Markdown

When MultiZarrToZarr is given paths to JSON reference files, the batch read ignored target_options, so gzip-compressed references failed to decode (#522). The same code built the filesystem with asynchronous=True and then called its blocking cat(), which raises "Loop is not running" for http, s3 and gcs paths. It also decoded the files in the sorted order cat() returns rather than the input order, so INDEX, list and filename coo_map selectors could be matched to the wrong dataset.

I now read the files through the filesystem of the OpenFiles, decompress each with its own compression, keep the input order, and pass target_options to ReferenceFileSystem in the per-file fallback.

New tests in test_combine.py cover gzip references, input order, and paths served from a local HTTP server. They fail on main and pass with this change.

Fixes #522

…aths

When the inputs are paths to JSON reference files, the batch read used a
filesystem created with asynchronous=True and called its blocking cat(),
which fails with "Loop is not running" for http/s3/gcs paths. It also
ignored the compression given in target_options, and decoded the files in
the sorted order returned by cat() rather than the order of the inputs,
so INDEX, list and filename-based coo_map selectors could be paired with
the wrong dataset.

Read the files through the filesystem of the OpenFiles, decompress with
the compression of each OpenFile, keep the input order, and pass
target_options to ReferenceFileSystem when falling back to opening each
path.

Fixes fsspec#522

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MultiZarrToZarr does not pass target_options to the reference filesystems

1 participant