Skip to content

Questions and deciding between scDataset and annbatch #90

Open

Description

@andrewni4313

Report

Hi!

This seems still in development so totally understandable if the docs are a little lacking right now on details but I was hoping you could answer a few questions that could help our team decide whether to use scDataset or annbatch for model training.

  • Does create_anndata_collection load all h5ads in backed mode and then shuffle? or does it have to read all the data into memory before shuffling to create the zarr-backed anndata format?
  • What does scDataset with "matched settings" mean in the benchmark?
  • What is the advantage over scDataset if you have to effectively duplicate the data (doubling storage) while being around the same speed (with whatever matched settings means)?
  • Will there be support for .h5ad objects with multiple layers or multi-modality between related .h5ad files?

Thanks so much!
Andrew

Versions

Activity

  1. ilan-gold commented on Dec 4, 2025

    @ilan-gold
    Collaborator

    Does create_anndata_collection load all h5ads in backed mode and then shuffle? or does it have to read all the data into memory before shuffling to create the zarr-backed anndata format?

    Not backed mode, but via dask, so memory efficiency should be pretty good modulo thorny bits of your dataset (for example the tahoe dataset has a categorical column with 3 million categories, which even in lazy mode is still a memory hog when you are working with only a fixed amount of memory). The best way to control memory before we merge #85 is via n_obs_per_dataset when shuffling. Even then, it is still the best way.

    What does scDataset with "matched settings" mean in the benchmark?

    This is a pure speed comparison - if scDataset were to use a block size of 256, how fast would that be? But unless you pre-shuffle your data, a block size of 256 for scDataset would probably not give the randomness you want for minibatch training. I would recommend just using scDataset's recommended settings for actual training if you don't pre-shuffle your data. That benchmark was only to highlight that choosing zarr over hdf5 is really the best decision if we are as generous as possible to hdf5.

    What is the advantage over scDataset if you have to effectively duplicate the data (doubling storage) while being around the same speed (with whatever matched settings means)?

    Pre-shuffling. You shouldn't use big block sizes (chunk_size as we call it) unless you pre-shuffle the data as noted above. And preshuffling hdf5 with this package takes about 3 times longer than preshuffling zarr, so there is really no advantage to it.

    Will there be support for .h5ad objects with multiple layers or multi-modality between related .h5ad files?

    Hmm I meant to create an issue for this. In short, yes, eventually and/or soon. But again, hdf5 is just slower than zarr (in broad strokes) so we are really encouraging zarr usage with this package.

    At the moment, the focus has been on performance, but as you can see, there are a lot of open API-related issues :)

    Feel free to ask follow-up questions, happy to help! And contributions are super-welcome!

  2. added
    questionFurther information is requested
    and removed
    bugSomething isn't working
    on Dec 4, 2025
  3. andrewni4313 commented on Dec 4, 2025

    @andrewni4313
    Author

    Thank you so much, that was very helpful! Just a few follow ups:

    Was the "matched settings" scDataset benchmark run on standard .h5ads or the zarr-backed anndata (as far as I understand scDataset can be run over any underlying format)? And what are the other settings set to (fetch_factor, num_workers, batch_size)?

  4. ilan-gold commented on Dec 4, 2025

    @ilan-gold
    Collaborator

    Was the "matched settings" scDataset benchmark run on standard .h5ads or the zarr-backed anndata (as far as I understand scDataset can be run over any underlying format)? And what are the other settings set to (fetch_factor, num_workers, batch_size)?

    It was run on hdf5 files. That's the correct understanding but anecdotally, I wouldn't expect much a speed up even using the rust accelerator with zarr. We have some very deep io tricks in our package :) As for the settings, the recommended ones were those from their suggestions and the matched ones were our attempt to match our own settings to give a pure speed hdf5 vs. zarr comparison (and zarr does win).

    See https://lamin.ai/laminlabs/arrayloader-benchmarks/transform/L6CAf9w0qdQj for defaults and https://lamin.ai/laminlabs/arrayloader-benchmarks/transform/e6Ry7noc4Y0d0004 for the actual run (this is an earlier version but should be the same or similar). It looks like the latest version is not on lamin (cc @felix0097 ).

    Overall, I think the overriding factor in the decision is whether you are willing to preshuffle your data. For us, it takes about 6-8 hours to preshuffle tahoe (100_000_000 cells, 60k genes) but when you consider the speedup from loading by using annbatch, that looks like much less.

    For example, if scDataset were to be double the performance we have reported, it would take ((100_000_000 / 4000) / 3600) ~ 7 hours of raw io for scDataset, not counting the time to put the data on the GPU. With our loader, the time per epoch of raw io would be according to our benchmarks (which are conservative since we don't put the data on the GPU and using preload_to_gpu=True actually speeds things up) ((100_000_000 / 32000) / 3600) ~ 50 minutes. And then there is every epoch after that! So you could comfortably shuffle the data and run two epochs of annbatch before 2 epochs of scDataset finishes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions