You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Questions and deciding between scDataset and annbatch #90
This seems still in development so totally understandable if the docs are a little lacking right now on details but I was hoping you could answer a few questions that could help our team decide whether to use scDataset or annbatch for model training.
Does create_anndata_collection load all h5ads in backed mode and then shuffle? or does it have to read all the data into memory before shuffling to create the zarr-backed anndata format?
What does scDataset with "matched settings" mean in the benchmark?
What is the advantage over scDataset if you have to effectively duplicate the data (doubling storage) while being around the same speed (with whatever matched settings means)?
Will there be support for .h5ad objects with multiple layers or multi-modality between related .h5ad files?
Does create_anndata_collection load all h5ads in backed mode and then shuffle? or does it have to read all the data into memory before shuffling to create the zarr-backed anndata format?
Not backed mode, but via dask, so memory efficiency should be pretty good modulo thorny bits of your dataset (for example the tahoe dataset has a categorical column with 3 million categories, which even in lazy mode is still a memory hog when you are working with only a fixed amount of memory). The best way to control memory before we merge #85 is via n_obs_per_dataset when shuffling. Even then, it is still the best way.
What does scDataset with "matched settings" mean in the benchmark?
This is a pure speed comparison - if scDataset were to use a block size of 256, how fast would that be? But unless you pre-shuffle your data, a block size of 256 for scDataset would probably not give the randomness you want for minibatch training. I would recommend just using scDataset's recommended settings for actual training if you don't pre-shuffle your data. That benchmark was only to highlight that choosing zarr over hdf5 is really the best decision if we are as generous as possible to hdf5.
What is the advantage over scDataset if you have to effectively duplicate the data (doubling storage) while being around the same speed (with whatever matched settings means)?
Pre-shuffling. You shouldn't use big block sizes (chunk_size as we call it) unless you pre-shuffle the data as noted above. And preshuffling hdf5 with this package takes about 3 times longer than preshuffling zarr, so there is really no advantage to it.
Will there be support for .h5ad objects with multiple layers or multi-modality between related .h5ad files?
Hmm I meant to create an issue for this. In short, yes, eventually and/or soon. But again, hdf5 is just slower than zarr (in broad strokes) so we are really encouraging zarr usage with this package.
At the moment, the focus has been on performance, but as you can see, there are a lot of open API-related issues :)
Feel free to ask follow-up questions, happy to help! And contributions are super-welcome!
Thank you so much, that was very helpful! Just a few follow ups:
Was the "matched settings" scDataset benchmark run on standard .h5ads or the zarr-backed anndata (as far as I understand scDataset can be run over any underlying format)? And what are the other settings set to (fetch_factor, num_workers, batch_size)?
Was the "matched settings" scDataset benchmark run on standard .h5ads or the zarr-backed anndata (as far as I understand scDataset can be run over any underlying format)? And what are the other settings set to (fetch_factor, num_workers, batch_size)?
It was run on hdf5 files. That's the correct understanding but anecdotally, I wouldn't expect much a speed up even using the rust accelerator with zarr. We have some very deep io tricks in our package :) As for the settings, the recommended ones were those from their suggestions and the matched ones were our attempt to match our own settings to give a pure speed hdf5 vs. zarr comparison (and zarr does win).
Overall, I think the overriding factor in the decision is whether you are willing to preshuffle your data. For us, it takes about 6-8 hours to preshuffle tahoe (100_000_000 cells, 60k genes) but when you consider the speedup from loading by using annbatch, that looks like much less.
For example, if scDataset were to be double the performance we have reported, it would take ((100_000_000 / 4000) / 3600) ~ 7 hours of raw io for scDataset, not counting the time to put the data on the GPU. With our loader, the time per epoch of raw io would be according to our benchmarks (which are conservative since we don't put the data on the GPU and using preload_to_gpu=True actually speeds things up) ((100_000_000 / 32000) / 3600) ~ 50 minutes. And then there is every epoch after that! So you could comfortably shuffle the data and run two epochs of annbatch before 2 epochs of scDataset finishes.
Activity
ilan-gold commentedon Dec 4, 2025
Not backed mode, but via dask, so memory efficiency should be pretty good modulo thorny bits of your dataset (for example the tahoe dataset has a categorical column with 3 million categories, which even in lazy mode is still a memory hog when you are working with only a fixed amount of memory). The best way to control memory before we merge #85 is via
n_obs_per_datasetwhen shuffling. Even then, it is still the best way.This is a pure speed comparison - if scDataset were to use a block size of 256, how fast would that be? But unless you pre-shuffle your data, a block size of 256 for scDataset would probably not give the randomness you want for minibatch training. I would recommend just using scDataset's recommended settings for actual training if you don't pre-shuffle your data. That benchmark was only to highlight that choosing zarr over hdf5 is really the best decision if we are as generous as possible to hdf5.
Pre-shuffling. You shouldn't use big block sizes (
chunk_sizeas we call it) unless you pre-shuffle the data as noted above. And preshuffling hdf5 with this package takes about 3 times longer than preshuffling zarr, so there is really no advantage to it.Hmm I meant to create an issue for this. In short, yes, eventually and/or soon. But again, hdf5 is just slower than zarr (in broad strokes) so we are really encouraging zarr usage with this package.
At the moment, the focus has been on performance, but as you can see, there are a lot of open API-related issues :)
Feel free to ask follow-up questions, happy to help! And contributions are super-welcome!
andrewni4313 commentedon Dec 4, 2025
Thank you so much, that was very helpful! Just a few follow ups:
Was the "matched settings" scDataset benchmark run on standard .h5ads or the zarr-backed anndata (as far as I understand scDataset can be run over any underlying format)? And what are the other settings set to (fetch_factor, num_workers, batch_size)?
ilan-gold commentedon Dec 4, 2025
It was run on hdf5 files. That's the correct understanding but anecdotally, I wouldn't expect much a speed up even using the rust accelerator with zarr. We have some very deep io tricks in our package :) As for the settings, the recommended ones were those from their suggestions and the matched ones were our attempt to match our own settings to give a pure speed hdf5 vs. zarr comparison (and zarr does win).
See https://lamin.ai/laminlabs/arrayloader-benchmarks/transform/L6CAf9w0qdQj for defaults and https://lamin.ai/laminlabs/arrayloader-benchmarks/transform/e6Ry7noc4Y0d0004 for the actual run (this is an earlier version but should be the same or similar). It looks like the latest version is not on lamin (cc @felix0097 ).
Overall, I think the overriding factor in the decision is whether you are willing to preshuffle your data. For us, it takes about 6-8 hours to preshuffle tahoe (100_000_000 cells, 60k genes) but when you consider the speedup from loading by using annbatch, that looks like much less.
For example, if scDataset were to be double the performance we have reported, it would take
((100_000_000 / 4000) / 3600) ~ 7hours of raw io for scDataset, not counting the time to put the data on the GPU. With our loader, the time per epoch of raw io would be according to our benchmarks (which are conservative since we don't put the data on the GPU and usingpreload_to_gpu=Trueactually speeds things up)((100_000_000 / 32000) / 3600) ~ 50minutes. And then there is every epoch after that! So you could comfortably shuffle the data and run two epochs ofannbatchbefore 2 epochs of scDataset finishes.