Repository navigation
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Description of feature
Hi,
I want to create a collection from one adata I have based on groupby iterations. Here is what I initially thought create_anndata_collection would support
# initial idea
create_anndata_collection(
[adata_lazy[gby.index] for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True)],
output_path="adata_grouped.zarr",
zarr_compressor=None
) # but this doesn't work since it expects a pathThere are two workarounds I thought of
First one is to use the add_to_collection function:
# idk if this works
gby_idxs = [gby.index for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True)]
def load_lazy_by_idx(path, idx):
adata = read_lazy(path)
return adata[idx]
create_anndata_collection(
"tahoe.h5ad",
load_func=lambda path: load_lazy_by_idx(path, 0),
output_path="adata_grouped.zarr",
)
for idx in gby_idxs[1:]:
add_to_collection(
"tahoe.h5ad",
load_func=lambda path: load_lazy_by_idx(path, idx),
output_path="adata_grouped.zarr",
)Second one is to write grouped adatas to disk and then create a collection from them:
for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True):
gby.write_zarr(f"adata_grouped_{gby.name}.zarr")
create_anndata_collection(
[f"adata_grouped_{gby.name}.zarr" for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True)],
output_path="adata_grouped.zarr",
)Since I have a custom code for my use case, I'd imagine it is faster than these solutions atm. But I created this issue to see if there can be features added to the library to support this use case more efficiently and conveniently.
Some solutions I thought of
1. Support lazy adata objects in create_anndata_collection
create_anndata_collection(
[adata_lazy[gby.index] for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True)],
output_path="adata_grouped.zarr",
zarr_compressor=None
)2. Just add a groupby argument to create_anndata_collection
create_anndata_collection(
adata_lazy,
groupby=["cell_line", "drug", "dosage"],
observed=True,
output_path="adata_grouped.zarr",
zarr_compressor=None
)I can start a PR with whatever your preference would be in this case.
cc: @ilan-gold @felix0097
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
felix0097 commentedon Dec 11, 2025
Hi @selmanozleyen,
I'm not sure option 1 is doing what we want in this case. I was thinking that this would create a directory structure something like this:
And then we have one Dataloader for each group, for which we need to somehow merge the input streams during batching.
Option 1 atm will just create on big zarr store as far as i understand.
I'd probably go with option 2, if you implement it like I described above
selmanozleyen commentedon Dec 11, 2025
By option2 do you mean the workarounds or suggested solutions? I agree if its on suggested solutions
felix0097 commentedon Dec 11, 2025
I mean this one:
And that the above code, creates the described directory structure. Instead of shuffling the whole dataset (current implementation). And if
groupby=None, you just create the store in the current way.What do you think?
selmanozleyen commentedon Dec 11, 2025
makes sense. However, to clarify it won't take adata_lazy object right? Just a path.
felix0097 commentedon Dec 11, 2025
It would take the paths of the input h5ad files as input (same as the current implementation)
ilan-gold commentedon Dec 11, 2025
I agree that this should be the goal. I think that before the feature is implemented we should try to understand what the data loading scheme would look like on top of that. To prototype I would do something like:
and then try out to make sure it is fast something like:
where you implement
get_random_batch_fromto get a random batch from the data loaders of interest.Also two things:
observedbeTrue? Wouldn't this just create an empty store.selmanozleyen commentedon Dec 13, 2025
Sorry for late response. I was focused on wrapping up the tahoe training on my part so I can focus on this project. Now I train with 20it/sec (same speed we had with in memory loader 7 months ago) with two tahoes with 0.5 replacement prob which is probably more random that what would be needed. Also the preprocessing and preparation is super smooth and quick. Since I can focus here, I hope I we can even make it better.
about the questions:
I am more familiar with h5ad than zarr. So in zarr related things I am open to be corrected.
Why would the data not be compressed? Because it has an overhead to encode/decode, I assume the data is dense for my case and I don't have trouble with disk space. Is there a benefit to compression other than disk space? I always think I am going to archive it if I need to move a zarr file anyway
Why would observed be True? Wouldn't this just create an empty store. No isn't it the opposite? it only iterates over observed categories. So no empty stores.
How to handle subcatgeories i.e.,
In my implementation I sort by categories then create a mapping for each category combination to the arrays. To get constant time access.
I know in no way it shouldn't be
adata[adata.obs["cat1"] == cat1]for sure since it isO(n)and it matters from my experience. Or we could sortadata.obs.sortthen use range indices so itsO(1). If we don't want to change the order then we could store the indices for each category combination. But then we would create the mapping structure I already mentioned and it would be worse thanO(1)when accessing even though we already have a mapping structure. I will write my propositions below.my proposal
Write collections as you wrote. In tahoe there will be
56827 + 50 = 56877 such adatas written. Since controls are hopefully null on the perturbation covariates (cell_lineA, null, null) would also be valid in this mapping. Then the researcher can have a list of which are these are source and which are target and can also find their mappings later (this part is the job of my codebase). But the researcher now has constant time access to each perturbation.
So the researcher has
src, tgt, and src2tgts. They can dopros:
cons:
So if I somehow have the folder structure like writing each combination as a zarr file, I don't even need a new feature. But it has been painful trying to run below script with the current API (see last section).
But I am not sure if a solution to my writing problem should be this projects concern. This project is a very nice low level layer. I think doing something on top might on another project might be a better idea. But more in-memory writing support would be cool. It's good to have out of core as option but I often prefer putting tahoe into memory and writing it once to later use it out of core with memory intensive tasks.
bonus: prototype script
I wrote a prototype script. In my ml pert toolset this writing part takes at most 20mins. But in this script for even one combination to be written out of 56827, I waited more than 20mins then I terminated the process. Maybe I am using it wrongly but I just wanted to share that.
ilan-gold commentedon Dec 15, 2025
A couple of things here
Sorry got this reversed in my head, you're right.
Watch
htopwhile the writing happens - writing is parallelized byzarrsunder the hood (https://zarrs-python.readthedocs.io/en/latest/) and you can usedirect_iowhich will could help your write throughput. If your cores dont look busy then you could consider parallelized writing but check this out first.#66 Contributions welcome :))))))
A couple of things about this
zarrsturned on in this script - did you usezarrs?create_anndata_collectionmight be overkill for what you are doing - you could just load each subset of interest and shuffle it if they are all small enough. Using this function comes with dask overhead which is probably unecessary (check out Generalize preprocessing to handle custom chunking /Xnot as a dask Array #66). Are your group-by categories spread across several files? Maybe not, in which case this function becomes even less necessary for your goal.Lastly I wonder about your batch size - covariate relationship. You're currently loading only one category per batch which is fine although I wonder if a larger batch size would enable more categories. What's the memory situation when you train? Potentially having a larger batch size could allow you to use more covariates at once which could cut down on the number of anndata files you create since you could elide one of the categoricals.
[-]Group by preshuffling isn't convinient to implement[/-][+]Group by preshuffling isn't convenient to implement[/+]selmanozleyen commentedon Jan 22, 2026
I am going to close this. Because I am now more familiar with the repository I can open more specific issues for my use cases