Skip to content

Group by preshuffling isn't convenient to implement #98

Closed

Description

@selmanozleyen

Description of feature

Hi,

I want to create a collection from one adata I have based on groupby iterations. Here is what I initially thought create_anndata_collection would support

# initial idea 
create_anndata_collection(
     [adata_lazy[gby.index] for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True)],
    output_path="adata_grouped.zarr",
    zarr_compressor=None
) # but this doesn't work since it expects a path

There are two workarounds I thought of

First one is to use the add_to_collection function:

# idk if this works
gby_idxs = [gby.index for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True)]
def load_lazy_by_idx(path, idx):
    adata = read_lazy(path)
    return adata[idx]

create_anndata_collection(
    "tahoe.h5ad",
    load_func=lambda path: load_lazy_by_idx(path, 0),
    output_path="adata_grouped.zarr",
)
for idx in gby_idxs[1:]:
    add_to_collection(
        "tahoe.h5ad",
        load_func=lambda path: load_lazy_by_idx(path, idx),
        output_path="adata_grouped.zarr",
    )

Second one is to write grouped adatas to disk and then create a collection from them:

for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True):
    gby.write_zarr(f"adata_grouped_{gby.name}.zarr")
create_anndata_collection(
    [f"adata_grouped_{gby.name}.zarr" for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True)],
    output_path="adata_grouped.zarr",
)

Since I have a custom code for my use case, I'd imagine it is faster than these solutions atm. But I created this issue to see if there can be features added to the library to support this use case more efficiently and conveniently.

Some solutions I thought of

1. Support lazy adata objects in create_anndata_collection

create_anndata_collection(
    [adata_lazy[gby.index] for gby in adata_lazy.obs.groupby(["cell_line", "drug", "dosage"], observed=True)],
    output_path="adata_grouped.zarr",
    zarr_compressor=None
)

2. Just add a groupby argument to create_anndata_collection

create_anndata_collection(
    adata_lazy,
    groupby=["cell_line", "drug", "dosage"],
    observed=True,
    output_path="adata_grouped.zarr",
    zarr_compressor=None
)

I can start a PR with whatever your preference would be in this case.

cc: @ilan-gold @felix0097

Activity

  1. felix0097 commented on Dec 11, 2025

    @felix0097
    Collaborator

    Hi @selmanozleyen,

    I'm not sure option 1 is doing what we want in this case. I was thinking that this would create a directory structure something like this:

    • group1 -> contains sharded zarr datasets for group1
    • group2 -> contains sharded zarr datasets for group2
    • ....

    And then we have one Dataloader for each group, for which we need to somehow merge the input streams during batching.

    Option 1 atm will just create on big zarr store as far as i understand.
    I'd probably go with option 2, if you implement it like I described above

  2. selmanozleyen commented on Dec 11, 2025

    @selmanozleyen
    MemberAuthor

    By option2 do you mean the workarounds or suggested solutions? I agree if its on suggested solutions

  3. felix0097 commented on Dec 11, 2025

    @felix0097
    Collaborator

    I mean this one:

    create_anndata_collection(
        adata_lazy,
        groupby=["cell_line", "drug", "dosage"],
        observed=True,
        output_path="/path/where/to/save/store",
        zarr_compressor=None
    )

    And that the above code, creates the described directory structure. Instead of shuffling the whole dataset (current implementation). And if groupby=None, you just create the store in the current way.

    What do you think?

  4. selmanozleyen commented on Dec 11, 2025

    @selmanozleyen
    MemberAuthor

    makes sense. However, to clarify it won't take adata_lazy object right? Just a path.

  5. felix0097 commented on Dec 11, 2025

    @felix0097
    Collaborator

    It would take the paths of the input h5ad files as input (same as the current implementation)

  6. ilan-gold commented on Dec 11, 2025

    @ilan-gold
    Collaborator

    I agree that this should be the goal. I think that before the feature is implemented we should try to understand what the data loading scheme would look like on top of that. To prototype I would do something like:

    for cat in groupy:
        create_anndata_collection(
            full_adata,
            load_adata: lamba adata: adata[adata.obs["cat"] == cat],
            output_path=f"/path/where/to/save/store/{cat}",
        )

    and then try out to make sure it is fast something like:

    for batch, cat in get_random_batch_from(ds_cat1, ds_cat2, ...):
        ctrl = adata_ctrl[adata_ctrl.obs["cat"] == cat].X
        do_thing(batch, ctrl)

    where you implement get_random_batch_from to get a random batch from the data loaders of interest.

    Also two things:

    1. Why would the data not be compressed?
    2. Why would observed be True? Wouldn't this just create an empty store.
    3. How to handle subcatgeories i.e., what is the concrete implementation going to do with nested categories? Do we create datasets per combination? Or do we sort based on all other categories that aren't the first one (and then the first one is the key on which we create separated on-disk datasets)?
  7. selmanozleyen commented on Dec 13, 2025

    @selmanozleyen
    MemberAuthor

    Sorry for late response. I was focused on wrapping up the tahoe training on my part so I can focus on this project. Now I train with 20it/sec (same speed we had with in memory loader 7 months ago) with two tahoes with 0.5 replacement prob which is probably more random that what would be needed. Also the preprocessing and preparation is super smooth and quick. Since I can focus here, I hope I we can even make it better.

    about the questions:

    I am more familiar with h5ad than zarr. So in zarr related things I am open to be corrected.

    • Why would the data not be compressed? Because it has an overhead to encode/decode, I assume the data is dense for my case and I don't have trouble with disk space. Is there a benefit to compression other than disk space? I always think I am going to archive it if I need to move a zarr file anyway

    • Why would observed be True? Wouldn't this just create an empty store. No isn't it the opposite? it only iterates over observed categories. So no empty stores.

    • How to handle subcatgeories i.e.,
      In my implementation I sort by categories then create a mapping for each category combination to the arrays. To get constant time access.

      # or something similar, this is my goal, a dumb dict was enough so far
      labels: {(cat1, cat2): 0}
      ranges: {0: (start, end)}
      data: array
      

      I know in no way it shouldn't be adata[adata.obs["cat1"] == cat1] for sure since it is O(n) and it matters from my experience. Or we could sort adata.obs.sort then use range indices so its O(1). If we don't want to change the order then we could store the indices for each category combination. But then we would create the mapping structure I already mentioned and it would be worse than O(1) when accessing even though we already have a mapping structure. I will write my propositions below.

    my proposal

    Write collections as you wrote. In tahoe there will be
    56827 + 50 = 56877 such adatas written. Since controls are hopefully null on the perturbation covariates (cell_lineA, null, null) would also be valid in this mapping. Then the researcher can have a list of which are these are source and which are target and can also find their mappings later (this part is the job of my codebase). But the researcher now has constant time access to each perturbation.

    So the researcher has src, tgt, and src2tgts. They can do

    dataloaders = {cats: DataLoader(f"mapping/{cats}.zarr") for cats in src+tgt} 
    for s in rng.choice(src):
        for t in rng.choice(src2tgts[s]):
            s_batch = dataloaders[s].sample()
            t_batch = dataloaders[t].sample()
            train(s_batch, t_batch)

    pros:

    • No changes
    • I think it's the fastest way it can be done for training time

    cons:

    • writing these adatas is going to be a nightmare since we have so many. Would need writing in parallel but is still going to be slow.

    So if I somehow have the folder structure like writing each combination as a zarr file, I don't even need a new feature. But it has been painful trying to run below script with the current API (see last section).

    But I am not sure if a solution to my writing problem should be this projects concern. This project is a very nice low level layer. I think doing something on top might on another project might be a better idea. But more in-memory writing support would be cool. It's good to have out of core as option but I often prefer putting tahoe into memory and writing it once to later use it out of core with memory intensive tasks.

    bonus: prototype script

    I wrote a prototype script. In my ml pert toolset this writing part takes at most 20mins. But in this script for even one combination to be written out of 56827, I waited more than 20mins then I terminated the process. Maybe I am using it wrongly but I just wanted to share that.

    from pathlib import Path
    import anndata as ad
    import h5py
    from annbatch import create_anndata_collection
    from anndata.experimental import read_lazy
    import time
    
    OUTPUT_PATH = Path("/data/annbatch_tahoe")
    OUTPUT_PATH.mkdir(parents=True, exist_ok=True)
    DATA_PATH = Path("/data/100m_int_indices.h5ad")
    
    with h5py.File(DATA_PATH, "r") as f:
        obs=ad.io.read_elem(f["obs"])
    t0 = time.time()
    covars = ["cell_line", "drug", "dosage"]
    def load_index(path: Path, idx):
        return read_lazy(path)[idx]
    for group in obs.groupby(covars, observed=True):
        cols = group[0]
        print(f"processing {cols[0]}_{cols[1]}_{cols[2]}")
        create_anndata_collection(
            adata_paths=[DATA_PATH],
            output_path=OUTPUT_PATH/f"{cols[0]}_{cols[1]}_{cols[2]}.zarr",
            load_adata=lambda path: ad.experimental.read_lazy(path)[group.index],
            zarr_compressor=None
        )
        print(f"processed {group[0]}_{group[1]}_{group[2]} in {time.time() - t0:.2f} seconds")
        t0 = time.time()

  8. ilan-gold commented on Dec 15, 2025

    @ilan-gold
    Collaborator

    Because it has an overhead to encode/decode, I assume the data is dense for my case and I don't have trouble with disk space. Is there a benefit to compression other than disk space? I always think I am going to archive it if I need to move a zarr file anyway

    A couple of things here

    1. Why is the data dense? Did you apply a transformation to it that would require this? If it's just the raw tahoe data, I would think the data should remain sparse (i.e., like the original data).
    2. There are benefits to compression besides disk space - if your "connection" is bottlenecked enough, like network connections or slow drives, compression decreases the file size so your throughput can go up if decompression is faster than the tradeoff from loading more data but not decompression. So decompression can help with throughput and usually it's safer to compress for this reason.

    Why would observed be True? Wouldn't this just create an empty store. No isn't it the opposite? it only iterates over observed categories. So no empty stores.

    Sorry got this reversed in my head, you're right.

    writing these adatas is going to be a nightmare since we have so many. Would need writing in parallel but is still going to be slow.

    Watch htop while the writing happens - writing is parallelized by zarrs under the hood (https://zarrs-python.readthedocs.io/en/latest/) and you can use direct_io which will could help your write throughput. If your cores dont look busy then you could consider parallelized writing but check this out first.

    But I am not sure if a solution to my writing problem should be this projects concern. This project is a very nice low level layer. I think doing something on top might on another project might be a better idea. But more in-memory writing support would be cool. It's good to have out of core as option but I often prefer putting tahoe into memory and writing it once to later use it out of core with memory intensive tasks.

    #66 Contributions welcome :))))))

    bonus: prototype script

    A couple of things about this

    1. Like I said above, compression can help (and I have no reason to believe that isnt the case here)
    2. There's no zarrs turned on in this script - did you use zarrs?
    3. create_anndata_collection might be overkill for what you are doing - you could just load each subset of interest and shuffle it if they are all small enough. Using this function comes with dask overhead which is probably unecessary (check out Generalize preprocessing to handle custom chunking / X not as a dask Array #66). Are your group-by categories spread across several files? Maybe not, in which case this function becomes even less necessary for your goal.

    Lastly I wonder about your batch size - covariate relationship. You're currently loading only one category per batch which is fine although I wonder if a larger batch size would enable more categories. What's the memory situation when you train? Potentially having a larger batch size could allow you to use more covariates at once which could cut down on the number of anndata files you create since you could elide one of the categoricals.

  9. changed the title [-]Group by preshuffling isn't convinient to implement[/-] [+]Group by preshuffling isn't convenient to implement[/+] on Dec 16, 2025
  10. selmanozleyen commented on Jan 22, 2026

    @selmanozleyen
    MemberAuthor

    I am going to close this. Because I am now more familiar with the repository I can open more specific issues for my use cases

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions