You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Give better errors when writing illegal column names #321
$ ./demo_load_maggie.py Variable names are not unique. To make them unique, call `.var_names_make_unique`.anndata/readwrite/write.py:301: UserWarning: Could not save field with key = "uns/condition_categories" to hdf5 file.'to hdf5 file.'.format(key))
Option A mangle/encode the names (e.g. replacing slashes with the unicode division slash /→∕), which modifies the names, so things aren’t as the user specified) and document this mangling in the format documentation.
I think mangling is inherently problematic, as it’ll result in subtle bugs arising from just a few columns not matching the names you put in. In this case it’s less bad because we wouldn’t mangle data (like gene names) but manually-addressed metadata, and we can unmangle again, so the bugs would be in other peoples’ code reading AnnData.
Option B store the names somewhere else (which makes the file less explorable since you now need code instead of a generic CLI tool to descend into the HDF5 file)
Option C leave things as is, fix all code that iterates over HDF5-stored dataframe columns, and document this so people wanting to read the thing don’t make a mistake
If it works, why does it matter if it’s internally represented as a tree of groups? The only problem is code that loops over dataframe columns, which would have to be changed to match and be a source of bugs for people wanting to read AnnData who don’t know this.
Option A mangle/encode the names (e.g. replacing slashes with the unicode division slash /→∕), which modifies the names, so things aren’t as the user specified) and document this mangling in the format documentation.
I was thinking we'd only mangle to and from disk, and it should be a in a completely reversible way. Basically, I was hoping we could just escape characters that filesystems treat as special.
I'm not sure if this can actually be done with hdf5. The current hdf5 user guide (which is conveniently only available as a pdf, so you get the old manual when you google) says group names must be ascii and can't a "." or "/". However, group names can contain a "." and be unicode (mention in other parts), so maybe there's a way for them to include "/'.
Option C ... fix all code that iterates over HDF5-stored dataframe columns
What are you thinking would be fixed here? Right now, I think it "works" for dataframes. It errors for values in the mapping attributes.
I was thinking we'd only mangle to and from disk, and it should be a in a completely reversible way.
You mean exactly the way I described above with the division slash? We can of course also escape the division slash by doubling it in the off chance someone uses one as column name, then stuff is reversible.
The current hdf5 user guide (which is conveniently only available as a pdf, so you get the old manual when you google)
fucking amazing, their online PDF viewer doesn’t even support Ctrl+F. Do they actively want to make developers’ lives harder?
I was hoping we could just escape characters that filesystems treat as special
HAhahahaHA “just” 🤪 Oh god, if we had to support filesystems we could either invest days into researching all the legacy crap that went into the 3 OS’s main file systems during decades of computing history or restrict ourselves to [A-Za-z_-] (or so! no idea if dashes and underscores are allowed everywhere! I mean, colons aren’t, so everything is possible 😵)
What are you thinking would be fixed here? Right now, I think it "works" for dataframes. It errors for values in the mapping attributes.
It does? We don’t have code that does for column_dataset in columns_group without checking if column_dataset is a group itself?
You mean exactly the way I described above with the division slash?
Not that, because it becomes non-obvious how to read the group. It would be really annoying for f["gene+/gene-"] to not work when it looks like it should. I was thinking more of escaping like \/.
Do they actively want to make developers’ lives harder?
Well, they have pivoted to emphasize consulting...
HAhahahaHA “just”
Well, I didn't want to have to do it, but figured this would be a common problem for projects like zarr, and we'd just be able to use that. There is a normalize_key argument, but it might just enforce lowercasing.
It does? We don’t have code that does for column_dataset in columns_group without checking if column_dataset is a group itself?
I’m pretty sure this won’t work. Escaping is something that depends on the interpreter. If HDF5 group names work like files system inode names in this regard it won’t interpret escape characters, it’ll simply do *group_names, value_name = value_path.split('/'), making it impossible to put a literal slash anywhere in a group name.
Not that, because it becomes non-obvious how to read the group. It would be really annoying for f["gene+/gene-"] to not work when it looks like it should.
As I said, if we go the mangling route, we should document whatever we do and de-mangle group names when reading. But yeah, it’s probably a good idea to not use lookalike characters. We should use an escape character that looks outlandishly unicody and is very unlikely to be used in an old anndata object so we don’t accidentally “unescape” something that was never escaped to begin with. Word uses some like that for their totally-not-LaTeX UnicodeMath (PDF)
Well, they have pivoted to emphasize consulting...
Well, did I ever hit the nail on the head.
Well, I didn't want to have to do it, but figured this would be a common problem for projects like zarr, and we'd just be able to use that.
Ah fuck, right, Zarr does that. Well, if we want to use it more prominently we have to stop using group names to store column names, plain and simple. The pain of figuring out and escaping all known file systems’ disallowed characters is one thing, but then there’s also device files (which on windows don’t have absolute paths and are named e.g. com1), different file systems are case sensitive while others aren’t, starting a column name with / might introduce a security risk, and I bet there’s more hidden nastiness. Just no.
Not that, because it becomes non-obvious how to read the group. It would be really annoying for f["gene+/gene-"] to not work when it looks like it should. I was thinking more of escaping like /.
I think cases like "gene+/gene-" would not be common because such names would appear as indices of the var dataframe, and not as column names of the var dataframe.
Activity
ivirshup commentedon Feb 16, 2020
Could be better, but I think it gives the important information (what element of the AnnData, and what key) at the end:
Example traceback:
I think this one should work fine now.
_indexwon't work, but I think we give a good error.To my surprise, this one works with dataframes, but not with values in mappings.
Traceback
AttributeError Traceback (most recent call last) <ipython-input-3-dc0e4fee41fa> in <module> 1 a.obsm["name/with/slashes"] = np.ones((a.shape[0], 10)) 2 a.write_h5ad("tmp.h5ad") ----> 3 b = ad.read_h5ad("tmp.h5ad") ~/github/anndata/anndata/_io/h5ad.py in read_h5ad(filename, backed, as_sparse, as_sparse_fmt, chunk_size) 427 _clean_uns(d) # backwards compat 428 --> 429 return AnnData(**d) 430 431 ~/github/anndata/anndata/_core/anndata.py in __init__(self, X, obs, var, uns, obsm, varm, layers, raw, dtype, shape, filename, filemode, asview, obsp, varp, oidx, vidx) 311 varp=varp, 312 filename=filename, --> 313 filemode=filemode, 314 ) 315 ~/github/anndata/anndata/_core/anndata.py in _init_as_actual(self, X, obs, var, uns, obsm, varm, varp, obsp, raw, layers, dtype, shape, filename, filemode) 492 493 # TODO: Think about consequences of making obsm a group in hdf --> 494 self._obsm = AxisArrays(self, 0, vals=convert_to_dict(obsm)) 495 self._varm = AxisArrays(self, 1, vals=convert_to_dict(varm)) 496 ~/github/anndata/anndata/_core/aligned_mapping.py in __init__(self, parent, axis, vals) 229 self._data = dict() 230 if vals is not None: --> 231 self.update(vals) 232 233 /usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/_collections_abc.py in update(*args, **kwds) 839 if isinstance(other, Mapping): 840 for key in other: --> 841 self[key] = other[key] 842 elif hasattr(other, "keys"): 843 for key in other.keys(): ~/github/anndata/anndata/_core/aligned_mapping.py in __setitem__(self, key, value) 148 149 def __setitem__(self, key: str, value: V): --> 150 value = self._validate_value(value, key) 151 self._data[key] = value 152 ~/github/anndata/anndata/_core/aligned_mapping.py in _validate_value(self, val, key) 212 f"value.index does not match parent’s axis {self.axes[0]} names" 213 ) --> 214 return super()._validate_value(val, key) 215 216 ~/github/anndata/anndata/_core/aligned_mapping.py in _validate_value(self, val, key) 47 """Raises an error if value is invalid""" 48 for i, axis in enumerate(self.axes): ---> 49 if self.parent.shape[axis] != val.shape[i]: 50 right_shape = tuple(self.parent.shape[a] for a in self.axes) 51 raise ValueError( AttributeError: 'dict' object has no attribute 'shape'I don't get an error for that one.
ivirshup commentedon Feb 16, 2020
Update on
name/with/slashesin dataframes:It works, but I'm not sure it's doing the right thing. Zarr does something similar. Truncated output of
h5ls -r tmp.h5ad:zarr["obs"].tree()
Should we normalize or error for these? I'm thinking normalize since I think slashes will be fairly common in column/ group names. E.g. "CD4+/CD8-".
It would be good to use some external standard for normalizing these names. I'm not sure where to find this kind of thing though.
flying-sheep commentedon Feb 17, 2020
Option A mangle/encode the names (e.g. replacing slashes with the unicode division slash /→∕), which modifies the names, so things aren’t as the user specified) and document this mangling in the format documentation.
I think mangling is inherently problematic, as it’ll result in subtle bugs arising from just a few columns not matching the names you put in. In this case it’s less bad because we wouldn’t mangle data (like gene names) but manually-addressed metadata, and we can unmangle again, so the bugs would be in other peoples’ code reading AnnData.
Option B store the names somewhere else (which makes the file less explorable since you now need code instead of a generic CLI tool to descend into the HDF5 file)
Option C leave things as is, fix all code that iterates over HDF5-stored dataframe columns, and document this so people wanting to read the thing don’t make a mistake
If it works, why does it matter if it’s internally represented as a tree of groups? The only problem is code that loops over dataframe columns, which would have to be changed to match and be a source of bugs for people wanting to read AnnData who don’t know this.
ivirshup commentedon Feb 18, 2020
I was thinking we'd only mangle to and from disk, and it should be a in a completely reversible way. Basically, I was hoping we could just escape characters that filesystems treat as special.
I'm not sure if this can actually be done with hdf5. The current hdf5 user guide (which is conveniently only available as a pdf, so you get the old manual when you google) says group names must be ascii and can't a "." or "/". However, group names can contain a "." and be unicode (mention in other parts), so maybe there's a way for them to include "/'.
What are you thinking would be fixed here? Right now, I think it "works" for dataframes. It errors for values in the mapping attributes.
flying-sheep commentedon Feb 18, 2020
You mean exactly the way I described above with the division slash? We can of course also escape the division slash by doubling it in the off chance someone uses one as column name, then stuff is reversible.
fucking amazing, their online PDF viewer doesn’t even support Ctrl+F. Do they actively want to make developers’ lives harder?
HAhahahaHA “just” 🤪 Oh god, if we had to support filesystems we could either invest days into researching all the legacy crap that went into the 3 OS’s main file systems during decades of computing history or restrict ourselves to
[A-Za-z_-](or so! no idea if dashes and underscores are allowed everywhere! I mean, colons aren’t, so everything is possible 😵)It does? We don’t have code that does
for column_dataset in columns_groupwithout checking ifcolumn_datasetis a group itself?ivirshup commentedon Feb 19, 2020
Not that, because it becomes non-obvious how to read the group. It would be really annoying for f["gene+/gene-"] to not work when it looks like it should. I was thinking more of escaping like
\/.Well, they have pivoted to emphasize consulting...
Well, I didn't want to have to do it, but figured this would be a common problem for projects like zarr, and we'd just be able to use that. There is a
normalize_keyargument, but it might just enforce lowercasing.Since the order of the columns is important for dataframes, we save the names in the correct order in the attributes, then do
{k: h5group[k] for k in colnames}. Here's the logic: https://github.com/theislab/anndata/blob/4440b90ff3dff213b4c512478e21426cf210368d/anndata/_io/h5ad.py#L460-L469flying-sheep commentedon Feb 19, 2020
I’m pretty sure this won’t work. Escaping is something that depends on the interpreter. If HDF5 group names work like files system inode names in this regard it won’t interpret escape characters, it’ll simply do
*group_names, value_name = value_path.split('/'), making it impossible to put a literal slash anywhere in a group name.As I said, if we go the mangling route, we should document whatever we do and de-mangle group names when reading. But yeah, it’s probably a good idea to not use lookalike characters. We should use an escape character that looks outlandishly unicody and is very unlikely to be used in an old anndata object so we don’t accidentally “unescape” something that was never escaped to begin with. Word uses some like that for their totally-not-LaTeX UnicodeMath (PDF)
Well, did I ever hit the nail on the head.
Ah fuck, right, Zarr does that. Well, if we want to use it more prominently we have to stop using group names to store column names, plain and simple. The pain of figuring out and escaping all known file systems’ disallowed characters is one thing, but then there’s also device files (which on windows don’t have absolute paths and are named e.g. com1), different file systems are case sensitive while others aren’t, starting a column name with
/might introduce a security risk, and I bet there’s more hidden nastiness. Just no.LucaMarconato commentedon Jan 20, 2025
I think cases like "gene+/gene-" would not be common because such names would appear as indices of the
vardataframe, and not as column names of thevardataframe.