You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The question is about writing ndtiff fast with python.
We use the ndtiff format as our main format in the lab. Mainly by micromanager, but with the availability of the ndtiff writer on python also as format after post processing.
Due to a Python project with high speed recordings, we need a high-performance file writer and we wish to use the ndtiff format.
Unfortunately, I stumbled over some limitations with the ndtiff Python implementation.
When a new ndtiff-file (ndtiff_file.py) is created, the file is first written with zeroes in maximum file size (Line 67 – 70). This takes quite some time. I guess this is done, to be able to work in read/write mode, to read file content while writing. Changing this to write only would eliminate this file creation time, but I guess the functionality to read while writing is lost?
The native file IO of python is blocking. And I couldn’t change this behaviour (by setting non-blocking flags). Therefore, the file writing process takes a lot of time, blocking in threaded mode. Running ndtiff in multiprocessing is not an option, because copying the data also takes some time. I used asyncio and aiofile to overcome this issue on very simplified file writing tests.
My Question is:
Is there interest in having a write only fast asynchronous ndtiff writer?
if yes, how should I proceed to implement one?
Should I try to implement one within the ndtiff package and make a pull request or
Fork the Projekt, for a separated write_only implementation or are there other thoughts?
Glad to hear you've found the format useful. At this point the python writer is the least optimized and least tested among the Java/Python readers/writers. I would not be surprised if there are substantial improvements to be had, and if you're interested in experimenting with it and finding them, pull requests welcome! So so long as it passes the tests changes are fair game
When a new ndtiff-file (ndtiff_file.py) is created, the file is first written with zeroes in maximum file size (Line 67 – 70). This takes quite some time.
This was copied from the Java NDTiff writer, where doing so greatly increased performance because repeated calls to resize were expensive. But it may not be the case with the python version. I don't see a problem with changing it.
The native file IO of python is blocking. And I couldn’t change this behaviour (by setting non-blocking flags). Therefore, the file writing process takes a lot of time, blocking in threaded mode
Hi!
Thanks for the feedback and advice!
I have multiple changes (tiny and "big"), should i make independent pull requests or on big one?
I tested Python 3.13 and it works fast when threaded, even without disabling the GIL. This is not the case up to Python 3.12.
AsyncIO is still a little bit faster, but i think not worth the hassle for a full and clean implementation.
The changes so far and well tested:
Commenting out the flush command to allocating the file size on the file system: The 4Gb file is then reserved without writing the full space.
Adding a Lock when a new file is generated. This was necessary to make the library thread save (i don't get why it is needed at all, but it works)
Tiny bugfix: somehow the tiff TAG for "samples per pixel" always gets 16 even the data is 8bit.
In question if wanted: orjson is much faster than json, specially with big metadata, but this would mean another external dependency.
Adding zlib compression (tiff comp. 8) option: This would be very useful for us. I tested also the compatibility with standard tiff libraries and it works.
Last: i still have to observe/debug a metadata read error i get when reading very large (multiple 4gb files) ndtiff datasets.
All sounds great! Either 1 big or multiple smaller ones -- whatever is easiest on your end. Have you run the NDTiff tests yet? you can set this up locally, but it might just be easier at first to open a PR and see the result in github actions
Ok, i don't know what's easier for me. I did run some tests more or less manually. Is there a good howto to set up the tests locally? I would like to learn that too. I guess i will need some additional days until i start making the pull requests.
I think you just need to navigate to python/ndstorage and do pytest -v .? You probably need to download the test data https://zenodo.org/records/7065552. I'm not sure I ever wrote down the exact sequence
Hi!
The pytest -v . worked find. For some reason i had to call it with python -m pytest -v . in the venv, also downloaded the test data and copied them to the test_data folder.
I tested python 3.13. It is noticeable faster. The non blocking file writes seemed to work in a "simple" threaded test. But in a more complex project with multiple threads it failed to be ether non blocking or stable.
I also tried syncio and implemented a syncio file writer into ndtiff storage. There the problem is, that this approach only works, if the whole python project is a syncio project. That is not a good solution for us. If syncio is running in a thread the non blocking features of syncio are neglected.
The next step for me now is to convert the python ndtiff storage into a cython project. I will let you know when i have something working.
I created some pull requests today, with the tiny things i encountered while working on the code. I also added zlib compression support which includes the tiny changes too.
Activity
henrypinkard commentedon Feb 23, 2025
Glad to hear you've found the format useful. At this point the python writer is the least optimized and least tested among the Java/Python readers/writers. I would not be surprised if there are substantial improvements to be had, and if you're interested in experimenting with it and finding them, pull requests welcome! So so long as it passes the tests changes are fair game
This was copied from the Java NDTiff writer, where doing so greatly increased performance because repeated calls to resize were expensive. But it may not be the case with the python version. I don't see a problem with changing it.
It be worth playing around with disabling the global interpreter lock, since this could allow the code to operate in a truly multithreaded way
LukasMicroscopy commentedon Mar 12, 2025
Hi!
Thanks for the feedback and advice!
I have multiple changes (tiny and "big"), should i make independent pull requests or on big one?
I tested Python 3.13 and it works fast when threaded, even without disabling the GIL. This is not the case up to Python 3.12.
AsyncIO is still a little bit faster, but i think not worth the hassle for a full and clean implementation.
The changes so far and well tested:
Commenting out the flush command to allocating the file size on the file system: The 4Gb file is then reserved without writing the full space.
Adding a Lock when a new file is generated. This was necessary to make the library thread save (i don't get why it is needed at all, but it works)
Tiny bugfix: somehow the tiff TAG for "samples per pixel" always gets 16 even the data is 8bit.
In question if wanted: orjson is much faster than json, specially with big metadata, but this would mean another external dependency.
Adding zlib compression (tiff comp. 8) option: This would be very useful for us. I tested also the compatibility with standard tiff libraries and it works.
Last: i still have to observe/debug a metadata read error i get when reading very large (multiple 4gb files) ndtiff datasets.
Thanks for your work and help!
henrypinkard commentedon Mar 12, 2025
All sounds great! Either 1 big or multiple smaller ones -- whatever is easiest on your end. Have you run the NDTiff tests yet? you can set this up locally, but it might just be easier at first to open a PR and see the result in github actions
LukasMicroscopy commentedon Mar 12, 2025
Ok, i don't know what's easier for me. I did run some tests more or less manually. Is there a good howto to set up the tests locally? I would like to learn that too. I guess i will need some additional days until i start making the pull requests.
henrypinkard commentedon Mar 12, 2025
I think you just need to navigate to
python/ndstorageand dopytest -v .? You probably need to download the test data https://zenodo.org/records/7065552. I'm not sure I ever wrote down the exact sequenceLukasMicroscopy commentedon Mar 27, 2025
Hi!
The
pytest -v .worked find. For some reason i had to call it withpython -m pytest -v .in the venv, also downloaded the test data and copied them to the test_data folder.I tested python 3.13. It is noticeable faster. The non blocking file writes seemed to work in a "simple" threaded test. But in a more complex project with multiple threads it failed to be ether non blocking or stable.
I also tried syncio and implemented a syncio file writer into ndtiff storage. There the problem is, that this approach only works, if the whole python project is a syncio project. That is not a good solution for us. If syncio is running in a thread the non blocking features of syncio are neglected.
The next step for me now is to convert the python ndtiff storage into a cython project. I will let you know when i have something working.
I created some pull requests today, with the tiny things i encountered while working on the code. I also added zlib compression support which includes the tiny changes too.
henrypinkard commentedon Mar 27, 2025
Nice work!