You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[FR]: Preallocate tiles with NumPy arrays for better performance and code clarity #280
Replace nested list pre-allocation of 'tiles' with NumPy arrays using np.empty(shape, dtype=object). Update all tile access patterns from tiles[i][j][k] to tiles[i, j, k] for consistency and improved code readability.
Use Cases
Enables more pythonic and efficient handling of multi-dimensional tiles and blocks.
Reduces memory allocation overhead and potentially improves execution performance.
Makes tile-based operations more readable and maintainable, especially for future features dependent on array shape.
Cleaner integration of code with scientific Python ecosystem and facilitates future optimizations leveraging NumPy.
I'd like to pick this up. One design question before diving in though:
The current np.empty(...) proposal does indeed make for the cleaner and more pythonic tiles[i, j, k] access pattern, but I want to flag a subtlety around the final compositing step:
np.block() recurses through lists/tuples to discover its grid, treating any ndarray as a single atomic block. This means that the object array can't be passed to np.block. In fact, it fails silently rather than raising.
We'd have to call np.block(tiles.tolist()), therefore rebuilding the nested lists. While that works I believe it would mostly reduce the object to array swap to a readability improvement, since np.block still reallocates and copies tiles over.
If we're optimizing for performance/memory, we could skip np.block entirely; preallocating the output and writing tiles straight into their slices as the thread pool returns them. This avoids the intermediate container and extra copies. It does however mean we'd have to compute each tile's pixel offsets as opposed to having np.block infer its placement in the output for us.
Which would you prefer? Happy to go either way, I just want to match expectations.
Activity
jdbcode commentedon Dec 18, 2025
See #254 from @pvrraju for suggested implementation, e.g.,
Xee/xee/ext.py
Line 1002 in 00b2e45
abidanBrito commentedon Jun 2, 2026
I'd like to pick this up. One design question before diving in though:
The current
np.empty(...)proposal does indeed make for the cleaner and more pythonictiles[i, j, k]access pattern, but I want to flag a subtlety around the final compositing step:np.block()recurses through lists/tuples to discover its grid, treating anyndarrayas a single atomic block. This means that the object array can't be passed tonp.block. In fact, it fails silently rather than raising.We'd have to call
np.block(tiles.tolist()), therefore rebuilding the nested lists. While that works I believe it would mostly reduce the object to array swap to a readability improvement, sincenp.blockstill reallocates and copies tiles over.If we're optimizing for performance/memory, we could skip
np.blockentirely; preallocating the output and writing tiles straight into their slices as the thread pool returns them. This avoids the intermediate container and extra copies. It does however mean we'd have to compute each tile's pixel offsets as opposed to havingnp.blockinfer its placement in the output for us.Which would you prefer? Happy to go either way, I just want to match expectations.