Skip to content

[FR]: Preallocate tiles with NumPy arrays for better performance and code clarity #280

Description

@jdbcode

Feature Summary

Replace nested list pre-allocation of 'tiles' with NumPy arrays using np.empty(shape, dtype=object). Update all tile access patterns from tiles[i][j][k] to tiles[i, j, k] for consistency and improved code readability.

Use Cases

  • Enables more pythonic and efficient handling of multi-dimensional tiles and blocks.
  • Reduces memory allocation overhead and potentially improves execution performance.
  • Makes tile-based operations more readable and maintainable, especially for future features dependent on array shape.
  • Cleaner integration of code with scientific Python ecosystem and facilitates future optimizations leveraging NumPy.

Activity

  1. added theissue type on Dec 18, 2025
  2. jdbcode commented on Dec 18, 2025

    @jdbcode
    MemberAuthor

    See #254 from @pvrraju for suggested implementation, e.g.,

    Xee/xee/ext.py

    Line 1002 in 00b2e45

    tiles = np.empty(shape, dtype=object)

  3. abidanBrito commented on Jun 2, 2026

    @abidanBrito

    I'd like to pick this up. One design question before diving in though:

    The current np.empty(...) proposal does indeed make for the cleaner and more pythonic tiles[i, j, k] access pattern, but I want to flag a subtlety around the final compositing step:

    np.block() recurses through lists/tuples to discover its grid, treating any ndarray as a single atomic block. This means that the object array can't be passed to np.block. In fact, it fails silently rather than raising.

    We'd have to call np.block(tiles.tolist()), therefore rebuilding the nested lists. While that works I believe it would mostly reduce the object to array swap to a readability improvement, since np.block still reallocates and copies tiles over.

    If we're optimizing for performance/memory, we could skip np.block entirely; preallocating the output and writing tiles straight into their slices as the thread pool returns them. This avoids the intermediate container and extra copies. It does however mean we'd have to compute each tile's pixel offsets as opposed to having np.block infer its placement in the output for us.

    Which would you prefer? Happy to go either way, I just want to match expectations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions