Skip to content

PyTorch 2.15 Support #9241

Open
Open

Description

@garciadias

PyTorch 2.15 was released on 2026-10-28 (release notes, blog). MONAI dev declares torch>=2.8.0 and CI tests 2.8.0 / 2.9.1 / 2.10.0 / 2.11.0. This issue tracks adding 2.15.0 to the supported and tested set.

Blocker: PyTorch 2.15 publishes no cp310 wheels

2.15 raises the minimum Python version for binaries to 3.11 (pytorch/pytorch#198233, pytorch/pytorch#199398). The 2.15.0 wheels on PyPI are cp311 / cp312 / cp313 / cp314 / cp315 only, so pip install torch==2.15.0 cannot resolve on Python 3.10. MONAI still declares requires-python = ">=3.10" and the min-dep matrix pairs Python 3.10 with the highest PyTorch version, so those cells cannot move to 2.15 as they are.

Either keep the Python 3.10 cells on 2.14.1 and add a 3.11+ cell for 2.15, or drop Python 3.10 support in the same change. The matrix comments ("Test PyTorch versions under Ubuntu with lowest Python version supported") need updating either way.

Version pairing

  • torchvision 0.29.2 is the matching release (requires torch>=2.14). TORCHVISION_VER1 (0.23.0, used with the lowest torch) is unaffected.
  • The default PyPI wheel is CUDA 13.2 (torch 2.15.0+cu132); CUDA 13.4 builds are also published.
  • Other pins that mention a torch version: Dockerfile (nvcr.io/nvidia/pytorch:25.12-py3), codeql-analysis.yml (PYTORCH_VER: 2.8.0), cron-ngc-bundle.yml and setupapp.yml (2.8.0), WIN_FULLDEP_VER (2.11.0 / torchvision 0.26.0).

Release-note items that touch MONAI code paths

  1. Rewritten NCCL backend is now the default ([c10d] Make nccl2 the default NCCL backend pytorch/pytorch#192281, #192895, #193237, #193241). Communicators are created eagerly in init_process_group/new_group, each group is bound to one device (device_id argument), isinstance(pg, ProcessGroupNCCL) is now False, and work.get_future() on an async all_gather returns the per-rank outputs instead of one stacked tensor. nccl-legacy and TORCH_DIST_USE_NCCL2=0 disappear in the next release. Relevant to monai.utils.dist, the *_dist tests and FL workflows.
  2. FSDP2 gradient dtype change ([FSDP2] Separate sharded and unsharded gradient dtypes pytorch/pytorch#194434): with MixedPrecisionPolicy(reduce_dtype=None) gradients are now accumulated and reduced in the parameter's grad_dtype (fp32 by default) instead of param_dtype. Affects users wrapping MONAI networks in fully_shard.
  3. torch.load(weights_only=True) no longer allowlists nested tensors (Remove NJT (nested jagged tensor) from weights_only safe globals pytorch/pytorch#194380). MONAI checkpoints are unaffected; worth a line in the bundle/checkpoint docs.
  4. torch.ops.* calls now use PyTorch's own exception and warning translation (Fix C++ warning propagation for torch.ops and custom ops pytorch/pytorch#193451): pybind11 exception translation is lost and TORCH_WARN from an op becomes a Python warning, which raises under -W error. Relevant to monai/_extensions and the compiled test job.
  5. Smaller ones: passing a set of parameters to an optimizer now emits a FutureWarning (Reject a bare set of params, consistent with set inside a param group pytorch/pytorch#198710), and torch.distributed.checkpoint output written by 2.15 cannot be read by 2.14 ([S615805][S694227][PyTorch] Safely preserve tensor strides (#194251) pytorch/pytorch#194251).

Local verification on 2.15.0

Environment: Python 3.13.18, torch 2.15.0+cu132, torchvision 0.29.2+cu132, MONAI dev (1.6.0rc1+91.g609a1108) installed from source with .[all,testing], RTX 5090 (driver 580.178.04). A second environment with the same dependency resolution but torch 2.14.1 was used as the baseline.

Run torch 2.15.0 torch 2.14.1
759 unit-test modules, QUICKTEST=True, CPU 14,755 tests not run (used for failing modules only)
759 unit-test modules, QUICKTEST=True, CUDA 13.2 17,910 tests not run (used for failing modules only)
759 unit-test modules, full (no QUICKTEST), CPU 14,805 tests not run (used for failing modules only)
759 unit-test modules, full, CUDA 13.2 18,150 tests not run (used for failing modules only)
17 *_dist.py modules (gloo/nccl, 1 GPU) 16 pass, 1 fail 16 pass, 1 fail
35 tests/integration modules CPU: all pass; CUDA: 33 pass, 2 fail CUDA: same 2 fail

No failure is specific to PyTorch 2.15. Every failing test was re-run serially in both environments and fails identically on 2.14.1:

Module 2.15.0 2.14.1 Cause
tests.inferers.test_zarr_avg_merger 20 errors 20 errors zarr 3.4.2 API
tests.data.test_image_rw (NIfTI cases) 4 failures, 4 errors identical nibabel affine/resample behaviour
tests.data.test_nifti_rw 1 failure, 1 error identical nibabel
tests.visualize.utils.test_matshow3d 3 failures identical matplotlib image comparison
tests.data.test_threadcontainer (test_plot) 1 failure identical matplotlib image comparison
tests.metrics.test_compute_fid_metric 1 error identical scipy sqrtm return/deprecation
tests.handlers.test_handler_mlflow 1 error identical test-data download hash check fails on this host
tests.losses.test_perceptual_loss (..._3_radimagenet_resnet50) 1 failure identical stale expectation: ValueError is not raised for 3D + non-MedicalNet, code and test disagree
tests.apps.test_cross_validation, test_decathlondataset pass after cleanup same failed earlier on a leftover root-owned ._Task04_Hippocampus in tests/testing_data; deleting it makes both pass on 2.15.0
tests.apps.test_mednistdataset 1 error identical leftover root-owned tests/testing_data/MedNIST from an earlier container run (needs sudo to delete), local artefact
tests/data/test_lmdbdataset_dist.py 1 failure identical lmdb error, pre-existing
tests/integration/test_auto3dseg_hpo.py 3 errors identical MLflow file-store backend removed/maintenance mode
tests/integration/test_integration_determinism.py loss 0.536044 vs 0.536134 identical GPU-dependent numerical drift on this hardware/cuDNN, not a 2.15 change

The first six rows are the unbounded-dependency-pin problem tracked in #9069, not a 2.15 problem.

All 17 failing modules were re-run serially on an idle GPU (nothing else holding VRAM) and produced the same results, so none of the above is a memory-pressure artefact or a parallel-run collision. A full GPU quick pass (17,910 tests) on the idle GPU leaves 35 failing tests, all inside the modules listed above.

Smoke checks that pass on 2.15.0: torch.jit.script and torch.jit.trace of a UNet, torch.compile(backend="eager"), torch.load(weights_only=True) on a MONAI-style checkpoint, torch.distributed.checkpoint import, torch.distributed gloo all_reduce, and torch.cuda.nccl availability. (torch.jit.* still works but emits its existing FutureWarning; that warning is present in 2.14 as well.)

Tasks

Activity

  1. siddharth07-code commented on Nov 3, 2026

    @siddharth07-code

    Hi @garciadias , I’d like to contribute to this issue.

    I’ve reviewed the current CI configuration and noticed that PyTorch 2.15 could be added to the min-dep matrix using Python 3.11, while retaining the existing Python 3.10 compatibility tests.

    Would a focused PR updating .github/workflows/cicd_tests.yml to include this configuration be welcome, or would you prefer to handle the broader compatibility updates together?

  2. garciadias commented on Nov 4, 2026

    @garciadias
    CollaboratorAuthor

    3. Hi @garciadias , I’d like to contribute to this issue.
    I’ve reviewed the current CI configuration and noticed that PyTorch 2.15 could be added to the min-dep matrix using Python 3.11, while retaining the existing Python 3.10 compatibility tests.
    Would a focused PR updating .github/workflows/cicd_tests.yml to include this configuration be welcome, or would you prefer to handle the broader compatibility updates together?

    Hi @siddharth07-code,

    Thank you for your interest in contributing.
    I would rather do extensive testing locally before making that change; otherwise, we may start seeing a lot of failures, and that will be annoying for unrelated PRs.

  3. siddharth07-code commented on Nov 4, 2026

    @siddharth07-code

    Understood, @garciadias . Would it be helpful if I ran additional local compatibility tests and shared the results here before proceeding with any CI changes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions