Skip to content

[BUG]: Illegal GPU memory access in test_heat_diffusion on Windows RTX Pro 6000 #3427

Open

Description

@jpascucci-nv

Is this a duplicate?

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

(Sol 5.1 high)

Illegal GPU memory access in test_heat_diffusion on Windows RTX Pro 6000

Failure

CI run 42725042120, job 129534476581:

  • Windows, MCDM
  • Python 3.13
  • CUDA 13.0.2 wheels
  • RTX Pro 6000

The first reported failure is:

tests/graph/test_graph_definition_integration.py::test_heat_diffusion[1-5]

It reports CUDA_ERROR_ILLEGAL_ADDRESS. Subsequent repetitions and other CUDA tests fail because the illegal access leaves the process in an inconsistent state. The process eventually crashes with a Windows access violation and exits with code 139.

The later failures may be cascading symptoms rather than independent regressions.

Baseline

The preceding nightly and the earlier push run at the same commit passed this configuration.

Nightlies repeat tests five times; push runs use one repetition. No direct connection to a recent code change has been established.

Suggested investigation

  • Run test_heat_diffusion in isolation on the same configuration.
  • Run it under Compute Sanitizer where supported.
  • If isolation passes, reproduce with preceding graph tests to check for shared state or lifetime problems.
  • Capture the first failing CUDA operation before subsequent errors obscure the original failure.

How to Reproduce

May be timing-dependent. Failed in CI.

Expected behavior

Successful CI run

Operating System

No response

nvidia-smi output

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions