Skip to content

jit: recover from torch extension locks left by killed builders - #26

Open
Orien-Xu wants to merge 1 commit into
MiniMax-AI:nv_devfrom
Orien-Xu:jit-stale-baton
Open

Orien-Xu wants to merge 1 commit into
MiniMax-AI:nv_devfrom
Orien-Xu:jit-stale-baton

Conversation

@Orien-Xu

@Orien-Xu Orien-Xu commented Oct 24, 2026 •

Copy link
Copy Markdown

Hi, I'm Orien from Parasail. This fixes an issue we ran into while serving MiniMax-M3.1-Flash (#23).

Fixes #23.

Problem

The NVFP4 dequant, Q8KV4 prefill attention, Q8KV4 decode indexer, indexer TopK and the
decode-attention host API are built with torch.utils.cpp_extension.load(). torch guards a
build directory with a FileBaton (an O_EXCL file named lock). A builder killed
mid-compile leaves that file, and every later process that loads the extension waits forever
without output.

Change

  • inference/msa_v1/_build_utils.py: new jit_build_lock(build_directory) context manager. It
    holds fcntl.flock on <dir>/build.lock (the mechanism attention/decode/q8kv4/jit.py already
    uses), and while holding it removes a leftover torch lock file with a warning. The kernel
    releases the flock when its holder exits, so a crash can no longer wedge the directory.
  • The five cpp_extension.load() call sites run inside jit_build_lock(<their build dir>). The
    decode host API uses torch's default build directory for its module name.
  • No change to cache keys, build flags or build directories. Builds of one directory were already
    serialized by the baton; they are now serialized by the flock.

Tests

  • tests/inference/msa_v1/test_build_utils.py: a spawned process takes the lock, leaves a torch
    lock file and exits with os._exit(137); the next jit_build_lock acquires immediately and the
    stale file is gone. Plus a fresh-directory case. (CPU only.)
  • Ran the helper logic standalone on CPU: recovery is immediate.
  • TODO before submission: run the existing JIT/correctness suites on SM100
    (tests/inference/msa_v1), including attention/decode/q8kv4/test_jit_concurrency.py.

Notes

fcntl locks need a filesystem with working POSIX locks (local disks and host-path mounts do).

GPU validation (NVIDIA B200, driver 595.71.05, CUDA 13.3 nvcc, Python 3.12, head 9f8480f, base nv_dev 35dc1be)

Deterministic repro (repro_baton.py: touch <dequant build dir>/lock, then JitSpec.build_and_load(), 300 s timeout):

torch nv_dev 35dc1be this PR
2.13.0+cu130 (FileBaton) hangs, killed by timeout after 300 s builds in 18.8 s, logs "Removing a stale torch extension lock …", lock gone
2.14.1+cu130 (torch's own OS lock) builds in 19.9 s builds in 18.8 s, lock left alone (no cleanup on torch >= 2.14)

Unit tests (tests/inference/msa_v1/test_build_utils.py, attention/decode/q8kv4/test_jit_concurrency.py): 13 passed on torch 2.13 and on torch 2.14.

Note: torch 2.14 replaced FileBaton with an OS-level lock (pytorch cpp_extension.py v2.14.0), so the hang affects torch <= 2.13 only; the PR is a no-op cleanup on newer torch.

The NVFP4 dequant, Q8KV4 prefill attention, Q8KV4 decode indexer, indexer
TopK and decode-attention host API builds go through
torch.utils.cpp_extension.load(), which guards the build directory with a
FileBaton: an O_EXCL "lock" file that other processes poll until it
disappears. When a builder is killed mid-compile (SIGKILL, OOM killer,
container eviction) the file stays, and every later process that loads the
extension waits forever without output.

Add _build_utils.jit_build_lock(): an fcntl lock on <dir>/build.lock (the
same mechanism the decode-attention JIT already uses) that the kernel
releases when its holder exits. All MSA builders of a directory take it
before cpp_extension.load(), so a baton that exists while it is held was
left by a dead builder and is removed with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Orien-Xu
Orien-Xu marked this pull request as ready for review October 24, 2026 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant