Repository navigation
Conversation
The NVFP4 dequant, Q8KV4 prefill attention, Q8KV4 decode indexer, indexer TopK and decode-attention host API builds go through torch.utils.cpp_extension.load(), which guards the build directory with a FileBaton: an O_EXCL "lock" file that other processes poll until it disappears. When a builder is killed mid-compile (SIGKILL, OOM killer, container eviction) the file stays, and every later process that loads the extension waits forever without output. Add _build_utils.jit_build_lock(): an fcntl lock on <dir>/build.lock (the same mechanism the decode-attention JIT already uses) that the kernel releases when its holder exits. All MSA builders of a directory take it before cpp_extension.load(), so a baton that exists while it is held was left by a dead builder and is removed with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Orien-Xu
marked this pull request as ready for review
October 24, 2026 00:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi, I'm Orien from Parasail. This fixes an issue we ran into while serving MiniMax-M3.1-Flash (#23).
Fixes #23.
Problem
The NVFP4 dequant, Q8KV4 prefill attention, Q8KV4 decode indexer, indexer TopK and the
decode-attention host API are built with
torch.utils.cpp_extension.load(). torch guards abuild directory with a
FileBaton(anO_EXCLfile namedlock). A builder killedmid-compile leaves that file, and every later process that loads the extension waits forever
without output.
Change
inference/msa_v1/_build_utils.py: newjit_build_lock(build_directory)context manager. Itholds
fcntl.flockon<dir>/build.lock(the mechanismattention/decode/q8kv4/jit.pyalreadyuses), and while holding it removes a leftover torch
lockfile with a warning. The kernelreleases the flock when its holder exits, so a crash can no longer wedge the directory.
cpp_extension.load()call sites run insidejit_build_lock(<their build dir>). Thedecode host API uses torch's default build directory for its module name.
serialized by the baton; they are now serialized by the flock.
Tests
tests/inference/msa_v1/test_build_utils.py: a spawned process takes the lock, leaves a torchlockfile and exits withos._exit(137); the nextjit_build_lockacquires immediately and thestale file is gone. Plus a fresh-directory case. (CPU only.)
(
tests/inference/msa_v1), includingattention/decode/q8kv4/test_jit_concurrency.py.Notes
fcntllocks need a filesystem with working POSIX locks (local disks and host-path mounts do).GPU validation (NVIDIA B200, driver 595.71.05, CUDA 13.3 nvcc, Python 3.12, head 9f8480f, base nv_dev 35dc1be)
Deterministic repro (
repro_baton.py: touch<dequant build dir>/lock, thenJitSpec.build_and_load(), 300 s timeout):lockgonelockleft alone (no cleanup on torch >= 2.14)Unit tests (
tests/inference/msa_v1/test_build_utils.py,attention/decode/q8kv4/test_jit_concurrency.py): 13 passed on torch 2.13 and on torch 2.14.Note: torch 2.14 replaced FileBaton with an OS-level lock (pytorch
cpp_extension.pyv2.14.0), so the hang affects torch <= 2.13 only; the PR is a no-op cleanup on newer torch.