Repository navigation
[None][fix] PyExecutor: create the user cache directory before run-time kernels are compiled - #22426
Draft
Wanli-Jiang wants to merge 1 commit into
Draft
Wanli-Jiang wants to merge 1 commit into
Wanli-Jiang wants to merge 1 commit into
Conversation
…me kernels are compiled The FMHA kernels that are compiled at run time are cached by name. When the kernel generator does not find the user cache directory ($XDG_CACHE_HOME, else $HOME/.cache) it keeps the compiled kernels in /tmp/jitify2_cache, which every process of the host shares. A generation kernel's name does not say whether it was built for one query token per sequence or for several, so an engine with draft tokens that starts after an engine without them loads the single-token kernel from that directory: NaN in its KV cache during warm-up, and token id 0 from small-batch decode steps for the life of the engine. With the user cache directory in existence the generator keeps its kernels in the process, as it does in an ordinary environment. The directory is missing when XDG_CACHE_HOME names a path nobody created, which test harnesses do to isolate runs. Create it when an executor is created, before the engine is built; warn and go on when it is unknown or cannot be created. This does not change how a kernel is named or cached. Two builds of one kernel name can still meet inside one process, and in a shared kernel cache where the directory cannot be created. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
An engine with draft tokens (MTP) can start in a state in which every small-batch decode step
returns non-finite logits. Its log has
and its answers continue with token id 0 after the first token (
The!!!!...). Large batchesstill look right, so an accuracy run at high concurrency can pass its threshold and hide it.
Cause. The FMHA generation kernels that are compiled at run time are cached by name, and
the name does not say whether the kernel was built for one query token per sequence (an engine
without draft tokens) or for several (an engine with draft tokens). The two engines ask for
the same name, for example
fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ16Kv256StaticSwapsAbForGen,with different options (
mSingleTokenQ,mMaxSeqLenQ,mIsCausalSpecDecodingGen,mNumSpecDecodingTokensamong them), and the two builds differ.When the kernel generator does not find the user cache directory (
$XDG_CACHE_HOME, else$HOME/.cache), it keeps the compiled kernels in/tmp/jitify2_cache, which every process onthe host shares. An engine with draft tokens that starts after an engine without them then
loads the single-token build under its own kernel's name. The reverse order is harmless: the
multi-token build also serves one query token.
The user cache directory is missing when
XDG_CACHE_HOMEnames a path that nobody created,which is what a test harness does when it exports a fresh path per run to isolate it. With the
directory in existence, or with the variable unset in an ordinary home, the generator keeps
its kernels in the process and nothing is shared.
Change.
create_py_executorcreates the user cache directory when it is missing, beforethe engine is built and so before any kernel is compiled
(
tensorrt_llm/_torch/pyexecutor/user_cache_directory.py). When the directory is not known orcannot be created, it warns and goes on.
What this does not change. The name and the cache key of a kernel are made in the prebuilt
kernel generator library and are untouched. Two builds of one name can still meet inside one
process that asks for the name first with one query token and later with several, and in the
shared directory when the user cache directory cannot be created.
Test Coverage
tests/unittest/_torch/pyexecutor/test_user_cache_directory.py(CPU only;l0_cpu.ymlliststhe directory): the directory
XDG_CACHE_HOMEnames is created; without the variable it is$HOME/.cache; an existing directory is left as it is; an unknown directory and one thatcannot be created warn without raising;
create_py_executormakes the directory before theengine is built.
Reproduction on GPUs.
XDG_CACHE_HOMEexported to a path that does not exist,/tmp/jitify2_cacheempty, one container. Start an engine of the Qwen3.8-Flash-Next NVFP4Modeling V2 target without draft tokens (
test_tp1_gsm8k), then one with one draft token(
test_tp1_mtp1_gsm8k_mmmu_and_video):test_tp1_mtp1_gsm8k_mmmu_and_videoaftertest_tp1_gsm8kb2f659aef8...PagedKvDenseP32{,MultiCtasKv,MultiCtasKvCga}VarSeqQ16Kv256StaticSwapsAbForGenkernels from/tmp/jitify2_cache(3.6 to 4.8 s each), logs the warning twice, answersThe!!!!...('red_rightward', 'the square is red!!!!!!...')b2f659aef8+ this change/tmp/jitify2_cacheis neither read nor writtenRun on GB300 (sm_103) at upstream main
b2f659aef8(2026-11-10), with the changecherry-picked. Not run at the commit this pull request is based on: the prebuilt kernel
generator archives under
cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/are identicalbetween the two commits.
The
l0_cpu.ymlentryunittest/_torch/pyexecutorwith-m cpu_only, started throughtest_unittests.py, atb2f659aef8+ this change: the five new cases pass. The session hasthree failures there
(
kv_cache/test_kv_cache_v2_vocab_size.py::test_branch_only_enables_multimodal_keys_when_needed),the same three as at
b2f659aef8without the change; main changed that test in [TRTLLM-16958][feat] Executor host path and hybrid GDN/Mamba perf improvements for RL #20916.More than one rank, on sm_107, on a tree that includes [None][feat] Qwen3.8-Flash-Next FP8 modeling_v2: one target for TP1/TP2/TEP2/DEP2/TP4/TEP4/DEP4 on GB300 #21621 (its Qwen3.8-Flash-Next FP8
target; so not plain main): engines without draft tokens first, then engines with draft
tokens at four ranks, same harness. Without the change the engines with draft tokens log
the warning; with it none does and
/tmp/jitify2_cachestays empty.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.