Skip to content

[None][fix] PyExecutor: create the user cache directory before run-time kernels are compiled - #22426

Draft
Wanli-Jiang wants to merge 1 commit into
NVIDIA:mainfrom
Wanli-Jiang:user/williamj/user-cache-dir
Draft

Wanli-Jiang wants to merge 1 commit into
NVIDIA:mainfrom
Wanli-Jiang:user/williamj/user-cache-dir

Conversation

@Wanli-Jiang

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

An engine with draft tokens (MTP) can start in a state in which every small-batch decode step
returns non-finite logits. Its log has

NaNs/Infs have been introduced to KVCache during warmup, KVCache was filled with zeros to avoid potential issues

and its answers continue with token id 0 after the first token (The!!!!...). Large batches
still look right, so an accuracy run at high concurrency can pass its threshold and hide it.

Cause. The FMHA generation kernels that are compiled at run time are cached by name, and
the name does not say whether the kernel was built for one query token per sequence (an engine
without draft tokens) or for several (an engine with draft tokens). The two engines ask for
the same name, for example
fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ16Kv256StaticSwapsAbForGen,
with different options (mSingleTokenQ, mMaxSeqLenQ, mIsCausalSpecDecodingGen,
mNumSpecDecodingTokens among them), and the two builds differ.

When the kernel generator does not find the user cache directory ($XDG_CACHE_HOME, else
$HOME/.cache), it keeps the compiled kernels in /tmp/jitify2_cache, which every process on
the host shares. An engine with draft tokens that starts after an engine without them then
loads the single-token build under its own kernel's name. The reverse order is harmless: the
multi-token build also serves one query token.

The user cache directory is missing when XDG_CACHE_HOME names a path that nobody created,
which is what a test harness does when it exports a fresh path per run to isolate it. With the
directory in existence, or with the variable unset in an ordinary home, the generator keeps
its kernels in the process and nothing is shared.

Change. create_py_executor creates the user cache directory when it is missing, before
the engine is built and so before any kernel is compiled
(tensorrt_llm/_torch/pyexecutor/user_cache_directory.py). When the directory is not known or
cannot be created, it warns and goes on.

What this does not change. The name and the cache key of a kernel are made in the prebuilt
kernel generator library and are untouched. Two builds of one name can still meet inside one
process that asks for the name first with one query token and later with several, and in the
shared directory when the user cache directory cannot be created.

Test Coverage

  • tests/unittest/_torch/pyexecutor/test_user_cache_directory.py (CPU only; l0_cpu.yml lists
    the directory): the directory XDG_CACHE_HOME names is created; without the variable it is
    $HOME/.cache; an existing directory is left as it is; an unknown directory and one that
    cannot be created warn without raising; create_py_executor makes the directory before the
    engine is built.

  • Reproduction on GPUs. XDG_CACHE_HOME exported to a path that does not exist,
    /tmp/jitify2_cache empty, one container. Start an engine of the Qwen3.8-Flash-Next NVFP4
    Modeling V2 target without draft tokens (test_tp1_gsm8k), then one with one draft token
    (test_tp1_mtp1_gsm8k_mmmu_and_video):

    Tree The engine with one draft token test_tp1_mtp1_gsm8k_mmmu_and_video after test_tp1_gsm8k
    main b2f659aef8 loads the three ...PagedKvDenseP32{,MultiCtasKv,MultiCtasKvCga}VarSeqQ16Kv256StaticSwapsAbForGen kernels from /tmp/jitify2_cache (3.6 to 4.8 s each), logs the warning twice, answers The!!!!... fails: ('red_rightward', 'the square is red!!!!!!...')
    b2f659aef8 + this change compiles its own (6.5 to 7.0 s each), no warning, answers right; /tmp/jitify2_cache is neither read nor written passes

    Run on GB300 (sm_103) at upstream main b2f659aef8 (2026-11-10), with the change
    cherry-picked. Not run at the commit this pull request is based on: the prebuilt kernel
    generator archives under cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/ are identical
    between the two commits.

  • The l0_cpu.yml entry unittest/_torch/pyexecutor with -m cpu_only, started through
    test_unittests.py, at b2f659aef8 + this change: the five new cases pass. The session has
    three failures there
    (kv_cache/test_kv_cache_v2_vocab_size.py::test_branch_only_enables_multimodal_keys_when_needed),
    the same three as at b2f659aef8 without the change; main changed that test in [TRTLLM-16958][feat] Executor host path and hybrid GDN/Mamba perf improvements for RL #20916.

  • More than one rank, on sm_107, on a tree that includes [None][feat] Qwen3.8-Flash-Next FP8 modeling_v2: one target for TP1/TP2/TEP2/DEP2/TP4/TEP4/DEP4 on GB300 #21621 (its Qwen3.8-Flash-Next FP8
    target; so not plain main): engines without draft tokens first, then engines with draft
    tokens at four ranks, same harness. Without the change the engines with draft tokens log
    the warning; with it none does and /tmp/jitify2_cache stays empty.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…me kernels are compiled

The FMHA kernels that are compiled at run time are cached by name. When the
kernel generator does not find the user cache directory ($XDG_CACHE_HOME, else
$HOME/.cache) it keeps the compiled kernels in /tmp/jitify2_cache, which every
process of the host shares. A generation kernel's name does not say whether it
was built for one query token per sequence or for several, so an engine with
draft tokens that starts after an engine without them loads the single-token
kernel from that directory: NaN in its KV cache during warm-up, and token id 0
from small-batch decode steps for the life of the engine.

With the user cache directory in existence the generator keeps its kernels in
the process, as it does in an ordinary environment. The directory is missing
when XDG_CACHE_HOME names a path nobody created, which test harnesses do to
isolate runs. Create it when an executor is created, before the engine is
built; warn and go on when it is unknown or cannot be created.

This does not change how a kernel is named or cached. Two builds of one kernel
name can still meet inside one process, and in a shared kernel cache where the
directory cannot be created.

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant