Skip to content

[Example] Add MemAgent long-context RL example (mem_agent) - #291

Merged
aoshen02 merged 2 commits into
vllm-project:mainfrom
CalvinXKY:examples/mem-agent
Jun 30, 2026
Merged

aoshen02 merged 2 commits into
vllm-project:mainfrom
CalvinXKY:examples/mem-agent

Conversation

@CalvinXKY

@CalvinXKY CalvinXKY commented Jun 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add `examples/mem_agent`, a vime port of MemAgent for long-context QA with RL:

  • Multi-turn rollout: chunk-wise memory update + final `\boxed{}` answer (HotpotQA reward)
  • Training: GRPO on Qwen3-4B via vLLM colocate (`run-qwen3-4b-train.sh`)
  • Evaluation: RULER-HQA offline eval via vLLM serve (`run-eval.sh`, `eval_ruler_hqa.py`)
  • Data prep: HuggingFace dataset `BytedTsinghua-SIA/hotpotqa` -> training JSONL + eval JSON download
    Also updates `examples/README.md` with a directory entry.

Files

File Description
`rollout.py` / `rollout_client.py` Multi-turn MemAgent rollout + vLLM router client
`custom_convert.py` Unroll trajectories for GRPO
`prepare_data.py` / `prepare-eval-data.sh` Training and eval data preparation
`eval_ruler_hqa.py` / `run-eval.sh` RULER-HQA evaluation
`run-qwen3-4b-train.sh` / `_common.sh` GRPO training launch scripts
`convert-to-hf.sh` Megatron checkpoint -> HuggingFace
See `examples/mem_agent/README.md` for quick start (data download, training, eval).

Test plan

  • Pre-commit passed (ruff, isort, black)
  • GRPO training smoke on H800 (Qwen3-4B, colocate vLLM)
  • RULER-HQA eval at n_docs = 50 / 200 / 800 (RL checkpoint vs base Qwen3-4B)
  • CI (await upstream checks)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the MemAgent long-context RL workflow, adding scripts and utilities for data preparation, multi-turn rollout generation, GRPO training, and RULER-HQA evaluation. The code review highlighted several critical improvements: skipping failed or aborted samples in custom_convert.py to prevent crashes, ensuring log probabilities are always appended to avoid list misalignment, and checking the status of the background vLLM process in run-eval.sh to prevent infinite loops. Additionally, feedback was provided to optimize performance by moving numpy imports to the module level in prepare_data.py, preserving case-sensitivity when extracting boxed answers in rollout.py and eval_ruler_hqa.py, and double-quoting bash array expansions in _common.sh to prevent word splitting.

Comment on lines +46 to +47
for i, sample in enumerate(samples):
meta = sample.train_metadata

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Failed or aborted samples should be skipped during conversion. If a rollout fails (e.g., due to network timeouts or OOM), its tokens and other attributes may be None or empty, which will cause training crashes or corrupt the training data if appended.

    for i, sample in enumerate(samples):
        if sample.status in (sample.status.FAILED, sample.status.ABORTED):
            continue
        meta = sample.train_metadata

Comment thread examples/mem_agent/custom_convert.py Outdated
Comment on lines +59 to +61
if sample.rollout_log_probs is not None:
rollout_log_probs_list.append(sample.rollout_log_probs)
has_rollout_log_probs = True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If some samples do not have log probs, skipping the append operation will cause rollout_log_probs_list to become shorter than tokens_list, leading to silent misalignment of log probs with tokens or shape mismatch crashes during training. Always append to the list, and check for presence to set has_rollout_log_probs.

Suggested change
if sample.rollout_log_probs is not None:
rollout_log_probs_list.append(sample.rollout_log_probs)
has_rollout_log_probs = True
rollout_log_probs_list.append(sample.rollout_log_probs)
if sample.rollout_log_probs is not None:
has_rollout_log_probs = True

Comment thread examples/mem_agent/custom_convert.py Outdated
Comment on lines +79 to +81
if turn.get("rollout_log_probs"):
rollout_log_probs_list.append(turn["rollout_log_probs"])
has_rollout_log_probs = True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Similarly to the sample-level log probs, always append the turn-level log probs to prevent list misalignment when some turns lack log probs.

Suggested change
if turn.get("rollout_log_probs"):
rollout_log_probs_list.append(turn["rollout_log_probs"])
has_rollout_log_probs = True
rollout_log_probs_list.append(turn.get("rollout_log_probs"))
if turn.get("rollout_log_probs"):
has_rollout_log_probs = True

Comment on lines +75 to +87
while true; do
resp=$(curl -sf --max-time 10 "${url}" 2>/dev/null || true)
if echo "${resp}" | grep -Fq "${MODEL_NAME}" 2>/dev/null; then
log "vLLM ready."
break
fi
attempts=$((attempts + 1))
if (( attempts % 6 == 0 )); then
found=$(echo "${resp}" | grep -o '"id":"[^"]*"' 2>/dev/null | head -3 || echo "(no response)")
log "Still waiting... models: ${found}"
fi
sleep 5
done

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

wait_for_server will loop infinitely if the background vllm serve process dies during startup (e.g., due to port conflicts or OOM). Checking if VLLM_PID is still running inside the loop prevents infinite hangs in automated pipelines.

Suggested change
while true; do
resp=$(curl -sf --max-time 10 "${url}" 2>/dev/null || true)
if echo "${resp}" | grep -Fq "${MODEL_NAME}" 2>/dev/null; then
log "vLLM ready."
break
fi
attempts=$((attempts + 1))
if (( attempts % 6 == 0 )); then
found=$(echo "${resp}" | grep -o '"id":"[^"]*"' 2>/dev/null | head -3 || echo "(no response)")
log "Still waiting... models: ${found}"
fi
sleep 5
done
while true; do
if ! kill -0 "${VLLM_PID}" 2>/dev/null; then
log "ERROR: vLLM process (PID ${VLLM_PID}) died."
exit 1
fi
resp=$(curl -sf --max-time 10 "${url}" 2>/dev/null || true)
if echo "${resp}" | grep -Fq "${MODEL_NAME}" 2>/dev/null; then
log "vLLM ready."
break
fi
attempts=$((attempts + 1))
if (( attempts % 6 == 0 )); then
found=$(echo "${resp}" | grep -o '"id":"[^"]*"' 2>/dev/null | head -3 || echo "(no response)")
log "Still waiting... models: ${found}"
fi
sleep 5
done

Comment thread examples/mem_agent/prepare_data.py Outdated
Comment on lines +42 to +57
def _to_json_safe(obj):
try:
import numpy as np
except ImportError:
np = None

if np is not None:
if isinstance(obj, np.ndarray):
return obj.tolist()
if isinstance(obj, (np.integer, np.floating, np.bool_)):
return obj.item()
if isinstance(obj, dict):
return {k: _to_json_safe(v) for k, v in obj.items()}
if isinstance(obj, list):
return [_to_json_safe(v) for v in obj]
return obj

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Importing numpy inside _to_json_safe is highly inefficient because this function is called for every single row in the dataset. Moving the import and check to the module level avoids this overhead.

Suggested change
def _to_json_safe(obj):
try:
import numpy as np
except ImportError:
np = None
if np is not None:
if isinstance(obj, np.ndarray):
return obj.tolist()
if isinstance(obj, (np.integer, np.floating, np.bool_)):
return obj.item()
if isinstance(obj, dict):
return {k: _to_json_safe(v) for k, v in obj.items()}
if isinstance(obj, list):
return [_to_json_safe(v) for v in obj]
return obj
try:
import numpy as np
except ImportError:
np = None
def _to_json_safe(obj):
if np is not None:
if isinstance(obj, np.ndarray):
return obj.tolist()
if isinstance(obj, (np.integer, np.floating, np.bool_)):
return obj.item()
if isinstance(obj, dict):
return {k: _to_json_safe(v) for k, v in obj.items()}
if isinstance(obj, list):
return [_to_json_safe(v) for v in obj]
return obj

Comment thread examples/mem_agent/rollout.py Outdated
Comment on lines +247 to +259
solution_str = final_output[-300:].lower()
pred = _extract_boxed(solution_str) or ""

score = 0.0
for gt in ground_truth:
gt_lower = gt.lower()
try:
boxed = _last_boxed_only_string(solution_str)
if boxed is not None:
answer = _remove_boxed(boxed)
if _is_equiv(answer, gt_lower):
score = 1.0
break

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Lowercasing the entire solution_str before extracting the boxed answer can corrupt case-sensitive LaTeX commands or case-sensitive answers. It is safer to extract the boxed answer from the original case string, and only lowercase it during the final comparison.

Suggested change
solution_str = final_output[-300:].lower()
pred = _extract_boxed(solution_str) or ""
score = 0.0
for gt in ground_truth:
gt_lower = gt.lower()
try:
boxed = _last_boxed_only_string(solution_str)
if boxed is not None:
answer = _remove_boxed(boxed)
if _is_equiv(answer, gt_lower):
score = 1.0
break
solution_str = final_output[-300:]
pred = _extract_boxed(solution_str) or ""
score = 0.0
for gt in ground_truth:
gt_lower = gt.lower()
try:
boxed = _last_boxed_only_string(solution_str)
if boxed is not None:
answer = _remove_boxed(boxed)
if _is_equiv(answer.lower(), gt_lower):
score = 1.0
break

Comment thread examples/mem_agent/eval_ruler_hqa.py Outdated
return None

gold = item["answers"][0] if item.get("answers") else ""
pred = _extract_boxed(response[-300:].lower()) or ""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Lowercasing the response before calling _extract_boxed is redundant and potentially harmful for case-sensitive patterns, especially since the metric functions (_normalize_answer) handle lowercasing themselves.

Suggested change
pred = _extract_boxed(response[-300:].lower()) or ""
pred = _extract_boxed(response[-300:]) or ""

Comment thread examples/mem_agent/_common.sh Outdated
if [[ "${RUN_TRAIN_DIRECT:-0}" == "1" && -n "${RUN_TRAIN_DIRECT_PY:-}" && -f "${RUN_TRAIN_DIRECT_PY}" ]]; then
export VIME_ROOT NCCL_NVLS_ENABLE MEM_CHUNK_TOKENS MEM_MAX_MEMORY MEM_MAX_FINAL MEM_MAX_CHUNKS
python3 "${RUN_TRAIN_DIRECT_PY}" \
${MODEL_ARGS[@]} \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In bash, expanding arrays without double quotes can cause word splitting if any element contains spaces. It is a standard best practice to always double-quote array expansions: "${MODEL_ARGS[@]}".

Suggested change
${MODEL_ARGS[@]} \
"${MODEL_ARGS[@]}" \

Comment thread examples/mem_agent/_common.sh Outdated
--runtime-env-json="${runtime_env}" \
-- python3 train.py \
--train-backend megatron \
${MODEL_ARGS[@]} \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Double-quote the array expansion to prevent word splitting.

Suggested change
${MODEL_ARGS[@]} \
"${MODEL_ARGS[@]}" \

@CalvinXKY

Copy link
Copy Markdown
Collaborator Author

Test result:

image image

Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>
Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>
@CalvinXKY
CalvinXKY force-pushed the examples/mem-agent branch from 49749f7 to 0747f8e Compare June 30, 2026 07:10
@aoshen02
aoshen02 merged commit d679c75 into vllm-project:main Jun 30, 2026
3 checks passed
aoshen02 pushed a commit to aoshen02/vime that referenced this pull request Jul 15, 2026
…ect#291)

* Add examples/mem_agent for long-context MemAgent RL

Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>

* Address review: harden mem_agent convert and eval scripts

Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>

---------

Signed-off-by: kaiyuan <kyxiezju@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
CalvinXKY added a commit that referenced this pull request Sep 27, 2026
* refactor(test): flatten CPU unit tests and centralize import stubs (#238)

* refactor(test): align CPU unit test layout with slime and add _unit_stubs

Move tests/unit/* into tests/ and tests/utils/, replace nested conftest stubs with shared helpers for optional deps, and register the moved files in cpu-unittest CI. Write generated workflow YAML as UTF-8 on Windows.

Signed-off-by: kaiyuan <kyxiezju@163.com>

* fix(vllm_engine): guard HTTPError.add_note for Python 3.10 CI

Exception.add_note() requires Python 3.11+; cpu-unittest runs on 3.10.

Signed-off-by: kaiyuan <kyxiezju@163.com>

* refactor(test): drop pytest pythonpath, bootstrap tests/ in test files

Remove redundant pythonpath from pyproject.toml; each test that imports
_unit_stubs prepends tests/ to sys.path (including test_vllm_rollout.py).

Signed-off-by: kaiyuan <kyxiezju@163.com>

* ci: bump cpu-unittest to Python 3.11 and revert add_note guard

Run cpu-unittest on 3.11 so HTTPError.add_note is available in CI without Py3.10 guards in vllm_engine or conditional __notes__ assertions in tests.

Signed-off-by: kaiyuan <kyxiezju@163.com>

---------

Signed-off-by: kaiyuan <kyxiezju@163.com>

* docs: align vLLM docs with current defaults (#240)

* docs: align vllm docs with current defaults

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: fix router policy examples

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: simplify ray multi-node wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore quick start hardware notes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: clarify cache-aware routing support

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove extra router policy example

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore cache aware policy example

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore hardware support wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove conda fallback note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: update docs logo

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: replace legacy logo assets

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: add README docs badges

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: replace logo image

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove redundant logo jpg

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: adjust hardware support wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: refine hardware support wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore hardware support notes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: add GB support guard notes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: clarify profiling flow

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: shorten profiling note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* tools: simplify rollout profiler helper

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore profiling request range

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove profiling note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: switch logo asset to jpg

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* update docs logo

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* update docs logo asset

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: move eplb note to faq

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: rephrase profiling wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove eplb note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore eplb example

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore eplb example in english

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>

docs: align vLLM docs with current defaults
CI: merged with buildkite/vime-ci failure (infra); GA checks passed. See PR #240 comment.

* refactor(weight-sync): use vLLM native /update_weights, remove worker extension (#246)

* refactor(weight-sync): use vLLM native /update_weights instead of worker extension

Remove the custom `vLLMColocateWorkerExtension` and `_VLLMHijack`
monkey-patch introduced in PR #104. These reimplemented vLLM's built-in
`IPCWeightTransferEngine.receive_weights()` (UUID routing, device_index
remapping, layerwise reload) inside a worker extension, bypassing the
native `/update_weights` endpoint in favor of `/collective_rpc`.

vLLM's native IPC engine already handles everything the extension did.
This change routes `update_weights_from_tensor` back through
`POST /update_weights` (the path that was working before PR #104) and
deletes ~160 lines of redundant code.

Changes:
- `update_weight_from_tensor.py`: delete `_VLLMHijack`,
  `vLLMColocateWorkerExtension`, and fix `ipc_handles` format to store
  `ipc_args` (not `(rebuild_func, ipc_args)`) for native API compat
- `vllm_engine.py`: revert `update_weights_from_tensor` from
  `/collective_rpc` to `/update_weights`, delete `update_weights_chunk`,
  remove `--worker-extension-cls` injection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(reloadable-pg): add missing gather_object patch; switch IPC gather to match slime

ReloadableProcessGroup monkey-patched all_gather_object but not
gather_object, causing "Group is not registered" after Megatron
offload/reload. This forced _send_to_colocated_engine to use the
less efficient all_gather_object (all ranks receive, only leader uses).

Fix: patch gather_object in the same list, then switch to
dist.gather_object (only leader allocates the receive buffer) to
align with slime's implementation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add Buildkite CI pipeline (CPU jobs + manual-gated GPU suites) and remove PR test on GHA (#239)

* ci: add Buildkite pipeline for always-on CPU jobs

Port the always-on jobs from .github/workflows/pr-test.yml.j2 (pre-commit
gate, plugin contracts, agent adapter, in-image unit tests) to a single
dynamically generated Buildkite pipeline targeting the vLLM elastic-stack
CPU queues. GitHub Actions keeps running in parallel and stays
authoritative; GPU suites are not migrated yet.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: replace Buildkite generator with static pipeline.yml

Drop generate_pipeline.py in favor of a plain static .buildkite/pipeline.yml
defining the four always-on CPU steps directly (pre-commit gate, plugin
contracts, agent adapter, in-image unit tests). Simpler to read and review for
a first cut; the GHA workflow stays authoritative and GPU suites are still out
of scope.

Pass GIT_CONFIG_PARAMETERS into every container so git (in pre-commit) doesn't
abort with "dubious ownership" on the host-owned checkout, and fix the
depends_on typo in the README.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): fix CPU test deps; add manual gate for GPU suites

plugin-contracts failed in build #3 on tests/utils/test_hf_checkpoint_saver.py
(ModuleNotFoundError: safetensors): the dep list predated the slime sync in
#232 which added requests/ray/safetensors to the GHA template. Mirror it.

GitHub PR labels can't trigger Buildkite jobs, so expose the run-ci-* GPU
suites behind a block step instead: unblocking offers a multi-select of suites
(short / vllm-config / megatron / precision / ckpt) and gpu_suites.py uploads
one step per test with the same gpu_lock_exec + docker invocations and
per-test DEEPEP/FP8/EVAL env combos as the GHA jobs. blocked_state: passed
keeps the commit status green when the gate is left untouched. GPU steps
target a new vime-gpu agent queue (self-hosted hosts; see README).

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): run GPU suites on mithril-h100-pool; pin gloo to loopback

Build #4's unblock test showed the CI cluster rejects uploads targeting a
nonexistent queue, and rather than minting a new queue, follow the pattern
vllm-omni already uses for mithril-h100-pool: each GPU job is a Kubernetes pod
(agent-stack-k8s kubernetes plugin) on an H100 SXM node with nvidia.com/gpu
limits (4 or 8), memory-backed /dev/shm, and /mnt/hf-cache mounted as HF_HOME.
vime tests hf-download their models, so the warm HF cache replaces the GHA
runners' /mnt/nvme0n1/vime_ci mounts; the docker-run wrapper goes away since
the pod runs the vime CI image directly.

Also pin GLOO/TP_SOCKET_IFNAME=lo in the plugin-contracts container:
test_metric_report_dist hung intermittently (build #4 timed out at 30 min)
because gloo can pick a non-loopback interface inside a bridge-network
container.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): expandable_segments for the borderline OOM short test

test_qwen3.5_0.8B_gsm8k_async_short OOMed in compute_log_probs on the mithril
pool's 80 GB H100s (build #6) with 7 GiB reserved-but-unallocated — the
allocator-fragmentation case PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
exists for. Scope it to this test's pod only (vLLM sleep-mode CuMemAllocator
can conflict with expandable segments) via verbatim pass-through of non-VIME
env overrides. The other short tests passed on H100 pods unchanged.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): soft-fail the two known-H100-incompatible GPU tests

Builds #6/#7 isolated two test-level failures on the mithril 80 GB H100s,
neither a pipeline issue:
- gsm8k_async_short OOMs as tuned (67 GiB live on the actor GPU after
  expandable_segments eliminated fragmentation; its sync twin passes).
- parallel_check's CP=2 grad norm diverges ~4% from the same-node baseline
  recording, a topology-sensitive numerical invariance question.

Mark exactly these two soft_fail so they keep running and stay visible on
Buildkite without failing the build; their authoritative gate remains the
GHA label jobs.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): keep the two H100-incompatible GPU tests failing loudly

Revert the soft_fail: per review, the gsm8k_async_short OOM and the
parallel_check CP-invariance divergence should stay visible as hard failures
on Buildkite until the underlying issues are fixed. Keep the diagnostic
comments and the test-scoped expandable_segments setting.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): soft-fail the two H100-incompatible GPU tests after all

Re-apply b334784 (reverted in 0a98010): per the follow-up decision, mark
gsm8k_async_short and parallel_check soft_fail so they keep running visibly
on mithril without failing the build, with the GHA label jobs as their
authoritative gate until the OOM tuning and CP-invariance questions are
resolved.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): resolve 0.8B async OOM on H100 by reducing max-tokens-per-gpu

Root cause: Qwen3.5's 248K vocab produces [T, 248320] fp32 logits tensors.
calculate_log_probs_and_entropy holds 5 copies simultaneously (2 clones +
2 intermediates + original). At max-tokens-per-gpu=9216, each copy is
~8.5 GB → 42.6 GB from logits alone, exceeding H100 80 GB with
activations and reserved pool fragmentation.

Fix: reduce max-tokens-per-gpu from 9216 to 2048. Peak drops from 117.6 GB
to 39.6 GB (measured on H200), well within H100's 80 GB. GSM8K's longest
sequence is ~1200 tokens, so 2048 still fits all samples.

Also removes gsm8k_async_short from SOFT_FAIL_ON_H100 (no longer needed)
and the expandable_segments workaround.

parallel_check remains soft-fail: ~11% flake rate on TP4+per-token-loss,
confirmed same behavior in slime (Megatron FP reduction-order issue).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style: format update_weight_from_tensor

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* remove github workflows

Signed-off-by: khluu <khluu000@gmail.com>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>

* [codex] update docs branding and fix Buildkite CPU tests (#249)

* docs: update docs favicon to Vime branding

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: fix unit test path in buildkite

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: force triton stub in CPU unit tests

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: align update_weights_from_tensor with native vllm

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: isolate deep_gemm and triton in cpu utils

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(arguments): include DP in TP auto-compute default (#252)

* fix(arguments): include DP in TP auto-compute default

The `vllm_parse_args` default for `vllm_tensor_parallel_size` was
computed as `rollout_num_gpus_per_engine // pp_size`, missing the
`dp_size` divisor. This made the default TP too large when
`--vllm-data-parallel-size > 1` (required for expert parallelism).

Align the formula with `_resolve_vllm_parallel_sizes` in
`vllm_engine.py`, which correctly uses `gpus // (pp * dp)`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test(arguments): add DP>1 cases for TP auto-compute

Cover the fix: TP = gpus_per_engine // (PP * DP).

- test_parse_args_tp_default_with_dp: 8 GPU, DP=4 → TP=2
- test_parse_args_tp_default_with_pp_and_dp: 8 GPU, PP=2, DP=2 → TP=2
- test_parse_args_tp_default_dp1_unchanged: DP=1 regression guard

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* Update arguments.py

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* chore: rerun Buildkite

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [Example] Add tau-bench multi-turn tool-use example (#142)

* feat(tau-bench): add vLLM multi-turn GRPO example with trainable agent

Introduce generate_with_tau custom rollout (render + /inference/v1/generate), trainable_agents for tau-bench env interaction, vllm_tool_parser, and TAU_CONFIGS-based user simulator configuration in generate_with_tau.py.

Signed-off-by: xky <kyxiezju@163.com>

* style(tau-bench): apply pre-commit formatting

Signed-off-by: xky <kyxiezju@163.com>

* ci: retrigger buildkite

Signed-off-by: xky <kyxiezju@163.com>

* ci: retrigger buildkite

Signed-off-by: xky <kyxiezju@163.com>

---------

Signed-off-by: xky <kyxiezju@163.com>

* [codex] drop stale unit CPU CI and tests (#268)

* fix buildkite cpu utils ci

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove gha references from buildkite ci

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* deps: add transformers and cloudpickle

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: stub optional utils deps in cpu ci

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: keep vllm http error notes py310 compatible

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: run cpu jobs on python 3.11

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: inline utils stubs in affected files

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docker: upgrade base to vLLM 0.23.0, remove CUDA 13 build path (#253)

* docker: upgrade base to vLLM 0.23.0, remove CUDA 13 build path

- Base image: v0.22.0-cu129-ubuntu2404 → v0.23.0-cu129-ubuntu2404
- Remove `ENABLE_CUDA_13` ARG and all conditional cu13 blocks:
  - cu13 apt dev headers (libcublas-dev-13-0, cuda-nvrtc-dev-13-0, etc.)
  - TE source build (cu13 wheel didn't exist; cu129 wheel works on arm64)
  - fzyzcjy triton source build (cu13 specific)
  - TMS_CUDA_MAJOR export (no longer needed)
- Simplify cublas-dev to unconditional libcublas-dev-12-9
- Simplify TE install to wheel-only
- justfile: remove `build-cu13` target and cu13 tag scheme
- vllm.patch: adapt line numbers for 0.23.0 (776/1896 vs 750/1844),
  preserve `with self.log_iteration_details(None):` wrapper

cu129 nvcc already supports sm100/sm120 (Blackwell), so cu13 build
path was unnecessary — it caused build failures on gb300 (cu13 apt
packages hijacked /etc/alternatives/cuda, breaking TE CMake).

Tested: built successfully on gb200 (arm64), h200 (x86), gb300 (arm64).
All three confirmed vLLM 0.23.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docker/patch: replace core.py sleep fix (already in v023) with all2all_utils weight-reload fix (#45989)

v0.23.0 already includes the sleep/scheduler guard from #44483, so the
core.py patch is no longer needed. Replace it with the FP8+DeepEP
weight-reload fix (vllm-project/vllm#45989): snapshot
max_num_batched_tokens from FusedMoEConfig instead of calling
get_current_vllm_config() during layerwise reload.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(vllm_engine): in-process launch + dataclass field introspection (#264)

* fix(vllm_engine): drop router_ip/port fallback to args in init()

PD mode passes router_ip=None to engine.init() intentionally — the
router launches AFTER engines are ready and collects their URLs. The
fallback `self.args.vllm_router_ip` reads a pre-allocated-but-not-yet-
listening address, causing ConnectionRefused on POST /workers.

Drop the fallback; assign directly like main does.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(arguments): unblock 6 skipped params for --vllm-* forwarding

skipped_args in arguments.py blocked 6 params that should be
user-configurable via --vllm-* CLI flags:
  - served_model_name, tokenizer, tokenizer_mode, tokenizer_revision
  - dtype (only hardcoded when --fp16, otherwise user should set)
  - tool_call_parser (was manually registered then skipped; let
    monkey-patch auto-register like slime does)

Also removes the redundant manual --vllm-tool-call-parser registration
since the monkey-patch now handles it automatically.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: fix Qwen3-4B download repo (#276)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Liron Kesem <52330564+LironKesem@users.noreply.github.com>

* test: fix CPU unit test failures introduced by PR #264 (#284)

PR #264 moved top-level and lazy vllm imports that break the bare
python:3.11 CPU CI image (no vllm installed).

Three changes — no production code touched:

1. `tests/_unit_stubs.py`: extend `install_vllm_cli_stubs()` with stubs
   for `vllm.utils.system_utils` (top-level import in vllm_engine.py)
   and the `vllm.entrypoints.*` hierarchy (lazy imports in arguments.py
   and vllm_engine._vllm_server_field_names).

2. `tests/utils/test_vllm_engine.py`: add an autouse fixture that patches
   `_VLLM_SERVER_FIELDS` to an empty frozenset, so tests calling
   `_compute_server_args` without an explicit monkeypatch don't trigger
   the real `_vllm_server_field_names()`.

3. `tests/utils/test_vllm_arguments.py`: evaluate `real_module_available`
   before calling `install_vllm_cli_stubs()` (which registers a fake vllm
   in sys.modules), then mark two tests that require real FrontendArgs
   field names with `@requires_vllm` so they skip on the CPU image.

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(data): reuse stored multimodal_inputs in length filter (#257)

* fix(data): reuse stored multimodal_inputs in length filter

filter_long_prompt re-extracted vision info from sample.prompt via
process_vision_info in the multimodal branch. When apply_chat_template
is set, sample.prompt is the rendered *string* (not a conversation
list), so process_vision_info -> qwen_vl_utils crashed with
"TypeError: string indices must be integers, not 'str'".

This made prompt-length filtering unusable for any VLM dataset: setting
--rollout-max-context-len (which derives rollout_max_prompt_len) or
--rollout-max-prompt-len / --eval-max-prompt-len activates the filter
and hits the crash.

Reuse the multimodal inputs already computed during dataset
construction (sample.multimodal_inputs) instead of recomputing them
from the string prompt.

Add CPU unit tests covering the multimodal branch and a mixed
text-only + multimodal dataset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>

* Delete tests/test_filter_long_prompt_multimodal.py

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* Update data.py

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>

* docker: rename published image from inferactinc/public to vllm/vime (#283)

Replace all references to `inferactinc/public:vime-*` with
`vllm/vime:*` across CI, docs, and the release justfile.

The image is now published under the official vllm DockerHub
namespace (`vllm/vime:latest`, `vllm/vime:test-latest`) as a
multi-arch manifest (amd64 + arm64).

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(examples/tau-bench): use RunConfig.agent_strategy in TAU_CONFIGS (#280)

tau-bench RunConfig defines agent_strategy, not agent. The old key was silently ignored by Pydantic, and resolve_tau_config accessed tau_config.agent which raises AttributeError at rollout time.

Signed-off-by: kaiyuan <kyxiezju@163.com>

* scripts: complete slime-exact port of most scripts except for gpt-oss 20B support (#260)

* restore: bring back deleted examples, scripts, and agent doc

Restore files that were either deleted by #126 ("trim examples to
qwen3 only") or never synced from slime:

**Reverted from pre-#126 (translated):**
- scripts/low_precision/run-qwen3-4b-fp8.sh
- scripts/low_precision/run-qwen3-30b-a3b-fp8.sh
- scripts/run-glm4-9B.sh
- scripts/run-moonlight-16B-A3B.sh
- scripts/run-qwen3-4B-base-sft.sh
- scripts/run-qwen3-32B.sh
- scripts/run-qwen3.5-35B-A3B-sft.sh

**New from slime@44d29ee (translated):**
- docs/en/get_started/agent.md
- examples/fully_async/run-qwen2.5-0.5B-fully_async.sh

All sglang engine flags translated to vllm equivalents (§2.4).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: unify pkill pattern to '[v]llm serve|VLL[M]::'

Standardize all scripts to use the bracket-escaped pkill pattern that
avoids matching pkill itself and also catches vLLM's renamed
subprocesses (VLLM::EngineCore, VLLM::Worker_TP*). Matches the
canonical pattern in command_utils.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* scripts: complete slime-exact translation of all 29 run scripts

Translate all slime scripts to vime following SGLANG_TO_VLLM_TRANSLATION.md:
- sglang→vllm prefix swap for CLI flags and variables
- _slime→_vime for checkpoint paths
- EP: --sglang-ep-size N → --vllm-enable-expert-parallel (boolean)
- Speculative: multi-param → --vllm-speculative-config JSON (§5.2)
- Delete genuinely sglang-coupled params (DP-attention, DeepEP, NSA, etc.)
- flashinfer → FLASHINFER case fix (§2.4)

23 new scripts + 6 existing updated to match slime@cutoff.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(scripts): correct model config source path in FP8 low_precision scripts

The FP8 scripts used `${SCRIPT_DIR}/../scripts/models/` which resolves
to `scripts/scripts/models/` (non-existent). Changed to `../models/`
to match the INT4 scripts. Same fix as slime PR #2094.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gpt-oss): fused BF16 format, bridge API patch, bshd qkv format

Three fixes needed to run GPT-OSS 20B RLHF on vLLM backend:

1. hf_weight_iterator_bridge: match Megatron-Bridge 0.5.0 API
   _patch_bridge_expert_cache_to_cpu monkey-patches GPTOSSBridge.
   maybe_modify_converted_hf_weight gained a 4th `hf_state_dict`
   parameter; the patched wrapper only accepted 3, causing TypeError
   during weight sync.

2. run-gpt-oss-20B: point --hf-checkpoint at fused BF16 format
   vLLM's _load_weights_other expects gate_up_proj [E, hidden, 2*ffn]
   (fused). The old per-expert split format (experts.{e}.gate_proj.weight)
   causes KeyError on bias loading. Use tools/convert_gpt_oss_to_fused.py
   to convert an existing per-expert checkpoint, or re-run
   preprocess_gpt_oss.py to produce fused format directly.

3. run-gpt-oss-20B: add --qkv-format bshd + fix seq-length
   GPT-OSS uses learnable softmax (sink attention). TransformerEngine
   disables all attention backends when softmax_type=learnable and
   qkv_format=thd (packed sequences). --qkv-format bshd avoids this.
   --use-dynamic-batch-size is incompatible with bshd; replaced with
   fixed --seq-length 10240 (covers 8192 max response + prompt headroom).

tools/convert_gpt_oss_to_fused.py: new tool to convert per-expert BF16
checkpoint (output of old preprocess_gpt_oss.py) to the fused HF format
expected by vLLM without re-running the slow MXFP4 dequantization.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(scripts): replace pkill -9 vllm with precise -f pattern (21 files)

pkill -9 vllm matches any process named "vllm" and can inadvertently
kill unrelated vllm processes (e.g. background services). Use the same
pattern as PR #220 which targets only vllm serve and Ray VLL[M]:: actors:

  pkill -9 -f '[v]llm serve|VLL[M]::'

Also updates the inline form used in multi-node SSH worker restart
commands (run-qwen3-235B-A22B*.sh, run-qwen3.5-27B.sh, etc.).

Skipped: scripts/run-gpt-oss-20B.sh (uses pkill -9 -f "vllm serve" already),
scripts/run-minimax-m2.sh and run-glm4.7-*.sh (already used -f "vllm serve").

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* chore(scripts): remove run-qwen3-4B-amd.sh from this PR

AMD-specific script is out of scope for the gb300-complete-port PR.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* sync(docs+scripts): port docs/examples from slime-44d29ee, fix script translations

- Add missing EN/ZH docs: low-precision, on-policy-distillation, get_started/agent,
  pd-disaggregation (heterogeneous server groups fix), examples zh docs
- Add missing examples: on_policy_distillation, eval_multi_task, delta_weight_sync,
  geo3k images
- Fix vLLM flag translations across all example docs:
  - --vllm-mem-fraction-static → --vllm-gpu-memory-utilization
  - Remove non-existent dp-attention flags (--vllm-enable-dp-attention, --vllm-dp-size,
    --vllm-moe-dense-tp-size, --vllm-enable-dp-lm-head, --vllm-ep-size)
  - --vllm-ep-num-redundant-experts → --vllm-eplb-config
  - --vllm-cuda-graph-bs → --vllm-max-cudagraph-capture-size
  - sglang speculative flags → --vllm-speculative-config JSON
  - GLM-4.7 MTP: method=eagle → method=mtp, num_speculative_tokens=4 → 3
  - sgl-router → vllm-router; THUDM/vime → vllm-project/vime
- Fix scripts: restore run-kimi-k2-Instruct/Thinking/qwen3-4B/qwen3-235B-A22B to
  slime-44d29ee-as-vime + pkill precision fix only; restore int4 python3 path

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* revert(scripts): pkill -9 -f pattern back to pkill -9 vllm, align with slime

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* revert(pkill): align all remaining kill patterns with slime (pkill -9 vllm)

Covers examples/, docs/, tests/, and vime/utils -- previously missed in
the scripts/ revert.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* chore: remove gpt-oss-20B script and convert tool (moved to separate PR)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [Doc] Fix broken Qwen3-4B example link in rollout_buffer README (#295)

The rollout_buffer README (English and Chinese) linked to
docs/{en,zh}/models/qwen3-4B.md, but there is no models/ directory —
the referenced doc lives under examples/. Point both links to
docs/{en,zh}/examples/qwen3-4B.md so the setup instructions resolve.

Signed-off-by: ajinkya-metica <ajinkya@metica.com>
Co-authored-by: ajinkya-metica <ajinkya@metica.com>

* sync(slime #2014..#2125): 3-way merge [WIP — 45 conflict files need review] (#286)

* sync(slime #2014..#2125): diff3 3-way merge, conflicts preserved

Mechanical commit 1 of 2 (per knowledge/rl/slime-to-vime-sync-sop.md §2).
diff3 translated 3-way merge on upstream/main (incl #260/#280/#283/#257):
  ours = vime@main, base = translate(slime@#2013), theirs = translate(slime@#2125)

Translation fixes vs prior attempt:
  - casing: SGLang->vLLM (prose) / SGLang<X>->VLLM<X> (identifiers); killed VLlm artifact (was 35 files)
  - dotted module refs slime.X->vime.X now translated (was leaking 'from slime.backends')
These resolved 9 spurious conflicts (46->37 files).

Results: 61 clean / 37 conflict (diff3 markers preserved) / 50 new-to-vime / 5 del.
Conflict markers use readable -L labels (ours/base/theirs). Resolve in commit 2.

Non-conflict provenance fix: vimerl/vime -> vllm/vime in 2 example docs.
Engine patch handling (docker/patch/) deferred to commit 2 per SOP §4.5.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* sync(slime #2014..#2125): resolve all conflicts (commit 2)

Resolved all 37 conflict files / 84 diff3 blocks per agent_run RESOLUTION_POLICY.
Principle: keep vime vLLM impl (ours) + incorporate slime's new features (theirs).

Highlights:
- vLLM API form kept everywhere: /inference/v1/generate, choices parsing,
  AsyncEngineArgs, vLLM flag names (--vllm-gpu-memory-utilization etc).
- Dropped all vllm.srt.* imports (non-existent in real vLLM).
- Accepted new slime features: delta-weight-sync CLI args, append_response_tokens
  (Sample), get_server_info/start_external_rollout_servers/get_rollout_num_engines,
  old-router(<=0.2.1) compat, TrajectoryManager adapter design (vime already adopted it).
- Kept vime-only: --rollout-external, add_router_arguments, _get_metrics_router_addr,
  reinit_wandb_primary_with_open_metrics, update_tracking_open_metrics, modal sandbox,
  VIME_AGENT_* env names, local-vLLM tau-bench user sim.
- Engine patches (docker/patch/): kept ours vllm.patch (22-line MoE fix), dropped
  theirs sglang 2674-line content; deleted sglang-only vllm-top_p.patch (per SOP 4.5).
- Dockerfile kept ours (vllm/vllm-openai base); version.txt accepted theirs nightly.
- run-deepseek-r1.sh: dropped /sgl-workspace dead-path env.

Deviations from policy (documented): vllm_rollout.py abort path kept ours
pause/drain (abort_servers_until_idle would break partial-rollout drain + leave
paused_workers unbound). README ecosystem section left empty (ours) pending
de-translation of provenance.

All changed .py py_compile clean; zero conflict markers; no sglang/slime leakage.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): pre-commit lint/format on resolved files

- re-add dropped `from vllm_router.launch_router import RouterArgs` import in
  vime/utils/arguments.py (add_router_arguments uses it; F821 from conflict resolution)
- drop unused base_top_p_token_ids/offsets in vllm_streaming_rollout.py (F841; came
  from theirs but ours's choices-parsing path doesn't use them)
- black/isort autoformat (anthropic.py, test_agent/*, arguments.py)
- pipeline.yml: agent tests moved to tests/test_agent/; wire new CPU tests

pre-commit: all hooks pass (ruff/autoflake/isort/black/yaml).
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): make CPU CI green (engine import, docker build, agent/utils tests)

Local CPU CI (pre-commit + plugin + agent + utils) all green on 8xH200 host
in python:3.11 containers. Fixes found while running it:

- vllm_engine.py: make 'import vllm_router'/'packaging.parse' lazy (inside
  _register_to_router); top-level import broke CPU import (vllm_router absent in
  CPU CI). The old-router(<=0.2.1) compat branch is theirs-accepted.
- docker/Dockerfile: TMS_CUDA_MAJOR=12 for torch_memory_saver pin (its build
  backend now requires it for CUDA wheels; base is cu129). Unblocks image build.
- agent/adapters/common.py: _run_turn called parse_model_output() without the
  required tokenizer= kwarg -> 500s in adapter tests. Pass tokenizer=tok.
- tests/test_agent/_fakes.py: FakeVLLMServer served sglang /generate + meta_info;
  retarget to vime /inference/v1/generate + choices shape + x-session-id header.
- tests/test_agent/test_adapters.py: parse_model_output(tokenizer=...) + assert
  vime body keys (token_ids/max_tokens).
- tests/utils/test_vllm_config.py: vLLMConfig->VllmConfig (4 sites); fake router
  returns 3-tuple (ip,port,prom) matching _start_router; drop spurious resolve().
- tests/test_megatron_argument_validation.py: add num_gpus_per_node=8 to the
  vime_validate_args fixture (vime colocate override needs it).
- .buildkite/pipeline.yml: agent tests -> tests/test_agent/*; +cispo_loss,
  +logprob_response_spans (CPU-safe); test_rollout_metrics stays GPU-only (imports vllm).

Engine patch verdict (PATCH_ASSESSMENT.md): P1-P7 vLLM doesn't need (NIXL/Mooncake
native); kept ours vllm.patch, dropped sglang content + top_p.patch.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): unblock GPU run (scipy pin, router arg dup, top-p-replay gate)

Found running GPU CI on 8xH200 with vllm/vime:latest:
- docker/Dockerfile: pin scipy<1.14 next to numpy<2 (scipy drifted to 1.18 which
  needs numpy>=2 and uses removed np.long -> 'import vllm' crash).
- vllm_utils/arguments.py: drop sync-added RouterArgs.add_cli_args + its import in
  add_vllm_router_arguments. main exposes the full router surface only in
  utils.add_router_arguments; the duplicate re-registered --router-request-timeout-secs
  -> argparse conflict at train startup.
- megatron_utils/loss.py: get_rollout_top_p_logprob_kwargs falls back to full-vocab
  logprob when top-p nucleus token ids are absent instead of raising. slime's
  top-p-replay needs engine-returned top-p tokens; vime's vLLM /inference/v1/generate
  does not expose them (sglang-only). Matches vime pre-sync behavior; flagged in
  OVERNIGHT_REPORT for review.

Image import smoke + Megatron ckpt load + 4x VLLMEngine bringup + NCCL weight
transfer all confirmed working in-image before this.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(gpu): drop deleted test_qwen2.5_0.5B_ppo_critic_only_short from short suite

slime deleted tests/test_qwen2.5_0.5B_ppo_critic_only_short.py this window (#2014..#2125);
gpu_suites.py still listed it -> 'no such file' exit 2. Critic-only path is still
covered by test_qwen3_4B_ppo_train_critic_only (megatron suite). Other 3 short tests
(gsm8k_async, gsm8k, fully_async) pass on 8xH200.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore base_log_probs init in streaming rollout (None+list crash)

GPU test_qwen3_4B_streaming_partial_rollout hit
'TypeError: unsupported operand +: NoneType and list' at vllm_streaming_rollout.py:234.
The conflict resolution changed base_log_probs from main's
`list(sample.rollout_log_probs or [])` to a None-able form; a fresh sample
(rollout_log_probs=None) then did None + call_log_probs. Restored main's form.
Real sync-resolution regression caught by GPU CI.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): streaming rollout uses append_response_tokens (was renamed update_from_meta_info)

slime #2110 renamed Sample.update_from_meta_info -> append_response_tokens; vllm_rollout
was updated but vllm_streaming_rollout still called the old name (AttributeError at
generate_streaming). Streaming already accumulates tokens incrementally for partial-rollout,
so call append_response_tokens(meta_info=meta) with tokens omitted -> metadata-only finalize
(no double-append). Caught by GPU test_qwen3_4B_streaming_partial_rollout.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore vime unconditional colocate rollout_num_gpus re-derive

3-way compare (slime / vime-main / PR) showed the merge made a broken hybrid:
it KEPT vime's num_gpus_per_node colocate override (which assumes rollout_num_gpus
is forced to actor_num_gpus_per_node*actor_num_nodes) but REPLACED vime's
unconditional re-derive (`!= -> re-derive`) with slime's `is None`-only form.
When a colocate test's rollout_num_gpus is non-None but mismatches, it was left
mis-sized -> engine/GPU misplacement -> mixed_offload IPC-UUID mismatch + ckpt
'Free memory < util'. slime passes (no override, self-consistent is-None); vime
main passes (override + unconditional re-derive, coupled). Restore vime's
re-derive; keep slime's new rollout_num_gpus==0 branch (checked first).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): slime-consistency review pass (docs, rollout routed_experts, comments)

Docs (EN+zh parity):
- fault-tolerance: revert junk 'server'->'engine' mistranslation (keep correct /health endpoint, verified vs vllm_engine.py)
- vllm-config: 'ServerArgs'->'EngineArgs' (sglang class -> vLLM AsyncEngineArgs u FrontendArgs); fix duplicated 'vllm-router (vllm-router)' alias
- customization: restore over-deleted 'custom_generate -> list[Sample]' section + signature (dropped only the vime-absent search-r1 example link)

Rollout:
- routed_experts now flows through the slime-identical Sample._apply_meta_info (single assignment site, torch.int32 tensor matching downstream) instead of an inline numpy assign; vLLM .npy-on-choice decode stays (engine wire-format delta). Both vllm_rollout and vllm_streaming_rollout.

Comments for future syncers:
- --opd-teacher-model + on_policy_distillation: engine-driven divergence (vLLM model field; sglang /generate has none)
- overrides / _vllm_server_field_names: AsyncEngineArgs u FrontendArgs == slime's sglang ServerArgs

Examples/docker/etc: drop vime-absent npu/retool/search-r1/tau-bench files; restore eval_multi_task; docker alignment with slime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): pre-commit green — define base in streaming MM render (F821) + isort/black on test_agent

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): correct two mistranslated CPU tests (colocate rollout-gpu re-derive; parse_model_output tokenizer)

- test_..._preserves_larger_rollout_gpus_under_colocate asserted slime behavior (==12); vime re-derives to actor*nodes=8 under colocate (commit 97013044). Renamed + assert ==8 + divergence note. vime-main never had the test; slime does.

- test_parse_model_output_plain_text_no_parsers called parse_model_output without the required tokenizer kwarg (#198 made it required for vLLM parsers). Pass tokenizer=None (unused on the no-parser path).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): streaming rollout posts to /inference/v1/generate, not sglang /generate

The slime diff3 merge took slime's sglang endpoint (/generate) for the streaming rollout POST instead of keeping vime's vLLM endpoint (/inference/v1/generate). main (7198547c) had the correct URL; the sync regressed it (and dropped the base var). Result: 404 Not Found at vllm_streaming_rollout.py:182 -> test_qwen3_4B_streaming_partial_rollout fails. Caught on a clean h200 node. The file's own docstrings already say /inference/v1/generate throughout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): thread rollout port cursor globally across multi-model engines

The diff3 merge adopted slime's deferred rollout-engine init (start_rollout_servers
now returns pending_init_handles awaited by the caller instead of ray.get-ing each
model's engines before the next). That broke an implicit invariant the per-model
port_cursors reset relied on: in main, model 0's engines were fully bound before
model 1 allocated ports, so the free-port bind-test in
_allocate_rollout_engine_addr_and_ports_normal skipped model 0's ports. With
deferred init, model 1 allocates while model 0 is unbound, the bind-test sees the
base ports free, and a second model (e.g. mixed_offload's frozen "ref") lands on
the same 15000-15003 as the actor. The actor's POST /update_weights to :15002 then
hits the never-started ref engine -> vLLM 500 "start_weight_update must be called
before update_weights" (test_vllm_config_mixed_offload[_ft]).

Fix: initialize port_cursors once before the model loop so the per-node next-free
cursor is monotonic across all models, keeping every engine's ports disjoint
regardless of bind timing. Single-model behaviour is unchanged; the per-model
reset only existed to scope cursors that are already node-keyed.

Caught on h200 GPU CI (new nightly-dev-20260618a image, which added the
start_weight_update-before-update_weights enforcement that exposed the collision).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore robust pkill pattern so ckpt cleanup kills vLLM ray actors

PR #260 ("complete slime-exact port") changed execute_train's pre-launch
cleanup from

    pkill -9 -f '[v]llm serve|VLL[M]::'

to

    pkill -9 vllm

as an over-literal sglang->vllm translation. But the vLLM rollout engine runs
as Ray actor processes whose process *name* is python/ray, with "VLLM::" only in
the command line — so `pkill -9 vllm` (name match, no -f) does not kill them.
Leftover engine processes from the ckpt test's save phase survive into the load
phase, holding ~115 GiB, so the load-phase engine starts with ~24/139 GiB free
and dies with "Free memory ... less than desired GPU memory utilization (0.8,
111.84 GiB)" (test_qwen3_4B_ckpt.py, both --async-save and not).

Restore the cmdline-match pattern `-f '[v]llm serve|VLL[M]::'`.

Bisected on h200: ckpt PASSES at 289ee6d / e62d44fa (old pattern, 4/4 runs) and
FAILS at 7198547c/main + PR (new pattern, 0/3), same old image -> code regression
in #260. Verified: pkill-fixed PR code + new pr286 image -> ckpt PASS (579s).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* chore(sync): replace all `pkill -9 vllm` with cmdline-match pattern

Same root cause as 764e1e18 (command_utils.py): `pkill -9 vllm` matches by
process *name*, but vLLM rollout engines run as Ray actor processes (python/ray
named, "VLLM::" only in the command line), so the name match never kills them.
Apply the robust cmdline pattern `pkill -9 -f '[v]llm serve|VLL[M]::'` everywhere
the bare `pkill -9 vllm` cleanup was used across run/example scripts, so leftover
engines don't squat GPUs across runs. No logic change beyond the kill pattern.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): restore vLLM core.py partial-wake sleep-guard (#44483)

Commit d41f0aa1 removed the core.py sleep-guard hunk from
docker/patch/latest/vllm.patch on the assumption it was "already in
v0.23.0". It is not: stock v0.23.0 `vllm/v1/engine/core.py` calls
`resume_scheduler()` and `execute_dummy_batch()` even during a partial
(weights-only) wake. So a colocate DP+EP pd_mooncake rollout, right after
`POST /wake_up?tags=weights` (KV cache still released under level-2 sleep),
has its DP busy-loop fire a decode-shaped dummy batch that touches freed
KV -> the scheduler_metadata write in flashattn_mla.py:234 (MLA, glm4.7)
and flash_attn.py:547 (FA3, qwen3.6) raises `CUDA error: invalid argument`.

Restore the guard (`if not self.model_executor.is_sleeping` around
resume_scheduler; `if not self.is_sleeping()` around execute_dummy_batch),
keeping the all2all_utils weight-reload fix. This is the #173 sleep-guard
patch re-expressed against v0.23.0 line numbers.

Verified: git-apply --check clean against stock v0.23.0; both guards land;
glm4.7/qwen3.6 pd_mooncake reproduced the crash without it (the 8-day-old
image that still carried the guard passes both).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): drop scipy<1.14 pin, mirror slime numpy<2 only

The scipy<1.14 pin (added in 0cda11c6 while chasing the pd_mooncake crash)
was a red herring: the real cause was the dropped core.py partial-wake
sleep-guard, now restored. slime-2125-as-vime pins only `numpy<2`; this
restores that exact line. numpy 1.26.4 + scipy resolved naturally matches
the working baseline.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): keep FlashQLA install gated behind INSTALL_FLASHQLA=0

slime installs FlashQLA unconditionally, but it is sm90/Hopper-only.
Restore vime's original gated form (default off; --qwen-gdn-backend fla
elsewhere). CI build passes --build-arg INSTALL_FLASHQLA=1 to include it.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(vllm-config): fix inference-only FAQ — vime launches engines in-process

The mechanical mirror of slime's answer steered users to vLLM's standalone
`vllm serve` (slime's `launch_server` analog) for inference-only. That is
misleading for vime: like slime, vime launches the vLLM engines in-process
from `--vllm-config` (same in-process path as training), so a rollout-only
run serves directly with no separate server process. Point standalone users
to `--rollout-external-engine-addrs` instead. EN + ZH.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(debug): restore INT4 / Compressed-Tensors checkpoint section

vime's debug.md was missing slime's "INT4 / Compressed-Tensors Quantization
Checkpoint Issues" section (slime #1642) — dropped in an earlier sync, not
present on main. Restore it (EN + ZH), translated sglang→vLLM / Megatron→vLLM.
Covers the quantization_config.ignore list, all-zero MoE router weights
(mlp.gate.weight) when mis-quantized, missing safetensors shards, and
diagnosis via --check-weight-update-equal / --debug-rollout-only.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(vllm-config): use _run_vllm_server for inference-only FAQ

Keep slime's wording; the only engine-coupled fix is the standalone launcher
name. slime's `launch_server` is its in-process engine entry; vime's analog
is `_run_vllm_server` (vllm_engine.py, launched via multiprocessing.Process),
not the standalone `vllm serve` CLI. EN + ZH.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): restore scipy pin (scipy<1.18) — vime base needs it

Reverts the scipy-pin removal in b359b245, which was wrong: vime's
vllm/vllm-openai base ships no scipy, so unpinned the build pulls scipy>=1.18,
which hard-requires numpy>=2 and uses np.long (removed numpy>=1.24) -> crashes
against the numpy<2 reinstall (Megatron needs numpy 1.x). slime's sglang base
resolves scipy 1.17.1 natively (numpy-1.x compatible), so slime needs no pin;
this is a base-image divergence, not a red herring. Pin boundary is 1.18
(slime runs 1.17.1), not the earlier 1.14 over-estimate.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(scripts): properly translate sglang args in glm5.2-744B + glm4.7-355B-delta

These two were prefix-swapped (--sglang-X -> --vllm-X) without semantic mapping,
leaving ~20 args that aren't vLLM AsyncEngineArgs (argparse would reject). Apply the
knowledge/rl/sglang-to-vllm-translation.md §5.5 mappings:
- dp-size->data-parallel-size, ep-size->enable-expert-parallel, max-running-requests->
  max-num-seqs, cuda-graph-max-bs->max-cudagraph-capture-size
- 5x/4x --speculative-* -> one --vllm-speculative-config JSON (§5.2)
- DeepEP: per-group deepep_mode auto/low_latency -> all2all_backend deepep_high_throughput/
  low_latency in the --vllm-config overrides (vLLM has no 'auto'; PD encodes it per-role)
- watchdog-timeout -> env VLLM_ENGINE_ITERATION_TIMEOUT_S
- drop sglang-only: dp-attention / dp-lm-head / moe-dense-tp / disable-overlap-schedule / NSA
  backends (vLLM selects DeepSeek sparse attn per model) / engine delta-receiver knobs
- flag PD mooncake transport (-> --vllm-kv-transfer-config) as fabric-specific TODO

These are 744B/355B scripts not runnable in CI — translations are SOP-mapped but
hardware-unvalidated (flagged inline).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(args): hard-guard unverified delta weight-sync mode

--update-weight-mode=delta (PR #278 lineage) is not yet validated on
vime+vLLM (vLLM exposes dense/sparse_flat only, not slime's gap-delta/
zstd encoding). Raise NotImplementedError at arg-validation so it fails
fast at startup instead of crashing mid weight-sync. Downstream delta
code is kept untouched; remove this raise once a real delta-load run
passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs,test: correct rollout engine endpoint to /inference/v1/generate

The agent rollout engine is reached at vime's vLLM ``/inference/v1/generate``
(see vllm_rollout.get_model_url default + common.call_vllm_generate), not the
bare ``/generate`` of sglang. Fix the imprecise path in adapter/test docstrings
and comments, and rewrite the vllm-config.md custom-rollout examples (en+zh):
they were still sglang-shaped (``/generate`` path + ``{"text":..., "return_logprob":
True}`` body). Use vime's real request schema instead -- ``{"model","token_ids",
"sampling_params"}`` with ``max_tokens``/``logprobs``, ``prompt_logprobs`` for
fixed-sequence scoring, and the ``choices[0]`` response shape.

No code/logic change: comments, docstrings, and doc examples only. The Megatron
training server's own ``/generate`` endpoint and the sglang citation in
arguments.py are correct and left untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style(args): collapse delta-guard message to one line (black)

The delta-guard NotImplementedError message was split across two adjacent
string literals; black on the CI (line-length 119) collapses/normalizes it.
Make it a single clean literal so pre-commit is green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test(args): assert delta weight-sync is guarded off (not per-condition)

The hard delta guard (7bb19e67) raises NotImplementedError at the top of the
delta branch, making the downstream colocate / unknown-transport rejections
unreachable. Replace test_update_weight_delta_rejects_colocate and
test_update_weight_delta_rejects_unknown_transport (whose ValueError paths no
longer fire) with a single test_update_weight_delta_disabled that asserts the
guard raises for any delta config. Breadcrumb left to restore the per-condition
tests when delta is verified and the guard is lifted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore dropped weight-sync metrics chain + delta dispatch (slime parity)

A mirror cross-check of the delta-weight-sync surface (slime #1806/#1991) found
the #2014..#2125 sync had silently dropped several slime-faithful pieces:

* extra_metrics logging chain — slime threads weight-update metrics from the
  actor through log_perf_data -> log_perf_data_raw. vime dropped the param at
  all three layers, so weight-update metrics were never logged. Restored:
  train_metric_utils.log_perf_data_raw(extra_metrics=...), data.log_perf_data
  passthrough, and actor passing self.weight_updater.pop_metrics().

* pop_metrics on UpdateWeightFromDistributed — the default (non-colocate, nccl)
  weight_updater. slime gives all three updaters a pop_metrics() stub so the
  actor can call it uniformly; vime kept it on tensor/disk but dropped it on
  distributed, which would AttributeError once the actor calls it. Restored the
  ~5-line stub (delta-specific plumbing stays dropped — vime+vLLM has no DeltaSpec).

* actor delta-mode dispatch branch — restores the elif selecting
  UpdateWeightFromDistributedDelta. Dead code behind the validation guard that
  rejects --update-weight-mode=delta, so vime mirrors slime with the guard as
  the single divergence.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* chore(sync): align comments to the mechanical mirror

Comment-only pass; no behavior change. Aligns vime comments to what a faithful
slime->vime translation would carry:

* Strip vime-divergence rationale markers (delta guard, opd-teacher-model,
  top-p fallback, colocate/delta tests). The rationale belongs in the
  divergence manifest, not inline; the guarded code + self-explanatory
  NotImplementedError messages stand on their own.
* De-verbose vLLM-specific comments to mirror scale: the router-args block,
  the AsyncEngineArgs u FrontendArgs docstring, and the MoE-replay /
  streaming-rollout blocks that slime does not carry at that length.
* Restore slime-original comments the sync had dropped or naively translated,
  with judgment translation of sglang-specific terms: session_id routing
  ("vLLM router", not the mechanical "Model Gateway"), "Prepare payload for
  vLLM server", the unique-session_id loop, and the pending-tasks wait.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(delta): note delta weight-sync not yet verified on vime+vLLM (PR #286 review)

Per review on PR #286: delta weight sync is documented here but the arg guard
disables --update-weight-mode=delta. Add a top-of-page note (en + zh) so users
see it before hitting NotImplementedError at argparse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Example] Add MemAgent long-context RL example (mem_agent) (#291)

* Add examples/mem_agent for long-context MemAgent RL

Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>

* Address review: harden mem_agent convert and eval scripts

Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>

---------

Signed-off-by: kaiyuan <kyxiezju@163.com>

* fix(rollout): abort vLLM rollout via delete-type /abort_requests (#296)

Under --partial-rollout, abort() (pause -> drain -> resume) deadlocks:
/pause?mode=abort puts the scheduler in PAUSED_NEW, and a /generate that races
in after the pause parks in the waiting queue and never returns until /resume,
which runs after the drain. Reordering to pause -> resume -> drain avoids the
hang, but resume reopens the whole queue so the long tail runs to COMPLETION --
breaking partial rollout's "truncate the tail, resume it next step" semantics.

Switch to a delete-type abort instead:

- vLLM: add POST /abort_requests to the RLHF api_router -> EngineClient.abort()
  (removes queued requests from the waiting queue and finish-aborts running
  ones, whose partial output returns on the original /generate stream). It does
  not pause the scheduler, so there is no /resume and no deadlock. Shipped as a
  build-time patch in docker/patch/latest/vllm.patch.

- vime: server_control.abort_inflight_requests() replaces the unused,
  slime-mirrored abort_servers_until_idle / _v1_loads helper (vLLM has neither
  /abort_request nor /v1/loads). abort() re-issues the sweep across drain waves
  and converges on state.pendings, with a timeout bounding how long a late
  multi-turn straggler can run before being truncated to partial.

- vllm_engine: drop the legacy version gate in _register_to_router. vime ships
  its own vllm-router, so only the /workers payload path is needed.

Adds delete-type abort unit tests.

AI-assisted change; reviewed by a human before submission.

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Josephasafg <ajgard7@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* [Bugfix][Rollout] Wire prefix_cache_hit_rate through vLLM usage (#303)

rollout/prefix_cache_hit_rate (and avg_cached_tokens_per_sample) were
structurally 0 due to three coupled gaps:

1. vLLM's non-streaming /inference/v1/generate builds a `usage` block but
   silently drops it: GenerateResponse (v0.23.0 serve/disagg/protocol.py)
   never declared a `usage` field and has no extra="allow", so pydantic
   discards it. Patched in docker/patch/latest/vllm.patch, mirroring
   GenerateStreamResponse. (Upstream fix filed against vllm-project/vllm.)
2. The rollout parser read usage.prompt_tokens/completion_tokens but never
   usage.prompt_tokens_details.cached_tokens -> PrefixCacheInfo numerator
   pinned to 0. Now read in both vllm_rollout and vllm_streaming_rollout.
3. Streaming additionally needs stream_options.include_usage=True for vLLM
   to emit the terminal usage SSE chunk.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docker): /abort_requests abort-all fix + cu13 image variant (#317)

* fix(docker): abort-all in the /abort_requests vLLM patch must abort by internal ids

The bundled /abort_requests endpoint (merged in #296) populated request_ids from output_processor.request_states (internal ids) but called engine.abort() with the default internal=False, so they were treated as external, matched nothing, and POST /abort_requests {} silently aborted no requests under default request-id randomization. Abort the all-in-flight list as internal. Mirrors vllm-project/vllm#47173.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style: reject malformed JSON in /abort_requests patch with 400

Match the sibling dev endpoints and the Rust frontend (400 on malformed JSON) instead of silently treating it as empty. Mirrors vllm-project/vllm#47173.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): abort-all patch must also abort parallel-sampling parents

The /abort_requests patch enumerated request_states (child internal ids
only), so with n>1 the ParentRequest entry leaked. Include
parent_requests keys in the abort-all set. Mirrors vllm PR #47173.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* feat(docker): cu13 image variant + TMS cu13 preload

Port the cu13 build support from #307: ENABLE_CUDA_13 branches the apt
dev headers, cublas header, TransformerEngine (source-built for cu13),
TMS_CUDA_MAJOR auto-detect, and the cudnn pin. justfile gains a
build-cu13 target and a VARIANT-prefixed manifest. actor_group preloads
the cu13 TMS .so.

Also switch the vLLM patch apply to --allow-empty so the build survives
once the patch is emptied upstream.

Excludes #307's NCCL_CUMEM_ENABLE default flip (0->1) and the glm5.2
scripts by request.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* Initial ROCm support for vime (#273)

* Added ROCm quick start guide

* changed wording

* changed docker image

* updated instructions

* Add initial working ROCm 7.0.2 build (Dockerfile.rocm + patch)

Initial working setup for building/running vime on ROCm 7.0.2 (gfx950).

* ROCm: re-base Dockerfile on vime mainline Megatron + shared patch

* ROCm: apply checkpoint-writer patch in HF->torch_dist converter

* ROCm: align async run script cleanup with run-qwen3-4B.sh format

* ROCm: move megatron.patch to docker/amd_patch/ (slime layout)

* ROCm: re-add fused-kernels init patch and AMD doc/template references

---------

Co-authored-by: pancake0003 <146360951+pancake0003@users.noreply.github.com>
Co-authored-by: indianspeedster <cp3793@nyu.edu>

* [Doc] Fix stale Qwen3-30B-A3B example: restore FP8-inference section + rewrite multi-node (#322)

* [Doc] Fix stale Qwen3-30B-A3B example: dataset download + align with actual script

The Qwen3-30B-A3B example documented an elaborate env-var-driven multi-node
script (ACTOR_NUM_NODES / MEGATRON_TP / ENABLE_R3 auto-behavior, "script skips
Ray", default batch 4/2/8, --vllm-moe-backend triton). None of that exists in
the current slime-exact scripts/run-qwen3-30B-A3B.sh — it was orphaned when the
script was reverted to the slime-exact single-node port. The real defaults are
rollout-batch-size 32 / n-samples 8 / global-batch 256.

Changes (zh + en, kept concise):
- Environment Preparation: add the concrete `hf download` commands for the
  model, dapo-math-17k train data, and aime-2024 eval data (previously the
  section only pointed at the Qwen3-4B doc).
- Restore the "BF16 training + FP8 inference" section (present upstream in
  slime, dropped here); matches the commented --hf-checkpoint line in the script.
- Replace the orphaned multi-node section with the concise, accurate
  slime-style guidance (manual modifications + vLLM EPLB example), fixing the
  broken zh anchor to the Chinese quick_start heading.

Doc now matches scripts/run-qwen3-30B-A3B.sh; no script changes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Expand Qwen3-30B-A3B multi-node section (code-grounded)

Replace the concise multi-node bullets with an actionable but verified guide:
Ray cross-node startup (from quick_start), the exact script edits
(--actor-num-nodes, colocate auto rollout-num-gpus), parallelism scaling as
constraints (defer concrete large-scale ratios to the GLM/DeepSeek examples
rather than fabricating a 2-node table), and the real pitfalls we hit
(keep each vLLM engine within a node to avoid cross-node TP=16 slowdown/
numerics, MASTER_ADDR / NCCL_SOCKET_IFNAME, global-batch identity,
--num-gpus-per-node for <8 GPUs/node). Every claim traces to vime/utils/
arguments.py, quick_start, or observed multi-node runs; no "tested" claims.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Drop model-download line from Qwen3-30B-A3B example

Model download is already covered by the referenced Qwen3-4B doc; keep only the
train/eval dataset download (dapo-math-17k, aime-2024) surfaced inline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Drop redundant dataset download; keep env-prep slime-exact

The intro sentence already covers env/model/data/ckpt via the Qwen3-4B
reference, and slime's 30B doc carries no dataset block. Remove it; keep the
30B-specific torchrun checkpoint conversion (differs from 4B's single-process
python). Env-prep is now byte-identical to upstream.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Multi-node: also skip the script's Ray-cleanup preamble

Codex review (P2): for the manual multi-node path, removing only the
`ray start --head` line is not enough — the script's initial cleanup block
(`ray stop --force` + `pkill -9 ray/python/redis`) would tear down the
manually-started head before `ray job submit`, failing submission and
orphaning the workers. Tell users to remove/comment both the cleanup block
and the `ray start --head` line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Fix H800→H100 typo in Qwen3-30B-A3B example (#325)

The page title says "8xH100" but the CPU-Adam bullet said "8xH800
environment", an inconsistency inherited from slime. Both are 80GB cards so
the point is unchanged; align the text with the title.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Add Ascend NPU platform tutorial and expand hardware support section (#334)

Signed-off-by: kaiyuan <kyxiezju@163.com>
Co-authored-by: XiaoxinWang <wangxiaoxin7@huawei.com>
Co-authored-by: FulinGao_HW <gaofulin1@huawei.com>
Co-authored-by: meihanc <zr010426ztt@outlook.com>
Co-authored-by: wuxiang <498160096@qq.com>
Co-authored-by: yuxinshan <syx_ctyg@126.com>
Co-authored-by: flb_dayo <floatlibai@gmail.com>
Co-authored-by: Windfeng8 <523758380@qq.com>
Co-authored-by: YZY <532183776@qq.com>

* sync: update vime through slime #2185 (#338)

Sync slime a897e1f4..680824dd using the mechanical merge plus vLLM adaptation workflow.

* revert: sync through slime #2185 (#338)

Revert the prematurely merged slime sync and restore the pre-#338 tree.

* [bugfix] Fix distributed update weights for pipeline parallel (#329)

* Fix pp for distributed update weights

Signed-off-by: knlnguyen1802 <knlnguyen180…
CalvinXKY added a commit that referenced this pull request Nov 5, 2026
* refactor(test): flatten CPU unit tests and centralize import stubs (#238)

* refactor(test): align CPU unit test layout with slime and add _unit_stubs

Move tests/unit/* into tests/ and tests/utils/, replace nested conftest stubs with shared helpers for optional deps, and register the moved files in cpu-unittest CI. Write generated workflow YAML as UTF-8 on Windows.

Signed-off-by: kaiyuan <kyxiezju@163.com>

* fix(vllm_engine): guard HTTPError.add_note for Python 3.10 CI

Exception.add_note() requires Python 3.11+; cpu-unittest runs on 3.10.

Signed-off-by: kaiyuan <kyxiezju@163.com>

* refactor(test): drop pytest pythonpath, bootstrap tests/ in test files

Remove redundant pythonpath from pyproject.toml; each test that imports
_unit_stubs prepends tests/ to sys.path (including test_vllm_rollout.py).

Signed-off-by: kaiyuan <kyxiezju@163.com>

* ci: bump cpu-unittest to Python 3.11 and revert add_note guard

Run cpu-unittest on 3.11 so HTTPError.add_note is available in CI without Py3.10 guards in vllm_engine or conditional __notes__ assertions in tests.

Signed-off-by: kaiyuan <kyxiezju@163.com>

---------

Signed-off-by: kaiyuan <kyxiezju@163.com>

* docs: align vLLM docs with current defaults (#240)

* docs: align vllm docs with current defaults

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: fix router policy examples

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: simplify ray multi-node wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore quick start hardware notes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: clarify cache-aware routing support

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove extra router policy example

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore cache aware policy example

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore hardware support wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove conda fallback note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: update docs logo

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: replace legacy logo assets

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: add README docs badges

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: replace logo image

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove redundant logo jpg

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: adjust hardware support wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: refine hardware support wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore hardware support notes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: add GB support guard notes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: clarify profiling flow

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: shorten profiling note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* tools: simplify rollout profiler helper

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore profiling request range

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove profiling note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: switch logo asset to jpg

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* update docs logo

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* update docs logo asset

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: move eplb note to faq

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: rephrase profiling wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove eplb note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore eplb example

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore eplb example in english

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>

docs: align vLLM docs with current defaults
CI: merged with buildkite/vime-ci failure (infra); GA checks passed. See PR #240 comment.

* refactor(weight-sync): use vLLM native /update_weights, remove worker extension (#246)

* refactor(weight-sync): use vLLM native /update_weights instead of worker extension

Remove the custom `vLLMColocateWorkerExtension` and `_VLLMHijack`
monkey-patch introduced in PR #104. These reimplemented vLLM's built-in
`IPCWeightTransferEngine.receive_weights()` (UUID routing, device_index
remapping, layerwise reload) inside a worker extension, bypassing the
native `/update_weights` endpoint in favor of `/collective_rpc`.

vLLM's native IPC engine already handles everything the extension did.
This change routes `update_weights_from_tensor` back through
`POST /update_weights` (the path that was working before PR #104) and
deletes ~160 lines of redundant code.

Changes:
- `update_weight_from_tensor.py`: delete `_VLLMHijack`,
  `vLLMColocateWorkerExtension`, and fix `ipc_handles` format to store
  `ipc_args` (not `(rebuild_func, ipc_args)`) for native API compat
- `vllm_engine.py`: revert `update_weights_from_tensor` from
  `/collective_rpc` to `/update_weights`, delete `update_weights_chunk`,
  remove `--worker-extension-cls` injection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(reloadable-pg): add missing gather_object patch; switch IPC gather to match slime

ReloadableProcessGroup monkey-patched all_gather_object but not
gather_object, causing "Group is not registered" after Megatron
offload/reload. This forced _send_to_colocated_engine to use the
less efficient all_gather_object (all ranks receive, only leader uses).

Fix: patch gather_object in the same list, then switch to
dist.gather_object (only leader allocates the receive buffer) to
align with slime's implementation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add Buildkite CI pipeline (CPU jobs + manual-gated GPU suites) and remove PR test on GHA (#239)

* ci: add Buildkite pipeline for always-on CPU jobs

Port the always-on jobs from .github/workflows/pr-test.yml.j2 (pre-commit
gate, plugin contracts, agent adapter, in-image unit tests) to a single
dynamically generated Buildkite pipeline targeting the vLLM elastic-stack
CPU queues. GitHub Actions keeps running in parallel and stays
authoritative; GPU suites are not migrated yet.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: replace Buildkite generator with static pipeline.yml

Drop generate_pipeline.py in favor of a plain static .buildkite/pipeline.yml
defining the four always-on CPU steps directly (pre-commit gate, plugin
contracts, agent adapter, in-image unit tests). Simpler to read and review for
a first cut; the GHA workflow stays authoritative and GPU suites are still out
of scope.

Pass GIT_CONFIG_PARAMETERS into every container so git (in pre-commit) doesn't
abort with "dubious ownership" on the host-owned checkout, and fix the
depends_on typo in the README.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): fix CPU test deps; add manual gate for GPU suites

plugin-contracts failed in build #3 on tests/utils/test_hf_checkpoint_saver.py
(ModuleNotFoundError: safetensors): the dep list predated the slime sync in
#232 which added requests/ray/safetensors to the GHA template. Mirror it.

GitHub PR labels can't trigger Buildkite jobs, so expose the run-ci-* GPU
suites behind a block step instead: unblocking offers a multi-select of suites
(short / vllm-config / megatron / precision / ckpt) and gpu_suites.py uploads
one step per test with the same gpu_lock_exec + docker invocations and
per-test DEEPEP/FP8/EVAL env combos as the GHA jobs. blocked_state: passed
keeps the commit status green when the gate is left untouched. GPU steps
target a new vime-gpu agent queue (self-hosted hosts; see README).

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): run GPU suites on mithril-h100-pool; pin gloo to loopback

Build #4's unblock test showed the CI cluster rejects uploads targeting a
nonexistent queue, and rather than minting a new queue, follow the pattern
vllm-omni already uses for mithril-h100-pool: each GPU job is a Kubernetes pod
(agent-stack-k8s kubernetes plugin) on an H100 SXM node with nvidia.com/gpu
limits (4 or 8), memory-backed /dev/shm, and /mnt/hf-cache mounted as HF_HOME.
vime tests hf-download their models, so the warm HF cache replaces the GHA
runners' /mnt/nvme0n1/vime_ci mounts; the docker-run wrapper goes away since
the pod runs the vime CI image directly.

Also pin GLOO/TP_SOCKET_IFNAME=lo in the plugin-contracts container:
test_metric_report_dist hung intermittently (build #4 timed out at 30 min)
because gloo can pick a non-loopback interface inside a bridge-network
container.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): expandable_segments for the borderline OOM short test

test_qwen3.5_0.8B_gsm8k_async_short OOMed in compute_log_probs on the mithril
pool's 80 GB H100s (build #6) with 7 GiB reserved-but-unallocated — the
allocator-fragmentation case PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
exists for. Scope it to this test's pod only (vLLM sleep-mode CuMemAllocator
can conflict with expandable segments) via verbatim pass-through of non-VIME
env overrides. The other short tests passed on H100 pods unchanged.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): soft-fail the two known-H100-incompatible GPU tests

Builds #6/#7 isolated two test-level failures on the mithril 80 GB H100s,
neither a pipeline issue:
- gsm8k_async_short OOMs as tuned (67 GiB live on the actor GPU after
  expandable_segments eliminated fragmentation; its sync twin passes).
- parallel_check's CP=2 grad norm diverges ~4% from the same-node baseline
  recording, a topology-sensitive numerical invariance question.

Mark exactly these two soft_fail so they keep running and stay visible on
Buildkite without failing the build; their authoritative gate remains the
GHA label jobs.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): keep the two H100-incompatible GPU tests failing loudly

Revert the soft_fail: per review, the gsm8k_async_short OOM and the
parallel_check CP-invariance divergence should stay visible as hard failures
on Buildkite until the underlying issues are fixed. Keep the diagnostic
comments and the test-scoped expandable_segments setting.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): soft-fail the two H100-incompatible GPU tests after all

Re-apply b334784 (reverted in 0a98010): per the follow-up decision, mark
gsm8k_async_short and parallel_check soft_fail so they keep running visibly
on mithril without failing the build, with the GHA label jobs as their
authoritative gate until the OOM tuning and CP-invariance questions are
resolved.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): resolve 0.8B async OOM on H100 by reducing max-tokens-per-gpu

Root cause: Qwen3.5's 248K vocab produces [T, 248320] fp32 logits tensors.
calculate_log_probs_and_entropy holds 5 copies simultaneously (2 clones +
2 intermediates + original). At max-tokens-per-gpu=9216, each copy is
~8.5 GB → 42.6 GB from logits alone, exceeding H100 80 GB with
activations and reserved pool fragmentation.

Fix: reduce max-tokens-per-gpu from 9216 to 2048. Peak drops from 117.6 GB
to 39.6 GB (measured on H200), well within H100's 80 GB. GSM8K's longest
sequence is ~1200 tokens, so 2048 still fits all samples.

Also removes gsm8k_async_short from SOFT_FAIL_ON_H100 (no longer needed)
and the expandable_segments workaround.

parallel_check remains soft-fail: ~11% flake rate on TP4+per-token-loss,
confirmed same behavior in slime (Megatron FP reduction-order issue).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style: format update_weight_from_tensor

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* remove github workflows

Signed-off-by: khluu <khluu000@gmail.com>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>

* [codex] update docs branding and fix Buildkite CPU tests (#249)

* docs: update docs favicon to Vime branding

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: fix unit test path in buildkite

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: force triton stub in CPU unit tests

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: align update_weights_from_tensor with native vllm

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: isolate deep_gemm and triton in cpu utils

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(arguments): include DP in TP auto-compute default (#252)

* fix(arguments): include DP in TP auto-compute default

The `vllm_parse_args` default for `vllm_tensor_parallel_size` was
computed as `rollout_num_gpus_per_engine // pp_size`, missing the
`dp_size` divisor. This made the default TP too large when
`--vllm-data-parallel-size > 1` (required for expert parallelism).

Align the formula with `_resolve_vllm_parallel_sizes` in
`vllm_engine.py`, which correctly uses `gpus // (pp * dp)`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test(arguments): add DP>1 cases for TP auto-compute

Cover the fix: TP = gpus_per_engine // (PP * DP).

- test_parse_args_tp_default_with_dp: 8 GPU, DP=4 → TP=2
- test_parse_args_tp_default_with_pp_and_dp: 8 GPU, PP=2, DP=2 → TP=2
- test_parse_args_tp_default_dp1_unchanged: DP=1 regression guard

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* Update arguments.py

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* chore: rerun Buildkite

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [Example] Add tau-bench multi-turn tool-use example (#142)

* feat(tau-bench): add vLLM multi-turn GRPO example with trainable agent

Introduce generate_with_tau custom rollout (render + /inference/v1/generate), trainable_agents for tau-bench env interaction, vllm_tool_parser, and TAU_CONFIGS-based user simulator configuration in generate_with_tau.py.

Signed-off-by: xky <kyxiezju@163.com>

* style(tau-bench): apply pre-commit formatting

Signed-off-by: xky <kyxiezju@163.com>

* ci: retrigger buildkite

Signed-off-by: xky <kyxiezju@163.com>

* ci: retrigger buildkite

Signed-off-by: xky <kyxiezju@163.com>

---------

Signed-off-by: xky <kyxiezju@163.com>

* [codex] drop stale unit CPU CI and tests (#268)

* fix buildkite cpu utils ci

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove gha references from buildkite ci

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* deps: add transformers and cloudpickle

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: stub optional utils deps in cpu ci

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: keep vllm http error notes py310 compatible

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: run cpu jobs on python 3.11

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: inline utils stubs in affected files

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docker: upgrade base to vLLM 0.23.0, remove CUDA 13 build path (#253)

* docker: upgrade base to vLLM 0.23.0, remove CUDA 13 build path

- Base image: v0.22.0-cu129-ubuntu2404 → v0.23.0-cu129-ubuntu2404
- Remove `ENABLE_CUDA_13` ARG and all conditional cu13 blocks:
  - cu13 apt dev headers (libcublas-dev-13-0, cuda-nvrtc-dev-13-0, etc.)
  - TE source build (cu13 wheel didn't exist; cu129 wheel works on arm64)
  - fzyzcjy triton source build (cu13 specific)
  - TMS_CUDA_MAJOR export (no longer needed)
- Simplify cublas-dev to unconditional libcublas-dev-12-9
- Simplify TE install to wheel-only
- justfile: remove `build-cu13` target and cu13 tag scheme
- vllm.patch: adapt line numbers for 0.23.0 (776/1896 vs 750/1844),
  preserve `with self.log_iteration_details(None):` wrapper

cu129 nvcc already supports sm100/sm120 (Blackwell), so cu13 build
path was unnecessary — it caused build failures on gb300 (cu13 apt
packages hijacked /etc/alternatives/cuda, breaking TE CMake).

Tested: built successfully on gb200 (arm64), h200 (x86), gb300 (arm64).
All three confirmed vLLM 0.23.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docker/patch: replace core.py sleep fix (already in v023) with all2all_utils weight-reload fix (#45989)

v0.23.0 already includes the sleep/scheduler guard from #44483, so the
core.py patch is no longer needed. Replace it with the FP8+DeepEP
weight-reload fix (vllm-project/vllm#45989): snapshot
max_num_batched_tokens from FusedMoEConfig instead of calling
get_current_vllm_config() during layerwise reload.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(vllm_engine): in-process launch + dataclass field introspection (#264)

* fix(vllm_engine): drop router_ip/port fallback to args in init()

PD mode passes router_ip=None to engine.init() intentionally — the
router launches AFTER engines are ready and collects their URLs. The
fallback `self.args.vllm_router_ip` reads a pre-allocated-but-not-yet-
listening address, causing ConnectionRefused on POST /workers.

Drop the fallback; assign directly like main does.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(arguments): unblock 6 skipped params for --vllm-* forwarding

skipped_args in arguments.py blocked 6 params that should be
user-configurable via --vllm-* CLI flags:
  - served_model_name, tokenizer, tokenizer_mode, tokenizer_revision
  - dtype (only hardcoded when --fp16, otherwise user should set)
  - tool_call_parser (was manually registered then skipped; let
    monkey-patch auto-register like slime does)

Also removes the redundant manual --vllm-tool-call-parser registration
since the monkey-patch now handles it automatically.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: fix Qwen3-4B download repo (#276)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Liron Kesem <52330564+LironKesem@users.noreply.github.com>

* test: fix CPU unit test failures introduced by PR #264 (#284)

PR #264 moved top-level and lazy vllm imports that break the bare
python:3.11 CPU CI image (no vllm installed).

Three changes — no production code touched:

1. `tests/_unit_stubs.py`: extend `install_vllm_cli_stubs()` with stubs
   for `vllm.utils.system_utils` (top-level import in vllm_engine.py)
   and the `vllm.entrypoints.*` hierarchy (lazy imports in arguments.py
   and vllm_engine._vllm_server_field_names).

2. `tests/utils/test_vllm_engine.py`: add an autouse fixture that patches
   `_VLLM_SERVER_FIELDS` to an empty frozenset, so tests calling
   `_compute_server_args` without an explicit monkeypatch don't trigger
   the real `_vllm_server_field_names()`.

3. `tests/utils/test_vllm_arguments.py`: evaluate `real_module_available`
   before calling `install_vllm_cli_stubs()` (which registers a fake vllm
   in sys.modules), then mark two tests that require real FrontendArgs
   field names with `@requires_vllm` so they skip on the CPU image.

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(data): reuse stored multimodal_inputs in length filter (#257)

* fix(data): reuse stored multimodal_inputs in length filter

filter_long_prompt re-extracted vision info from sample.prompt via
process_vision_info in the multimodal branch. When apply_chat_template
is set, sample.prompt is the rendered *string* (not a conversation
list), so process_vision_info -> qwen_vl_utils crashed with
"TypeError: string indices must be integers, not 'str'".

This made prompt-length filtering unusable for any VLM dataset: setting
--rollout-max-context-len (which derives rollout_max_prompt_len) or
--rollout-max-prompt-len / --eval-max-prompt-len activates the filter
and hits the crash.

Reuse the multimodal inputs already computed during dataset
construction (sample.multimodal_inputs) instead of recomputing them
from the string prompt.

Add CPU unit tests covering the multimodal branch and a mixed
text-only + multimodal dataset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>

* Delete tests/test_filter_long_prompt_multimodal.py

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* Update data.py

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>

* docker: rename published image from inferactinc/public to vllm/vime (#283)

Replace all references to `inferactinc/public:vime-*` with
`vllm/vime:*` across CI, docs, and the release justfile.

The image is now published under the official vllm DockerHub
namespace (`vllm/vime:latest`, `vllm/vime:test-latest`) as a
multi-arch manifest (amd64 + arm64).

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(examples/tau-bench): use RunConfig.agent_strategy in TAU_CONFIGS (#280)

tau-bench RunConfig defines agent_strategy, not agent. The old key was silently ignored by Pydantic, and resolve_tau_config accessed tau_config.agent which raises AttributeError at rollout time.

Signed-off-by: kaiyuan <kyxiezju@163.com>

* scripts: complete slime-exact port of most scripts except for gpt-oss 20B support (#260)

* restore: bring back deleted examples, scripts, and agent doc

Restore files that were either deleted by #126 ("trim examples to
qwen3 only") or never synced from slime:

**Reverted from pre-#126 (translated):**
- scripts/low_precision/run-qwen3-4b-fp8.sh
- scripts/low_precision/run-qwen3-30b-a3b-fp8.sh
- scripts/run-glm4-9B.sh
- scripts/run-moonlight-16B-A3B.sh
- scripts/run-qwen3-4B-base-sft.sh
- scripts/run-qwen3-32B.sh
- scripts/run-qwen3.5-35B-A3B-sft.sh

**New from slime@44d29ee (translated):**
- docs/en/get_started/agent.md
- examples/fully_async/run-qwen2.5-0.5B-fully_async.sh

All sglang engine flags translated to vllm equivalents (§2.4).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: unify pkill pattern to '[v]llm serve|VLL[M]::'

Standardize all scripts to use the bracket-escaped pkill pattern that
avoids matching pkill itself and also catches vLLM's renamed
subprocesses (VLLM::EngineCore, VLLM::Worker_TP*). Matches the
canonical pattern in command_utils.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* scripts: complete slime-exact translation of all 29 run scripts

Translate all slime scripts to vime following SGLANG_TO_VLLM_TRANSLATION.md:
- sglang→vllm prefix swap for CLI flags and variables
- _slime→_vime for checkpoint paths
- EP: --sglang-ep-size N → --vllm-enable-expert-parallel (boolean)
- Speculative: multi-param → --vllm-speculative-config JSON (§5.2)
- Delete genuinely sglang-coupled params (DP-attention, DeepEP, NSA, etc.)
- flashinfer → FLASHINFER case fix (§2.4)

23 new scripts + 6 existing updated to match slime@cutoff.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(scripts): correct model config source path in FP8 low_precision scripts

The FP8 scripts used `${SCRIPT_DIR}/../scripts/models/` which resolves
to `scripts/scripts/models/` (non-existent). Changed to `../models/`
to match the INT4 scripts. Same fix as slime PR #2094.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gpt-oss): fused BF16 format, bridge API patch, bshd qkv format

Three fixes needed to run GPT-OSS 20B RLHF on vLLM backend:

1. hf_weight_iterator_bridge: match Megatron-Bridge 0.5.0 API
   _patch_bridge_expert_cache_to_cpu monkey-patches GPTOSSBridge.
   maybe_modify_converted_hf_weight gained a 4th `hf_state_dict`
   parameter; the patched wrapper only accepted 3, causing TypeError
   during weight sync.

2. run-gpt-oss-20B: point --hf-checkpoint at fused BF16 format
   vLLM's _load_weights_other expects gate_up_proj [E, hidden, 2*ffn]
   (fused). The old per-expert split format (experts.{e}.gate_proj.weight)
   causes KeyError on bias loading. Use tools/convert_gpt_oss_to_fused.py
   to convert an existing per-expert checkpoint, or re-run
   preprocess_gpt_oss.py to produce fused format directly.

3. run-gpt-oss-20B: add --qkv-format bshd + fix seq-length
   GPT-OSS uses learnable softmax (sink attention). TransformerEngine
   disables all attention backends when softmax_type=learnable and
   qkv_format=thd (packed sequences). --qkv-format bshd avoids this.
   --use-dynamic-batch-size is incompatible with bshd; replaced with
   fixed --seq-length 10240 (covers 8192 max response + prompt headroom).

tools/convert_gpt_oss_to_fused.py: new tool to convert per-expert BF16
checkpoint (output of old preprocess_gpt_oss.py) to the fused HF format
expected by vLLM without re-running the slow MXFP4 dequantization.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(scripts): replace pkill -9 vllm with precise -f pattern (21 files)

pkill -9 vllm matches any process named "vllm" and can inadvertently
kill unrelated vllm processes (e.g. background services). Use the same
pattern as PR #220 which targets only vllm serve and Ray VLL[M]:: actors:

  pkill -9 -f '[v]llm serve|VLL[M]::'

Also updates the inline form used in multi-node SSH worker restart
commands (run-qwen3-235B-A22B*.sh, run-qwen3.5-27B.sh, etc.).

Skipped: scripts/run-gpt-oss-20B.sh (uses pkill -9 -f "vllm serve" already),
scripts/run-minimax-m2.sh and run-glm4.7-*.sh (already used -f "vllm serve").

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* chore(scripts): remove run-qwen3-4B-amd.sh from this PR

AMD-specific script is out of scope for the gb300-complete-port PR.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* sync(docs+scripts): port docs/examples from slime-44d29ee, fix script translations

- Add missing EN/ZH docs: low-precision, on-policy-distillation, get_started/agent,
  pd-disaggregation (heterogeneous server groups fix), examples zh docs
- Add missing examples: on_policy_distillation, eval_multi_task, delta_weight_sync,
  geo3k images
- Fix vLLM flag translations across all example docs:
  - --vllm-mem-fraction-static → --vllm-gpu-memory-utilization
  - Remove non-existent dp-attention flags (--vllm-enable-dp-attention, --vllm-dp-size,
    --vllm-moe-dense-tp-size, --vllm-enable-dp-lm-head, --vllm-ep-size)
  - --vllm-ep-num-redundant-experts → --vllm-eplb-config
  - --vllm-cuda-graph-bs → --vllm-max-cudagraph-capture-size
  - sglang speculative flags → --vllm-speculative-config JSON
  - GLM-4.7 MTP: method=eagle → method=mtp, num_speculative_tokens=4 → 3
  - sgl-router → vllm-router; THUDM/vime → vllm-project/vime
- Fix scripts: restore run-kimi-k2-Instruct/Thinking/qwen3-4B/qwen3-235B-A22B to
  slime-44d29ee-as-vime + pkill precision fix only; restore int4 python3 path

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* revert(scripts): pkill -9 -f pattern back to pkill -9 vllm, align with slime

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* revert(pkill): align all remaining kill patterns with slime (pkill -9 vllm)

Covers examples/, docs/, tests/, and vime/utils -- previously missed in
the scripts/ revert.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* chore: remove gpt-oss-20B script and convert tool (moved to separate PR)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [Doc] Fix broken Qwen3-4B example link in rollout_buffer README (#295)

The rollout_buffer README (English and Chinese) linked to
docs/{en,zh}/models/qwen3-4B.md, but there is no models/ directory —
the referenced doc lives under examples/. Point both links to
docs/{en,zh}/examples/qwen3-4B.md so the setup instructions resolve.

Signed-off-by: ajinkya-metica <ajinkya@metica.com>
Co-authored-by: ajinkya-metica <ajinkya@metica.com>

* sync(slime #2014..#2125): 3-way merge [WIP — 45 conflict files need review] (#286)

* sync(slime #2014..#2125): diff3 3-way merge, conflicts preserved

Mechanical commit 1 of 2 (per knowledge/rl/slime-to-vime-sync-sop.md §2).
diff3 translated 3-way merge on upstream/main (incl #260/#280/#283/#257):
  ours = vime@main, base = translate(slime@#2013), theirs = translate(slime@#2125)

Translation fixes vs prior attempt:
  - casing: SGLang->vLLM (prose) / SGLang<X>->VLLM<X> (identifiers); killed VLlm artifact (was 35 files)
  - dotted module refs slime.X->vime.X now translated (was leaking 'from slime.backends')
These resolved 9 spurious conflicts (46->37 files).

Results: 61 clean / 37 conflict (diff3 markers preserved) / 50 new-to-vime / 5 del.
Conflict markers use readable -L labels (ours/base/theirs). Resolve in commit 2.

Non-conflict provenance fix: vimerl/vime -> vllm/vime in 2 example docs.
Engine patch handling (docker/patch/) deferred to commit 2 per SOP §4.5.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* sync(slime #2014..#2125): resolve all conflicts (commit 2)

Resolved all 37 conflict files / 84 diff3 blocks per agent_run RESOLUTION_POLICY.
Principle: keep vime vLLM impl (ours) + incorporate slime's new features (theirs).

Highlights:
- vLLM API form kept everywhere: /inference/v1/generate, choices parsing,
  AsyncEngineArgs, vLLM flag names (--vllm-gpu-memory-utilization etc).
- Dropped all vllm.srt.* imports (non-existent in real vLLM).
- Accepted new slime features: delta-weight-sync CLI args, append_response_tokens
  (Sample), get_server_info/start_external_rollout_servers/get_rollout_num_engines,
  old-router(<=0.2.1) compat, TrajectoryManager adapter design (vime already adopted it).
- Kept vime-only: --rollout-external, add_router_arguments, _get_metrics_router_addr,
  reinit_wandb_primary_with_open_metrics, update_tracking_open_metrics, modal sandbox,
  VIME_AGENT_* env names, local-vLLM tau-bench user sim.
- Engine patches (docker/patch/): kept ours vllm.patch (22-line MoE fix), dropped
  theirs sglang 2674-line content; deleted sglang-only vllm-top_p.patch (per SOP 4.5).
- Dockerfile kept ours (vllm/vllm-openai base); version.txt accepted theirs nightly.
- run-deepseek-r1.sh: dropped /sgl-workspace dead-path env.

Deviations from policy (documented): vllm_rollout.py abort path kept ours
pause/drain (abort_servers_until_idle would break partial-rollout drain + leave
paused_workers unbound). README ecosystem section left empty (ours) pending
de-translation of provenance.

All changed .py py_compile clean; zero conflict markers; no sglang/slime leakage.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): pre-commit lint/format on resolved files

- re-add dropped `from vllm_router.launch_router import RouterArgs` import in
  vime/utils/arguments.py (add_router_arguments uses it; F821 from conflict resolution)
- drop unused base_top_p_token_ids/offsets in vllm_streaming_rollout.py (F841; came
  from theirs but ours's choices-parsing path doesn't use them)
- black/isort autoformat (anthropic.py, test_agent/*, arguments.py)
- pipeline.yml: agent tests moved to tests/test_agent/; wire new CPU tests

pre-commit: all hooks pass (ruff/autoflake/isort/black/yaml).
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): make CPU CI green (engine import, docker build, agent/utils tests)

Local CPU CI (pre-commit + plugin + agent + utils) all green on 8xH200 host
in python:3.11 containers. Fixes found while running it:

- vllm_engine.py: make 'import vllm_router'/'packaging.parse' lazy (inside
  _register_to_router); top-level import broke CPU import (vllm_router absent in
  CPU CI). The old-router(<=0.2.1) compat branch is theirs-accepted.
- docker/Dockerfile: TMS_CUDA_MAJOR=12 for torch_memory_saver pin (its build
  backend now requires it for CUDA wheels; base is cu129). Unblocks image build.
- agent/adapters/common.py: _run_turn called parse_model_output() without the
  required tokenizer= kwarg -> 500s in adapter tests. Pass tokenizer=tok.
- tests/test_agent/_fakes.py: FakeVLLMServer served sglang /generate + meta_info;
  retarget to vime /inference/v1/generate + choices shape + x-session-id header.
- tests/test_agent/test_adapters.py: parse_model_output(tokenizer=...) + assert
  vime body keys (token_ids/max_tokens).
- tests/utils/test_vllm_config.py: vLLMConfig->VllmConfig (4 sites); fake router
  returns 3-tuple (ip,port,prom) matching _start_router; drop spurious resolve().
- tests/test_megatron_argument_validation.py: add num_gpus_per_node=8 to the
  vime_validate_args fixture (vime colocate override needs it).
- .buildkite/pipeline.yml: agent tests -> tests/test_agent/*; +cispo_loss,
  +logprob_response_spans (CPU-safe); test_rollout_metrics stays GPU-only (imports vllm).

Engine patch verdict (PATCH_ASSESSMENT.md): P1-P7 vLLM doesn't need (NIXL/Mooncake
native); kept ours vllm.patch, dropped sglang content + top_p.patch.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): unblock GPU run (scipy pin, router arg dup, top-p-replay gate)

Found running GPU CI on 8xH200 with vllm/vime:latest:
- docker/Dockerfile: pin scipy<1.14 next to numpy<2 (scipy drifted to 1.18 which
  needs numpy>=2 and uses removed np.long -> 'import vllm' crash).
- vllm_utils/arguments.py: drop sync-added RouterArgs.add_cli_args + its import in
  add_vllm_router_arguments. main exposes the full router surface only in
  utils.add_router_arguments; the duplicate re-registered --router-request-timeout-secs
  -> argparse conflict at train startup.
- megatron_utils/loss.py: get_rollout_top_p_logprob_kwargs falls back to full-vocab
  logprob when top-p nucleus token ids are absent instead of raising. slime's
  top-p-replay needs engine-returned top-p tokens; vime's vLLM /inference/v1/generate
  does not expose them (sglang-only). Matches vime pre-sync behavior; flagged in
  OVERNIGHT_REPORT for review.

Image import smoke + Megatron ckpt load + 4x VLLMEngine bringup + NCCL weight
transfer all confirmed working in-image before this.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(gpu): drop deleted test_qwen2.5_0.5B_ppo_critic_only_short from short suite

slime deleted tests/test_qwen2.5_0.5B_ppo_critic_only_short.py this window (#2014..#2125);
gpu_suites.py still listed it -> 'no such file' exit 2. Critic-only path is still
covered by test_qwen3_4B_ppo_train_critic_only (megatron suite). Other 3 short tests
(gsm8k_async, gsm8k, fully_async) pass on 8xH200.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore base_log_probs init in streaming rollout (None+list crash)

GPU test_qwen3_4B_streaming_partial_rollout hit
'TypeError: unsupported operand +: NoneType and list' at vllm_streaming_rollout.py:234.
The conflict resolution changed base_log_probs from main's
`list(sample.rollout_log_probs or [])` to a None-able form; a fresh sample
(rollout_log_probs=None) then did None + call_log_probs. Restored main's form.
Real sync-resolution regression caught by GPU CI.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): streaming rollout uses append_response_tokens (was renamed update_from_meta_info)

slime #2110 renamed Sample.update_from_meta_info -> append_response_tokens; vllm_rollout
was updated but vllm_streaming_rollout still called the old name (AttributeError at
generate_streaming). Streaming already accumulates tokens incrementally for partial-rollout,
so call append_response_tokens(meta_info=meta) with tokens omitted -> metadata-only finalize
(no double-append). Caught by GPU test_qwen3_4B_streaming_partial_rollout.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore vime unconditional colocate rollout_num_gpus re-derive

3-way compare (slime / vime-main / PR) showed the merge made a broken hybrid:
it KEPT vime's num_gpus_per_node colocate override (which assumes rollout_num_gpus
is forced to actor_num_gpus_per_node*actor_num_nodes) but REPLACED vime's
unconditional re-derive (`!= -> re-derive`) with slime's `is None`-only form.
When a colocate test's rollout_num_gpus is non-None but mismatches, it was left
mis-sized -> engine/GPU misplacement -> mixed_offload IPC-UUID mismatch + ckpt
'Free memory < util'. slime passes (no override, self-consistent is-None); vime
main passes (override + unconditional re-derive, coupled). Restore vime's
re-derive; keep slime's new rollout_num_gpus==0 branch (checked first).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): slime-consistency review pass (docs, rollout routed_experts, comments)

Docs (EN+zh parity):
- fault-tolerance: revert junk 'server'->'engine' mistranslation (keep correct /health endpoint, verified vs vllm_engine.py)
- vllm-config: 'ServerArgs'->'EngineArgs' (sglang class -> vLLM AsyncEngineArgs u FrontendArgs); fix duplicated 'vllm-router (vllm-router)' alias
- customization: restore over-deleted 'custom_generate -> list[Sample]' section + signature (dropped only the vime-absent search-r1 example link)

Rollout:
- routed_experts now flows through the slime-identical Sample._apply_meta_info (single assignment site, torch.int32 tensor matching downstream) instead of an inline numpy assign; vLLM .npy-on-choice decode stays (engine wire-format delta). Both vllm_rollout and vllm_streaming_rollout.

Comments for future syncers:
- --opd-teacher-model + on_policy_distillation: engine-driven divergence (vLLM model field; sglang /generate has none)
- overrides / _vllm_server_field_names: AsyncEngineArgs u FrontendArgs == slime's sglang ServerArgs

Examples/docker/etc: drop vime-absent npu/retool/search-r1/tau-bench files; restore eval_multi_task; docker alignment with slime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): pre-commit green — define base in streaming MM render (F821) + isort/black on test_agent

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): correct two mistranslated CPU tests (colocate rollout-gpu re-derive; parse_model_output tokenizer)

- test_..._preserves_larger_rollout_gpus_under_colocate asserted slime behavior (==12); vime re-derives to actor*nodes=8 under colocate (commit 97013044). Renamed + assert ==8 + divergence note. vime-main never had the test; slime does.

- test_parse_model_output_plain_text_no_parsers called parse_model_output without the required tokenizer kwarg (#198 made it required for vLLM parsers). Pass tokenizer=None (unused on the no-parser path).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): streaming rollout posts to /inference/v1/generate, not sglang /generate

The slime diff3 merge took slime's sglang endpoint (/generate) for the streaming rollout POST instead of keeping vime's vLLM endpoint (/inference/v1/generate). main (7198547c) had the correct URL; the sync regressed it (and dropped the base var). Result: 404 Not Found at vllm_streaming_rollout.py:182 -> test_qwen3_4B_streaming_partial_rollout fails. Caught on a clean h200 node. The file's own docstrings already say /inference/v1/generate throughout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): thread rollout port cursor globally across multi-model engines

The diff3 merge adopted slime's deferred rollout-engine init (start_rollout_servers
now returns pending_init_handles awaited by the caller instead of ray.get-ing each
model's engines before the next). That broke an implicit invariant the per-model
port_cursors reset relied on: in main, model 0's engines were fully bound before
model 1 allocated ports, so the free-port bind-test in
_allocate_rollout_engine_addr_and_ports_normal skipped model 0's ports. With
deferred init, model 1 allocates while model 0 is unbound, the bind-test sees the
base ports free, and a second model (e.g. mixed_offload's frozen "ref") lands on
the same 15000-15003 as the actor. The actor's POST /update_weights to :15002 then
hits the never-started ref engine -> vLLM 500 "start_weight_update must be called
before update_weights" (test_vllm_config_mixed_offload[_ft]).

Fix: initialize port_cursors once before the model loop so the per-node next-free
cursor is monotonic across all models, keeping every engine's ports disjoint
regardless of bind timing. Single-model behaviour is unchanged; the per-model
reset only existed to scope cursors that are already node-keyed.

Caught on h200 GPU CI (new nightly-dev-20260618a image, which added the
start_weight_update-before-update_weights enforcement that exposed the collision).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore robust pkill pattern so ckpt cleanup kills vLLM ray actors

PR #260 ("complete slime-exact port") changed execute_train's pre-launch
cleanup from

    pkill -9 -f '[v]llm serve|VLL[M]::'

to

    pkill -9 vllm

as an over-literal sglang->vllm translation. But the vLLM rollout engine runs
as Ray actor processes whose process *name* is python/ray, with "VLLM::" only in
the command line — so `pkill -9 vllm` (name match, no -f) does not kill them.
Leftover engine processes from the ckpt test's save phase survive into the load
phase, holding ~115 GiB, so the load-phase engine starts with ~24/139 GiB free
and dies with "Free memory ... less than desired GPU memory utilization (0.8,
111.84 GiB)" (test_qwen3_4B_ckpt.py, both --async-save and not).

Restore the cmdline-match pattern `-f '[v]llm serve|VLL[M]::'`.

Bisected on h200: ckpt PASSES at 289ee6d / e62d44fa (old pattern, 4/4 runs) and
FAILS at 7198547c/main + PR (new pattern, 0/3), same old image -> code regression
in #260. Verified: pkill-fixed PR code + new pr286 image -> ckpt PASS (579s).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* chore(sync): replace all `pkill -9 vllm` with cmdline-match pattern

Same root cause as 764e1e18 (command_utils.py): `pkill -9 vllm` matches by
process *name*, but vLLM rollout engines run as Ray actor processes (python/ray
named, "VLLM::" only in the command line), so the name match never kills them.
Apply the robust cmdline pattern `pkill -9 -f '[v]llm serve|VLL[M]::'` everywhere
the bare `pkill -9 vllm` cleanup was used across run/example scripts, so leftover
engines don't squat GPUs across runs. No logic change beyond the kill pattern.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): restore vLLM core.py partial-wake sleep-guard (#44483)

Commit d41f0aa1 removed the core.py sleep-guard hunk from
docker/patch/latest/vllm.patch on the assumption it was "already in
v0.23.0". It is not: stock v0.23.0 `vllm/v1/engine/core.py` calls
`resume_scheduler()` and `execute_dummy_batch()` even during a partial
(weights-only) wake. So a colocate DP+EP pd_mooncake rollout, right after
`POST /wake_up?tags=weights` (KV cache still released under level-2 sleep),
has its DP busy-loop fire a decode-shaped dummy batch that touches freed
KV -> the scheduler_metadata write in flashattn_mla.py:234 (MLA, glm4.7)
and flash_attn.py:547 (FA3, qwen3.6) raises `CUDA error: invalid argument`.

Restore the guard (`if not self.model_executor.is_sleeping` around
resume_scheduler; `if not self.is_sleeping()` around execute_dummy_batch),
keeping the all2all_utils weight-reload fix. This is the #173 sleep-guard
patch re-expressed against v0.23.0 line numbers.

Verified: git-apply --check clean against stock v0.23.0; both guards land;
glm4.7/qwen3.6 pd_mooncake reproduced the crash without it (the 8-day-old
image that still carried the guard passes both).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): drop scipy<1.14 pin, mirror slime numpy<2 only

The scipy<1.14 pin (added in 0cda11c6 while chasing the pd_mooncake crash)
was a red herring: the real cause was the dropped core.py partial-wake
sleep-guard, now restored. slime-2125-as-vime pins only `numpy<2`; this
restores that exact line. numpy 1.26.4 + scipy resolved naturally matches
the working baseline.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): keep FlashQLA install gated behind INSTALL_FLASHQLA=0

slime installs FlashQLA unconditionally, but it is sm90/Hopper-only.
Restore vime's original gated form (default off; --qwen-gdn-backend fla
elsewhere). CI build passes --build-arg INSTALL_FLASHQLA=1 to include it.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(vllm-config): fix inference-only FAQ — vime launches engines in-process

The mechanical mirror of slime's answer steered users to vLLM's standalone
`vllm serve` (slime's `launch_server` analog) for inference-only. That is
misleading for vime: like slime, vime launches the vLLM engines in-process
from `--vllm-config` (same in-process path as training), so a rollout-only
run serves directly with no separate server process. Point standalone users
to `--rollout-external-engine-addrs` instead. EN + ZH.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(debug): restore INT4 / Compressed-Tensors checkpoint section

vime's debug.md was missing slime's "INT4 / Compressed-Tensors Quantization
Checkpoint Issues" section (slime #1642) — dropped in an earlier sync, not
present on main. Restore it (EN + ZH), translated sglang→vLLM / Megatron→vLLM.
Covers the quantization_config.ignore list, all-zero MoE router weights
(mlp.gate.weight) when mis-quantized, missing safetensors shards, and
diagnosis via --check-weight-update-equal / --debug-rollout-only.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(vllm-config): use _run_vllm_server for inference-only FAQ

Keep slime's wording; the only engine-coupled fix is the standalone launcher
name. slime's `launch_server` is its in-process engine entry; vime's analog
is `_run_vllm_server` (vllm_engine.py, launched via multiprocessing.Process),
not the standalone `vllm serve` CLI. EN + ZH.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): restore scipy pin (scipy<1.18) — vime base needs it

Reverts the scipy-pin removal in b359b245, which was wrong: vime's
vllm/vllm-openai base ships no scipy, so unpinned the build pulls scipy>=1.18,
which hard-requires numpy>=2 and uses np.long (removed numpy>=1.24) -> crashes
against the numpy<2 reinstall (Megatron needs numpy 1.x). slime's sglang base
resolves scipy 1.17.1 natively (numpy-1.x compatible), so slime needs no pin;
this is a base-image divergence, not a red herring. Pin boundary is 1.18
(slime runs 1.17.1), not the earlier 1.14 over-estimate.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(scripts): properly translate sglang args in glm5.2-744B + glm4.7-355B-delta

These two were prefix-swapped (--sglang-X -> --vllm-X) without semantic mapping,
leaving ~20 args that aren't vLLM AsyncEngineArgs (argparse would reject). Apply the
knowledge/rl/sglang-to-vllm-translation.md §5.5 mappings:
- dp-size->data-parallel-size, ep-size->enable-expert-parallel, max-running-requests->
  max-num-seqs, cuda-graph-max-bs->max-cudagraph-capture-size
- 5x/4x --speculative-* -> one --vllm-speculative-config JSON (§5.2)
- DeepEP: per-group deepep_mode auto/low_latency -> all2all_backend deepep_high_throughput/
  low_latency in the --vllm-config overrides (vLLM has no 'auto'; PD encodes it per-role)
- watchdog-timeout -> env VLLM_ENGINE_ITERATION_TIMEOUT_S
- drop sglang-only: dp-attention / dp-lm-head / moe-dense-tp / disable-overlap-schedule / NSA
  backends (vLLM selects DeepSeek sparse attn per model) / engine delta-receiver knobs
- flag PD mooncake transport (-> --vllm-kv-transfer-config) as fabric-specific TODO

These are 744B/355B scripts not runnable in CI — translations are SOP-mapped but
hardware-unvalidated (flagged inline).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(args): hard-guard unverified delta weight-sync mode

--update-weight-mode=delta (PR #278 lineage) is not yet validated on
vime+vLLM (vLLM exposes dense/sparse_flat only, not slime's gap-delta/
zstd encoding). Raise NotImplementedError at arg-validation so it fails
fast at startup instead of crashing mid weight-sync. Downstream delta
code is kept untouched; remove this raise once a real delta-load run
passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs,test: correct rollout engine endpoint to /inference/v1/generate

The agent rollout engine is reached at vime's vLLM ``/inference/v1/generate``
(see vllm_rollout.get_model_url default + common.call_vllm_generate), not the
bare ``/generate`` of sglang. Fix the imprecise path in adapter/test docstrings
and comments, and rewrite the vllm-config.md custom-rollout examples (en+zh):
they were still sglang-shaped (``/generate`` path + ``{"text":..., "return_logprob":
True}`` body). Use vime's real request schema instead -- ``{"model","token_ids",
"sampling_params"}`` with ``max_tokens``/``logprobs``, ``prompt_logprobs`` for
fixed-sequence scoring, and the ``choices[0]`` response shape.

No code/logic change: comments, docstrings, and doc examples only. The Megatron
training server's own ``/generate`` endpoint and the sglang citation in
arguments.py are correct and left untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style(args): collapse delta-guard message to one line (black)

The delta-guard NotImplementedError message was split across two adjacent
string literals; black on the CI (line-length 119) collapses/normalizes it.
Make it a single clean literal so pre-commit is green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test(args): assert delta weight-sync is guarded off (not per-condition)

The hard delta guard (7bb19e67) raises NotImplementedError at the top of the
delta branch, making the downstream colocate / unknown-transport rejections
unreachable. Replace test_update_weight_delta_rejects_colocate and
test_update_weight_delta_rejects_unknown_transport (whose ValueError paths no
longer fire) with a single test_update_weight_delta_disabled that asserts the
guard raises for any delta config. Breadcrumb left to restore the per-condition
tests when delta is verified and the guard is lifted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore dropped weight-sync metrics chain + delta dispatch (slime parity)

A mirror cross-check of the delta-weight-sync surface (slime #1806/#1991) found
the #2014..#2125 sync had silently dropped several slime-faithful pieces:

* extra_metrics logging chain — slime threads weight-update metrics from the
  actor through log_perf_data -> log_perf_data_raw. vime dropped the param at
  all three layers, so weight-update metrics were never logged. Restored:
  train_metric_utils.log_perf_data_raw(extra_metrics=...), data.log_perf_data
  passthrough, and actor passing self.weight_updater.pop_metrics().

* pop_metrics on UpdateWeightFromDistributed — the default (non-colocate, nccl)
  weight_updater. slime gives all three updaters a pop_metrics() stub so the
  actor can call it uniformly; vime kept it on tensor/disk but dropped it on
  distributed, which would AttributeError once the actor calls it. Restored the
  ~5-line stub (delta-specific plumbing stays dropped — vime+vLLM has no DeltaSpec).

* actor delta-mode dispatch branch — restores the elif selecting
  UpdateWeightFromDistributedDelta. Dead code behind the validation guard that
  rejects --update-weight-mode=delta, so vime mirrors slime with the guard as
  the single divergence.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* chore(sync): align comments to the mechanical mirror

Comment-only pass; no behavior change. Aligns vime comments to what a faithful
slime->vime translation would carry:

* Strip vime-divergence rationale markers (delta guard, opd-teacher-model,
  top-p fallback, colocate/delta tests). The rationale belongs in the
  divergence manifest, not inline; the guarded code + self-explanatory
  NotImplementedError messages stand on their own.
* De-verbose vLLM-specific comments to mirror scale: the router-args block,
  the AsyncEngineArgs u FrontendArgs docstring, and the MoE-replay /
  streaming-rollout blocks that slime does not carry at that length.
* Restore slime-original comments the sync had dropped or naively translated,
  with judgment translation of sglang-specific terms: session_id routing
  ("vLLM router", not the mechanical "Model Gateway"), "Prepare payload for
  vLLM server", the unique-session_id loop, and the pending-tasks wait.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(delta): note delta weight-sync not yet verified on vime+vLLM (PR #286 review)

Per review on PR #286: delta weight sync is documented here but the arg guard
disables --update-weight-mode=delta. Add a top-of-page note (en + zh) so users
see it before hitting NotImplementedError at argparse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Example] Add MemAgent long-context RL example (mem_agent) (#291)

* Add examples/mem_agent for long-context MemAgent RL

Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>

* Address review: harden mem_agent convert and eval scripts

Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>

---------

Signed-off-by: kaiyuan <kyxiezju@163.com>

* fix(rollout): abort vLLM rollout via delete-type /abort_requests (#296)

Under --partial-rollout, abort() (pause -> drain -> resume) deadlocks:
/pause?mode=abort puts the scheduler in PAUSED_NEW, and a /generate that races
in after the pause parks in the waiting queue and never returns until /resume,
which runs after the drain. Reordering to pause -> resume -> drain avoids the
hang, but resume reopens the whole queue so the long tail runs to COMPLETION --
breaking partial rollout's "truncate the tail, resume it next step" semantics.

Switch to a delete-type abort instead:

- vLLM: add POST /abort_requests to the RLHF api_router -> EngineClient.abort()
  (removes queued requests from the waiting queue and finish-aborts running
  ones, whose partial output returns on the original /generate stream). It does
  not pause the scheduler, so there is no /resume and no deadlock. Shipped as a
  build-time patch in docker/patch/latest/vllm.patch.

- vime: server_control.abort_inflight_requests() replaces the unused,
  slime-mirrored abort_servers_until_idle / _v1_loads helper (vLLM has neither
  /abort_request nor /v1/loads). abort() re-issues the sweep across drain waves
  and converges on state.pendings, with a timeout bounding how long a late
  multi-turn straggler can run before being truncated to partial.

- vllm_engine: drop the legacy version gate in _register_to_router. vime ships
  its own vllm-router, so only the /workers payload path is needed.

Adds delete-type abort unit tests.

AI-assisted change; reviewed by a human before submission.

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Josephasafg <ajgard7@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* [Bugfix][Rollout] Wire prefix_cache_hit_rate through vLLM usage (#303)

rollout/prefix_cache_hit_rate (and avg_cached_tokens_per_sample) were
structurally 0 due to three coupled gaps:

1. vLLM's non-streaming /inference/v1/generate builds a `usage` block but
   silently drops it: GenerateResponse (v0.23.0 serve/disagg/protocol.py)
   never declared a `usage` field and has no extra="allow", so pydantic
   discards it. Patched in docker/patch/latest/vllm.patch, mirroring
   GenerateStreamResponse. (Upstream fix filed against vllm-project/vllm.)
2. The rollout parser read usage.prompt_tokens/completion_tokens but never
   usage.prompt_tokens_details.cached_tokens -> PrefixCacheInfo numerator
   pinned to 0. Now read in both vllm_rollout and vllm_streaming_rollout.
3. Streaming additionally needs stream_options.include_usage=True for vLLM
   to emit the terminal usage SSE chunk.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docker): /abort_requests abort-all fix + cu13 image variant (#317)

* fix(docker): abort-all in the /abort_requests vLLM patch must abort by internal ids

The bundled /abort_requests endpoint (merged in #296) populated request_ids from output_processor.request_states (internal ids) but called engine.abort() with the default internal=False, so they were treated as external, matched nothing, and POST /abort_requests {} silently aborted no requests under default request-id randomization. Abort the all-in-flight list as internal. Mirrors vllm-project/vllm#47173.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style: reject malformed JSON in /abort_requests patch with 400

Match the sibling dev endpoints and the Rust frontend (400 on malformed JSON) instead of silently treating it as empty. Mirrors vllm-project/vllm#47173.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): abort-all patch must also abort parallel-sampling parents

The /abort_requests patch enumerated request_states (child internal ids
only), so with n>1 the ParentRequest entry leaked. Include
parent_requests keys in the abort-all set. Mirrors vllm PR #47173.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* feat(docker): cu13 image variant + TMS cu13 preload

Port the cu13 build support from #307: ENABLE_CUDA_13 branches the apt
dev headers, cublas header, TransformerEngine (source-built for cu13),
TMS_CUDA_MAJOR auto-detect, and the cudnn pin. justfile gains a
build-cu13 target and a VARIANT-prefixed manifest. actor_group preloads
the cu13 TMS .so.

Also switch the vLLM patch apply to --allow-empty so the build survives
once the patch is emptied upstream.

Excludes #307's NCCL_CUMEM_ENABLE default flip (0->1) and the glm5.2
scripts by request.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* Initial ROCm support for vime (#273)

* Added ROCm quick start guide

* changed wording

* changed docker image

* updated instructions

* Add initial working ROCm 7.0.2 build (Dockerfile.rocm + patch)

Initial working setup for building/running vime on ROCm 7.0.2 (gfx950).

* ROCm: re-base Dockerfile on vime mainline Megatron + shared patch

* ROCm: apply checkpoint-writer patch in HF->torch_dist converter

* ROCm: align async run script cleanup with run-qwen3-4B.sh format

* ROCm: move megatron.patch to docker/amd_patch/ (slime layout)

* ROCm: re-add fused-kernels init patch and AMD doc/template references

---------

Co-authored-by: pancake0003 <146360951+pancake0003@users.noreply.github.com>
Co-authored-by: indianspeedster <cp3793@nyu.edu>

* [Doc] Fix stale Qwen3-30B-A3B example: restore FP8-inference section + rewrite multi-node (#322)

* [Doc] Fix stale Qwen3-30B-A3B example: dataset download + align with actual script

The Qwen3-30B-A3B example documented an elaborate env-var-driven multi-node
script (ACTOR_NUM_NODES / MEGATRON_TP / ENABLE_R3 auto-behavior, "script skips
Ray", default batch 4/2/8, --vllm-moe-backend triton). None of that exists in
the current slime-exact scripts/run-qwen3-30B-A3B.sh — it was orphaned when the
script was reverted to the slime-exact single-node port. The real defaults are
rollout-batch-size 32 / n-samples 8 / global-batch 256.

Changes (zh + en, kept concise):
- Environment Preparation: add the concrete `hf download` commands for the
  model, dapo-math-17k train data, and aime-2024 eval data (previously the
  section only pointed at the Qwen3-4B doc).
- Restore the "BF16 training + FP8 inference" section (present upstream in
  slime, dropped here); matches the commented --hf-checkpoint line in the script.
- Replace the orphaned multi-node section with the concise, accurate
  slime-style guidance (manual modifications + vLLM EPLB example), fixing the
  broken zh anchor to the Chinese quick_start heading.

Doc now matches scripts/run-qwen3-30B-A3B.sh; no script changes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Expand Qwen3-30B-A3B multi-node section (code-grounded)

Replace the concise multi-node bullets with an actionable but verified guide:
Ray cross-node startup (from quick_start), the exact script edits
(--actor-num-nodes, colocate auto rollout-num-gpus), parallelism scaling as
constraints (defer concrete large-scale ratios to the GLM/DeepSeek examples
rather than fabricating a 2-node table), and the real pitfalls we hit
(keep each vLLM engine within a node to avoid cross-node TP=16 slowdown/
numerics, MASTER_ADDR / NCCL_SOCKET_IFNAME, global-batch identity,
--num-gpus-per-node for <8 GPUs/node). Every claim traces to vime/utils/
arguments.py, quick_start, or observed multi-node runs; no "tested" claims.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Drop model-download line from Qwen3-30B-A3B example

Model download is already covered by the referenced Qwen3-4B doc; keep only the
train/eval dataset download (dapo-math-17k, aime-2024) surfaced inline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Drop redundant dataset download; keep env-prep slime-exact

The intro sentence already covers env/model/data/ckpt via the Qwen3-4B
reference, and slime's 30B doc carries no dataset block. Remove it; keep the
30B-specific torchrun checkpoint conversion (differs from 4B's single-process
python). Env-prep is now byte-identical to upstream.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Multi-node: also skip the script's Ray-cleanup preamble

Codex review (P2): for the manual multi-node path, removing only the
`ray start --head` line is not enough — the script's initial cleanup block
(`ray stop --force` + `pkill -9 ray/python/redis`) would tear down the
manually-started head before `ray job submit`, failing submission and
orphaning the workers. Tell users to remove/comment both the cleanup block
and the `ray start --head` line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Fix H800→H100 typo in Qwen3-30B-A3B example (#325)

The page title says "8xH100" but the CPU-Adam bullet said "8xH800
environment", an inconsistency inherited from slime. Both are 80GB cards so
the point is unchanged; align the text with the title.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Add Ascend NPU platform tutorial and expand hardware support section (#334)

Signed-off-by: kaiyuan <kyxiezju@163.com>
Co-authored-by: XiaoxinWang <wangxiaoxin7@huawei.com>
Co-authored-by: FulinGao_HW <gaofulin1@huawei.com>
Co-authored-by: meihanc <zr010426ztt@outlook.com>
Co-authored-by: wuxiang <498160096@qq.com>
Co-authored-by: yuxinshan <syx_ctyg@126.com>
Co-authored-by: flb_dayo <floatlibai@gmail.com>
Co-authored-by: Windfeng8 <523758380@qq.com>
Co-authored-by: YZY <532183776@qq.com>

* sync: update vime through slime #2185 (#338)

Sync slime a897e1f4..680824dd using the mechanical merge plus vLLM adaptation workflow.

* revert: sync through slime #2185 (#338)

Revert the prematurely merged slime sync and restore the pre-#338 tree.

* [bugfix] Fix distributed update weights for pipeline parallel (#329)

* Fix pp for distributed update weights

Signed-off-by: knlnguyen1802 <knlnguyen1802@g…
CalvinXKY added a commit that referenced this pull request Nov 19, 2026
* refactor(test): flatten CPU unit tests and centralize import stubs (#238)

* refactor(test): align CPU unit test layout with slime and add _unit_stubs

Move tests/unit/* into tests/ and tests/utils/, replace nested conftest stubs with shared helpers for optional deps, and register the moved files in cpu-unittest CI. Write generated workflow YAML as UTF-8 on Windows.

Signed-off-by: kaiyuan <kyxiezju@163.com>

* fix(vllm_engine): guard HTTPError.add_note for Python 3.10 CI

Exception.add_note() requires Python 3.11+; cpu-unittest runs on 3.10.

Signed-off-by: kaiyuan <kyxiezju@163.com>

* refactor(test): drop pytest pythonpath, bootstrap tests/ in test files

Remove redundant pythonpath from pyproject.toml; each test that imports
_unit_stubs prepends tests/ to sys.path (including test_vllm_rollout.py).

Signed-off-by: kaiyuan <kyxiezju@163.com>

* ci: bump cpu-unittest to Python 3.11 and revert add_note guard

Run cpu-unittest on 3.11 so HTTPError.add_note is available in CI without Py3.10 guards in vllm_engine or conditional __notes__ assertions in tests.

Signed-off-by: kaiyuan <kyxiezju@163.com>

---------

Signed-off-by: kaiyuan <kyxiezju@163.com>

* docs: align vLLM docs with current defaults (#240)

* docs: align vllm docs with current defaults

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: fix router policy examples

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: simplify ray multi-node wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore quick start hardware notes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: clarify cache-aware routing support

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove extra router policy example

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore cache aware policy example

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore hardware support wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove conda fallback note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: update docs logo

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: replace legacy logo assets

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: add README docs badges

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: replace logo image

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove redundant logo jpg

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: adjust hardware support wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: refine hardware support wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore hardware support notes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: add GB support guard notes

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: clarify profiling flow

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: shorten profiling note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* tools: simplify rollout profiler helper

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore profiling request range

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove profiling note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: switch logo asset to jpg

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* update docs logo

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* update docs logo asset

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: move eplb note to faq

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: rephrase profiling wording

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove eplb note

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore eplb example

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: restore eplb example in english

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>

docs: align vLLM docs with current defaults
CI: merged with buildkite/vime-ci failure (infra); GA checks passed. See PR #240 comment.

* refactor(weight-sync): use vLLM native /update_weights, remove worker extension (#246)

* refactor(weight-sync): use vLLM native /update_weights instead of worker extension

Remove the custom `vLLMColocateWorkerExtension` and `_VLLMHijack`
monkey-patch introduced in PR #104. These reimplemented vLLM's built-in
`IPCWeightTransferEngine.receive_weights()` (UUID routing, device_index
remapping, layerwise reload) inside a worker extension, bypassing the
native `/update_weights` endpoint in favor of `/collective_rpc`.

vLLM's native IPC engine already handles everything the extension did.
This change routes `update_weights_from_tensor` back through
`POST /update_weights` (the path that was working before PR #104) and
deletes ~160 lines of redundant code.

Changes:
- `update_weight_from_tensor.py`: delete `_VLLMHijack`,
  `vLLMColocateWorkerExtension`, and fix `ipc_handles` format to store
  `ipc_args` (not `(rebuild_func, ipc_args)`) for native API compat
- `vllm_engine.py`: revert `update_weights_from_tensor` from
  `/collective_rpc` to `/update_weights`, delete `update_weights_chunk`,
  remove `--worker-extension-cls` injection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(reloadable-pg): add missing gather_object patch; switch IPC gather to match slime

ReloadableProcessGroup monkey-patched all_gather_object but not
gather_object, causing "Group is not registered" after Megatron
offload/reload. This forced _send_to_colocated_engine to use the
less efficient all_gather_object (all ranks receive, only leader uses).

Fix: patch gather_object in the same list, then switch to
dist.gather_object (only leader allocates the receive buffer) to
align with slime's implementation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add Buildkite CI pipeline (CPU jobs + manual-gated GPU suites) and remove PR test on GHA (#239)

* ci: add Buildkite pipeline for always-on CPU jobs

Port the always-on jobs from .github/workflows/pr-test.yml.j2 (pre-commit
gate, plugin contracts, agent adapter, in-image unit tests) to a single
dynamically generated Buildkite pipeline targeting the vLLM elastic-stack
CPU queues. GitHub Actions keeps running in parallel and stays
authoritative; GPU suites are not migrated yet.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: replace Buildkite generator with static pipeline.yml

Drop generate_pipeline.py in favor of a plain static .buildkite/pipeline.yml
defining the four always-on CPU steps directly (pre-commit gate, plugin
contracts, agent adapter, in-image unit tests). Simpler to read and review for
a first cut; the GHA workflow stays authoritative and GPU suites are still out
of scope.

Pass GIT_CONFIG_PARAMETERS into every container so git (in pre-commit) doesn't
abort with "dubious ownership" on the host-owned checkout, and fix the
depends_on typo in the README.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): fix CPU test deps; add manual gate for GPU suites

plugin-contracts failed in build #3 on tests/utils/test_hf_checkpoint_saver.py
(ModuleNotFoundError: safetensors): the dep list predated the slime sync in
#232 which added requests/ray/safetensors to the GHA template. Mirror it.

GitHub PR labels can't trigger Buildkite jobs, so expose the run-ci-* GPU
suites behind a block step instead: unblocking offers a multi-select of suites
(short / vllm-config / megatron / precision / ckpt) and gpu_suites.py uploads
one step per test with the same gpu_lock_exec + docker invocations and
per-test DEEPEP/FP8/EVAL env combos as the GHA jobs. blocked_state: passed
keeps the commit status green when the gate is left untouched. GPU steps
target a new vime-gpu agent queue (self-hosted hosts; see README).

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): run GPU suites on mithril-h100-pool; pin gloo to loopback

Build #4's unblock test showed the CI cluster rejects uploads targeting a
nonexistent queue, and rather than minting a new queue, follow the pattern
vllm-omni already uses for mithril-h100-pool: each GPU job is a Kubernetes pod
(agent-stack-k8s kubernetes plugin) on an H100 SXM node with nvidia.com/gpu
limits (4 or 8), memory-backed /dev/shm, and /mnt/hf-cache mounted as HF_HOME.
vime tests hf-download their models, so the warm HF cache replaces the GHA
runners' /mnt/nvme0n1/vime_ci mounts; the docker-run wrapper goes away since
the pod runs the vime CI image directly.

Also pin GLOO/TP_SOCKET_IFNAME=lo in the plugin-contracts container:
test_metric_report_dist hung intermittently (build #4 timed out at 30 min)
because gloo can pick a non-loopback interface inside a bridge-network
container.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): expandable_segments for the borderline OOM short test

test_qwen3.5_0.8B_gsm8k_async_short OOMed in compute_log_probs on the mithril
pool's 80 GB H100s (build #6) with 7 GiB reserved-but-unallocated — the
allocator-fragmentation case PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
exists for. Scope it to this test's pod only (vLLM sleep-mode CuMemAllocator
can conflict with expandable segments) via verbatim pass-through of non-VIME
env overrides. The other short tests passed on H100 pods unchanged.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): soft-fail the two known-H100-incompatible GPU tests

Builds #6/#7 isolated two test-level failures on the mithril 80 GB H100s,
neither a pipeline issue:
- gsm8k_async_short OOMs as tuned (67 GiB live on the actor GPU after
  expandable_segments eliminated fragmentation; its sync twin passes).
- parallel_check's CP=2 grad norm diverges ~4% from the same-node baseline
  recording, a topology-sensitive numerical invariance question.

Mark exactly these two soft_fail so they keep running and stay visible on
Buildkite without failing the build; their authoritative gate remains the
GHA label jobs.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): keep the two H100-incompatible GPU tests failing loudly

Revert the soft_fail: per review, the gsm8k_async_short OOM and the
parallel_check CP-invariance divergence should stay visible as hard failures
on Buildkite until the underlying issues are fixed. Keep the diagnostic
comments and the test-scoped expandable_segments setting.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(buildkite): soft-fail the two H100-incompatible GPU tests after all

Re-apply b334784 (reverted in 0a98010): per the follow-up decision, mark
gsm8k_async_short and parallel_check soft_fail so they keep running visibly
on mithril without failing the build, with the GHA label jobs as their
authoritative gate until the OOM tuning and CP-invariance questions are
resolved.

https://claude.ai/code/session_01BSqHKH1FafdRobA7ZeRzBQ
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): resolve 0.8B async OOM on H100 by reducing max-tokens-per-gpu

Root cause: Qwen3.5's 248K vocab produces [T, 248320] fp32 logits tensors.
calculate_log_probs_and_entropy holds 5 copies simultaneously (2 clones +
2 intermediates + original). At max-tokens-per-gpu=9216, each copy is
~8.5 GB → 42.6 GB from logits alone, exceeding H100 80 GB with
activations and reserved pool fragmentation.

Fix: reduce max-tokens-per-gpu from 9216 to 2048. Peak drops from 117.6 GB
to 39.6 GB (measured on H200), well within H100's 80 GB. GSM8K's longest
sequence is ~1200 tokens, so 2048 still fits all samples.

Also removes gsm8k_async_short from SOFT_FAIL_ON_H100 (no longer needed)
and the expandable_segments workaround.

parallel_check remains soft-fail: ~11% flake rate on TP4+per-token-loss,
confirmed same behavior in slime (Megatron FP reduction-order issue).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style: format update_weight_from_tensor

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* remove github workflows

Signed-off-by: khluu <khluu000@gmail.com>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>

* [codex] update docs branding and fix Buildkite CPU tests (#249)

* docs: update docs favicon to Vime branding

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: fix unit test path in buildkite

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: force triton stub in CPU unit tests

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: align update_weights_from_tensor with native vllm

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: isolate deep_gemm and triton in cpu utils

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(arguments): include DP in TP auto-compute default (#252)

* fix(arguments): include DP in TP auto-compute default

The `vllm_parse_args` default for `vllm_tensor_parallel_size` was
computed as `rollout_num_gpus_per_engine // pp_size`, missing the
`dp_size` divisor. This made the default TP too large when
`--vllm-data-parallel-size > 1` (required for expert parallelism).

Align the formula with `_resolve_vllm_parallel_sizes` in
`vllm_engine.py`, which correctly uses `gpus // (pp * dp)`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test(arguments): add DP>1 cases for TP auto-compute

Cover the fix: TP = gpus_per_engine // (PP * DP).

- test_parse_args_tp_default_with_dp: 8 GPU, DP=4 → TP=2
- test_parse_args_tp_default_with_pp_and_dp: 8 GPU, PP=2, DP=2 → TP=2
- test_parse_args_tp_default_dp1_unchanged: DP=1 regression guard

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* Update arguments.py

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* chore: rerun Buildkite

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [Example] Add tau-bench multi-turn tool-use example (#142)

* feat(tau-bench): add vLLM multi-turn GRPO example with trainable agent

Introduce generate_with_tau custom rollout (render + /inference/v1/generate), trainable_agents for tau-bench env interaction, vllm_tool_parser, and TAU_CONFIGS-based user simulator configuration in generate_with_tau.py.

Signed-off-by: xky <kyxiezju@163.com>

* style(tau-bench): apply pre-commit formatting

Signed-off-by: xky <kyxiezju@163.com>

* ci: retrigger buildkite

Signed-off-by: xky <kyxiezju@163.com>

* ci: retrigger buildkite

Signed-off-by: xky <kyxiezju@163.com>

---------

Signed-off-by: xky <kyxiezju@163.com>

* [codex] drop stale unit CPU CI and tests (#268)

* fix buildkite cpu utils ci

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs: remove gha references from buildkite ci

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* deps: add transformers and cloudpickle

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: stub optional utils deps in cpu ci

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix: keep vllm http error notes py310 compatible

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci: run cpu jobs on python 3.11

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test: inline utils stubs in affected files

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docker: upgrade base to vLLM 0.23.0, remove CUDA 13 build path (#253)

* docker: upgrade base to vLLM 0.23.0, remove CUDA 13 build path

- Base image: v0.22.0-cu129-ubuntu2404 → v0.23.0-cu129-ubuntu2404
- Remove `ENABLE_CUDA_13` ARG and all conditional cu13 blocks:
  - cu13 apt dev headers (libcublas-dev-13-0, cuda-nvrtc-dev-13-0, etc.)
  - TE source build (cu13 wheel didn't exist; cu129 wheel works on arm64)
  - fzyzcjy triton source build (cu13 specific)
  - TMS_CUDA_MAJOR export (no longer needed)
- Simplify cublas-dev to unconditional libcublas-dev-12-9
- Simplify TE install to wheel-only
- justfile: remove `build-cu13` target and cu13 tag scheme
- vllm.patch: adapt line numbers for 0.23.0 (776/1896 vs 750/1844),
  preserve `with self.log_iteration_details(None):` wrapper

cu129 nvcc already supports sm100/sm120 (Blackwell), so cu13 build
path was unnecessary — it caused build failures on gb300 (cu13 apt
packages hijacked /etc/alternatives/cuda, breaking TE CMake).

Tested: built successfully on gb200 (arm64), h200 (x86), gb300 (arm64).
All three confirmed vLLM 0.23.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docker/patch: replace core.py sleep fix (already in v023) with all2all_utils weight-reload fix (#45989)

v0.23.0 already includes the sleep/scheduler guard from #44483, so the
core.py patch is no longer needed. Replace it with the FP8+DeepEP
weight-reload fix (vllm-project/vllm#45989): snapshot
max_num_batched_tokens from FusedMoEConfig instead of calling
get_current_vllm_config() during layerwise reload.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor(vllm_engine): in-process launch + dataclass field introspection (#264)

* fix(vllm_engine): drop router_ip/port fallback to args in init()

PD mode passes router_ip=None to engine.init() intentionally — the
router launches AFTER engines are ready and collects their URLs. The
fallback `self.args.vllm_router_ip` reads a pre-allocated-but-not-yet-
listening address, causing ConnectionRefused on POST /workers.

Drop the fallback; assign directly like main does.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(arguments): unblock 6 skipped params for --vllm-* forwarding

skipped_args in arguments.py blocked 6 params that should be
user-configurable via --vllm-* CLI flags:
  - served_model_name, tokenizer, tokenizer_mode, tokenizer_revision
  - dtype (only hardcoded when --fp16, otherwise user should set)
  - tool_call_parser (was manually registered then skipped; let
    monkey-patch auto-register like slime does)

Also removes the redundant manual --vllm-tool-call-parser registration
since the monkey-patch now handles it automatically.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: fix Qwen3-4B download repo (#276)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Liron Kesem <52330564+LironKesem@users.noreply.github.com>

* test: fix CPU unit test failures introduced by PR #264 (#284)

PR #264 moved top-level and lazy vllm imports that break the bare
python:3.11 CPU CI image (no vllm installed).

Three changes — no production code touched:

1. `tests/_unit_stubs.py`: extend `install_vllm_cli_stubs()` with stubs
   for `vllm.utils.system_utils` (top-level import in vllm_engine.py)
   and the `vllm.entrypoints.*` hierarchy (lazy imports in arguments.py
   and vllm_engine._vllm_server_field_names).

2. `tests/utils/test_vllm_engine.py`: add an autouse fixture that patches
   `_VLLM_SERVER_FIELDS` to an empty frozenset, so tests calling
   `_compute_server_args` without an explicit monkeypatch don't trigger
   the real `_vllm_server_field_names()`.

3. `tests/utils/test_vllm_arguments.py`: evaluate `real_module_available`
   before calling `install_vllm_cli_stubs()` (which registers a fake vllm
   in sys.modules), then mark two tests that require real FrontendArgs
   field names with `@requires_vllm` so they skip on the CPU image.

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(data): reuse stored multimodal_inputs in length filter (#257)

* fix(data): reuse stored multimodal_inputs in length filter

filter_long_prompt re-extracted vision info from sample.prompt via
process_vision_info in the multimodal branch. When apply_chat_template
is set, sample.prompt is the rendered *string* (not a conversation
list), so process_vision_info -> qwen_vl_utils crashed with
"TypeError: string indices must be integers, not 'str'".

This made prompt-length filtering unusable for any VLM dataset: setting
--rollout-max-context-len (which derives rollout_max_prompt_len) or
--rollout-max-prompt-len / --eval-max-prompt-len activates the filter
and hits the crash.

Reuse the multimodal inputs already computed during dataset
construction (sample.multimodal_inputs) instead of recomputing them
from the string prompt.

Add CPU unit tests covering the multimodal branch and a mixed
text-only + multimodal dataset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>

* Delete tests/test_filter_long_prompt_multimodal.py

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* Update data.py

Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>

* docker: rename published image from inferactinc/public to vllm/vime (#283)

Replace all references to `inferactinc/public:vime-*` with
`vllm/vime:*` across CI, docs, and the release justfile.

The image is now published under the official vllm DockerHub
namespace (`vllm/vime:latest`, `vllm/vime:test-latest`) as a
multi-arch manifest (amd64 + arm64).

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(examples/tau-bench): use RunConfig.agent_strategy in TAU_CONFIGS (#280)

tau-bench RunConfig defines agent_strategy, not agent. The old key was silently ignored by Pydantic, and resolve_tau_config accessed tau_config.agent which raises AttributeError at rollout time.

Signed-off-by: kaiyuan <kyxiezju@163.com>

* scripts: complete slime-exact port of most scripts except for gpt-oss 20B support (#260)

* restore: bring back deleted examples, scripts, and agent doc

Restore files that were either deleted by #126 ("trim examples to
qwen3 only") or never synced from slime:

**Reverted from pre-#126 (translated):**
- scripts/low_precision/run-qwen3-4b-fp8.sh
- scripts/low_precision/run-qwen3-30b-a3b-fp8.sh
- scripts/run-glm4-9B.sh
- scripts/run-moonlight-16B-A3B.sh
- scripts/run-qwen3-4B-base-sft.sh
- scripts/run-qwen3-32B.sh
- scripts/run-qwen3.5-35B-A3B-sft.sh

**New from slime@44d29ee (translated):**
- docs/en/get_started/agent.md
- examples/fully_async/run-qwen2.5-0.5B-fully_async.sh

All sglang engine flags translated to vllm equivalents (§2.4).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: unify pkill pattern to '[v]llm serve|VLL[M]::'

Standardize all scripts to use the bracket-escaped pkill pattern that
avoids matching pkill itself and also catches vLLM's renamed
subprocesses (VLLM::EngineCore, VLLM::Worker_TP*). Matches the
canonical pattern in command_utils.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* scripts: complete slime-exact translation of all 29 run scripts

Translate all slime scripts to vime following SGLANG_TO_VLLM_TRANSLATION.md:
- sglang→vllm prefix swap for CLI flags and variables
- _slime→_vime for checkpoint paths
- EP: --sglang-ep-size N → --vllm-enable-expert-parallel (boolean)
- Speculative: multi-param → --vllm-speculative-config JSON (§5.2)
- Delete genuinely sglang-coupled params (DP-attention, DeepEP, NSA, etc.)
- flashinfer → FLASHINFER case fix (§2.4)

23 new scripts + 6 existing updated to match slime@cutoff.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(scripts): correct model config source path in FP8 low_precision scripts

The FP8 scripts used `${SCRIPT_DIR}/../scripts/models/` which resolves
to `scripts/scripts/models/` (non-existent). Changed to `../models/`
to match the INT4 scripts. Same fix as slime PR #2094.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(gpt-oss): fused BF16 format, bridge API patch, bshd qkv format

Three fixes needed to run GPT-OSS 20B RLHF on vLLM backend:

1. hf_weight_iterator_bridge: match Megatron-Bridge 0.5.0 API
   _patch_bridge_expert_cache_to_cpu monkey-patches GPTOSSBridge.
   maybe_modify_converted_hf_weight gained a 4th `hf_state_dict`
   parameter; the patched wrapper only accepted 3, causing TypeError
   during weight sync.

2. run-gpt-oss-20B: point --hf-checkpoint at fused BF16 format
   vLLM's _load_weights_other expects gate_up_proj [E, hidden, 2*ffn]
   (fused). The old per-expert split format (experts.{e}.gate_proj.weight)
   causes KeyError on bias loading. Use tools/convert_gpt_oss_to_fused.py
   to convert an existing per-expert checkpoint, or re-run
   preprocess_gpt_oss.py to produce fused format directly.

3. run-gpt-oss-20B: add --qkv-format bshd + fix seq-length
   GPT-OSS uses learnable softmax (sink attention). TransformerEngine
   disables all attention backends when softmax_type=learnable and
   qkv_format=thd (packed sequences). --qkv-format bshd avoids this.
   --use-dynamic-batch-size is incompatible with bshd; replaced with
   fixed --seq-length 10240 (covers 8192 max response + prompt headroom).

tools/convert_gpt_oss_to_fused.py: new tool to convert per-expert BF16
checkpoint (output of old preprocess_gpt_oss.py) to the fused HF format
expected by vLLM without re-running the slow MXFP4 dequantization.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* fix(scripts): replace pkill -9 vllm with precise -f pattern (21 files)

pkill -9 vllm matches any process named "vllm" and can inadvertently
kill unrelated vllm processes (e.g. background services). Use the same
pattern as PR #220 which targets only vllm serve and Ray VLL[M]:: actors:

  pkill -9 -f '[v]llm serve|VLL[M]::'

Also updates the inline form used in multi-node SSH worker restart
commands (run-qwen3-235B-A22B*.sh, run-qwen3.5-27B.sh, etc.).

Skipped: scripts/run-gpt-oss-20B.sh (uses pkill -9 -f "vllm serve" already),
scripts/run-minimax-m2.sh and run-glm4.7-*.sh (already used -f "vllm serve").

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* chore(scripts): remove run-qwen3-4B-amd.sh from this PR

AMD-specific script is out of scope for the gb300-complete-port PR.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* sync(docs+scripts): port docs/examples from slime-44d29ee, fix script translations

- Add missing EN/ZH docs: low-precision, on-policy-distillation, get_started/agent,
  pd-disaggregation (heterogeneous server groups fix), examples zh docs
- Add missing examples: on_policy_distillation, eval_multi_task, delta_weight_sync,
  geo3k images
- Fix vLLM flag translations across all example docs:
  - --vllm-mem-fraction-static → --vllm-gpu-memory-utilization
  - Remove non-existent dp-attention flags (--vllm-enable-dp-attention, --vllm-dp-size,
    --vllm-moe-dense-tp-size, --vllm-enable-dp-lm-head, --vllm-ep-size)
  - --vllm-ep-num-redundant-experts → --vllm-eplb-config
  - --vllm-cuda-graph-bs → --vllm-max-cudagraph-capture-size
  - sglang speculative flags → --vllm-speculative-config JSON
  - GLM-4.7 MTP: method=eagle → method=mtp, num_speculative_tokens=4 → 3
  - sgl-router → vllm-router; THUDM/vime → vllm-project/vime
- Fix scripts: restore run-kimi-k2-Instruct/Thinking/qwen3-4B/qwen3-235B-A22B to
  slime-44d29ee-as-vime + pkill precision fix only; restore int4 python3 path

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* revert(scripts): pkill -9 -f pattern back to pkill -9 vllm, align with slime

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* revert(pkill): align all remaining kill patterns with slime (pkill -9 vllm)

Covers examples/, docs/, tests/, and vime/utils -- previously missed in
the scripts/ revert.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* chore: remove gpt-oss-20B script and convert tool (moved to separate PR)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [Doc] Fix broken Qwen3-4B example link in rollout_buffer README (#295)

The rollout_buffer README (English and Chinese) linked to
docs/{en,zh}/models/qwen3-4B.md, but there is no models/ directory —
the referenced doc lives under examples/. Point both links to
docs/{en,zh}/examples/qwen3-4B.md so the setup instructions resolve.

Signed-off-by: ajinkya-metica <ajinkya@metica.com>
Co-authored-by: ajinkya-metica <ajinkya@metica.com>

* sync(slime #2014..#2125): 3-way merge [WIP — 45 conflict files need review] (#286)

* sync(slime #2014..#2125): diff3 3-way merge, conflicts preserved

Mechanical commit 1 of 2 (per knowledge/rl/slime-to-vime-sync-sop.md §2).
diff3 translated 3-way merge on upstream/main (incl #260/#280/#283/#257):
  ours = vime@main, base = translate(slime@#2013), theirs = translate(slime@#2125)

Translation fixes vs prior attempt:
  - casing: SGLang->vLLM (prose) / SGLang<X>->VLLM<X> (identifiers); killed VLlm artifact (was 35 files)
  - dotted module refs slime.X->vime.X now translated (was leaking 'from slime.backends')
These resolved 9 spurious conflicts (46->37 files).

Results: 61 clean / 37 conflict (diff3 markers preserved) / 50 new-to-vime / 5 del.
Conflict markers use readable -L labels (ours/base/theirs). Resolve in commit 2.

Non-conflict provenance fix: vimerl/vime -> vllm/vime in 2 example docs.
Engine patch handling (docker/patch/) deferred to commit 2 per SOP §4.5.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* sync(slime #2014..#2125): resolve all conflicts (commit 2)

Resolved all 37 conflict files / 84 diff3 blocks per agent_run RESOLUTION_POLICY.
Principle: keep vime vLLM impl (ours) + incorporate slime's new features (theirs).

Highlights:
- vLLM API form kept everywhere: /inference/v1/generate, choices parsing,
  AsyncEngineArgs, vLLM flag names (--vllm-gpu-memory-utilization etc).
- Dropped all vllm.srt.* imports (non-existent in real vLLM).
- Accepted new slime features: delta-weight-sync CLI args, append_response_tokens
  (Sample), get_server_info/start_external_rollout_servers/get_rollout_num_engines,
  old-router(<=0.2.1) compat, TrajectoryManager adapter design (vime already adopted it).
- Kept vime-only: --rollout-external, add_router_arguments, _get_metrics_router_addr,
  reinit_wandb_primary_with_open_metrics, update_tracking_open_metrics, modal sandbox,
  VIME_AGENT_* env names, local-vLLM tau-bench user sim.
- Engine patches (docker/patch/): kept ours vllm.patch (22-line MoE fix), dropped
  theirs sglang 2674-line content; deleted sglang-only vllm-top_p.patch (per SOP 4.5).
- Dockerfile kept ours (vllm/vllm-openai base); version.txt accepted theirs nightly.
- run-deepseek-r1.sh: dropped /sgl-workspace dead-path env.

Deviations from policy (documented): vllm_rollout.py abort path kept ours
pause/drain (abort_servers_until_idle would break partial-rollout drain + leave
paused_workers unbound). README ecosystem section left empty (ours) pending
de-translation of provenance.

All changed .py py_compile clean; zero conflict markers; no sglang/slime leakage.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): pre-commit lint/format on resolved files

- re-add dropped `from vllm_router.launch_router import RouterArgs` import in
  vime/utils/arguments.py (add_router_arguments uses it; F821 from conflict resolution)
- drop unused base_top_p_token_ids/offsets in vllm_streaming_rollout.py (F841; came
  from theirs but ours's choices-parsing path doesn't use them)
- black/isort autoformat (anthropic.py, test_agent/*, arguments.py)
- pipeline.yml: agent tests moved to tests/test_agent/; wire new CPU tests

pre-commit: all hooks pass (ruff/autoflake/isort/black/yaml).
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): make CPU CI green (engine import, docker build, agent/utils tests)

Local CPU CI (pre-commit + plugin + agent + utils) all green on 8xH200 host
in python:3.11 containers. Fixes found while running it:

- vllm_engine.py: make 'import vllm_router'/'packaging.parse' lazy (inside
  _register_to_router); top-level import broke CPU import (vllm_router absent in
  CPU CI). The old-router(<=0.2.1) compat branch is theirs-accepted.
- docker/Dockerfile: TMS_CUDA_MAJOR=12 for torch_memory_saver pin (its build
  backend now requires it for CUDA wheels; base is cu129). Unblocks image build.
- agent/adapters/common.py: _run_turn called parse_model_output() without the
  required tokenizer= kwarg -> 500s in adapter tests. Pass tokenizer=tok.
- tests/test_agent/_fakes.py: FakeVLLMServer served sglang /generate + meta_info;
  retarget to vime /inference/v1/generate + choices shape + x-session-id header.
- tests/test_agent/test_adapters.py: parse_model_output(tokenizer=...) + assert
  vime body keys (token_ids/max_tokens).
- tests/utils/test_vllm_config.py: vLLMConfig->VllmConfig (4 sites); fake router
  returns 3-tuple (ip,port,prom) matching _start_router; drop spurious resolve().
- tests/test_megatron_argument_validation.py: add num_gpus_per_node=8 to the
  vime_validate_args fixture (vime colocate override needs it).
- .buildkite/pipeline.yml: agent tests -> tests/test_agent/*; +cispo_loss,
  +logprob_response_spans (CPU-safe); test_rollout_metrics stays GPU-only (imports vllm).

Engine patch verdict (PATCH_ASSESSMENT.md): P1-P7 vLLM doesn't need (NIXL/Mooncake
native); kept ours vllm.patch, dropped sglang content + top_p.patch.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): unblock GPU run (scipy pin, router arg dup, top-p-replay gate)

Found running GPU CI on 8xH200 with vllm/vime:latest:
- docker/Dockerfile: pin scipy<1.14 next to numpy<2 (scipy drifted to 1.18 which
  needs numpy>=2 and uses removed np.long -> 'import vllm' crash).
- vllm_utils/arguments.py: drop sync-added RouterArgs.add_cli_args + its import in
  add_vllm_router_arguments. main exposes the full router surface only in
  utils.add_router_arguments; the duplicate re-registered --router-request-timeout-secs
  -> argparse conflict at train startup.
- megatron_utils/loss.py: get_rollout_top_p_logprob_kwargs falls back to full-vocab
  logprob when top-p nucleus token ids are absent instead of raising. slime's
  top-p-replay needs engine-returned top-p tokens; vime's vLLM /inference/v1/generate
  does not expose them (sglang-only). Matches vime pre-sync behavior; flagged in
  OVERNIGHT_REPORT for review.

Image import smoke + Megatron ckpt load + 4x VLLMEngine bringup + NCCL weight
transfer all confirmed working in-image before this.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* ci(gpu): drop deleted test_qwen2.5_0.5B_ppo_critic_only_short from short suite

slime deleted tests/test_qwen2.5_0.5B_ppo_critic_only_short.py this window (#2014..#2125);
gpu_suites.py still listed it -> 'no such file' exit 2. Critic-only path is still
covered by test_qwen3_4B_ppo_train_critic_only (megatron suite). Other 3 short tests
(gsm8k_async, gsm8k, fully_async) pass on 8xH200.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore base_log_probs init in streaming rollout (None+list crash)

GPU test_qwen3_4B_streaming_partial_rollout hit
'TypeError: unsupported operand +: NoneType and list' at vllm_streaming_rollout.py:234.
The conflict resolution changed base_log_probs from main's
`list(sample.rollout_log_probs or [])` to a None-able form; a fresh sample
(rollout_log_probs=None) then did None + call_log_probs. Restored main's form.
Real sync-resolution regression caught by GPU CI.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): streaming rollout uses append_response_tokens (was renamed update_from_meta_info)

slime #2110 renamed Sample.update_from_meta_info -> append_response_tokens; vllm_rollout
was updated but vllm_streaming_rollout still called the old name (AttributeError at
generate_streaming). Streaming already accumulates tokens incrementally for partial-rollout,
so call append_response_tokens(meta_info=meta) with tokens omitted -> metadata-only finalize
(no double-append). Caught by GPU test_qwen3_4B_streaming_partial_rollout.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore vime unconditional colocate rollout_num_gpus re-derive

3-way compare (slime / vime-main / PR) showed the merge made a broken hybrid:
it KEPT vime's num_gpus_per_node colocate override (which assumes rollout_num_gpus
is forced to actor_num_gpus_per_node*actor_num_nodes) but REPLACED vime's
unconditional re-derive (`!= -> re-derive`) with slime's `is None`-only form.
When a colocate test's rollout_num_gpus is non-None but mismatches, it was left
mis-sized -> engine/GPU misplacement -> mixed_offload IPC-UUID mismatch + ckpt
'Free memory < util'. slime passes (no override, self-consistent is-None); vime
main passes (override + unconditional re-derive, coupled). Restore vime's
re-derive; keep slime's new rollout_num_gpus==0 branch (checked first).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): slime-consistency review pass (docs, rollout routed_experts, comments)

Docs (EN+zh parity):
- fault-tolerance: revert junk 'server'->'engine' mistranslation (keep correct /health endpoint, verified vs vllm_engine.py)
- vllm-config: 'ServerArgs'->'EngineArgs' (sglang class -> vLLM AsyncEngineArgs u FrontendArgs); fix duplicated 'vllm-router (vllm-router)' alias
- customization: restore over-deleted 'custom_generate -> list[Sample]' section + signature (dropped only the vime-absent search-r1 example link)

Rollout:
- routed_experts now flows through the slime-identical Sample._apply_meta_info (single assignment site, torch.int32 tensor matching downstream) instead of an inline numpy assign; vLLM .npy-on-choice decode stays (engine wire-format delta). Both vllm_rollout and vllm_streaming_rollout.

Comments for future syncers:
- --opd-teacher-model + on_policy_distillation: engine-driven divergence (vLLM model field; sglang /generate has none)
- overrides / _vllm_server_field_names: AsyncEngineArgs u FrontendArgs == slime's sglang ServerArgs

Examples/docker/etc: drop vime-absent npu/retool/search-r1/tau-bench files; restore eval_multi_task; docker alignment with slime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): pre-commit green — define base in streaming MM render (F821) + isort/black on test_agent

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(ci): correct two mistranslated CPU tests (colocate rollout-gpu re-derive; parse_model_output tokenizer)

- test_..._preserves_larger_rollout_gpus_under_colocate asserted slime behavior (==12); vime re-derives to actor*nodes=8 under colocate (commit 97013044). Renamed + assert ==8 + divergence note. vime-main never had the test; slime does.

- test_parse_model_output_plain_text_no_parsers called parse_model_output without the required tokenizer kwarg (#198 made it required for vLLM parsers). Pass tokenizer=None (unused on the no-parser path).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): streaming rollout posts to /inference/v1/generate, not sglang /generate

The slime diff3 merge took slime's sglang endpoint (/generate) for the streaming rollout POST instead of keeping vime's vLLM endpoint (/inference/v1/generate). main (7198547c) had the correct URL; the sync regressed it (and dropped the base var). Result: 404 Not Found at vllm_streaming_rollout.py:182 -> test_qwen3_4B_streaming_partial_rollout fails. Caught on a clean h200 node. The file's own docstrings already say /inference/v1/generate throughout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): thread rollout port cursor globally across multi-model engines

The diff3 merge adopted slime's deferred rollout-engine init (start_rollout_servers
now returns pending_init_handles awaited by the caller instead of ray.get-ing each
model's engines before the next). That broke an implicit invariant the per-model
port_cursors reset relied on: in main, model 0's engines were fully bound before
model 1 allocated ports, so the free-port bind-test in
_allocate_rollout_engine_addr_and_ports_normal skipped model 0's ports. With
deferred init, model 1 allocates while model 0 is unbound, the bind-test sees the
base ports free, and a second model (e.g. mixed_offload's frozen "ref") lands on
the same 15000-15003 as the actor. The actor's POST /update_weights to :15002 then
hits the never-started ref engine -> vLLM 500 "start_weight_update must be called
before update_weights" (test_vllm_config_mixed_offload[_ft]).

Fix: initialize port_cursors once before the model loop so the per-node next-free
cursor is monotonic across all models, keeping every engine's ports disjoint
regardless of bind timing. Single-model behaviour is unchanged; the per-model
reset only existed to scope cursors that are already node-keyed.

Caught on h200 GPU CI (new nightly-dev-20260618a image, which added the
start_weight_update-before-update_weights enforcement that exposed the collision).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore robust pkill pattern so ckpt cleanup kills vLLM ray actors

PR #260 ("complete slime-exact port") changed execute_train's pre-launch
cleanup from

    pkill -9 -f '[v]llm serve|VLL[M]::'

to

    pkill -9 vllm

as an over-literal sglang->vllm translation. But the vLLM rollout engine runs
as Ray actor processes whose process *name* is python/ray, with "VLLM::" only in
the command line — so `pkill -9 vllm` (name match, no -f) does not kill them.
Leftover engine processes from the ckpt test's save phase survive into the load
phase, holding ~115 GiB, so the load-phase engine starts with ~24/139 GiB free
and dies with "Free memory ... less than desired GPU memory utilization (0.8,
111.84 GiB)" (test_qwen3_4B_ckpt.py, both --async-save and not).

Restore the cmdline-match pattern `-f '[v]llm serve|VLL[M]::'`.

Bisected on h200: ckpt PASSES at 289ee6d / e62d44fa (old pattern, 4/4 runs) and
FAILS at 7198547c/main + PR (new pattern, 0/3), same old image -> code regression
in #260. Verified: pkill-fixed PR code + new pr286 image -> ckpt PASS (579s).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* chore(sync): replace all `pkill -9 vllm` with cmdline-match pattern

Same root cause as 764e1e18 (command_utils.py): `pkill -9 vllm` matches by
process *name*, but vLLM rollout engines run as Ray actor processes (python/ray
named, "VLLM::" only in the command line), so the name match never kills them.
Apply the robust cmdline pattern `pkill -9 -f '[v]llm serve|VLL[M]::'` everywhere
the bare `pkill -9 vllm` cleanup was used across run/example scripts, so leftover
engines don't squat GPUs across runs. No logic change beyond the kill pattern.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): restore vLLM core.py partial-wake sleep-guard (#44483)

Commit d41f0aa1 removed the core.py sleep-guard hunk from
docker/patch/latest/vllm.patch on the assumption it was "already in
v0.23.0". It is not: stock v0.23.0 `vllm/v1/engine/core.py` calls
`resume_scheduler()` and `execute_dummy_batch()` even during a partial
(weights-only) wake. So a colocate DP+EP pd_mooncake rollout, right after
`POST /wake_up?tags=weights` (KV cache still released under level-2 sleep),
has its DP busy-loop fire a decode-shaped dummy batch that touches freed
KV -> the scheduler_metadata write in flashattn_mla.py:234 (MLA, glm4.7)
and flash_attn.py:547 (FA3, qwen3.6) raises `CUDA error: invalid argument`.

Restore the guard (`if not self.model_executor.is_sleeping` around
resume_scheduler; `if not self.is_sleeping()` around execute_dummy_batch),
keeping the all2all_utils weight-reload fix. This is the #173 sleep-guard
patch re-expressed against v0.23.0 line numbers.

Verified: git-apply --check clean against stock v0.23.0; both guards land;
glm4.7/qwen3.6 pd_mooncake reproduced the crash without it (the 8-day-old
image that still carried the guard passes both).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): drop scipy<1.14 pin, mirror slime numpy<2 only

The scipy<1.14 pin (added in 0cda11c6 while chasing the pd_mooncake crash)
was a red herring: the real cause was the dropped core.py partial-wake
sleep-guard, now restored. slime-2125-as-vime pins only `numpy<2`; this
restores that exact line. numpy 1.26.4 + scipy resolved naturally matches
the working baseline.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): keep FlashQLA install gated behind INSTALL_FLASHQLA=0

slime installs FlashQLA unconditionally, but it is sm90/Hopper-only.
Restore vime's original gated form (default off; --qwen-gdn-backend fla
elsewhere). CI build passes --build-arg INSTALL_FLASHQLA=1 to include it.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(vllm-config): fix inference-only FAQ — vime launches engines in-process

The mechanical mirror of slime's answer steered users to vLLM's standalone
`vllm serve` (slime's `launch_server` analog) for inference-only. That is
misleading for vime: like slime, vime launches the vLLM engines in-process
from `--vllm-config` (same in-process path as training), so a rollout-only
run serves directly with no separate server process. Point standalone users
to `--rollout-external-engine-addrs` instead. EN + ZH.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(debug): restore INT4 / Compressed-Tensors checkpoint section

vime's debug.md was missing slime's "INT4 / Compressed-Tensors Quantization
Checkpoint Issues" section (slime #1642) — dropped in an earlier sync, not
present on main. Restore it (EN + ZH), translated sglang→vLLM / Megatron→vLLM.
Covers the quantization_config.ignore list, all-zero MoE router weights
(mlp.gate.weight) when mis-quantized, missing safetensors shards, and
diagnosis via --check-weight-update-equal / --debug-rollout-only.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(vllm-config): use _run_vllm_server for inference-only FAQ

Keep slime's wording; the only engine-coupled fix is the standalone launcher
name. slime's `launch_server` is its in-process engine entry; vime's analog
is `_run_vllm_server` (vllm_engine.py, launched via multiprocessing.Process),
not the standalone `vllm serve` CLI. EN + ZH.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): restore scipy pin (scipy<1.18) — vime base needs it

Reverts the scipy-pin removal in b359b245, which was wrong: vime's
vllm/vllm-openai base ships no scipy, so unpinned the build pulls scipy>=1.18,
which hard-requires numpy>=2 and uses np.long (removed numpy>=1.24) -> crashes
against the numpy<2 reinstall (Megatron needs numpy 1.x). slime's sglang base
resolves scipy 1.17.1 natively (numpy-1.x compatible), so slime needs no pin;
this is a base-image divergence, not a red herring. Pin boundary is 1.18
(slime runs 1.17.1), not the earlier 1.14 over-estimate.

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(scripts): properly translate sglang args in glm5.2-744B + glm4.7-355B-delta

These two were prefix-swapped (--sglang-X -> --vllm-X) without semantic mapping,
leaving ~20 args that aren't vLLM AsyncEngineArgs (argparse would reject). Apply the
knowledge/rl/sglang-to-vllm-translation.md §5.5 mappings:
- dp-size->data-parallel-size, ep-size->enable-expert-parallel, max-running-requests->
  max-num-seqs, cuda-graph-max-bs->max-cudagraph-capture-size
- 5x/4x --speculative-* -> one --vllm-speculative-config JSON (§5.2)
- DeepEP: per-group deepep_mode auto/low_latency -> all2all_backend deepep_high_throughput/
  low_latency in the --vllm-config overrides (vLLM has no 'auto'; PD encodes it per-role)
- watchdog-timeout -> env VLLM_ENGINE_ITERATION_TIMEOUT_S
- drop sglang-only: dp-attention / dp-lm-head / moe-dense-tp / disable-overlap-schedule / NSA
  backends (vLLM selects DeepSeek sparse attn per model) / engine delta-receiver knobs
- flag PD mooncake transport (-> --vllm-kv-transfer-config) as fabric-specific TODO

These are 744B/355B scripts not runnable in CI — translations are SOP-mapped but
hardware-unvalidated (flagged inline).

Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(args): hard-guard unverified delta weight-sync mode

--update-weight-mode=delta (PR #278 lineage) is not yet validated on
vime+vLLM (vLLM exposes dense/sparse_flat only, not slime's gap-delta/
zstd encoding). Raise NotImplementedError at arg-validation so it fails
fast at startup instead of crashing mid weight-sync. Downstream delta
code is kept untouched; remove this raise once a real delta-load run
passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs,test: correct rollout engine endpoint to /inference/v1/generate

The agent rollout engine is reached at vime's vLLM ``/inference/v1/generate``
(see vllm_rollout.get_model_url default + common.call_vllm_generate), not the
bare ``/generate`` of sglang. Fix the imprecise path in adapter/test docstrings
and comments, and rewrite the vllm-config.md custom-rollout examples (en+zh):
they were still sglang-shaped (``/generate`` path + ``{"text":..., "return_logprob":
True}`` body). Use vime's real request schema instead -- ``{"model","token_ids",
"sampling_params"}`` with ``max_tokens``/``logprobs``, ``prompt_logprobs`` for
fixed-sequence scoring, and the ``choices[0]`` response shape.

No code/logic change: comments, docstrings, and doc examples only. The Megatron
training server's own ``/generate`` endpoint and the sglang citation in
arguments.py are correct and left untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style(args): collapse delta-guard message to one line (black)

The delta-guard NotImplementedError message was split across two adjacent
string literals; black on the CI (line-length 119) collapses/normalizes it.
Make it a single clean literal so pre-commit is green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* test(args): assert delta weight-sync is guarded off (not per-condition)

The hard delta guard (7bb19e67) raises NotImplementedError at the top of the
delta branch, making the downstream colocate / unknown-transport rejections
unreachable. Replace test_update_weight_delta_rejects_colocate and
test_update_weight_delta_rejects_unknown_transport (whose ValueError paths no
longer fire) with a single test_update_weight_delta_disabled that asserts the
guard raises for any delta config. Breadcrumb left to restore the per-condition
tests when delta is verified and the guard is lifted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(sync): restore dropped weight-sync metrics chain + delta dispatch (slime parity)

A mirror cross-check of the delta-weight-sync surface (slime #1806/#1991) found
the #2014..#2125 sync had silently dropped several slime-faithful pieces:

* extra_metrics logging chain — slime threads weight-update metrics from the
  actor through log_perf_data -> log_perf_data_raw. vime dropped the param at
  all three layers, so weight-update metrics were never logged. Restored:
  train_metric_utils.log_perf_data_raw(extra_metrics=...), data.log_perf_data
  passthrough, and actor passing self.weight_updater.pop_metrics().

* pop_metrics on UpdateWeightFromDistributed — the default (non-colocate, nccl)
  weight_updater. slime gives all three updaters a pop_metrics() stub so the
  actor can call it uniformly; vime kept it on tensor/disk but dropped it on
  distributed, which would AttributeError once the actor calls it. Restored the
  ~5-line stub (delta-specific plumbing stays dropped — vime+vLLM has no DeltaSpec).

* actor delta-mode dispatch branch — restores the elif selecting
  UpdateWeightFromDistributedDelta. Dead code behind the validation guard that
  rejects --update-weight-mode=delta, so vime mirrors slime with the guard as
  the single divergence.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* chore(sync): align comments to the mechanical mirror

Comment-only pass; no behavior change. Aligns vime comments to what a faithful
slime->vime translation would carry:

* Strip vime-divergence rationale markers (delta guard, opd-teacher-model,
  top-p fallback, colocate/delta tests). The rationale belongs in the
  divergence manifest, not inline; the guarded code + self-explanatory
  NotImplementedError messages stand on their own.
* De-verbose vLLM-specific comments to mirror scale: the router-args block,
  the AsyncEngineArgs u FrontendArgs docstring, and the MoE-replay /
  streaming-rollout blocks that slime does not carry at that length.
* Restore slime-original comments the sync had dropped or naively translated,
  with judgment translation of sglang-specific terms: session_id routing
  ("vLLM router", not the mechanical "Model Gateway"), "Prepare payload for
  vLLM server", the unique-session_id loop, and the pending-tasks wait.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* docs(delta): note delta weight-sync not yet verified on vime+vLLM (PR #286 review)

Per review on PR #286: delta weight sync is documented here but the arg guard
disables --update-weight-mode=delta. Add a top-of-page note (en + zh) so users
see it before hitting NotImplementedError at argparse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Example] Add MemAgent long-context RL example (mem_agent) (#291)

* Add examples/mem_agent for long-context MemAgent RL

Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>

* Address review: harden mem_agent convert and eval scripts

Co-authored-by: Cursor
Signed-off-by: kaiyuan <kyxiezju@163.com>

---------

Signed-off-by: kaiyuan <kyxiezju@163.com>

* fix(rollout): abort vLLM rollout via delete-type /abort_requests (#296)

Under --partial-rollout, abort() (pause -> drain -> resume) deadlocks:
/pause?mode=abort puts the scheduler in PAUSED_NEW, and a /generate that races
in after the pause parks in the waiting queue and never returns until /resume,
which runs after the drain. Reordering to pause -> resume -> drain avoids the
hang, but resume reopens the whole queue so the long tail runs to COMPLETION --
breaking partial rollout's "truncate the tail, resume it next step" semantics.

Switch to a delete-type abort instead:

- vLLM: add POST /abort_requests to the RLHF api_router -> EngineClient.abort()
  (removes queued requests from the waiting queue and finish-aborts running
  ones, whose partial output returns on the original /generate stream). It does
  not pause the scheduler, so there is no /resume and no deadlock. Shipped as a
  build-time patch in docker/patch/latest/vllm.patch.

- vime: server_control.abort_inflight_requests() replaces the unused,
  slime-mirrored abort_servers_until_idle / _v1_loads helper (vLLM has neither
  /abort_request nor /v1/loads). abort() re-issues the sweep across drain waves
  and converges on state.pendings, with a timeout bounding how long a late
  multi-turn straggler can run before being truncated to partial.

- vllm_engine: drop the legacy version gate in _register_to_router. vime ships
  its own vllm-router, so only the /workers payload path is needed.

Adds delete-type abort unit tests.

AI-assisted change; reviewed by a human before submission.

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Josephasafg <ajgard7@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* [Bugfix][Rollout] Wire prefix_cache_hit_rate through vLLM usage (#303)

rollout/prefix_cache_hit_rate (and avg_cached_tokens_per_sample) were
structurally 0 due to three coupled gaps:

1. vLLM's non-streaming /inference/v1/generate builds a `usage` block but
   silently drops it: GenerateResponse (v0.23.0 serve/disagg/protocol.py)
   never declared a `usage` field and has no extra="allow", so pydantic
   discards it. Patched in docker/patch/latest/vllm.patch, mirroring
   GenerateStreamResponse. (Upstream fix filed against vllm-project/vllm.)
2. The rollout parser read usage.prompt_tokens/completion_tokens but never
   usage.prompt_tokens_details.cached_tokens -> PrefixCacheInfo numerator
   pinned to 0. Now read in both vllm_rollout and vllm_streaming_rollout.
3. Streaming additionally needs stream_options.include_usage=True for vLLM
   to emit the terminal usage SSE chunk.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docker): /abort_requests abort-all fix + cu13 image variant (#317)

* fix(docker): abort-all in the /abort_requests vLLM patch must abort by internal ids

The bundled /abort_requests endpoint (merged in #296) populated request_ids from output_processor.request_states (internal ids) but called engine.abort() with the default internal=False, so they were treated as external, matched nothing, and POST /abort_requests {} silently aborted no requests under default request-id randomization. Abort the all-in-flight list as internal. Mirrors vllm-project/vllm#47173.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* style: reject malformed JSON in /abort_requests patch with 400

Match the sibling dev endpoints and the Rust frontend (400 on malformed JSON) instead of silently treating it as empty. Mirrors vllm-project/vllm#47173.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* fix(docker): abort-all patch must also abort parallel-sampling parents

The /abort_requests patch enumerated request_states (child internal ids
only), so with n>1 the ParentRequest entry leaked. Include
parent_requests keys in the abort-all set. Mirrors vllm PR #47173.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

* feat(docker): cu13 image variant + TMS cu13 preload

Port the cu13 build support from #307: ENABLE_CUDA_13 branches the apt
dev headers, cublas header, TransformerEngine (source-built for cu13),
TMS_CUDA_MAJOR auto-detect, and the cudnn pin. justfile gains a
build-cu13 target and a VARIANT-prefixed manifest. actor_group preloads
the cu13 TMS .so.

Also switch the vLLM patch apply to --allow-empty so the build survives
once the patch is emptied upstream.

Excludes #307's NCCL_CUMEM_ENABLE default flip (0->1) and the glm5.2
scripts by request.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

---------

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* Initial ROCm support for vime (#273)

* Added ROCm quick start guide

* changed wording

* changed docker image

* updated instructions

* Add initial working ROCm 7.0.2 build (Dockerfile.rocm + patch)

Initial working setup for building/running vime on ROCm 7.0.2 (gfx950).

* ROCm: re-base Dockerfile on vime mainline Megatron + shared patch

* ROCm: apply checkpoint-writer patch in HF->torch_dist converter

* ROCm: align async run script cleanup with run-qwen3-4B.sh format

* ROCm: move megatron.patch to docker/amd_patch/ (slime layout)

* ROCm: re-add fused-kernels init patch and AMD doc/template references

---------

Co-authored-by: pancake0003 <146360951+pancake0003@users.noreply.github.com>
Co-authored-by: indianspeedster <cp3793@nyu.edu>

* [Doc] Fix stale Qwen3-30B-A3B example: restore FP8-inference section + rewrite multi-node (#322)

* [Doc] Fix stale Qwen3-30B-A3B example: dataset download + align with actual script

The Qwen3-30B-A3B example documented an elaborate env-var-driven multi-node
script (ACTOR_NUM_NODES / MEGATRON_TP / ENABLE_R3 auto-behavior, "script skips
Ray", default batch 4/2/8, --vllm-moe-backend triton). None of that exists in
the current slime-exact scripts/run-qwen3-30B-A3B.sh — it was orphaned when the
script was reverted to the slime-exact single-node port. The real defaults are
rollout-batch-size 32 / n-samples 8 / global-batch 256.

Changes (zh + en, kept concise):
- Environment Preparation: add the concrete `hf download` commands for the
  model, dapo-math-17k train data, and aime-2024 eval data (previously the
  section only pointed at the Qwen3-4B doc).
- Restore the "BF16 training + FP8 inference" section (present upstream in
  slime, dropped here); matches the commented --hf-checkpoint line in the script.
- Replace the orphaned multi-node section with the concise, accurate
  slime-style guidance (manual modifications + vLLM EPLB example), fixing the
  broken zh anchor to the Chinese quick_start heading.

Doc now matches scripts/run-qwen3-30B-A3B.sh; no script changes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Expand Qwen3-30B-A3B multi-node section (code-grounded)

Replace the concise multi-node bullets with an actionable but verified guide:
Ray cross-node startup (from quick_start), the exact script edits
(--actor-num-nodes, colocate auto rollout-num-gpus), parallelism scaling as
constraints (defer concrete large-scale ratios to the GLM/DeepSeek examples
rather than fabricating a 2-node table), and the real pitfalls we hit
(keep each vLLM engine within a node to avoid cross-node TP=16 slowdown/
numerics, MASTER_ADDR / NCCL_SOCKET_IFNAME, global-batch identity,
--num-gpus-per-node for <8 GPUs/node). Every claim traces to vime/utils/
arguments.py, quick_start, or observed multi-node runs; no "tested" claims.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Drop model-download line from Qwen3-30B-A3B example

Model download is already covered by the referenced Qwen3-4B doc; keep only the
train/eval dataset download (dapo-math-17k, aime-2024) surfaced inline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Drop redundant dataset download; keep env-prep slime-exact

The intro sentence already covers env/model/data/ckpt via the Qwen3-4B
reference, and slime's 30B doc carries no dataset block. Remove it; keep the
30B-specific torchrun checkpoint conversion (differs from 4B's single-process
python). Env-prep is now byte-identical to upstream.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Multi-node: also skip the script's Ray-cleanup preamble

Codex review (P2): for the manual multi-node path, removing only the
`ray start --head` line is not enough — the script's initial cleanup block
(`ray stop --force` + `pkill -9 ray/python/redis`) would tear down the
manually-started head before `ray job submit`, failing submission and
orphaning the workers. Tell users to remove/comment both the cleanup block
and the `ray start --head` line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Fix H800→H100 typo in Qwen3-30B-A3B example (#325)

The page title says "8xH100" but the CPU-Adam bullet said "8xH800
environment", an inconsistency inherited from slime. Both are 80GB cards so
the point is unchanged; align the text with the title.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [Doc] Add Ascend NPU platform tutorial and expand hardware support section (#334)

Signed-off-by: kaiyuan <kyxiezju@163.com>
Co-authored-by: XiaoxinWang <wangxiaoxin7@huawei.com>
Co-authored-by: FulinGao_HW <gaofulin1@huawei.com>
Co-authored-by: meihanc <zr010426ztt@outlook.com>
Co-authored-by: wuxiang <498160096@qq.com>
Co-authored-by: yuxinshan <syx_ctyg@126.com>
Co-authored-by: flb_dayo <floatlibai@gmail.com>
Co-authored-by: Windfeng8 <523758380@qq.com>
Co-authored-by: YZY <532183776@qq.com>

* sync: update vime through slime #2185 (#338)

Sync slime a897e1f4..680824dd using the mechanical merge plus vLLM adaptation workflow.

* revert: sync through slime #2185 (#338)

Revert the prematurely merged slime sync and restore the pre-#338 tree.

* [bugfix] Fix distributed update weights for pipeline parallel (#329)

* Fix pp for distributed update weights

Signed-off-by: knlnguyen1802 <knlngu…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants