Repository navigation
weekly sync: update vime through slime #2185 - #343
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for Gemma4 text models (dense and MoE), adds the SWE-bench Verified grading protocol, refactors agent harness execution to use a detached exec_and_wait mechanism, and implements a fused PPO log-probability and entropy calculation. The review feedback highlights several critical issues: a guaranteed crash in the attention forward pass due to an invalid enable_gqa argument in PyTorch's scaled_dot_product_attention; a potential NameError in the server's update handler under task cancellation; unsafe in-place tensor mutation in the PPO backward pass; a hardcoded 10-second sleep bottleneck in the generation loop; and potential cleanup failures in the Dockerfile and sandbox execution.
| else: | ||
| cu_seqlens_list = cu_seqlens.tolist() |
There was a problem hiding this comment.
PyTorch's native torch.nn.functional.scaled_dot_product_attention does not accept an enable_gqa argument. Passing this argument will cause a guaranteed TypeError and crash the attention forward pass. PyTorch natively handles Grouped-Query Attention (GQA) broadcasting internally based on the head dimensions of the inputs.
scale=scale,| env: dict[str, str] | None = None, | ||
| timeout: int = 120, | ||
| check: bool = False, | ||
| idempotent: bool = True, |
There was a problem hiding this comment.
The exec method (which runs arbitrary shell commands in the sandbox) defaults to idempotent=True. Arbitrary shell commands (such as process spawning or file appending) are generally not idempotent. Retrying them automatically on transient network errors can lead to dangerous double-execution side effects. It is safer to default idempotent to False and only explicitly set it to True for known read-only or safe commands.
| idempotent: bool = True, | |
| idempotent: bool = False, |
| timeout_s = _get_update_timeout_s(payload, args) | ||
| update_state["in_progress"] = True | ||
| result = None | ||
| error = None | ||
| try: | ||
| before_loads = await _wait_until_idle(sample_manager, timeout_s) | ||
| update_result = await asyncio.to_thread(update_from_disk_fn, model_path) | ||
| after_loads = await _ray_get(sample_manager.get_loads.remote()) | ||
| except TimeoutError as e: | ||
| return _json_error(str(e), 503) | ||
| error = {"ok": False, "status": 503, "error": str(e)} | ||
| except Exception as e: | ||
| return _json_error(f"update_from_disk failed: {e}", 500) | ||
| error = {"ok": False, "status": 500, "error": f"update_from_disk failed: {e}"} |
There was a problem hiding this comment.
If the main update task is cancelled (e.g., due to a client disconnecting), an asyncio.CancelledError is raised. Since CancelledError inherits from BaseException (not Exception), it bypasses the except Exception blocks, leaving update_result unassigned. When the finally block is entered, constructing result with update_result will raise a NameError, masking the cancellation and potentially leaving coalesced tasks hung. Initializing the variables and catching BaseException to cleanly handle cancellation prevents this.
timeout_s = _get_update_timeout_s(payload, args)
result = None
error = None
before_loads = None
after_loads = None
update_result = None
try:
before_loads = await _wait_until_idle(sample_manager, timeout_s)
update_result = await asyncio.to_thread(update_from_disk_fn, model_path)
after_loads = await _ray_get(sample_manager.get_loads.remote())
except TimeoutError as e:
error = {"ok": False, "status": 503, "error": str(e)}
except Exception as e:
error = {"ok": False, "status": 500, "error": f"update_from_disk failed: {e}"}
except BaseException as e:
error = {"ok": False, "status": 500, "error": f"update_from_disk interrupted: {type(e).__name__}"}
raise| grad_entropy_input.mul_(grad_entropy.reshape(-1, 1)) | ||
|
|
||
| vocab_parallel_size = log_prob_softmax.size(-1) | ||
| grad_input = log_prob_softmax.neg_() |
There was a problem hiding this comment.
Using log_prob_softmax.neg_() mutates a saved tensor in-place during the backward pass. In PyTorch, mutating saved tensors in-place is highly discouraged as it can corrupt the autograd history, break double-backward/gradcheck, or raise runtime errors. Using the out-of-place .neg() is safer and avoids side effects.
| grad_input = log_prob_softmax.neg_() | |
| grad_input = log_prob_softmax.neg() |
| RUN git clone https://github.com/Dao-AILab/flash-attention.git && \ | ||
| cd flash-attention/ && git checkout fbf24f67cf7f6442c5cfb2c1057f4bfc57e72d89 && git submodule update --init && cd hopper/ && \ | ||
| MAX_JOBS=96 python setup.py install && \ | ||
| export python_path=`python -c "import site; print(site.getsitepackages()[0])"` && \ | ||
| mkdir -p $python_path/flash_attn_3 && \ | ||
| cp flash_attn_interface.py $python_path/flash_attn_3/flash_attn_interface.py && \ | ||
| rm -rf flash-attention/ | ||
| cd flash-attention/ && git checkout 002cce0a1068f8c07dfccb5a1d232b9a3276947c && git submodule update --init && \ | ||
| cd hopper/ && \ | ||
| FLASH_ATTENTION_FORCE_BUILD=TRUE MAX_JOBS=96 pip -v install . --no-build-isolation && \ | ||
| cd /root/ && rm -rf flash-attention/ |
There was a problem hiding this comment.
The flash-attention repository is cloned in the current working directory, but the cleanup command cd /root/ && rm -rf flash-attention/ attempts to delete it relative to /root/. If the working directory is not /root/, this leaves the cloned repository intact, bloating the Docker image. Deleting it relative to its parent directory avoids this issue.
RUN git clone https://github.com/Dao-AILab/flash-attention.git && \
cd flash-attention/ && git checkout 002cce0a1068f8c07dfccb5a1d232b9a3276947c && git submodule update --init && \
cd hopper/ && \
FLASH_ATTENTION_FORCE_BUILD=TRUE MAX_JOBS=96 pip -v install . --no-build-isolation && \
cd ../.. && rm -rf flash-attention/
| await state.adapter.drop_session(session_id, wait_timeout=30) # cleanup only, idempotent | ||
| await asyncio.sleep(10) |
There was a problem hiding this comment.
A hardcoded 10-second sleep in the finally block of generate() will block the completion of every single generation/rollout task. In reinforcement learning training loops with thousands of rollouts, this introduces a massive throughput bottleneck. Consider removing this sleep or making it conditional/configurable.
| await state.adapter.drop_session(session_id, wait_timeout=30) # cleanup only, idempotent | |
| await asyncio.sleep(10) | |
| await state.adapter.drop_session(session_id, wait_timeout=30) # cleanup only, idempotent |
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
7f23d79 to
02df2fe
Compare
Signed-off-by: aoshen02 <aoshen@inferact.ai>
|
Overall: LGTM as the #338 redo with stronger validation. |
The ci 610 failed for I don't know what reason, but ci 609 does work, I post the pictures there: |
* sync: mechanically merge slime through 680824dd Signed-off-by: aoshen02 <aoshen@inferact.ai> * sync: adapt slime changes for vLLM Signed-off-by: aoshen02 <aoshen@inferact.ai> * ci: register synced CPU and GPU tests Signed-off-by: aoshen02 <aoshen@inferact.ai> * update Signed-off-by: aoshen02 <aoshen@inferact.ai> * update Signed-off-by: aoshen02 <aoshen@inferact.ai> * update Signed-off-by: aoshen02 <aoshen@inferact.ai> * sync: tighten vLLM adaptations after review Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix: preserve vLLM top-p fallback Signed-off-by: aoshen02 <aoshen@inferact.ai> * sync: complete slime adaptation audit Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(ci): update external PD test for native vLLM * fix(ci): use NIXL default UCX device selection * fix(ci): configure Qwen3.6 Mamba state layout Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(ci): honor explicit NCCL NVLS override * fix(ci): preserve explicit UCX device override --------- Signed-off-by: aoshen02 <aoshen@inferact.ai>


Summary
680824dd(slime PR #2185)Review structure
aoshen02:codex/slime-2185-mechanical-review(0317536c)aoshen02:codex/slime-2185-vime-review(4026f440)main, then review adaptation-only changes between the two branches.Validation
CI
4026f440test_qwen3_4B_ppo.pyattempt hit a node-localCUDA error 999during NCCLall_reduce; an unchanged retry on another H100 node passed. The failing GPU still had 49.92 GiB free, so this was not OOM or a deterministic code failure.100-step convergence run
5pjx92rfqwen3-30b-a3b-base-single-node-scaled32p-n16-r20k-pack64k-100step-20260713_093256Z02df2fe78bac9667a3d1eab824efe78b3c26f53a; later PR commits only adjust CI/runtime environment handling and are covered by Buildkite #609.0..99; the final checkpoint is iteration 99 because rollout IDs are zero-based.Reproduction environment
Qwen/Qwen3-30B-A3B-Base02df2fe78bac9667a3d1eab824efe78b3c26f53ah200-1vllm/vime:pr343-02df2fe-cu13-h2001-backup(sha256:7c6a...); the unqualified tag now points to a later rebuildsha256:7c6a18bae213d50206ec3c917d2e4b4cb2e57d012f240d078d507afefad5d6d1vllm/vllm-openai@sha256:bfdefe75b5c3fb83f4f0fcaae8f39fac87941cbadb05cd2203f44a1689236c712.11.0+cu130/13.0/2.28.90.24.0/5.12.12.56.0/0.28.0NCCL_NVLS_ENABLE=0; the test H200 fabric does not expose NVLink multicastImage build
docker build --progress=plain -f docker/Dockerfile \ --build-arg BASE_IMAGE=vllm/vllm-openai@sha256:bfdefe75b5c3fb83f4f0fcaae8f39fac87941cbadb05cd2203f44a1689236c71 \ --build-arg ENABLE_CUDA_13=1 \ --build-arg INSTALL_FLASHQLA=1 \ --build-arg VIME_COMMIT=02df2fe78bac9667a3d1eab824efe78b3c26f53a \ -t vllm/vime:pr343-02df2fe-cu13 .Dataset preparation
The run used the official ByteDance DAPO/AIME parquet data. Rows are deduplicated by prompt and reward metadata; prompts with conflicting ground-truth rewards are removed.
dapo-math-17k.parquet534375d6bb8630d22ab46a56e11f2ffec1d288d8f7d04099bc82d68948705941dapo-math-17k-cleaned-vime.jsonlb912727ac69dea9f1edcdd6c06e6ddf85dc2532a3d7d9e73e5b54f37e74c77e7aime-2024.parquet12154e38a716d12db5731f9a022ae69a610c4f7d0e0dcc04e902887a686877e7aime-2024-cleaned-vime.jsonl488aba4c72a0c874012f94d28eb4ba835a3ab2dc848c41d06e06939aa831709aDataset conversion script
Model conversion
Exact training profile
0..99)dapo, keyscore, correct+1, incorrect-11e-6, constant, 10 warmup iterations0.20, high0.280.70.7Actual launcher script
This is the task-owned launcher used for the Base-model run family. The completed run relied on the default full weight-sync mode; the preserved script now pins
--update-weight-mode fullexplicitly so future defaults cannot change the reproduction. OverrideWORKTREE, data paths, checkpoint paths, W&B group, and Ray ports as needed.Launch it with:
Notes
MAX_TOKENS_PER_GPU=65536is the packed-token training budget, not a 64K per-sample response limit. Individual samples remain bounded by the 32K total context and 20K response cap.