Skip to content

Add Qwen3-VL loading and tool-use examples - #493

Open
Meihan-chen wants to merge 7 commits into
vllm-project:mainfrom
Meihan-chen:feat/qwen3-vl-and-examples
Open

Meihan-chen wants to merge 7 commits into
vllm-project:mainfrom
Meihan-chen:feat/qwen3-vl-and-examples

Conversation

@Meihan-chen

@Meihan-chen Meihan-chen commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Add the missing HF-to-Megatron loading path and native training model for Qwen3-VL, plus ReTool and Search-R1 rollout examples. Also fix weight-transfer engine-group reconnection so teardown targets the previous group.

Changes

  • Qwen3-VL: register an HF loader that reuses the Qwen language-weight mapping, handles the vision tower and tied output embeddings, and complements the existing Megatron-to-HF exporter. Add the native model with image/video embedding injection, packed MRoPE positions, and DeepStack features. Remove vocabulary padding from embedding/output tensors whose names carry the language_model. prefix.
  • Engine reconnection: disconnect the previous rollout engines before replacing self.rollout_engines. Previously, reconnecting sent teardown to the incoming engines and left the previous group connected. Update the regression test to check which engines receive teardown.
  • Examples: add ReTool generation, Python-tool execution, reward logic, data preparation and SFT/RL scripts; add Search-R1 generation and scoring with local-retrieval and Google Search services. Include English/Chinese documentation and propagate worker/parent identifiers through multi-agent sample metadata.

Validation

The following tests pass against the proposed code in an isolated Python 3.12 CPU environment:

python -m pytest -q \
  tests/test_hf_to_megatron.py \
  tests/test_qwen3_vl_native.py \
  tests/utils/test_update_weight_from_distributed.py

90 passed, 11 skipped. The 11 native Qwen3-VL tests require megatron.core.tensor_parallel, which is absent from that environment. Passing tests cover weight mapping/round trips, tied embeddings, vocabulary padding, and engine-group replacement. git diff --check also passes.

Current limits

  • The native Qwen3-VL provider requires THD packed sequences, PP=1, CP=1, and the Transformer Engine layer spec; native MTP is unsupported.
  • Full Qwen3-VL training and the two example workflows have not been validated end to end on this branch.
  • The example setup instructions currently describe an Ascend environment. Hardware integration and Docker changes are outside this diff.

Meihan-chen and others added 5 commits October 20, 2026 11:21
main can already export Qwen3-VL to HF: megatron_to_hf/qwen3_vl.py and
its dispatch entry are there. The loading direction is missing, so a
Qwen3-VL checkpoint cannot be brought in to begin with.

Add the loader, register it, and add the native model. The loader reads
the language tower through a thin view that rewrites model.* to
model.language_model.*, leaves model.visual.* alone, and resolves a tied
output layer back to the language embedding, so the existing qwen
loader is reused rather than duplicated.

Also fix vocab-padding removal on export. remove_padding() matched
embedding.word_embeddings.weight and output_layer.weight exactly, which
a Qwen3-VL parameter never is: its names carry a language_model. prefix,
so padded rows were exported as real vocabulary. The prefix is stripped
before the comparison, which is a no-op for every other model.

test_loader_scope_stays_explicit pins the registered loader set on
purpose, so it is updated here rather than relaxed.

Verified: tests/test_hf_to_megatron.py and tests/test_qwen3_vl_native.py
total 85 passed, including a language-tower round trip, the tied-output
case, and padding removal checked for both prefixed and unprefixed names.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
…g it

connect_rollout_engines() assigned self.rollout_engines to the incoming
engines and then called disconnect_rollout_engines(), which reads that
same attribute. On a reconnect the teardown therefore went to the new
engines, which had never joined a group, while the group being replaced
stayed connected and its trainer kept its peers.

Swap the two statements. The first connect is unaffected: __init__ sets
_trainer to None, so disconnect short-circuits before reading anything.

test_connect_replaces_existing_trainer asserted the old behavior, by
checking the teardown reached the incoming engine. It now gives the
updater a previous engine and asserts the teardown reaches that one and
not the new one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Three fixes in the deepscaler rule-based reward:

  - "\boxed" written as a Python literal is a backspace character, so
    responses can arrive with the marker already destroyed. Restore it
    before parsing rather than scoring the answer as absent.
  - With enable_thinking=false the turn ends at <|im_end|> and there is
    no </think>. The span up to it is graded, so an earlier boxed answer
    is not dropped because an explanation follows.
  - split("###Response") is bounded to one split, so a response
    containing the marker twice no longer changes which part is graded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
…r ids

Two example bodies with their data preparation, serving helpers and run
scripts, plus the rollout and reward code they need.

examples/multi_agent threads a worker id through generate_response() and
records it, with the role and the parent sample's group and index, in
each generated sample's metadata, so a multi-agent rollout can be traced
back to the agent and the parent that produced it.

Examples only; nothing under vime/ changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
pytest.importorskip("megatron.core") is satisfied by the stub an earlier
test installs, so in a whole-suite run these eleven tests reached the
import and errored on a missing tensor_parallel instead of skipping.
Guard on megatron.core.tensor_parallel, which is what the native model
imports.

Also drop a VIME_PLATFORM setenv carried over from a platform layer that
does not exist here, and which nothing reads.

CI runs test files individually, so this changes nothing there: the file
is 11 passed either way. In a single-process run of the whole suite it is
now 11 skipped rather than 11 errors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
@read-the-docs-community

read-the-docs-community Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several new features and examples, including a multi-agent system with rollout ID sharing, a tool-enabled language model generation framework (Retool) with a safe Python sandbox, a Search-R1 lite implementation supporting local and Google search backends, and native Qwen3-VL model support. Key feedback includes addressing a critical security vulnerability in the Retool sandbox by replacing fragile regex-based safety checks with AST parsing, fixing JSON parsing errors caused by newline replacements, correcting nested dictionary retrieval in the local search server to prevent discarding search results, and resolving a potential runtime AttributeError in the Qwen3-VL model by using the configuration's parameter data type instead of accessing a module's non-existent dtype attribute.

Comment on lines +113 to +166
def _check_code_safety(self, code: str) -> tuple[bool, str]:
"""Check code safety by scanning for dangerous patterns"""
# Check for dangerous operations
dangerous_patterns = [
r"import\s+os",
r"import\s+sys",
r"import\s+subprocess",
r"import\s+shutil",
r"import\s+glob",
r"import\s+pathlib",
r"__import__",
r"eval\s*\(",
r"exec\s*\(",
r"open\s*\(",
r"file\s*\(",
r"input\s*\(",
r"raw_input\s*\(",
r"compile\s*\(",
r"execfile\s*\(",
r"getattr\s*\(",
r"setattr\s*\(",
r"delattr\s*\(",
r"hasattr\s*\(",
r"globals\s*\(",
r"locals\s*\(",
r"vars\s*\(",
r"dir\s*\(",
r"type\s*\(",
r"isinstance\s*\(",
r"issubclass\s*\(",
r"super\s*\(",
r"property\s*\(",
r"staticmethod\s*\(",
r"classmethod\s*\(",
r"__\w+__", # double underscore methods
]

for pattern in dangerous_patterns:
if re.search(pattern, code, re.IGNORECASE):
return False, f"Code contains dangerous pattern: {pattern}"

# Check imported modules
import_pattern = r"import\s+(\w+)"
from_pattern = r"from\s+(\w+)"

imports = re.findall(import_pattern, code)
froms = re.findall(from_pattern, code)

all_imports = set(imports + froms)
for imp in all_imports:
if imp not in self.allowed_modules:
return False, f"Import of '{imp}' is not allowed"

return True, "Code is safe"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-critical critical

The _check_code_safety method uses regular expressions to detect dangerous imports and operations. However, regex-based parsing of Python code is extremely fragile and easy to bypass. For example, import math, os bypasses both the dangerous_patterns check (which expects import\s+os) and the import_pattern check (which only extracts the first module math due to the \s+(\w+) pattern). This allows an attacker to import and execute arbitrary code (e.g., os.system) on the host system, leading to a complete sandbox escape.\n\nUsing Python's built-in ast (Abstract Syntax Tree) module is the standard, robust, and secure way to parse and inspect Python code structure safely.

    def _check_code_safety(self, code: str) -> tuple[bool, str]:\n        \"\"\"Check code safety by parsing the AST to prevent sandbox escape.\"\"\"\n        import ast\n\n        try:\n            tree = ast.parse(code)\n        except SyntaxError as e:\n            return False, f\"Syntax error: {e}\"\n\n        for node in ast.walk(tree):\n            # Check imports (e.g., import os, or import math, os)\n            if isinstance(node, ast.Import):\n                for alias in node.names:\n                    base_module = alias.name.split(\".\")[0]\n                    if base_module not in self.allowed_modules:\n                        return False, f\"Import of '{alias.name}' is not allowed\"\n            elif isinstance(node, ast.ImportFrom):\n                if node.module:\n                    base_module = node.module.split(\".\")[0]\n                    if base_module not in self.allowed_modules:\n                        return False, f\"Import from '{node.module}' is not allowed\"\n\n            # Check for dangerous builtins\n            if isinstance(node, ast.Name):\n                if node.id in {\n                    \"eval\",\n                    \"exec\",\n                    \"open\",\n                    \"input\",\n                    \"raw_input\",\n                    \"compile\",\n                    \"globals\",\n                    \"locals\",\n                    \"__import__\",\n                    \"getattr\",\n                    \"setattr\",\n                    \"delattr\",\n                    \"hasattr\",\n                }:\n                    return False, f\"Use of builtin '{node.id}' is not allowed\"\n\n            # Check for dunder attributes/methods access\n            if isinstance(node, ast.Attribute):\n                if node.attr.startswith(\"__\") and node.attr.endswith(\"__\"):\n                    return False, f\"Access to double underscore attribute '{node.attr}' is not allowed\"\n\n        return True, \"Code is safe\"

Comment on lines +126 to +128
# Replace newlines in string values with \n
json_str = json_str.replace("\n", "\\n")
tool_call_data = json.loads(json_str)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The code attempts to clean up the JSON string by replacing all newlines with \\n (json_str = json_str.replace(\"\\n\", \"\\\\n\")). However, this replaces newlines outside of JSON string literals (e.g., between keys or after braces in pretty-printed JSON), turning valid JSON into completely invalid JSON (e.g., {\n becomes {\\n which is a syntax error). This causes json.loads to always fail with a JSONDecodeError whenever the model outputs formatted or pretty-printed JSON.\n\nUsing json.loads(json_str, strict=False) is the standard and robust way to allow unescaped control characters (like raw newlines) inside JSON string values without breaking the overall JSON structure.

Suggested change
# Replace newlines in string values with \n
json_str = json_str.replace("\n", "\\n")
tool_call_data = json.loads(json_str)
tool_call_data = json.loads(json_str, strict=False)

Comment on lines +84 to +94
if isinstance(item, dict):
# Access the document dict first, then get contents
content = item.get("contents", "")

if content:
# The contents are already in the correct format: '"Title"\nText content...'
# Just pass through as-is to match google_search format
contexts.append({"document": {"contents": content}})
else:
# Empty content case - provide default values
contexts.append({"document": {"contents": '"No title."\nNo snippet available.'}})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The code attempts to extract search contents using content = item.get(\"contents\", \"\"). However, the comment on line 83 states that the retrieval server returns a nested structure: {\"document\": {\"id\": \"...\", \"contents\": \"...\"}}. If the structure is nested, item.get(\"contents\") will return None (or \"\"), causing the code to fall back to the else block and append \"No title.\"\\nNo snippet available. for every single search result. This effectively discards all retrieved search results.\n\nWe should safely handle both flat and nested structures by checking if \"document\" is in item.

        if isinstance(item, dict):\n            # Access the document dict first if nested, then get contents\n            document = item.get(\"document\", item) if \"document\" in item else item\n            content = document.get(\"contents\", \"\") if isinstance(document, dict) else \"\"\n\n            if content:

continue
if grids is None:
raise ValueError("Qwen3-VL pixel values require matching grid_thw")
output = self.model.visual(values.to(dtype=self.model.visual.dtype), grid_thw=grids)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The code attempts to cast values to self.model.visual.dtype. However, standard PyTorch nn.Module objects (such as Qwen3VLVisionModel) do not have a .dtype attribute by default, which can raise an AttributeError at runtime. Since self.config.params_dtype is already initialized and used elsewhere in the file, it is much safer and more robust to use self.config.params_dtype directly.

Suggested change
output = self.model.visual(values.to(dtype=self.model.visual.dtype), grid_thw=grids)
output = self.model.visual(values.to(dtype=self.config.params_dtype), grid_thw=grids)

@Meihan-chen Meihan-chen changed the title [Training][Rollout][Example] Add Qwen3-VL loading and tool-use examples Add Qwen3-VL loading and tool-use examples Oct 23, 2026
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant