Skip to content

[Test][Ascend] Validate Qwen3-0.6B disk delta updates against full checkpoint - #505

Draft
wangx700 wants to merge 11 commits into
vllm-project:ascendfrom
wangx700:test/disk-delta-30b-oracle
Draft

wangx700 wants to merge 11 commits into
vllm-project:ascendfrom
wangx700:test/disk-delta-30b-oracle

Conversation

@wangx700

@wangx700 wangx700 commented Oct 28, 2026 •

Copy link
Copy Markdown

Strengthen the Qwen3-0.6B disk-delta test with an independent full-checkpoint and inference comparison.

Test flow:

  1. Run two real training/update rounds for both XOR and overwrite.
  2. Check nonempty deltas, version progression, and checksums; compare all reconstructed tensor bytes against an independently exported HF checkpoint.
  3. Compare fixed greedy inference from the updated engine with a fresh engine loaded from the full checkpoint. Token IDs, per-token logprobs, and finish reason must match exactly.

Also add model-response validation, UTF-8 JSON reads, and safe fresh-server startup and cleanup.

Validation summary (condensed from test logs):

Unit tests: 3 passed
XOR: 2 updates passed; 238 delta checksums verified
Overwrite: 2 updates passed; 242 delta checksums verified
Both encodings: all 310 checkpoint tensors match byte-for-byte
Token IDs / logprobs / finish reason: exact match
Maximum absolute logprob error: 0.0
E2E_EXIT:0

Ruff, Black, isort, Python compilation, and whitespace checks passed.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request upgrades the disk delta weight synchronization E2E smoke test to target the Qwen3-30B-A3B model, transitioning it into a robust E2E and full-checkpoint inference oracle. It also adds comprehensive documentation detailing the background, architecture, and encoding rules for incremental weight synchronization on shared disks. The reviewer feedback is highly constructive, highlighting several opportunities to improve the robustness of the test script. Key recommendations include adding defensive checks for empty model lists, explicitly specifying UTF-8 encoding when reading text files, checking directory existence before deletion, avoiding the use of private attributes like _actor_handlers, and addressing a potential TOCTOU race condition when reserving ports.

Comment thread tests/test_delta_disk_weight_update.py Resolved
Comment thread tests/test_delta_disk_weight_update.py Resolved
Comment thread tests/test_delta_disk_weight_update.py Resolved
Comment thread tests/test_delta_disk_weight_update.py Resolved
Comment thread tests/test_delta_disk_weight_update.py Resolved
Comment thread tests/test_delta_disk_weight_update.py Resolved
@wangx700 wangx700 changed the title [Test][Ascend] Validate 30B disk delta updates against full checkpoint [Test][Ascend] Validate Qwen3-0.6B disk delta updates against full checkpoint Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant