Repository navigation
Conversation
Passes TrainConfig.dp_balance to twinkle's TransformersModel, which by default reorders each Ray slice_dp batch so the DP ranks get similar workloads. --dp_balance false turns it off. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Pass the default-on dp_balance switch through Transformers, Unsloth, SentenceTransformer and Megatron construction. Initialize independent sentence-model constructors with the shared base state. Remove recipe-level planners, permutations and balance_batch configuration. Forward streaming lengths without fetching feature refs, retain model metric collection, and cover switch propagation and unchanged recipe signal ordering.
lincq2000
force-pushed
the
feat/dp-balance-switch
branch
from
November 12, 2026 09:36
29ac326 to
cc7d95e
Compare
2 of 4 tasks
tastelikefeet
approved these changes
Nov 13, 2026
Upstream converged recipe-level balancing into batching.balance_update and added balance_lanes for the RL lane forwards. This branch keeps balancing in the Twinkle model instead, so the merge drops recipe/batching.py and StepState.dp_balance, and the recipes submit every forward in the caller's order with input_lengths taken from the batch summary's row lengths. RFT now carries prompt + response row lengths for the update; the response lengths still feed the reward metric. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR type
PR 说明
关联 issue:无。依赖 modelscope/twinkle#367;替换 #10517 / #10546 在 recipe 层的接入。
统一使用
TrainableModel内置、默认开启的 DP 负载均衡,移除各 recipe 中独立的规划和重排逻辑。本 PR 负责 Swift 接入;planner、后端布局估算、loss 安全检查、引用调度及输出顺序还原统一由 Twinkle 实现。整体方案
TrainConfig.dp_balance默认True,统一使用--dp_balance false关闭。Transformers 系列及 Megatron builder 均将该配置传给模型,覆盖普通 HF、Unsloth 和 SentenceTransformer 路径;独立的 SentenceTransformer、multi-LoRA SentenceTransformer 构造函数也初始化公共基类状态。共用接口不代表强制对 embedding 等不适用目标重排。recipe/batching.py,以及 GRPO / PPO / RFT / 蒸馏中的 planner 调用、逐样本信号手动重排和 recipe 单独维护的均衡指标。recipe 按原始顺序提交已对齐的 optimizer batch,由模型在 DP 分发前统一规划,并在返回时还原逐样本输出顺序。RLConfig.balance_batch、对应的仅 Ray / 在线训练可用校验,以及StepState中重复的开关状态。原--balance_batch用户迁移到--dp_balance;local / Ray 的适用性由模型、后端和 loss 的公共约束判断。RolloutSample/DistillRow的token_lengths,通过轻量公共函数生成模型调用的input_lengths,不拉取远端编码特征。GRPO、PPO 策略/价值模型、RFT 和蒸馏按同一原始顺序传递元数据与训练信号,引用列表的分片和 worker 侧解析由 Twinkle 负责。model.calculate_metric返回的dp_balance/*指标进入原有训练日志;PPO 原有的价值模型指标收集路径继续保留。各训练任务的生效范围
reduction='sum')、RFT与 #10517 / #10546 的关系
本 PR 替换这两个 PR 在 recipe 层接入的均衡逻辑,同时保留异步样本长度元数据的传递能力,统一为模型层的一套实现和一个开关。迁移应与 Twinkle #367 配套完成,无需单独兼容。
与 #10517 的差异:#10517 对 PPO 价值模型和 GKD / OPSD / MOPD 学生 rollout 也做均衡,但不保证 loss 归一化不变;本 PR 对这些按 rank 局部 token 求均值的 loss 默认保持原顺序,避免开关默认开启时改变训练梯度。
验证
在
swift/dev/tests下运行(配合 Twinkle #367 分支):结果:240 passed。
覆盖默认开启/显式关闭的 CLI 解析,HF / Unsloth / SentenceTransformer / Megatron 构造参数传递,异步引用身份与长度元数据、逐样本训练信号对齐、replay / GA 更新窗口、PPO 策略/价值模型调用、蒸馏组装及既有指标日志路径。
git diff --check通过;相较 rebase 后的分支基线,无新增 Ruff 诊断。配套 Twinkle PR 的后端布局、安全性、Ray 与 Gloo 相关测试共 292 项通过。本 PR 使用了 Claude Code 辅助完成。
Experiment results
环境:4 × 昇腾 910B,Python 3.12,torch 2.10.0 / torch_npu 2.10.0.post4,transformers 5.14.1,vLLM 0.31.0 / vllm-ascend 0.19.1rc2,megatron-core 0.18.2 / MindSpeed 26.2.0,flash-linear-attention 0.5.2,ray 2.59.0
配置:Qwen3.5-4B GRPO,GSM8K,LoRA(r=16,alpha=32,all-linear),bf16,DP=2。每轮 rollout 8 条 prompt × 8 个采样,每个优化步 32 样本;最大生成长度 2048。FSDP2 使用动态 batching(每个 micro-batch 8192 token);Megatron 使用固定 micro-batch(每 rank 2 行,梯度累积 8 步)。采用同一随机种子,仅变化
dp_balance。