Skip to content

megatro-swift npu环境 qwen3.5-122b grpo server模式时采样长度直接重复感叹号到最大采样长度 #10595

Open

Description

@machaoyang1999

Checklist / 检查清单

  • I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。

Bug Description / Bug 描述

达到最大采样长度:
Image
感叹号效果:
Image

How to Reproduce / 如何复现

npu 910b环境
ms-swift 4.5.3
transformers=5.17.0
训练脚本:
服务1:

ASCEND_RT_VISIBLE_DEVICES=0,1
vllm serve /npu_data/models/Qwen/Qwen3.5-35B-A3B
--host 0.0.0.0
--port 8080
--data_parallel_size 1
--tensor_parallel_size 2
--max_model_len 16384
--gpu_memory_utilization 0.8
--served-model-name Qwen3.5-35B-A3B
--compilation-config '{"cudagraph_capture_sizes": [1, 2, 4, 8, 16, 32, 64]}'
服务2:

ASCEND_RT_VISIBLE_DEVICES=2,3
NPROC_PER_NODE=2
swift rollout
--model /npu_data/models/Qwen/Qwen3.5-35B-A3B
--vllm_data_parallel_size 1
--vllm_tensor_parallel_size 2
--vllm_max_model_len 16384
--vllm_gpu_memory_utilization 0.8
--vllm_engine_kwargs '{"compilation_config": {"cudagraph_capture_sizes": [1, 2, 4, 8, 16, 32, 64]}}'
服务3:

ASCEND_RT_VISIBLE_DEVICES=4,5,6,7
NPROC_PER_NODE=4
MASTER_PORT=29600
GENRM_API_BASE=http://localhost:8080/v1
megatron rlhf
--rlhf_type grpo
--add_version false
--model /npu_data/models/Qwen/Qwen3.5-35B-A3B
--split_dataset_ratio 0.1
--gradient_accumulation_fusion false
--save_safetensors true
--context_parallel_size 1
--tensor_model_parallel_size 1
--pipeline_model_parallel_size 2
--expert_model_parallel_size 2
--recompute_granularity full
--recompute_method uniform
--optimizer_cpu_offload true
--recompute_num_layers 1
--dataset /npu_data/dataset_grpo.jsonl
--num_train_epochs 5
--global_batch_size 16
--micro_batch_size 1
--steps_per_generation 1
--num_generations 8
--external_plugins plugin.py
--use_vllm true
--vllm_mode server
--vllm_server_host 127.0.0.1
--vllm_server_port 8000
--max_length 4096
--max_completion_length 4096
--tuner_type lora
--lr 5e-5
--beta 0.00
--importance_sampling_level sequence
--epsilon 3e-4
--epsilon_high 4e-4
--dynamic_sample false
--overlong_filter true
--loss_type grpo
--sleep_level 2
--finetune
--dataloader_num_workers 8
--dataset_num_proc 8
--no_save_optim
--no_save_rng
--attention_backend flash
--temperature 1.0
--padding_free true
--log_completions true
--logging_steps 1
--eval_steps 2
--save_steps 2
--eval_iters=-1
--save_total_limit 3
--output_dir /npu_data/output_grpo
--reward_funcs async_semantic_genrm
依赖的plugin.py 也可直接使用--reward_funcs accuracy format替代

plugin.py

Additional Information / 补充信息

No response

Activity

  1. hjh0119 commented on Nov 18, 2026

    @hjh0119
    Collaborator

    can you retry with swift 4.6.1?

  2. machaoyang1999 commented on Nov 18, 2026

    @machaoyang1999
    Author

    ok i will try

  3. machaoyang1999 commented on Nov 19, 2026

    @machaoyang1999
    Author
    image 安装完上述依赖及ms-swift=4.6.1 还是有这个问题
  4. hjh0119 commented on Nov 19, 2026

    @hjh0119
    Collaborator

    大概率是训崩了,换个数据集/调整下参数试试

  5. hjh0119 commented on Nov 19, 2026

    @hjh0119
    Collaborator

    可以设置 log_rollout_offpolicy_metrics true 观察下指标,排除权重同步的问题

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions