Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 20 additions & 6 deletions docs/source/Instruction/Export-and-push.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,17 @@ SWIFT支持AWQ、GPTQ、FP8、BNB模型的量化导出。其中使用AWQ、GPTQ
| -------- | ------ | -------- | -------- |
| FP8 | ✅ | ✅ | ✅ |
| GPTQ | ✅ | ✅ | ✅ |
| AWQ | ✅ | ✅ | ✅ |
| AWQ | ❌ | ✅ | ✅ |
| BNB | ❌ | ✅ | ✅ |


除SWIFT安装外,需要安装以下额外依赖:
```shell
# 使用awq量化:
# autoawq和cuda版本有对应关系,请按照`https://github.com/casper-hansen/AutoAWQ`选择版本
# 如果出现torch依赖冲突,请额外增加指令`--no-deps`
pip install autoawq -U
# 使用 AWQ 量化(从源码目录安装):
pip install -e '.[awq]'

# 已发布的 ms-swift 包提供该选项后,也可以使用:
# pip install 'ms-swift[awq]'

# 使用gptq量化:
pip install gptqmodel optimum -U
Expand All @@ -34,13 +35,26 @@ pip install gptqmodel optimum -U
pip install bitsandbytes -U
```

`awq` 是独立的可选依赖,不包含在基础安装或 `all` 选项中。安装前请检查环境中的依赖约束,安装后运行 `python -m pip check`。以下是已验证的特定版本,不能推断其他版本也兼容:

| 组件 | 测试版本或已知约束 |
| --- | --- |
| AWQ 导出测试环境 | Python 3.12、PyTorch 2.11.0+cu130、Transformers 5.14.1、GPTQModel 7.3.2 |
| GPTQModel 7.3.2 | 要求 Python >=3.10;Python 3.12 下要求 NumPy 2.2.6 和 protobuf >=7.34.0 |
| SwanLab 0.9.2 | 要求 protobuf <7,与 GPTQModel 7.3.2 的要求冲突 |
| ms-opencompass 0.1.6 | 要求 NumPy <2,与 GPTQModel 7.3.2 的要求冲突 |

如果同一环境需要 AWQ、SwanLab 或 ms-opencompass,先确认所选版本的依赖声明存在共同解;上述冲突版本不能通过调整安装顺序解决。若暂时没有共同解,可以将量化、训练和评测放在各自的环境中。不要使用 `--no-deps` 将 `pip check` 中的冲突视为已解决。实际依赖可能随版本和 Python 版本变化,还应验证 PyTorch、CUDA、Transformers 和其他已安装组件。上述测试镜像自身仍有其他依赖冲突,AWQ 导出测试通过不代表整个镜像兼容。

我们提供了一系列脚本展现SWIFT的量化导出能力:
- 支持[AWQ](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/awq.sh)/[GPTQ](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/gptq.sh)/[GPTQ v2](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/gptq_v2.sh)/[BNB](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/bnb.sh)量化导出。
- 多模态量化: 支持使用GPTQ和AWQ对多模态模型进行量化,其中AWQ支持的多模态模型有限。参考[这里](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/mllm)。
- 多模态量化: GPTQ支持多模态模型;AWQ目前不支持多模态校准。参考[这里](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/mllm)。
- 更多系列模型的支持: 支持[Bert](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/bert),[Reward Model](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/reward_model)的量化导出。
- 使用SWIFT量化导出的模型支持使用vllm/sglang/lmdeploy进行推理加速;也支持使用QLoRA继续进行SFT/RLHF。


AWQ 使用 GPTQModel 生成带零点的 GEMM 格式权重。目前支持文本校准,使用 safetensors 和未量化的模型;LoRA 需要先使用 `--merge_lora true` 合并。AWQ 不支持在导出时添加特殊 token、指定 `--device_map` 或设置自定义模型加载参数。模型架构须受 GPTQModel 支持,安装时请检查与现有 PyTorch、Transformers 和其他依赖的兼容性。

使用 GPTQ v2 量化时,后端可能将对称量化权重保存为 GPTQ v1 格式。导出配置会记录实际保存的格式,以便正确重新加载模型。

## 推送模型
Expand Down
26 changes: 20 additions & 6 deletions docs/source_en/Instruction/Export-and-push.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,17 @@ SWIFT supports quantization exports for AWQ, GPTQ, FP8, and BNB models. AWQ and
| ---------------------- | ---------- | ---------------------- | ------------------ |
| FP8 | ✅ | ✅ | ✅ |
| GPTQ | ✅ | ✅ | ✅ |
| AWQ | ✅ | ✅ | ✅ |
| AWQ | ❌ | ✅ | ✅ |
| BNB | ❌ | ✅ | ✅ |

In addition to the SWIFT installation, the following additional dependencies need to be installed:

```shell
# For AWQ quantization:
# The versions of autoawq and CUDA are correlated; please choose the version according to `https://github.com/casper-hansen/AutoAWQ`.
# If there are dependency conflicts with torch, please add the `--no-deps` option.
pip install autoawq -U
# For AWQ quantization (from the source checkout):
pip install -e '.[awq]'

# Once a published ms-swift release provides this extra:
# pip install 'ms-swift[awq]'

# For GPTQ quantization:
pip install gptqmodel optimum -U
Expand All @@ -33,14 +34,27 @@ pip install gptqmodel optimum -U
pip install bitsandbytes -U
```

`awq` is an independent optional extra; it is not included in the base installation or the `all` extra. Check the environment's dependency constraints before installing and run `python -m pip check` afterward. The following observations apply to the specific versions tested, not to all releases:

| Component | Tested version or known constraint |
| --- | --- |
| AWQ export test environment | Python 3.12, PyTorch 2.11.0+cu130, Transformers 5.14.1, GPTQModel 7.3.2 |
| GPTQModel 7.3.2 | Requires Python >=3.10; on Python 3.12, requires NumPy 2.2.6 and protobuf >=7.34.0 |
| SwanLab 0.9.2 | Requires protobuf <7, conflicting with GPTQModel 7.3.2 |
| ms-opencompass 0.1.6 | Requires NumPy <2, conflicting with GPTQModel 7.3.2 |

If AWQ, SwanLab, or ms-opencompass are needed in the same environment, first confirm that the chosen versions' declared requirements have a common solution; changing the installation order cannot resolve the conflicts above. If no common solution is available, use separate environments for quantization, training, and evaluation. Do not treat `--no-deps` as a resolution of `pip check` conflicts. Requirements may change by package and Python version; also verify PyTorch, CUDA, Transformers, and other installed components. The test image has other dependency conflicts, so successful AWQ export does not establish compatibility of the entire image.

We provide a series of scripts to demonstrate SWIFT's quantization export capabilities:

- Supports [AWQ](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/awq.sh)/[GPTQ](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/gptq.sh)/[GPTQ v2](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/gptq_v2.sh)/[BNB](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/bnb.sh) quantization exports.
- Multimodal quantization: Supports quantizing multimodal models using GPTQ and AWQ, with limited multimodal models supported by AWQ. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/mllm).
- Multimodal quantization: GPTQ supports multimodal models; AWQ does not currently support multimodal calibration. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/mllm).
- Support for more model series: Supports quantization exports for [BERT](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/bert) and [Reward Model](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/reward_model).
- Models exported with SWIFT's quantization support inference acceleration using vllm/sglang/lmdeploy; they also support further SFT/RLHF using QLoRA.


AWQ uses GPTQModel to produce GEMM weights with zero points. It supports text calibration, safetensors, and unquantized source models; merge LoRA adapters first using `--merge_lora true`. Adding special tokens, setting `--device_map`, or passing custom model loading arguments during AWQ export are not supported. The model architecture must be supported by GPTQModel. Check compatibility with the existing PyTorch, Transformers, and other dependencies when installing it.

GPTQ backends may save symmetric weights in GPTQ v1 format even when GPTQ v2 is used during quantization. The exported configuration records the saved format so the model reloads correctly.

## Push Model
Expand Down
1 change: 1 addition & 0 deletions requirements/awq.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
gptqmodel>=7.3.2
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ def gen_packages_items():
extra_requires['eval'], _ = parse_requirements('requirements/eval.txt')
extra_requires['swanlab'], _ = parse_requirements('requirements/swanlab.txt')
extra_requires['ray'], _ = parse_requirements('requirements/ray.txt')
extra_requires['awq'], _ = parse_requirements('requirements/awq.txt')
all_requires.extend(install_requires)
all_requires.extend(extra_requires['eval'])
all_requires.extend(extra_requires['swanlab'])
Expand Down
105 changes: 58 additions & 47 deletions swift/pipelines/export/quant.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,42 @@ class QuantEngine(ProcessorMixin):

def __init__(self, args: ExportArguments):
self.args = args
kwargs = {}
if args.quant_method == 'awq':
from awq import AutoAWQForCausalLM
kwargs['auto_model_cls'] = AutoAWQForCausalLM
self.model, self.template = prepare_model_template(args, **kwargs)
if args.adapters:
raise ValueError('AWQ requires merged adapters. Use --merge_lora true first.')
if not args.safe_serialization:
raise ValueError('GPTQModel AWQ export requires --safe_serialization true.')
if args.quant_bits is None:
raise ValueError('Please set quant_bits for AWQ export.')
if args.new_special_tokens:
raise ValueError('AWQ does not support adding special tokens during export.')
if args.device_map is not None:
raise ValueError('GPTQModel AWQ selects its own device; --device_map is not supported.')
if args.task_type != 'causal_lm' or args.tuner_backend == 'unsloth':
raise ValueError('GPTQModel AWQ supports causal language models without the unsloth backend.')
for name in ('model_kwargs', 'rope_scaling', 'max_memory', 'attn_impl', 'experts_impl', 'max_model_len',
'init_strategy'):
if getattr(args, name, None):
raise ValueError(f'GPTQModel AWQ does not support --{name} during export.')
if getattr(args.model_info.config, 'quantization_config', None):
raise ValueError('AWQ requires an unquantized model; the input is already quantized.')
try:
from gptqmodel import AWQConfig, GPTQModel
except ImportError as exc:
raise ImportError('AWQ export requires GPTQModel >= 7.3.2: pip install "gptqmodel>=7.3.2"') from exc
_, processor = args.get_model_processor(load_model=False)
self.template = args.get_template(processor)
quant_config = AWQConfig(bits=args.quant_bits, group_size=args.group_size, sym=False)
self.awq_model = GPTQModel.load(
args.model_dir, quantize_config=quant_config, trust_remote_code=True, dtype=args.torch_dtype)
if self.awq_model.quantized:
raise ValueError('AWQ requires an unquantized model; the input is already quantized.')
self.model = self.awq_model.model
self.model.model_meta = processor.model_meta
self.model.model_info = processor.model_info
self.model.model_dir = args.model_dir
else:
self.model, self.template = prepare_model_template(args)
self.template.set_mode('train')
self.model.config.use_cache = False
HfConfigFactory.set_config_attr(self.model.config, 'use_cache', False)
Expand All @@ -38,10 +69,9 @@ def quantize(self):
if args.quant_bits is None and args.quant_method not in {'fp8', 'entropack'}:
raise ValueError(f'Please set the quant_bits. args.quant_bits: {args.quant_bits}')
if args.quant_method == 'awq':
self.template.model = self.model.model
self.awq_model_quantize()
self.model.save_quantized(
args.output_dir, safetensors=args.safe_serialization, shard_size=args.max_shard_size)
self.template.model = self.model
awq_model = self.awq_model_quantize()
awq_model.save(args.output_dir, max_shard_size=args.max_shard_size)
elif args.quant_method in {'gptq', 'gptq_v2'}:
self.template.model = self.model
gptq_quantizer = self.gptq_model_quantize(v2=(args.quant_method == 'gptq_v2'))
Expand Down Expand Up @@ -134,48 +164,29 @@ def _get_quant_dataset(self, *args, **kwargs):
res.append(torch.tensor(input_ids)[None])
return res

@staticmethod
@contextmanager
def _patch_awq_move_embed(awq_model):
_origin_move_embed = awq_model.move_embed

def _move_embed(model, device: str):
if hasattr(model, '_hf_hook') and device != 'cpu':
return
_origin_move_embed(model, device)

awq_model.move_embed = _move_embed
try:
yield
finally:
awq_model.move_embed = _origin_move_embed

def awq_model_quantize(self) -> None:
from awq.quantize import quantizer

def awq_model_quantize(self):
args = self.args
if getattr(self.model.config, 'quantization_config', None):
raise ValueError('AWQ requires an unquantized model; the input is already quantized.')
if any(str(param.dtype).startswith('torch.float8') for param in self.model.parameters()):
raise ValueError('AWQ requires an unquantized model; FP8 weights are not supported.')
if args.adapters:
raise ValueError('AWQ requires merged adapters. Use --merge_lora true first.')
if not args.safe_serialization:
raise ValueError('GPTQModel AWQ export requires --safe_serialization true.')

logger.info(f'Quantization dataset: {args.dataset}')
_origin_get_calib_dataset = quantizer.get_calib_dataset
quantizer.get_calib_dataset = self._get_quant_dataset
quant_config = {
'zero_point': True,
'q_group_size': args.group_size,
'w_bit': args.quant_bits,
'version': 'GEMM'
}
if self.model.model_info.is_moe_model:
quant_config['modules_to_not_convert'] = self.args.get_modules_to_not_convert()
logger.info(f'quant_config: {quant_config}')
if self.model.model_meta.is_multimodal:
raise ValueError('GPTQModel AWQ export does not currently support multimodal calibration.')
calibration = [{
'input_ids': block.reshape(-1).tolist()
} for block in self._get_quant_dataset() if block.numel()]
if not calibration:
raise ValueError('AWQ calibration dataset contains no usable tokens.')
logger.info('Start quantizing the model...')
with self._patch_awq_move_embed(self.model):
self.model.quantize(
self.tokenizer, quant_config=quant_config, n_parallel_calib_samples=args.quant_batch_size)
quantizer.get_calib_dataset = _origin_get_calib_dataset # recover
if self.model.quant_config.modules_to_not_convert:
model_arch = args.model_meta.model_arch
lm_head_key = getattr(model_arch, 'lm_head', None) or 'lm_head'
if lm_head_key not in self.model.quant_config.modules_to_not_convert:
self.model.quant_config.modules_to_not_convert.append(lm_head_key)
batch_size = args.quant_batch_size if args.quant_batch_size is not None else len(calibration)
self.awq_model.quantize(calibration=calibration, batch_size=batch_size)
return self.awq_model

@contextmanager
def _patch_gptq(self):
Expand Down
24 changes: 24 additions & 0 deletions tests/export/test_awq_dependency_metadata.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Copyright (c) ModelScope Contributors. All rights reserved.
import subprocess
import sys
from importlib.metadata import PathDistribution
from packaging.requirements import Requirement
from pathlib import Path


def test_awq_backend_is_an_opt_in_dependency(tmp_path):
root = Path(__file__).resolve().parents[2]
subprocess.run([sys.executable, 'setup.py', 'egg_info', '--egg-base',
str(tmp_path)],
cwd=root,
check=True,
capture_output=True,
text=True)
metadata = PathDistribution(next(tmp_path.glob('*.egg-info')))
requirements = [Requirement(item) for item in metadata.requires or []]
gptqmodel = [item for item in requirements if item.name.lower() == 'gptqmodel']
assert len(gptqmodel) == 1
assert str(gptqmodel[0].specifier) == '>=7.3.2'
assert gptqmodel[0].marker.evaluate({'extra': 'awq'})
assert not gptqmodel[0].marker.evaluate({'extra': ''})
assert not gptqmodel[0].marker.evaluate({'extra': 'all'})
Loading