Skip to content

Add GPTQModel-backed AWQ export - #10491

Open
sfeng1996 wants to merge 3 commits into
modelscope:mainfrom
sfeng1996:awq-gptqmodel
Open

sfeng1996 wants to merge 3 commits into
modelscope:mainfrom
sfeng1996:awq-gptqmodel

Conversation

@sfeng1996

Copy link
Copy Markdown

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Refs #10307

This PR replaces the AutoAWQ export path with GPTQModel while preserving the existing --quant_method awq interface.

It also:

  • Adds ms-swift[awq] as an optional dependency.
  • Supports image calibration for Qwen2-VL.
  • Preserves the --quant_batch_size -1 behavior.
  • Documents known dependency conflicts and current export limitations.

Issue #10307 concerns dependency conflicts during AWQ model fine-tuning. This PR is related to that issue but does not claim to resolve the fine-tuning problem.

Current limitations

This implementation is not yet a drop-in replacement for every previous AWQ use case:

  • Explicit --device_map and some custom model-loading options are not supported.
  • Multimodal AWQ calibration currently supports Qwen2-VL images only.
  • Qwen2-VL was tested with a randomly initialized small model, not a pretrained model or an accuracy benchmark.
  • The optional dependency does not resolve the declared conflicts between GPTQModel, SwanLab, and ms-opencompass.

Experiment results

Tests were performed on an NVIDIA GeForce RTX 4090 D with Python 3.12, PyTorch 2.11.0+cu130, Transformers 5.14.1, GPTQModel 7.3.2, and vLLM 0.25.1.

  • Qwen3-0.6B AWQ, GPTQ, and GPTQ v2 GPU exports completed. vLLM loaded each exported model and generated text.

  • The tested GPTQ v2 export was converted to GPTQ v1 during saving; the exported configuration recorded the actual format.

  • A nonzero LoRA adapter was merged before AWQ export.

  • A synthetic small Qwen2-VL model completed image calibration, AWQ export, reload, and generation with image inputs.

  • --quant_batch_size -1 completed an AWQ GPU export.

  • The opt-in GPU integration suite passed for AWQ, GPTQ, and GPTQ v2:

    SWIFT_TEST_QUANT_MODEL=/path/to/Qwen3-0.6B \
    SWIFT_TEST_QUANT_DATASET=/path/to/c4_demo.jsonl \
    python -m pytest -q tests/export/test_quant_gpu_integration.py
  • AWQ unit tests, GPTQ saved-format tests, the optional-dependency metadata test, and pre-commit run --all-files passed.

The diagnostic image still had dependency conflicts: GPTQModel 7.3.2 requires protobuf >=7.34 and NumPy 2.2.6 on Python 3.12, while SwanLab 0.9.2 requires protobuf <7 and ms-opencompass 0.1.6 requires NumPy <2. Therefore, pip check did not pass for the complete diagnostic image.

AI assistance was used to develop and test this change.

Checklist

  • I have linked the related issue above, if there is one.
  • I have reviewed and understand every changed line, and I can explain how the change works and why it is correct.
  • I have run the relevant tests or reproductions locally, and included the commands and results above (if applicable).
  • I am responsible for the content of this PR, regardless of how it was produced.

sunfeng added 3 commits October 30, 2026 21:02
Use GPTQModel public loading API, add Qwen2-VL image calibration and document the optional dependencies and known compatibility limits.

AI-Assisted-by: OpenAI Codex
AI-Assisted-by: OpenAI Codex
AI-Assisted-by: OpenAI Codex

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant