Skip to content

feat(model): support SmolVLM and SmolVLM2, fix the idefics3 template - #10591

Open
Zephyr4772 wants to merge 1 commit into
modelscope:mainfrom
Zephyr4772:feat/smolvlm-support
Open

Zephyr4772 wants to merge 1 commit into
modelscope:mainfrom
Zephyr4772:feat/smolvlm-support

Conversation

@Zephyr4772

Copy link
Copy Markdown

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Fixes #10590. Refs #2989, #6419.

The idefics3 template only worked for Idefics3-8B, and even there it drifted from the model's chat template. This PR:

  • Expands each <image>/<video> with the HF processor, in order of appearance, using the processor's own image token (Idefics3-8B: 128257, SmolVLM: 49190, SmolVLM-Instruct: 49153) instead of a hardcoded id. Video uses the processor's frame sampling (1 fps, up to 64 frames), as its chat template does.
  • Uses the tokenizer's BOS (<|begin_of_text|> / <|im_start|>) and the chat template's spacing: Role: text, but Role:<image> when a turn starts with an image; the response gets the leading space because the generation prompt ends with Assistant:.
  • Pads pixel_values/pixel_attention_mask across samples, so batch size > 1 works (Error finetuning SmolVLM (idefics3 template) with batch size > 1 #6419). The model drops all-zero patches; batched logits match one-by-one logits (max diff ~3e-4 in fp32).
  • Registers SmolVLM-256M/500M/Instruct (+Base) and SmolVLM2-2.2B-Base under idefics3, and adds smolvlm2 (SmolVLMForConditionalGeneration) for SmolVLM2-256M/500M-Video-Instruct and 2.2B-Instruct, tags vision + video. SmolVLM2 needs num2words.
  • Removes align_image_inputs, which only this template used.
  • Adds both families to the supported-models docs (en/zh).

vLLM/LMDeploy keep the plain placeholders (not tested here).

Experiment results

tests/general/test_idefics3_template.py (processor files only, no weights): input_ids, labels, pixel_values and pixel_attention_mask equal the HF processor + chat template for system + multi-turn, two images inside text, text-only and SmolVLM2 video; the collator pads patches. 9 passed with this PR, 9 failed on main.

pytest tests/general/test_idefics3_template.py   # 9 passed

Same comparison on the processors of Idefics3-8B, SmolVLM-Instruct (2.2B) and SmolVLM2-2.2B-Instruct: all match. On main, Idefics3-8B differs on every text turn, e.g. User:hello ... Assistant:world vs User: hello ... Assistant: world.

Greedy inference (TransformersEngine) gives exactly the same answers as plain transformers generate() with the processor's chat template, for image, text and video (SmolVLM-256M, SmolVLM2-256M). Batched inference with mixed image sizes and a video equals one-by-one inference.

Training (RTX 4060, LoRA):

  • SmolVLM-256M, batch size 4, mixed image sizes: loss 0.80 → 0.10, eval token acc 1.0 (fails on main).
  • SmolVLM2-256M, batch size 4, mixed video/image batches: loss 0.85 → 0.27.
  • DPO with images (SmolVLM-256M): loss 0.693 → 0.426, reward accuracy 1.0.

Checklist

  • I have linked the related issue above, if there is one.
  • I have reviewed and understand every changed line, and I can explain how the change works and why it is correct.
  • I have run the relevant tests or reproductions locally, and included the commands and results above (if applicable).
  • I am responsible for the content of this PR, regardless of how it was produced.

This PR was prepared with AI assistance (Claude Code), which wrote the change and ran the tests and checks above. I am responsible for it and will answer review questions.

- Expand each <image>/<video> with the HF processor, in order, using the
  processor's own image token instead of the hardcoded Idefics3-8B id.
- Use the tokenizer's BOS and the chat template's `Role: text` /
  `Role:<image>` spacing, so encodings match the processor exactly.
- Pad pixel_values/pixel_attention_mask across samples (batch size > 1).
- Register the SmolVLM models under idefics3 and add smolvlm2
  (SmolVLMForConditionalGeneration, image and video).
- Remove align_image_inputs, which is no longer used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hjh0119 hjh0119 self-assigned this Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support SmolVLM / SmolVLM2 (idefics3 template breaks for them and drifts from the chat template for Idefics3-8B)

2 participants