Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/source/Instruction/Supported-models-and-datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -1158,6 +1158,16 @@
|[AI-ModelScope/Florence-2-large](https://modelscope.cn/models/AI-ModelScope/Florence-2-large)|florence|florence||-|✘|vision|[microsoft/Florence-2-large](https://huggingface.co/microsoft/Florence-2-large)|
|[AI-ModelScope/Florence-2-large-ft](https://modelscope.cn/models/AI-ModelScope/Florence-2-large-ft)|florence|florence||-|✘|vision|[microsoft/Florence-2-large-ft](https://huggingface.co/microsoft/Florence-2-large-ft)|
|[AI-ModelScope/Idefics3-8B-Llama3](https://modelscope.cn/models/AI-ModelScope/Idefics3-8B-Llama3)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceM4/Idefics3-8B-Llama3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3)|
|[HuggingFaceTB/SmolVLM-256M-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-256M-Instruct)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-256M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct)|
|[HuggingFaceTB/SmolVLM-500M-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-500M-Instruct)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct)|
|[HuggingFaceTB/SmolVLM-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-Instruct)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct)|
|[HuggingFaceTB/SmolVLM-256M-Base](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-256M-Base)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-256M-Base](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Base)|
|[HuggingFaceTB/SmolVLM-500M-Base](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-500M-Base)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-500M-Base](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Base)|
|[HuggingFaceTB/SmolVLM-Base](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-Base)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-Base](https://huggingface.co/HuggingFaceTB/SmolVLM-Base)|
|[HuggingFaceTB/SmolVLM2-2.2B-Base](https://modelscope.cn/models/HuggingFaceTB/SmolVLM2-2.2B-Base)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM2-2.2B-Base](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Base)|
|[HuggingFaceTB/SmolVLM2-256M-Video-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM2-256M-Video-Instruct)|smolvlm2|idefics3||transformers>=4.50, num2words|✘|vision, video|[HuggingFaceTB/SmolVLM2-256M-Video-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-256M-Video-Instruct)|
|[HuggingFaceTB/SmolVLM2-500M-Video-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM2-500M-Video-Instruct)|smolvlm2|idefics3||transformers>=4.50, num2words|✘|vision, video|[HuggingFaceTB/SmolVLM2-500M-Video-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-Instruct)|
|[HuggingFaceTB/SmolVLM2-2.2B-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM2-2.2B-Instruct)|smolvlm2|idefics3||transformers>=4.50, num2words|✘|vision, video|[HuggingFaceTB/SmolVLM2-2.2B-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct)|
|[AI-ModelScope/paligemma-3b-pt-224](https://modelscope.cn/models/AI-ModelScope/paligemma-3b-pt-224)|paligemma|paligemma||transformers>=4.41|✘|vision|[google/paligemma-3b-pt-224](https://huggingface.co/google/paligemma-3b-pt-224)|
|[AI-ModelScope/paligemma-3b-pt-448](https://modelscope.cn/models/AI-ModelScope/paligemma-3b-pt-448)|paligemma|paligemma||transformers>=4.41|✘|vision|[google/paligemma-3b-pt-448](https://huggingface.co/google/paligemma-3b-pt-448)|
|[AI-ModelScope/paligemma-3b-pt-896](https://modelscope.cn/models/AI-ModelScope/paligemma-3b-pt-896)|paligemma|paligemma||transformers>=4.41|✘|vision|[google/paligemma-3b-pt-896](https://huggingface.co/google/paligemma-3b-pt-896)|
Expand Down
10 changes: 10 additions & 0 deletions docs/source_en/Instruction/Supported-models-and-datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -1159,6 +1159,16 @@ The table below introduces the models integrated with ms-swift:
|[AI-ModelScope/Florence-2-large](https://modelscope.cn/models/AI-ModelScope/Florence-2-large)|florence|florence||-|✘|vision|[microsoft/Florence-2-large](https://huggingface.co/microsoft/Florence-2-large)|
|[AI-ModelScope/Florence-2-large-ft](https://modelscope.cn/models/AI-ModelScope/Florence-2-large-ft)|florence|florence||-|✘|vision|[microsoft/Florence-2-large-ft](https://huggingface.co/microsoft/Florence-2-large-ft)|
|[AI-ModelScope/Idefics3-8B-Llama3](https://modelscope.cn/models/AI-ModelScope/Idefics3-8B-Llama3)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceM4/Idefics3-8B-Llama3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3)|
|[HuggingFaceTB/SmolVLM-256M-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-256M-Instruct)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-256M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct)|
|[HuggingFaceTB/SmolVLM-500M-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-500M-Instruct)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct)|
|[HuggingFaceTB/SmolVLM-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-Instruct)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct)|
|[HuggingFaceTB/SmolVLM-256M-Base](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-256M-Base)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-256M-Base](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Base)|
|[HuggingFaceTB/SmolVLM-500M-Base](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-500M-Base)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-500M-Base](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Base)|
|[HuggingFaceTB/SmolVLM-Base](https://modelscope.cn/models/HuggingFaceTB/SmolVLM-Base)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM-Base](https://huggingface.co/HuggingFaceTB/SmolVLM-Base)|
|[HuggingFaceTB/SmolVLM2-2.2B-Base](https://modelscope.cn/models/HuggingFaceTB/SmolVLM2-2.2B-Base)|idefics3|idefics3||transformers>=4.45|✘|vision|[HuggingFaceTB/SmolVLM2-2.2B-Base](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Base)|
|[HuggingFaceTB/SmolVLM2-256M-Video-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM2-256M-Video-Instruct)|smolvlm2|idefics3||transformers>=4.50, num2words|✘|vision, video|[HuggingFaceTB/SmolVLM2-256M-Video-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-256M-Video-Instruct)|
|[HuggingFaceTB/SmolVLM2-500M-Video-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM2-500M-Video-Instruct)|smolvlm2|idefics3||transformers>=4.50, num2words|✘|vision, video|[HuggingFaceTB/SmolVLM2-500M-Video-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-Instruct)|
|[HuggingFaceTB/SmolVLM2-2.2B-Instruct](https://modelscope.cn/models/HuggingFaceTB/SmolVLM2-2.2B-Instruct)|smolvlm2|idefics3||transformers>=4.50, num2words|✘|vision, video|[HuggingFaceTB/SmolVLM2-2.2B-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct)|
|[AI-ModelScope/paligemma-3b-pt-224](https://modelscope.cn/models/AI-ModelScope/paligemma-3b-pt-224)|paligemma|paligemma||transformers>=4.41|✘|vision|[google/paligemma-3b-pt-224](https://huggingface.co/google/paligemma-3b-pt-224)|
|[AI-ModelScope/paligemma-3b-pt-448](https://modelscope.cn/models/AI-ModelScope/paligemma-3b-pt-448)|paligemma|paligemma||transformers>=4.41|✘|vision|[google/paligemma-3b-pt-448](https://huggingface.co/google/paligemma-3b-pt-448)|
|[AI-ModelScope/paligemma-3b-pt-896](https://modelscope.cn/models/AI-ModelScope/paligemma-3b-pt-896)|paligemma|paligemma||transformers>=4.41|✘|vision|[google/paligemma-3b-pt-896](https://huggingface.co/google/paligemma-3b-pt-896)|
Expand Down
1 change: 1 addition & 0 deletions swift/model/constant.py
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,7 @@ class MLLMModelType:
phi4_multimodal = 'phi4_multimodal'
florence = 'florence'
idefics3 = 'idefics3'
smolvlm2 = 'smolvlm2'
paligemma = 'paligemma'
molmo = 'molmo'
molmo2 = 'molmo2'
Expand Down
27 changes: 27 additions & 0 deletions swift/model/models/mllm.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,15 @@ def get_model(self, model_dir: str, *args, **kwargs) -> PreTrainedModel:
ModelGroup([
Model('AI-ModelScope/Idefics3-8B-Llama3', 'HuggingFaceM4/Idefics3-8B-Llama3'),
]),
ModelGroup([
Model('HuggingFaceTB/SmolVLM-256M-Instruct', 'HuggingFaceTB/SmolVLM-256M-Instruct'),
Model('HuggingFaceTB/SmolVLM-500M-Instruct', 'HuggingFaceTB/SmolVLM-500M-Instruct'),
Model('HuggingFaceTB/SmolVLM-Instruct', 'HuggingFaceTB/SmolVLM-Instruct'),
Model('HuggingFaceTB/SmolVLM-256M-Base', 'HuggingFaceTB/SmolVLM-256M-Base'),
Model('HuggingFaceTB/SmolVLM-500M-Base', 'HuggingFaceTB/SmolVLM-500M-Base'),
Model('HuggingFaceTB/SmolVLM-Base', 'HuggingFaceTB/SmolVLM-Base'),
Model('HuggingFaceTB/SmolVLM2-2.2B-Base', 'HuggingFaceTB/SmolVLM2-2.2B-Base'),
]),
],
Idefics3Loader,
template=TemplateType.idefics3,
Expand All @@ -42,6 +51,24 @@ def get_model(self, model_dir: str, *args, **kwargs) -> PreTrainedModel:
requires=['transformers>=4.45'],
))

register_model(
ModelMeta(
MLLMModelType.smolvlm2,
[
ModelGroup([
Model('HuggingFaceTB/SmolVLM2-256M-Video-Instruct', 'HuggingFaceTB/SmolVLM2-256M-Video-Instruct'),
Model('HuggingFaceTB/SmolVLM2-500M-Video-Instruct', 'HuggingFaceTB/SmolVLM2-500M-Video-Instruct'),
Model('HuggingFaceTB/SmolVLM2-2.2B-Instruct', 'HuggingFaceTB/SmolVLM2-2.2B-Instruct'),
]),
],
Idefics3Loader,
template=TemplateType.idefics3,
model_arch=ModelArch.idefics3,
architectures=['SmolVLMForConditionalGeneration'],
tags=['vision', 'video'],
requires=['transformers>=4.50', 'num2words'],
))


class PixtralLoader(ModelLoader):

Expand Down
92 changes: 79 additions & 13 deletions swift/template/templates/idefics3.py
Original file line number Diff line number Diff line change
@@ -1,37 +1,103 @@
# Copyright (c) ModelScope Contributors. All rights reserved.
from typing import Any, Dict
import torch
from typing import Any, Dict, List, Literal

from ..base import Template
from ..constant import MLLMTemplateType
from ..register import TemplateMeta, register_template
from ..template_inputs import StdTemplateInputs
from ..utils import align_image_inputs
from ..utils import Context
from ..vision_utils import load_video_hf


class Idefics3Template(Template):
"""Idefics3, SmolVLM and SmolVLM2. Each `<image>`/`<video>` is expanded by the HF processor, in order."""
placeholder_tokens = ['<image>']

def replace_tag(self, media_type: Literal['image', 'video', 'audio'], index: int,
inputs: StdTemplateInputs) -> List[Context]:
if self.mode in {'vllm', 'lmdeploy'} or media_type not in {'image', 'video'}:
return super().replace_tag(media_type, index, inputs)
processor = self.processor
if media_type == 'image':
media_inputs = processor(
text='<image>', images=[[inputs.images[index]]], return_tensors='pt', add_special_tokens=False)
else:
videos, video_metadata = load_video_hf([inputs.videos[index]])
video_processor = processor.video_processor
# Same frame sampling as the processor's chat template.
media_inputs = processor(
text='<video>',
videos=[videos],
video_metadata=[video_metadata],
do_sample_frames=True,
num_frames=video_processor.num_frames,
fps=video_processor.fps,
return_tensors='pt',
add_special_tokens=False)
if not hasattr(inputs, '_idefics3_media'):
inputs._idefics3_media = []
inputs._idefics3_media.append(media_inputs)
input_ids = media_inputs['input_ids'][0].tolist()
if media_type == 'video':
# The video prompt starts and ends with plain text ("You are provided ...", "\n\n"), which the
# processor tokenizes together with the surrounding text, so return it as text to be merged.
return [processor.tokenizer.decode(input_ids)]
return [input_ids]

def _swift_prepare_inputs(self, inputs: StdTemplateInputs):
super()._swift_prepare_inputs(inputs)
if self.template_backend != 'swift':
return
# The chat template writes `Role: text`, but `Role:<image>` when the turn starts with an image.
# The generation prompt ends with `Assistant:`, so the space belongs to the response.
for message in inputs.messages:
content = message['content']
if message['role'] not in {'user', 'assistant', 'tool'}:
continue
# Consecutive assistant messages are merged into a list.
first = content[0] if isinstance(content, list) and content else content
if not isinstance(first, str):
continue
if message['role'] == 'assistant' or not first.startswith('<image>'):
first = ' ' + first
if isinstance(content, list):
content[0] = first
else:
message['content'] = first

def _encode(self, inputs: StdTemplateInputs) -> Dict[str, Any]:
inputs._idefics3_media = []
encoded = super()._encode(inputs)
images = inputs.images or []
processor = self.processor
prompt = self.processor.decode(encoded['input_ids'])
if images:
image_inputs = processor(text=prompt, images=images, return_tensors='pt', add_special_tokens=False)
image_token = 128257 # <image>
encoded['input_ids'], encoded['labels'] = align_image_inputs(encoded['input_ids'], encoded['labels'],
image_inputs['input_ids'][0], image_token)
encoded['pixel_values'] = image_inputs['pixel_values']
media_inputs = inputs._idefics3_media
if media_inputs:
# Shape (1, num_patches, 3, H, W), in order of appearance.
encoded['pixel_values'] = torch.concat([x['pixel_values'] for x in media_inputs], dim=1)
encoded['pixel_attention_mask'] = torch.concat([x['pixel_attention_mask'] for x in media_inputs], dim=1)
return encoded

def _data_collator_mm_data(self, batch: List[Dict[str, Any]]) -> Dict[str, Any]:
keys = {'pixel_values', 'pixel_attention_mask'}
res = super()._data_collator_mm_data([{k: v for k, v in b.items() if k not in keys} for b in batch])
pixel_values = [b['pixel_values'] for b in batch if b.get('pixel_values') is not None]
if pixel_values:
# Samples have different numbers of patches; pad with all-zero patches, which the model drops.
max_num = max(x.shape[1] for x in pixel_values)
res['pixel_values'] = torch.concat(
[torch.nn.functional.pad(x, (0, 0, 0, 0, 0, 0, 0, max_num - x.shape[1])) for x in pixel_values])
masks = [b['pixel_attention_mask'] for b in batch if b.get('pixel_values') is not None]
res['pixel_attention_mask'] = torch.concat(
[torch.nn.functional.pad(x, (0, 0, 0, 0, 0, max_num - x.shape[1])) for x in masks])
return res


register_template(
TemplateMeta(
MLLMTemplateType.idefics3,
prefix=['<|begin_of_text|>'],
prefix=[['bos_token_id']],
prompt=['User:{{QUERY}}<end_of_utterance>\nAssistant:'],
chat_sep=['<end_of_utterance>\n'],
suffix=['<end_of_utterance>'],
system_prefix=['System:{{SYSTEM}}<end_of_utterance>\n'],
system_prefix=[['bos_token_id'], 'System: {{SYSTEM}}<end_of_utterance>\n'],
template_cls=Idefics3Template,
))
37 changes: 0 additions & 37 deletions swift/template/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,43 +97,6 @@ def findall(token_list: List[int], sub_token_list: Union[int, List[int]]) -> Lis
return res


def align_image_inputs(input_ids: List[int], labels: List[int], new_input_ids,
image_token: int) -> Tuple[List[int], List[int]]:
if isinstance(new_input_ids, torch.Tensor):
new_input_ids = new_input_ids.tolist()

# Find the tokens after the image_token in input_ids, and then align them.
i, j = 0, 0
while i < len(input_ids):
x = input_ids[i]
if x == image_token:
assert i + 1 < len(input_ids), f'input_ids[-10:]: {input_ids[-10:]}'
assert i - 1 >= 0, f'input_ids[:10]: {input_ids[:10]}'
# [1, 2, 3(i-1), image_token(i), 4(i+1) ,5, 6]
# [1, 2, 3(j_begin), a(j'), a, a, a, 4(j) ,5, 6]
j_begin = j - 1
for k in range(5): # Increase robustness.
if j_begin + k < len(new_input_ids) and new_input_ids[j_begin + k] == input_ids[i - 1]:
j_begin += k
break
if j_begin - k >= 0 and new_input_ids[j_begin - k] == input_ids[i - 1]:
j_begin -= k
break
else:
raise ValueError(f'new_input_ids: {new_input_ids}, input_ids: {input_ids}')
j_begin += 1
while j < len(new_input_ids) and new_input_ids[j] != input_ids[i + 1]:
j += 1
input_ids = input_ids[:i] + new_input_ids[j_begin:j] + input_ids[i + 1:]
if labels:
labels = labels[:i] + [-100] * (j - j_begin) + labels[i + 1:]
i += j - j_begin
else:
j += 1
i += 1
return input_ids, labels


def _split_str_by_regex(text: str, regex_delimiters: List[str]) -> List[str]:
combined_pattern = '|'.join(f'({pattern})' for pattern in regex_delimiters)
parts = re.split(combined_pattern, text, flags=re.DOTALL)
Expand Down
Loading