为什么VLLM推理的时间会比transformer慢那么多、输出格式也不一样

[2535853007]

transformer：
model = Qwen2VLForConditionalGeneration.from_pretrained(
"/root/autodl-tmp/OS-Copilot/OS-Atlas-Base-7B", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("/root/autodl-tmp/OS-Copilot/OS-Atlas-Base-7B")

messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "/root/autodl-tmp/web_6f93090a-81f6-489e-bb35-1a2838b18c01.png",
"min_pixels": 56 * 56,
"max_pixels": 1280 * 28 * 28,
},
{"type": "text", "text": "In this UI screenshot, what is the position of the element corresponding to the command "switch language of current page" (with bbox)?"},
],
}
]

Preparation for inference

text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")

Inference: Generation of the output

generated_ids = model.generate(**inputs, max_new_tokens=128)

generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]

output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=False, clean_up_tokenization_spaces=False
)
print(output_text)

结果1：运行时间 12.7s
['<|object_ref_start|>language switch<|object_ref_end|><|box_start|>(573,12),(591,39)<|box_end|><|im_end|>']

vllm：

MODEL_PATH = "/root/autodl-tmp/OS-Copilot/OS-Atlas-Base-7B"

llm = LLM(
model=MODEL_PATH,
trust_remote_code=True,
limit_mm_per_prompt={"image": 1, "video": 1},
)

sampling_params = SamplingParams(
temperature=0.1,
top_p=0.001,
repetition_penalty=1.05,
max_tokens=256,
stop_token_ids=[],
)

messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "/root/autodl-tmp/web_6f93090a-81f6-489e-bb35-1a2838b18c01.png",
"min_pixels": 56 * 56,
"max_pixels": 1280 * 28 * 28,
},
{"type": "text", "text": "In this UI screenshot, what is the position of the element corresponding to the command "switch language of current page" (with bbox)?"},
],
}
]

processor = AutoProcessor.from_pretrained(MODEL_PATH)
prompt = processor.apply_chat_template(
messages,
tokenize=False,
use_fast=False,
add_generation_prompt=True,
)
image_inputs, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)

mm_data = {}
if image_inputs is not None:
mm_data["image"] = image_inputs
if video_inputs is not None:
mm_data["video"] = video_inputs

llm_inputs = {
"prompt": prompt,
"multi_modal_data": mm_data,

# FPS will be returned in video_kwargs
"mm_processor_kwargs": video_kwargs,

}

outputs = llm.generate([llm_inputs], sampling_params=sampling_params)
generated_text = outputs[0].outputs[0].text

print(generated_text)
结果2：运行时间 1m21s
language switch(573,12),(590,42)

VLLM：0.8.4
cuda：12.1
transformers：4.51.3
gpu：4080