Deepseek-v4-pro cannot see images in SaaS

[juanmichelini]

Please see screenshot:

notice that gpt-5.5 behaves fine with same test

[juanmichelini]

This comment was created by an AI agent (OpenHands) at the user's request.

I inspected the SWE-bench Multimodal run artifact:

https://results.eval.all-hands.dev/swebenchmultimodal/litellm_proxy-deepseek-deepseek-v4-pro/25967365988/results.tar.gz

Findings:

• The run config used model: litellm_proxy/deepseek/deepseek-v4-pro with disable_vision: null and force_string_serializer: null, so vision was not explicitly disabled in the OpenHands config.
• The saved conversation events do include image inputs. The initial user message is structured as a text block plus an image block, e.g. type: "image" with image_urls: [...].
• Across the readable conversation archives I inspected:
  • 63 conversation archives were readable.
  • 60/63 had image inputs.
  • There were 93 total attached image URLs.
  • Distribution: 38 tasks had 1 image, 15 had 2, 5 had 3, 1 had 4, and 1 had 6.
• Example: markedjs__marked-1739 had a text block plus an image block containing the three GitHub user-image PNG URLs from the issue.

However, I did not find convincing behavioral evidence that the LLM actually perceived/understood the image pixels:

• I did not find a clear case where the model described visual details that were unavailable from the surrounding issue text.
• Some phrases like "the screenshot shows..." were explainable from the issue description itself.
• In at least one clear case (markedjs__marked-1739), despite image URLs being attached, the model said: "The user shows screenshots (which I can't see)...".
• Several later "can't view image/screenshot" statements also appeared, though some of those referred to browser-generated verification screenshots rather than the original issue images.

Important caveat: log_completions was false, so the artifact does not contain the raw LiteLLM/provider request payload. From this archive alone, I can confirm that OpenHands conversation objects contained image blocks, but I cannot prove that LiteLLM/proxy/provider received them in the expected multimodal format.

Practical conclusion: the image-input plumbing appears present on the OpenHands conversation side, but this run does not demonstrate that deepseek-v4-pro is actually receiving and using the images correctly. A good next check would be to enable raw request/completion logging or run a canary image task where the answer requires information only visible in the image and absent from text.

[all-hands-bot]

Issue #3768 reports DeepSeek v4-pro cannot see images in SaaS. This is a new specific bug (vision/image support) that naturally belongs under the existing DeepSeek v4-pro umbrella tracker #3267, which already tracks 4 other compatibility issues but does not cover vision. The root cause mechanism (_supports_vision detection) is shared with #3767 (Kimi-k2.6) and #3495 (step-3.7-flash), but the model-specific context makes this distinct. Filed by a maintainer who manages #3267, so likely intentionally separate, but linking would help consolidate tracking.

Found 3 possible duplicate issues:

1. #3267 — [Bug]: DeepSeek v4-pro/v4-flash compatibility issues (model list, provider routing, reasoning_content)
2. #3767 — Kimi-k2.6 is not being recognized as a vision model in the SDK
3. #3495 — step-3.7-flash: vision support not detected at runtime (LiteLLM proxy /v1/model/info lookup returns None)

These may not be exact duplicates, but the scope appears to overlap enough that keeping discussion in one place may be more useful.

This comment was created by an AI assistant (OpenHands) on behalf of the repository maintainer.