Kimi-k2.6 is not being recognized as a vision model in the SDK

[juanmichelini]

See screenshot:

Notice that images were uploaded correctly during benchmark of multimodal https://results.eval.all-hands.dev/swebenchmultimodal/litellm_proxy-moonshot-kimi-k2-6/25924060044/results.tar.gz

[juanmichelini]

Yes, Kimi K2.6 IS receiving and understanding images correctly in this benchmark.

I analyzed the results archive and found clear evidence that images are being sent as vision inputs and the model is interpreting them:

1. Images are inlined as base64 before sending

The eval metadata shows "inline_image_urls": true, and the eval was explicitly triggered on commit 04c9fe35 (feat(llm): inline http(s) image URLs as base64 for models that require it). The benchmark log confirms: "Sending instruction with 2 valid images" / "Sending instruction with 3 valid images" per instance.

2. Token counts are consistent with image payloads

Initial prompt token counts are very large (e.g., 17,562 tokens for Automattic__wp-calypso-26286 with 2 images; 20,504 tokens for chartjs__Chart.js-9027 with 3 images). These numbers are far too high for text-only prompts and align with base64-encoded image data.

3. The LLM describes specific visual content from the images

The agent output shows the model referencing exact details visible in screenshots — not just the surrounding text:

Image 1: Shows the Activity Log page with a banner saying "Stats at your fingertips" — this shouldn't appear on Activity Log
Image 2: Shows Redux state with section name as "stats" and path as "/stats"

Image 1 shows the first call to getNumberFormat with options = {notation: undefined, minimumFractionDigits: 0, maximumFractionDigits: 0}
Image 2 shows the second call with options = {minimumFractionDigits: NaN, maximumFractionDigits: NaN}
Image 3 shows intermediate values during chart construction where availableHeight = -31, height = 0, and chartArea = {h: NaN, w: NaN}

These are precise visual readings of screenshot content that the model could not infer from the text description alone.

4. No image/vision API errors

There are zero errors in the logs related to invalid image format, unsupported vision features, or failed image loading. The only errors are unrelated Python runtime errors (generator didn't stop after throw()).

---

This comment was created by an AI agent (OpenHands) on behalf of the user.

[all-hands-bot]

Both #3767 (kimi-k2.6) and #3495 (step-3.7-flash) report the SDK failing to detect vision support at runtime. They share the same code path (_supports_vision / LiteLLM proxy model_info lookup) and would both benefit from the same general fix (e.g., explicit supports_vision LLM config or fixing the proxy lookup). However, the specific root causes differ: for kimi-k2.6, litellm.supports_vision returns True with the 'moonshot/' prefix but False without it, suggesting a model name transformation issue; for step-3.7-flash, LiteLLM has no static metadata at all. They are overlapping-scope rather than exact duplicates.

Found 3 possible duplicate issues:

1. #3495 — step-3.7-flash: vision support not detected at runtime (LiteLLM proxy /v1/model/info lookup returns None)
2. #3428 — LiteLLM proxy model_info lookup misses when proxy uses short aliases (claude-opus-4-8 vision still off)
3. #3002 — Kimi-k2-6 shows 87% LLMBadRequestError on swebench multimodal but no swebench

These may not be exact duplicates, but the scope appears to overlap enough that keeping discussion in one place may be more useful.

This comment was created by an AI assistant (OpenHands) on behalf of the repository maintainer.