[Question]: Multimodal not supported?

[mpetruc]

Question

When trying to upload an image i'm getting a pop-up saying: Multimodal not supported form the server of model. The model i'm using (ibm-granite-docling-258m) IS multimodal, and llama.ui was built from source.

What am ai doing wrong? Do i need to enable vision anywhere? Thanks.

Steps Taken

npm ci; npm run build; npm start
${llama_server} -m ${models}/granite-docling-258M-bf16.gguf --mmproj ${models}/mmproj-ibm-granite-docling-258M-f16.gguf

Environment

wsl in windows 11; Application Version: 2.36.0

Additional Context

No response

[olegshulyakov]

@mpetruc What is ${llama_server}? Is it llama.cpp?
Why do you do npm ci; npm run build; npm start?

[mpetruc]

it's a preset for llama-server (they call it "macro") in llama-swap: It's basically a llama-server with a bunch of routine parameters, so I don't have to type them for every single model.

Of course i don't enter those commands like that in bash! I just wanted to make it clear that i'm following the instructions to build llama.ui:

npm ci
npm run build

then start the server with npm start

[olegshulyakov]

@mpetruc Have you tried to launch llama.cpp directly? llama-swap does not provide /props endpoint where llama.cpp returns modalities. See https://github.com/olegshulyakov/llama.ui/blob/main/src/api/providers/LlamaCppProvider.ts#L116
I've submitted a ticket some time ago about it mostlygeek/llama-swap#267.

Please check with direct llama.cpp launch.

[mpetruc]

ok, it seems that vision works only with Inference Provider = llama.cpp, but not with Openai Compatible. Doesn't matter if i hit the llama-swap endpoint (port 9292), litellm (port 4000) or directly llama-server (port 5050), if the provider is llama.cpp it works, if it's openai compat it doesn't.

[mpetruc]

In Jan and Cherry vision works even when configuring the model as openai compatible. However they allow the end-user to toggle vision and tools capabilities: