optillm documentation update

[DIGist]

So this isn't really a problem in how Optillm is packaged for Harbor, but might be useful to mention in the optillm notes until it's fixed upstream.

I decided to try optillm and ran into problems. It's working great now, but the environment variables don't exactly align the way it's referenced in either documentation (harbor or optillm upstream).

I had to set these environment variables to get them to work with my harbor.llamacpp:

OPTILLM_APPROACH="auto"
OPTILLM_BASE_URL="http://harborhostname:34401/v1"
OPTILLM_OPENAI_API_KEY="xxx"
OPTILLM_LOG="debug"
OPENAI_API_KEY="xxx"

The noteable changes being the documentation suggests "OPTILLM_BASE-URL" when it only works with "OPTILLM_BASE_URL" and I had to set "OPEN_API_KEY" as "OPTILLM_OPENAPI_API_KEY" didn't seem to work.

[av]

Hi, I just tested OptiLLM on a fresh install with llama.cpp with following steps:

harbor build optillm
harbor up optillm llamacpp

Then, to query:

POST {{host}}/v1/chat/completions
Content-Type: application/json

{
  "model": "llama3.1:8b-instruct-q8_0",
  "messages": [
    {
      "role": "user",
      "content": "Which is heavier, 1 kilogram of feathers or 1 pound of steel?"
    }
  ],
  "optillm_approach": "z3"
}

Where model is the same Model ID as served by the llamacpp instance, and It appears to work as expected. Where is the BASE-URL coming from? That definitely looks like invalid env variable name, but I can't pinpoint it in either Harbor or OptiLLM repos. The way Harbor integrates OptiLLM with inference backends is via .x. cross-service files, for example, for llamacpp it looks like this:

services:
  optillm:
    environment:
      - OPTILLM_BASE_URL=http://llamacpp:8080/v1
      - OPENAI_API_KEY=sk-llamacpp

So, when the two (optillm and llamacpp) are running together (harbor up optillm llamacpp) - these env vars will be added to the optillm to point to llamacpp instance.

• It uses internal address on a Harbor network: llamacpp:8080
• llamacpp is expected to be up and running before first query from optillm

Additionally, I think that OptiLLM only supports streaming formally, by sending back content in one large chunk, so integration with various Chat UIs can be problematic.

Please let me know if any of this makes sense, thanks!