Model-loading: pre-painted dialog, throttled cold loads, keep_alive setting, startup warm

[mrjeeves]

Makes loading a local model far less disruptive — especially on laptops where a cold load could freeze the whole machine.

1. Cold-start "loading the model" dialog

A non-blocking dialog over the chat surface while the model loads, with a Cancel button. Adapts copy for local vs. mesh ("Loading <model>…" / "Waiting for <host>…").

• Pre-painted before the freeze: before sending, we check Ollama's /api/ps to see if the model is resident. On a predicted cold start we show the dialog and force an actual paint (tick + double rAF) before firing the request — so it's on screen ahead of any load freeze. Warm sends keep a lightweight 5s reactive fallback timer.
• Tears down on the first agent frame; not re-armed for warm later turns.

2. Throttle the load so the machine stays usable

The freeze happens inside the Ollama server while it pages multi-GB weights in from disk. Since the app spawns that server itself, we now throttle it:

• Lower IO priority of the spawned server — Linux ionice -c3 (idle) + a small renice; macOS taskpolicy -b; Windows BelowNormal priority class. IO is the real lever (loading is disk-bound), so this keeps the desktop responsive with negligible impact on inference (compute-bound). Best-effort; only applies when we own the process (not a system/tray Ollama).
• Cap memory pressure on spawn via OLLAMA_MAX_LOADED_MODELS=1 and OLLAMA_NUM_PARALLEL=1, cutting the swap thrash behind the hardest freezes.

3. Warm the model proactively at startup

After launch we warm the active chat model in the background (throttled), so the one-time cold load happens at a predictable moment — with the dialog shown — instead of on the first message. Skipped when the model isn't downloaded or keep_alive is 0.

4. Configurable keep_alive (the repeated-cold-start fix)

Chat requests never set Ollama's keep_alive, so they relied on Ollama's 5-minute default; idle past that and the model unloaded, making the next message a cold start.

• Chat requests (streaming + one-shot) now send keep_alive, read in Rust from config.
• Configurable in Settings → Hardware → Performance → Model memory, default 30m, from "Unload immediately" (frees RAM/VRAM for transcription on tight machines) to "Until the app quits".

5. Live resource readout in the dialog

Shows live CPU / RAM / GPU + disk free so the user can see why a load is slow (e.g. RAM near full → paging from disk).

• Reuses the existing usage_live_snapshot command; the LiveSnapshot type was promoted to types.ts and is now shared with the Usage settings tab.
• Polled only while the dialog is visible (1.2s), with interval cleanup on teardown.

New backend surface

• ollama_model_loaded(model) → bool (via /api/ps)
• ollama_warm(model) (ensures running, then warms)
• process::lower_priority(pid) — cross-platform best-effort throttle
• ollama_keep_alive config field (default 30m)

Verification

• pnpm run check — 0 errors, 0 warnings
• pnpm run build — succeeds
• cargo check + cargo clippy — my code clean (only pre-existing mesh/ unused-import warnings)
• cargo test ... ollama — 4 passed

Notes / possible follow-ups

• Throttling/mem-caps only apply when MyOwnLLM spawns Ollama. If Ollama runs as a system/tray service (common on Windows/macOS), we don't own the PID — could be extended to discover and throttle it.
• The dialog's resource readout is whole-machine, not Ollama-process-specific; an /api/ps-based "model X — 100% GPU, 3.2 GB VRAM" line is a possible follow-up.

https://claude.ai/code/session_01Eze77o5msnfo5CBnJjd3Sd