v0.4.1 — VLM Engine (MLXSwiftEngine branch via MLXVLM)

.github/workflows/ci.yml

@@ -40,13 +40,17 @@ env:
 jobs:
   spm:
     name: SPM Build & Test
-    runs-on: macos-15
+    runs-on: macos-26
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v5
 
-      - name: Pin Xcode 16.4
-        run: sudo xcode-select -s /Applications/Xcode_16.4.app
+      # Xcode 26.4.1 ships Swift 6.3, which (unlike 6.0 in Xcode 16.4)
+      # tolerates mlx-swift-lm's `private let context = CIContext()` in
+      # MLXVLM/MediaProcessing.swift under strict concurrency. Pinning to
+      # the GA build (not the 26.5 beta) so toolchain churn stays low.
+      - name: Pin Xcode 26.4.1
+        run: sudo xcode-select -s /Applications/Xcode_26.4.1.app
 
       - name: Cache SPM checkouts + build
         uses: actions/cache@v5
@@ -81,13 +85,17 @@ jobs:
 
   app:
     name: Xcode App Build
-    runs-on: macos-15
+    runs-on: macos-26
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v5
 
-      - name: Pin Xcode 16.4
-        run: sudo xcode-select -s /Applications/Xcode_16.4.app
+      # Xcode 26.4.1 ships Swift 6.3, which (unlike 6.0 in Xcode 16.4)
+      # tolerates mlx-swift-lm's `private let context = CIContext()` in
+      # MLXVLM/MediaProcessing.swift under strict concurrency. Pinning to
+      # the GA build (not the 26.5 beta) so toolchain churn stays low.
+      - name: Pin Xcode 26.4.1
+        run: sudo xcode-select -s /Applications/Xcode_26.4.1.app
 
       - name: Cache Xcode DerivedData
         uses: actions/cache@v5

.github/workflows/release.yml

@@ -16,7 +16,7 @@ on:
 jobs:
   build:
     name: Build, Sign & Release
-    runs-on: macos-15
+    runs-on: macos-26
     timeout-minutes: 60
 
     permissions:
@@ -29,8 +29,11 @@ jobs:
           fetch-depth: 0
           token: ${{ secrets.GITHUB_TOKEN }}
 
-      - name: Pin Xcode 16.4
-        run: sudo xcode-select -s /Applications/Xcode_16.4.app
+      # Match ci.yml — Xcode 26.4.1 GA (Swift 6.3) tolerates the
+      # CIContext Sendable annotation in mlx-swift-lm's MLXVLM
+      # (Xcode 16.4 / Swift 6.0 fails with strict-concurrency error).
+      - name: Pin Xcode 26.4.1
+        run: sudo xcode-select -s /Applications/Xcode_26.4.1.app
 
       - name: Show toolchain
         run: |

CHANGELOG.md

@@ -59,6 +59,31 @@ Versioning follows [Semantic Versioning](https://semver.org/).
   `macmlx serve` and `macmlx run`). Drop into Claude Desktop's
   `claude_desktop_config.json` as
   `{ "mcpServers": { "macmlx": { "command": "macmlx", "args": ["mcp", "serve"] } } }`.
+- **VLM Engine** (v0.4.1, part 2 of 3). MLXSwiftEngine now branches
+  on `model.format` to load text-only models through
+  `MLXLLM.LLMModelFactory` and vision-language models through
+  `MLXVLM.VLMModelFactory`. Runtime modality stored in a new
+  `LoadedSupport` enum (`.none / .llm / .vlm`).
+  - `runGeneration(_:)` splits into `runLLMGeneration` (existing
+    prompt-cache flow — hot/cold KV tier, suffix prefill, save
+    extended cache after stream) and `runVLMGeneration` (fresh KV
+    cache per call; bypasses the prompt cache for now since
+    multimodal cache keys would need to fold image bytes into the
+    chained hash).
+  - `Chat.Message` mapping respects modality: VLM models receive
+    `ChatMessage.images` as `UserInput.Image.url(URL)` so the VLM's
+    `UserInputProcessor` can inject image tokens; LLM models drop
+    accidental attachments with a debug-level Pulse warning.
+  - `MLXVLM` added to `MacMLXCore` package dependencies (sibling
+    product of `MLXLLM` already in our `mlx-swift-lm` 3.31.x pin —
+    no new SPM dependency tree).
+  - Three new unit tests cover unsupported-format rejection
+    (gguf, unknown, missing-VLM-directory). 111/111 Core tests
+    green. Real VLM smoke (loading e.g. SmolVLM-Instruct-4bit) is
+    a manual-QA item — multi-GB download.
+  - Image picker, multimodal HTTP, and conversation persistence
+    land in the v0.4.1 part-3 PR. Plan:
+    `docs/superpowers/plans/2026-05-10-v0.4.1-vlm.md`.
 - **VLM Foundation** (v0.4.1, part 1 of 3). Pure-Swift Core changes
   for vision-language model support. No MLX integration yet, no UI,
   no HTTP changes.

MacMLXCore/Package.swift

@@ -20,6 +20,7 @@ let package = Package(
             name: "MacMLXCore",
             dependencies: [
                 .product(name: "MLXLLM", package: "mlx-swift-lm"),
+                .product(name: "MLXVLM", package: "mlx-swift-lm"),
                 .product(name: "MLXLMCommon", package: "mlx-swift-lm"),
                 .product(name: "Transformers", package: "swift-transformers"),
                 .product(name: "Pulse", package: "Pulse"),