diff --git a/docs/dev/cookbook.md b/docs/dev/cookbook.md index 4bcc9cb..d417065 100644 --- a/docs/dev/cookbook.md +++ b/docs/dev/cookbook.md @@ -16,8 +16,10 @@ user can do the same via the Cookbook modal. Storage layout follows `memories` deliberately — recipes are user-owned notes that the LLM can also author, same row-level security posture, same "freeform text column as source of truth" -philosophy. No embeddings pipeline: ILIKE on `title` is fast enough -at cookbook scale (tens to low hundreds of rows per user). +philosophy. Embeddings pipeline added later (see "Embeddings" +below) for the drawer-side recipe search; the LLM tool path +(`recipe_list`, `recipe_search`) still uses ILIKE-on-title and is +unaffected. ## Files @@ -207,6 +209,36 @@ sync` is a no-op. user opted in to keeping every revision so the History panel reads as a complete diary. +## Embeddings + +The drawer's recipe search runs through the same embed-then-merge +pipeline as the wiki and journal sidebars (see `./embeddings.md`). +Added after the original `recipes` design, which omitted embeddings +on the rationale that ILIKE-on-title is enough for a small +single-user cookbook. That holds for the LLM tool path - the model +knows exactly what title it wrote - but the human drawer is a +different problem: a fuzzy query like "fluffy potato side" should +find "Mashed Potatoes" by meaning. + +Columns: `embedding vector(2048)`, `embedding_model text`, +`embedding_claim_holder text`, `embedding_claim_expires timestamptz` +on `public.recipes`. A `clear_recipe_embedding_on_change` trigger +nulls the embedding (and the claim) whenever `title`, `cooklang`, +or `source` change; `recipe_update_with_version` updates these +columns inside its own RPC so the trigger fires automatically. + +RPCs: `claim_next_pending_recipe`, `save_recipe_embedding_if_claimed`, +`search_recipes_by_embedding`. Same shape as the wiki RPCs and the +same `for update skip locked` claim discipline. The worker adapter +is `src/lib/embeddings/sources/recipes.ts`, registered in +`src/lib/embeddings/worker.ts` alongside the other five sources. + +Search wrapper: `SupabaseService.searchRecipes({query, queryEmbedding, +limit})` merges semantic hits (RPC, cosine order) and ILIKE hits +(title only) deduped by id and capped at `limit`. The sidebar +(`src/components/RecipeList.svelte`) calls it on debounced +keystrokes; the LLM tool path keeps using `listRecipes`. + ## Interactions - **Tools** (`./tools.md`) — five recipe tools registered in @@ -248,10 +280,14 @@ as a complete diary. add textarea treats each line as an item — a "saucepan" row would sit in the shopping list permanently. Instructions and ingredients are the useful parts for transfer. -- **No embeddings.** ILIKE on `title` is fine at cookbook scale. If - a user ever needs semantic search, the escape hatch mirrors - memories exactly: add `embedding vector(2048)` + claim columns + - the worker-claim RPC pattern. Don't pre-emptively wire it. +- **Drawer search is semantic; LLM tool path is not.** The + embedding pipeline (added after the original "no embeddings" + design - see "Embeddings" below) feeds `searchRecipes` only. + The `recipe_list` and `recipe_search` tools the model uses + still go through ILIKE-on-title. Two reasons: the model knows + exactly what title it just wrote so meaning matching adds no + value on the tool path, and keeping the tool's behaviour + deterministic avoids fuzz on a path the agent reasons about. - **Cooklang source is the source of truth.** The parsed ingredients list you see in the render is derived, not stored. This means a future parser bug is a pure read-path issue — diff --git a/docs/dev/embeddings.md b/docs/dev/embeddings.md index ce3cf36..cf2ac9b 100644 --- a/docs/dev/embeddings.md +++ b/docs/dev/embeddings.md @@ -42,9 +42,11 @@ under-ranked until the worker catches up. - `src/lib/embeddings/types.ts` — `EmbeddingSource` interface and shared constants. - `src/lib/embeddings/sources/memories.ts`, - `sources/threads.ts` — per-table adapters. Each knows how to - claim one pending row, build the input string for Venice, - and save the result under a guard. + `sources/threads.ts`, `sources/journal.ts`, + `sources/wiki.ts`, `sources/samskara-substrate.ts`, + `sources/recipes.ts` — per-table adapters. Each knows how + to claim one pending row, build the input string for + Venice, and save the result under a guard. - `supabase/schema.sql` — `worker_leases` table, lease RPCs, `claim_next_pending_memory` / `claim_next_pending_thread_embedding`, @@ -165,6 +167,13 @@ extension. `clear_journal_embedding_on_change` trigger fires when `content | topics | mood` change so an edit reselects the row. See `./journal.md`. +- **Cookbook** — `recipes` joined the registered-source list + so the drawer's recipe search can rank by meaning. The + `clear_recipe_embedding_on_change` trigger fires when + `title | cooklang | source` change. The LLM-facing + `recipe_list` tool still runs ILIKE-on-title only - the + embedding pipeline is for the human drawer search. See + `./cookbook.md`. - **Auth-session** — worker startup is gated on an active Supabase session; `manager.start()` pulls the session from the Supabase client and passes tokens to the Worker. A diff --git a/docs/user/README.md b/docs/user/README.md index 338a8d2..faaf6bd 100644 --- a/docs/user/README.md +++ b/docs/user/README.md @@ -51,8 +51,8 @@ You can reach these pages two ways: alongside your conversations, plus space for your own entries. - [Cookbook](./cookbook.md) — a personal store for Cooklang recipes, with a dedicated modal and a tab in the conversation drawer. -- [Search](./search.md) — finding a thread or a message across your - history. +- [Search](./search.md) — finding a thread, recipe, journal day, or + wiki article by meaning across your history. - [Keyboard shortcuts](./shortcuts.md) — the keys that save you the most time. diff --git a/docs/user/search.md b/docs/user/search.md index e06053a..7c87c61 100644 --- a/docs/user/search.md +++ b/docs/user/search.md @@ -1,19 +1,78 @@ # Search -Finding a thread or a specific message across your history. The search -field in the drawer matches thread titles and message bodies. +Every drawer tab in Nak - Chats, Recipes, Journal, Wiki - has a +search box at the top. They all work the same way: type a query, +wait briefly while Nak embeds your phrase, and the matches come +back in order of closeness rather than alphabetical or +chronological order. -## Searching from the drawer +## How matching works -## What matches and what doesn't +Each search runs in two passes and merges the results: -## Jumping to a matched message +1. **Meaning match.** Your query is sent to the embedding model + and compared to a stored vector for every item in that tab. + The closer the meaning, the higher the rank. This is how + "fluffy potato side" finds *Mashed Potatoes* and "anxious about + work last week" finds the right journal day. +2. **Word match.** A simple case-insensitive substring search + over the obvious fields (thread title, recipe title, article + body, journal content / mood). These appear after the meaning + matches, so a freshly added item that hasn't been embedded yet + is still findable - just not yet rankable by meaning. + +The two passes are deduplicated by id and capped per tab. + +## Loading feedback + +When you type, Nak debounces briefly (so it isn't embedding every +keystroke), then shows a small scanner animation in place of the +list while the embedding round-trip and the database query run. +The actual results appear once they arrive, replacing the scanner. +Searches typically complete in under a second. + +If your network is offline, or the embedding model is briefly +unreachable, Nak falls back to the substring-only pass quietly - +your search will still return matches, just ranked by recency +rather than meaning. + +## Ordering + +| Tab | Empty query | Active query | +| --- | --- | --- | +| **Chats** | Recent / Older / Archived buckets | Exact title hits first, then by similarity | +| **Recipes** | Most-recent or by rating (your choice) | By similarity to your query | +| **Journal** | Days newest-first | Days ranked by their best-matching entry | +| **Wiki** | Alphabetical by title | By similarity to your query | + +The Wiki list switches from alphabetical to relevance order only +while there's text in the search box. Clearing the box returns it +to the alphabetical browse view. + +The Recipe list hides its sort picker while you're searching - the +similarity ranking is the sort during a search. + +The Journal list rolls multiple matching entries from the same day +into one row; days are ordered by their single best match, so a +day with one strong hit can rank above a day with three weak ones. + +## Why a search might miss something + +- The item was just added and the background embedding worker + hasn't reached it yet. Substring-on-the-obvious-field still + catches it; meaning matches will catch up within minutes. +- The phrase you typed is too short or too generic for the model + to anchor on. Try adding a noun or a context word. +- You typed a long query and the substring pass found nothing + literal; meaning matches still ranked but none were close. ## Where to go next -- [Threads](./threads.md) — the drawer that hosts the search field. -- [Keyboard shortcuts](./shortcuts.md) — the shortcut for focusing - search. +- [Chats](./threads.md) - the conversation drawer. +- [Recipes](./cookbook.md) - the cookbook drawer. +- [Journal](./journal.md) - the journal drawer. +- [Wiki](./wiki.md) - the wiki drawer. +- [Keyboard shortcuts](./shortcuts.md) - focusing the search box. --- Back to the [index](./README.md). diff --git a/src/components/JournalList.svelte b/src/components/JournalList.svelte index 22139ea..fb752ca 100644 --- a/src/components/JournalList.svelte +++ b/src/components/JournalList.svelte @@ -5,11 +5,28 @@ * all open the same day-view, so collapsing to a date index is the * right granularity for a browse surface. * + * Empty query: aggregates the eagerly-loaded `journal.entries` by + * `entry_date`, newest day first. Substring fallback is intentionally + * NOT used - the embed-then-search pipeline below replaces it so the + * sidebar matches the assistant's `journal_search` tool (and the + * wiki sidebar) shape. + * + * Active query: embeds the query via Venice and calls + * `supabase.searchJournalEntries` (semantic-first merge with ILIKE + * fallback inside Supabase). The returned rows are aggregated by + * date, ordered by the best per-date similarity score so the closest + * day floats to the top. Loading state replaces the listing with a + * Scanner. + * * Clicking a date calls navigate({ journal_date: 'YYYY-MM-DD' }), which * switches the main panel to JournalPanel showing that day's entries. */ + import { app } from '$lib/state.svelte'; import { navigate, route } from '$lib/routing.svelte'; import { journal } from '$lib/journal-store.svelte'; + import { VENICE_EMBEDDING_MODEL, padEmbeddingForStorage } from '$lib/models'; + import type { JournalEntry } from '$lib/supabase'; + import Scanner from './Scanner.svelte'; // Parent (Chat shell) passes a callback that dismisses the mobile // drawer once the main panel has navigated to the chosen day. @@ -21,36 +38,137 @@ const { onSelect }: Props = $props(); let query = $state(''); + let searchResults = $state([]); + let searchBusy = $state(false); + let searchError = $state(null); + let searchAbort: AbortController | null = null; + + const SEARCH_DEBOUNCE_MS = 200; + // Match the journal_search tool's max so the sidebar surfaces every + // day the assistant could reach. Aggregated by date downstream so + // the actual row count in the drawer is the unique-date count. + const JOURNAL_SEARCH_LIMIT = 50; + + $effect(() => { + const q = query.trim(); + if (q.length === 0) { + searchResults = []; + searchBusy = false; + searchError = null; + if (searchAbort) searchAbort.abort(); + searchAbort = null; + return; + } + if (!app.supabase) return; + const timer = setTimeout(() => { + void runJournalSearch(q); + }, SEARCH_DEBOUNCE_MS); + return () => clearTimeout(timer); + }); - // Filters entries by content / topics / mood substring match, then - // aggregates to one row per entry_date. journal.entries is sorted - // newest-day-first; preserve that order via insertion-ordered Map. - const visibleJournal = $derived.by< - { entry_date: string; count: number; matchId: string }[] - >(() => { - const q = query.trim().toLowerCase(); - const matches = q.length === 0 - ? journal.entries - : journal.entries.filter((e) => { - if (e.content.toLowerCase().includes(q)) return true; - if (e.mood && e.mood.toLowerCase().includes(q)) return true; - for (const t of e.topics) { - if (t.toLowerCase().includes(q)) return true; + async function runJournalSearch(q: string): Promise { + if (!app.supabase) return; + // Supersede any in-flight search so a slow embed call from a + // stale query can't clobber a newer one. + if (searchAbort) searchAbort.abort(); + const ctl = new AbortController(); + searchAbort = ctl; + searchBusy = true; + searchError = null; + try { + let queryEmbedding: number[] | null = null; + if (app.venice) { + try { + const resp = await app.venice.embed({ + model: VENICE_EMBEDDING_MODEL, + input: q, + signal: ctl.signal, + }); + const raw = resp.data[0]?.embedding; + if (raw && raw.length > 0) { + queryEmbedding = padEmbeddingForStorage(raw); } - return false; - }); - const byDay = new Map(); - for (const e of matches) { - const cur = byDay.get(e.entry_date); - if (cur) cur.count += 1; - else byDay.set(e.entry_date, { count: 1, matchId: e.id }); + } catch { + // Best-effort: ILIKE-only is still useful. The supabase + // method treats a null embedding as "skip the vector RPC". + } + } + if (ctl.signal.aborted) return; + const hits = await app.supabase.searchJournalEntries({ + query: q, + queryEmbedding, + limit: JOURNAL_SEARCH_LIMIT, + }); + if (ctl.signal.aborted) return; + searchResults = hits; + } catch (err) { + if (!ctl.signal.aborted) { + searchError = err instanceof Error ? err.message : String(err); + } + } finally { + if (searchAbort === ctl) { + searchAbort = null; + searchBusy = false; + } + } + } + + interface DayRow { + entry_date: string; + count: number; + } + + // Empty query: aggregate the eagerly-loaded list by date, newest + // first. journal.entries is already sorted newest-day-first; the + // insertion-ordered Map preserves that. + const browseDays = $derived.by(() => { + const byDay = new Map(); + for (const e of journal.entries) { + byDay.set(e.entry_date, (byDay.get(e.entry_date) ?? 0) + 1); + } + return Array.from(byDay, ([entry_date, count]) => ({ entry_date, count })); + }); + + // Active query: aggregate the server hits by date, ranking days by + // their best per-date similarity. ILIKE-only rows lack `similarity`; + // they fall to the bottom (similarity -Infinity proxy). Within a day + // we take the max so a day with one strong match outranks a day + // with several weak ones. + const searchDays = $derived.by(() => { + interface Bucket { + count: number; + best: number; + firstIdx: number; } - return Array.from(byDay, ([entry_date, v]) => ({ + const byDay = new Map(); + searchResults.forEach((e, idx) => { + const sim = typeof e.similarity === 'number' ? e.similarity : -Infinity; + const cur = byDay.get(e.entry_date); + if (cur) { + cur.count += 1; + if (sim > cur.best) cur.best = sim; + } else { + byDay.set(e.entry_date, { count: 1, best: sim, firstIdx: idx }); + } + }); + return Array.from(byDay, ([entry_date, b]) => ({ entry_date, - count: v.count, - matchId: v.matchId, - })); + count: b.count, + best: b.best, + firstIdx: b.firstIdx, + })) + .sort((a, b) => { + // Higher similarity first. Tie-break on the position the + // first matching entry came back in (preserves the supabase + // method's exact-vs-semantic merge order for unscored rows). + if (a.best !== b.best) return b.best - a.best; + return a.firstIdx - b.firstIdx; + }) + .map(({ entry_date, count }) => ({ entry_date, count })); }); + + const isSearching = $derived(query.trim().length > 0); + const visibleDays = $derived(isSearching ? searchDays : browseDays);
@@ -61,24 +179,41 @@ placeholder="Search journal" aria-label="Search journal" bind:value={query} + autocomplete="off" + spellcheck="false" />
- {#if journal.loading && !journal.loaded} -

Loading journal…

- {:else if journal.error} + {#if isSearching && searchBusy} + +
+ +
+ {:else if isSearching && searchError} +

+ Search failed: {searchError} +

+ {:else if !isSearching && journal.loading && !journal.loaded} +
+ +
+ {:else if !isSearching && journal.error}

Couldn't load journal: {journal.error}

- {:else if visibleJournal.length === 0} + {:else if visibleDays.length === 0}

- {#if journal.entries.length === 0} + {#if isSearching} + No matches. + {:else if journal.entries.length === 0} No journal entries yet. Use the panel to add one. {:else} No matches. {/if}

{:else} - {#each visibleJournal as day (day.entry_date)} + {#each visibleDays as day (day.entry_date)}
- {#if cookbook.loading && cookbook.recipes.length === 0} -

Loading recipes…

+ {#if isSearching && searchBusy} + +
+ +
+ {:else if isSearching && searchError} +

+ Search failed: {searchError} +

+ {:else if !isSearching && cookbook.loading && cookbook.recipes.length === 0} +
+ +
{:else if visibleRecipes.length === 0}

- {#if cookbook.recipes.length === 0} + {#if isSearching} + No matches. + {:else if cookbook.recipes.length === 0} No recipes yet. Use the panel to add one. {:else} No matches. diff --git a/src/components/WikiList.svelte b/src/components/WikiList.svelte index 8fd3ade..b619c2f 100644 --- a/src/components/WikiList.svelte +++ b/src/components/WikiList.svelte @@ -17,6 +17,7 @@ import { app } from '$lib/state.svelte'; import { route, navigate } from '$lib/routing.svelte'; import { wikiStore, runWikiSearch } from '$lib/wiki-store.svelte'; + import Scanner from './Scanner.svelte'; interface Props { onSelect?: () => void; @@ -45,17 +46,17 @@ }; }); - // Sort by title case-insensitively. Semantic-search results come back - // in similarity order; alphabetising at the view layer keeps the - // drawer reading as a wiki listing rather than a relevance ranking. - // Substring-only and empty-query results land here already alpha- - // sorted but a re-sort is a no-op on those, so this stays the only - // place that owns the order. - const sorted = $derived( - [...wikiStore.results].sort((a, b) => + // Empty query: alphabetical, so the drawer reads as a wiki listing + // rather than a relevance ranking. Active query: pass server-returned + // order through verbatim - semantic hits come back ordered by cosine + // similarity ascending (closest first), then ILIKE hits the vector + // pass missed. That's what "ordered by closest match" means here. + const sorted = $derived.by(() => { + if (wikiStore.query.trim().length > 0) return wikiStore.results; + return [...wikiStore.results].sort((a, b) => a.title.toLowerCase().localeCompare(b.title.toLowerCase()), - ), - ); + ); + }); function pickArticle(id: string): void { navigate({ wiki_article_id: id }); @@ -75,8 +76,19 @@ spellcheck="false" /> - {#if wikiStore.loading && wikiStore.results.length === 0} -

Loading wiki…

+ {#if wikiStore.loading} + +
+ 0 + ? 'Searching wiki' + : 'Loading wiki'} + size={0.9} + /> +
{:else if wikiStore.error}

Couldn't load wiki: {wikiStore.error} diff --git a/src/lib/embeddings/sources/recipes.ts b/src/lib/embeddings/sources/recipes.ts new file mode 100644 index 0000000..ee19308 --- /dev/null +++ b/src/lib/embeddings/sources/recipes.ts @@ -0,0 +1,68 @@ +/** + * EmbeddingSource adapter for the `recipes` table. Lets the background + * embeddings worker populate the `embedding` column on new and edited + * recipes so the drawer's recipe search can rank by meaning rather + * than title-substring alone. + * + * Thin wrapper over two SupabaseService RPCs (`claimNextPendingRecipe` + * and `saveRecipeEmbedding`); same shape as the wiki and journal + * adapters. The generic loop in `../loop.ts` drives every source the + * worker registers. + */ +import type { SupabaseService } from '../../supabase'; +import type { EmbeddingSource, PendingItem } from '../types'; + +/** + * Defensive ceiling on the embed input. A user can paste an enormous + * recipe (long ingredient list plus a chapter of instructions) and + * Venice rejects requests past its embedding-model token window. The + * `recipes` table has no application-side length cap of its own + * (cooklang is the source of truth and can run several kilobytes), + * so the truncation lives here instead. Matches the order of + * magnitude the journal and wiki adapters cap at. + */ +const MAX_RECIPE_EMBED_CHARS = 16000; + +/** + * Compose the text Venice embeds. Title leads (a recipe titled + * "kombucha" should match "fermented tea" via the title path) and is + * followed by the optional free-form `source` (e.g. "NYT Cooking - + * Alison Roman") and then the cooklang body. Double-newline between + * blocks gives the embedding model a soft boundary so the title + * doesn't smear into the source line. + */ +function buildRecipeEmbedInput( + title: string, + source: string | null, + cooklang: string +): string { + const blocks: string[] = [title]; + if (source && source.trim().length > 0) blocks.push(source.trim()); + blocks.push(cooklang); + const joined = blocks.join('\n\n'); + return joined.length > MAX_RECIPE_EMBED_CHARS + ? joined.slice(0, MAX_RECIPE_EMBED_CHARS) + : joined; +} + +export function createRecipesSource(supabase: SupabaseService): EmbeddingSource { + return { + name: 'recipes', + async claimNext(holderId: string, ttlSeconds: number): Promise { + const row = await supabase.claimNextPendingRecipe(holderId, ttlSeconds); + if (!row) return null; + return { + id: row.id, + input: buildRecipeEmbedInput(row.title, row.source, row.cooklang), + }; + }, + async save( + id: string, + holderId: string, + embedding: number[], + model: string + ): Promise { + return supabase.saveRecipeEmbedding(id, holderId, embedding, model); + }, + }; +} diff --git a/src/lib/embeddings/worker.ts b/src/lib/embeddings/worker.ts index f5420da..1a69f4d 100644 --- a/src/lib/embeddings/worker.ts +++ b/src/lib/embeddings/worker.ts @@ -32,6 +32,7 @@ import { createThreadsSource } from './sources/threads'; import { createSamskaraSubstrateSource } from './sources/samskara-substrate'; import { createJournalSource } from './sources/journal'; import { createWikiSource } from './sources/wiki'; +import { createRecipesSource } from './sources/recipes'; import { LeaseCoordinator } from './lease'; import { runOneCycle, @@ -177,6 +178,7 @@ async function runWorker(msg: StartMessage, signal: AbortSignal): Promise createSamskaraSubstrateSource(supabase), createJournalSource(supabase), createWikiSource(supabase), + createRecipesSource(supabase), ]; const napConfig: NapConfig = { leasePollMs: msg.leasePollMs, diff --git a/src/lib/supabase.ts b/src/lib/supabase.ts index b62ac2c..eb00cd0 100644 --- a/src/lib/supabase.ts +++ b/src/lib/supabase.ts @@ -301,6 +301,8 @@ export interface Recipe { rating: number | null; created_at: string; updated_at: string; + /** Populated only by `search_recipes_by_embedding`. */ + similarity?: number; } /** @@ -1693,10 +1695,14 @@ export class SupabaseService { // // Same RLS posture as memories: every query is scoped to the signed-in // user automatically; only inserts need an explicit user_id because the - // with_check policy has no default to fall back on. No embedding - // pipeline — the cookbook stays small enough that ILIKE on `title` - // is cheap, and the model doesn't need semantic search to find a - // recipe it just wrote. + // with_check policy has no default to fall back on. + // + // Embedding pipeline: the cookbook stays small enough that the LLM + // tool path (`recipe_list`, `recipe_search`) gets by on ILIKE alone, + // but the human-facing drawer search (`RecipeList.svelte`) wires + // through the shared embeddings worker so a fuzzy query ("fluffy + // potato side") can find a recipe by meaning rather than title + // substring. Same claim/save/search RPC trio as wiki/journal. /** * List recipes, optionally filtered by a case-insensitive `title` @@ -1753,6 +1759,121 @@ export class SupabaseService { return (data as Recipe | null) ?? null; } + /** + * Semantic + substring search over recipes. Same merge contract as + * `searchWikiArticles` and `searchJournalEntries`: vector hits first + * (RPC, ordered by cosine similarity), then ILIKE hits the vector + * pass missed, deduped by id and capped at `limit`. Empty `query` + * falls back to `listRecipes` (most-recently-updated first) so + * callers don't need to special-case the no-query branch. + * `queryEmbedding` may be null - callers without Venice get ILIKE- + * only results. + * + * The ILIKE side runs on title only; the semantic side has the + * full `title + source + cooklang` blob folded into the embedding + * by the worker, so a meaning match can reach ingredient or + * technique text the title alone misses. + */ + async searchRecipes(opts: { + query: string; + queryEmbedding: number[] | null; + limit?: number; + }): Promise { + const query = opts.query.trim(); + const limit = opts.limit ?? 50; + if (query.length === 0) return this.listRecipes('', limit); + + const safe = query.replace(/([,()])/g, '\\$1'); + const pattern = `%${safe}%`; + + const ilikePromise = this.client + .from('recipes') + .select( + 'id, title, source, source_url, cooklang, rating, created_at, updated_at' + ) + .ilike('title', pattern) + .order('updated_at', { ascending: false }) + .limit(limit); + + const semanticPromise = opts.queryEmbedding + ? this.client.rpc('search_recipes_by_embedding', { + query_embedding: opts.queryEmbedding, + match_limit: limit, + }) + : Promise.resolve({ data: [] as unknown[], error: null }); + + const [ilikeRes, semRes] = await Promise.all([ilikePromise, semanticPromise]); + if (ilikeRes.error) throw new SupabaseError(ilikeRes.error.message); + const ilikeRows = ((ilikeRes.data ?? []) as unknown[]) as Recipe[]; + const semanticRows: Recipe[] = + semRes.error !== null ? [] : (((semRes.data ?? []) as unknown[]) as Recipe[]); + + const out: Recipe[] = []; + const seen = new Set(); + for (const r of semanticRows) { + if (seen.has(r.id)) continue; + seen.add(r.id); + out.push(r); + if (out.length >= limit) return out; + } + for (const r of ilikeRows) { + if (seen.has(r.id)) continue; + seen.add(r.id); + out.push(r); + if (out.length >= limit) return out; + } + return out; + } + + /** + * Claim the next recipe whose embedding column is null (or whose + * prior claim has expired). Used by the embeddings worker via the + * `createRecipesSource` adapter. Returns null when the queue is + * empty. Mirrors `claimNextPendingWikiArticle`. + */ + async claimNextPendingRecipe( + holderId: string, + ttlSeconds: number + ): Promise<{ + id: string; + title: string; + source: string | null; + cooklang: string; + } | null> { + const { data, error } = await this.client.rpc('claim_next_pending_recipe', { + p_holder_id: holderId, + p_ttl_seconds: ttlSeconds, + }); + if (error) throw new SupabaseError(error.message); + const rows = (data ?? []) as { + id: string; + title: string; + source: string | null; + cooklang: string; + }[]; + if (rows.length === 0) return null; + return rows[0]; + } + + async saveRecipeEmbedding( + id: string, + holderId: string, + embedding: number[], + model: string + ): Promise { + const { data, error } = await this.client.rpc( + 'save_recipe_embedding_if_claimed', + { + p_id: id, + p_holder_id: holderId, + p_embedding: embedding, + p_embedding_model: model, + } + ); + if (error) throw new SupabaseError(error.message); + return data === true; + } + /** * Create a recipe and snapshot the initial state into * `recipe_versions` atomically via the diff --git a/src/screens/Chat.svelte b/src/screens/Chat.svelte index 3d01233..edd668a 100644 --- a/src/screens/Chat.svelte +++ b/src/screens/Chat.svelte @@ -4262,9 +4262,12 @@ {#if searchQuery.trim().length > 0} - {#if searchBusy && searchResults.length === 0} + An in-flight search renders a Scanner in place of any + prior result list - the spinner-replaces-entries idiom + that the wiki / recipe / journal sidebars also use, so + every search surface in the app gives the same kind of + progress feedback. --> + {#if searchBusy}

diff --git a/supabase/schema.sql b/supabase/schema.sql index 2c69347..bdd0678 100644 --- a/supabase/schema.sql +++ b/supabase/schema.sql @@ -886,11 +886,14 @@ create policy "memory_relations are self-deletable" -- machine-readable URL when the model imported it from the web. Both -- nullable because hand-typed recipes often have neither. -- --- No embedding column: a personal cookbook is small (tens to low --- hundreds of rows). ILIKE on `title` is fast enough and keeps us off --- the embeddings worker's critical path. If cookbook sizes grow past a --- few hundred rows, the escape hatch mirrors memories exactly — add --- vector(2048) + claim columns + the same RPC shape. +-- Embedding column added later (see "Recipe embeddings" section below). +-- Original design omitted it on the rationale that ILIKE-on-title is +-- enough for a single-user cookbook; that holds for the LLM tool path +-- but the drawer's recipe search is a human surface where a fuzzy +-- "fluffy potato side" should find "Mashed Potatoes." Vector storage +-- mirrors memories / wiki: 2048-padded, written by the shared +-- embeddings worker. The default ILIKE-on-title still works for +-- callers that pass no embedding (e.g. the `recipe_list` tool). create table if not exists public.recipes ( id uuid primary key default gen_random_uuid(), @@ -1998,6 +2001,137 @@ begin order by l.position; end $$; +-- Recipe embeddings ------------------------------------------------------ +-- +-- Late add to the recipes table: the file-level comment further up +-- claims "no embedding column" on the rationale that ILIKE-on-title is +-- enough for a single-user cookbook. That holds for the LLM tool path +-- (the model already knows what title it just wrote), but the drawer's +-- recipe search is a human-facing surface where "fluffy potato side" +-- should find "Mashed Potatoes with Olive Oil." The sidebar wires +-- through the same embed-then-merge pipeline wiki and journal use, so +-- a column + the standard claim/save/search RPC trio joins recipes +-- without disturbing the existing recipe_*_with_version RPCs. +-- +-- Same 2048-padded vector storage as memories / journal_entries / wiki +-- so the single embeddings worker can share a pool and no per-source +-- dim plumbing is needed. +alter table public.recipes + add column if not exists embedding vector(2048); +alter table public.recipes + add column if not exists embedding_model text; +alter table public.recipes + add column if not exists embedding_claim_holder text; +alter table public.recipes + add column if not exists embedding_claim_expires timestamptz; + +-- Invalidate the embedding whenever the text that produced it changes. +-- The embedding source builds its input from title + source + cooklang +-- (see src/lib/embeddings/sources/recipes.ts), so any of those three +-- diverging means the stored vector is stale. Null the claim columns +-- too so an in-flight worker save (which guards on holder + expires > +-- now()) cannot land a stale vector against the new text. +create or replace function public.clear_recipe_embedding_on_change() + returns trigger language plpgsql as $$ +begin + if new.title is distinct from old.title + or new.cooklang is distinct from old.cooklang + or new.source is distinct from old.source then + new.embedding := null; + new.embedding_model := null; + new.embedding_claim_holder := null; + new.embedding_claim_expires := null; + end if; + return new; +end $$; + +drop trigger if exists clear_recipe_embedding_on_change on public.recipes; +create trigger clear_recipe_embedding_on_change + before update on public.recipes + for each row execute function public.clear_recipe_embedding_on_change(); + +-- Claim the next recipe whose embedding is null or whose prior claim +-- has expired. Same skip-locked fairness and claim shape as the wiki +-- and journal pipelines. Returns (id, title, source, cooklang) so the +-- worker can build the embedding input without a second round-trip. +drop function if exists public.claim_next_pending_recipe(text, int); +create or replace function public.claim_next_pending_recipe( + p_holder_id text, + p_ttl_seconds int +) returns table (id uuid, title text, source text, cooklang text) +language sql security invoker as $$ + with candidate as ( + select r.id + from public.recipes r + where r.user_id = auth.uid() + and r.embedding is null + and (r.embedding_claim_expires is null + or r.embedding_claim_expires < now()) + order by r.updated_at desc + limit 1 + for update skip locked + ) + update public.recipes r + set embedding_claim_holder = p_holder_id, + embedding_claim_expires = now() + make_interval(secs => p_ttl_seconds) + from candidate c + where r.id = c.id + returning r.id, r.title, r.source, r.cooklang; +$$; + +drop function if exists public.save_recipe_embedding_if_claimed(uuid, text, vector, text); +create or replace function public.save_recipe_embedding_if_claimed( + p_id uuid, + p_holder_id text, + p_embedding vector(2048), + p_embedding_model text +) returns boolean +language plpgsql security invoker as $$ +declare + updated int; +begin + update public.recipes + set embedding = p_embedding, + embedding_model = p_embedding_model, + embedding_claim_holder = null, + embedding_claim_expires = null + where id = p_id + and user_id = auth.uid() + and embedding_claim_holder = p_holder_id + and embedding_claim_expires > now(); + get diagnostics updated = row_count; + return updated > 0; +end $$; + +-- Cosine similarity search. Same shape as the wiki RPC; the sidebar +-- merges these hits with an ILIKE pass on the client side so freshly +-- written recipes that the worker has not embedded yet still appear. +drop function if exists public.search_recipes_by_embedding(vector, int); +create or replace function public.search_recipes_by_embedding( + query_embedding vector(2048), + match_limit int +) returns table ( + id uuid, + title text, + source text, + source_url text, + cooklang text, + rating smallint, + created_at timestamptz, + updated_at timestamptz, + similarity real +) +language sql stable security invoker as $$ + select id, title, source, source_url, cooklang, rating, + created_at, updated_at, + (1 - (embedding <=> query_embedding))::real as similarity + from public.recipes + where user_id = auth.uid() + and embedding is not null + order by embedding <=> query_embedding asc + limit match_limit +$$; + -- worker_leases ---------------------------------------------------------- -- -- Singleton per user per worker kind: at most one worker of a given kind