feat: add local semantic memory with embedding + vector search

[0xbyt4]

Summary

OM1 currently has short-term memory (LLMHistoryManager: last N messages + summarization) but no long-term memory - the robot cannot learn from previous interactions. This PR adds local semantic memory using embedding-based vector search, allowing the robot to store experiences and automatically retrieve relevant memories during conversations.

How It Works

Tick Start
    |
    v
Fuser.fuse()
    |-- Collect sensor inputs (inputs_fused)
    |-- SemanticMemoryProvider.retrieve(inputs_fused, mode)    <-- NEW
    |-- Inject RELEVANT MEMORIES section into prompt            <-- NEW
    |-- Return fused_prompt
    |
    v
LLM.ask(prompt)
    |-- @update_history() decorator
    |   |-- LLM generates response
    |   |-- SemanticMemoryProvider.store(input, response, mode) <-- NEW
    |
    v
Actions executed

Each tick, the Fuser queries semantic memory with the current sensor input. ChromaDB returns the top-k most similar past experiences (filtered by cosine similarity threshold). These are injected into the prompt as a RELEVANT MEMORIES section between inputs and actions. After the LLM responds, the input-response pair is stored as a new memory.

Technical Details

• Embedding Model: all-MiniLM-L6-v2 - 384-dim embeddings, ~22M params, runs on CPU
• Vector Store: ChromaDB PersistentClient - SQLite-backed, persists to config/memory/embeddings/
• Similarity: Cosine distance with configurable threshold (default: 0.3)
• Mode Isolation: Each robot mode gets its own ChromaDB collection (om1_{mode_name})
• Lazy Loading: Model and ChromaDB only initialize when semantic_memory_enabled: true
• Singleton Pattern: Consistent with existing providers (IOProvider, etc.)

Resource Budget (Apple M3, 8GB RAM)

Component	RAM	Latency
all-MiniLM-L6-v2	~90 MB	~10ms/encode
ChromaDB	~20-50 MB	<5ms/query
Total	~140 MB	~20ms/tick

Configuration

Disabled by default. Enable per config:

cortex_llm: {
    type: "OpenAILLM",
    config: {
        agent_name: "Spot",
        history_length: 10,
        semantic_memory_enabled: true,   // default: false
        semantic_memory_top_k: 3,        // default: 3
        semantic_memory_threshold: 0.3,  // default: 0.3
    },
},

Changes

File	Change
src/providers/semantic_memory_provider.py	New singleton provider: store/retrieve/clear_mode
src/llm/__init__.py	3 new LLMConfig fields
src/runtime/cortex.py	Provider init in _initialize_mode()
src/providers/llm_history_manager.py	Store hook in update_history() decorator
src/fuser/__init__.py	Retrieve + RELEVANT MEMORIES prompt injection
tests/providers/test_semantic_memory_provider.py	20 unit tests
pyproject.toml	sentence-transformers, chromadb, onnxruntime deps

Verified Manually

Tested with MockInput (WebSocket) + conversation config. Robot successfully:

1. Stored facts ("My name is Oxbye", "I work as a blockchain developer", "I love cats")
2. Retrieved correct memories when asked ("Where am I from?" -> Jupiter, "What do I do?" -> blockchain developer)
3. Persisted memories across restarts (ChromaDB PersistentClient)

[codecov]

Codecov Report

❌ Patch coverage is 85.15625% with 19 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/providers/semantic_memory_provider.py	81.90%	19 Missing ⚠️

📢 Thoughts on this report? Let us know!