feat(memory): pluggable storage backends

[erma07]

Summary

Redesign the openfang-memory crate with pluggable storage backends. The main storage backend (SQLite or PostgreSQL) and the semantic/vector backend (SQLite, PostgreSQL, Qdrant, HTTP) are now independently configurable, allowing mix-and-match deployments like PostgreSQL for structured data with Qdrant for vector search.

Architecture

The orchestration layer (substrate.rs) is now 100% backend-agnostic — zero database-specific imports. All storage is abstracted through 9 backend traits with implementations per database:

Trait	SQLite	PostgreSQL	Qdrant	HTTP
Structured	x	x
Semantic	x	x	x	x
Knowledge	x	x
Session	x	x
Usage	x	x
PairedDevices	x	x
TaskQueue	x	x
Consolidation	x	x
Audit	x	x

Each backend lives in its own folder (sqlite/, postgres/, qdrant/, http/) with identical file structure (11 files each for SQLite and PostgreSQL). Shared serialization and parsing logic is extracted into helpers.rs to eliminate cross-backend duplication.

SessionBackend uses Rust default trait implementations for 5 methods (create_session, create_session_with_label, append_canonical, canonical_context, store_llm_summary) — new backends only need to implement the storage primitives.

Changes

• New backend traits: PairedDevicesBackend, TaskQueueBackend, ConsolidationBackend, AuditBackend (added to existing StructuredBackend, SemanticBackend, KnowledgeBackend, SessionBackend, UsageBackend)
• PostgreSQL: full parity with SQLite — all 9 backends implemented with pgvector for vectors, versioned migrations matching SQLite v1–v9
• Qdrant: semantic-only backend via gRPC with auto-collection creation and cosine similarity search
• HTTP: semantic-only backend routing to a remote memory-api gateway with automatic fallback to local storage
• Folder restructure: SQLite code moved from top-level into sqlite/, all database-specific code isolated in its backend folder
• External callers migrated: kernel, API routes, and runtime now use trait-based APIs (Arc<dyn UsageBackend>, AuditBackend) — no more leaked rusqlite::Connection types
• New config field: semantic_backend allows independent vector backend selection
• Docker Compose: added pgvector/pgvector:pg18 and qdrant/qdrant:latest services for integration testing

Testing

cargo clippy --workspace --all-targets -- -D warnings passes
cargo test --workspace passes (65 tests: 40 unit + 25 integration)
Integration tested against live PostgreSQL (pgvector/pg18) and Qdrant
All SQLite, PostgreSQL, and Qdrant backends verified with identical test suite

Security

No new unsafe code (existing sqlite-vec FFI registration unchanged)
No secrets or API keys in diff
User input validated at boundaries
usage_conn() removed — no more raw database connection leaks to external crates

Configuration

[memory]
backend = "sqlite"              # main storage: "sqlite" or "postgres"
semantic_backend = "qdrant"     # vector search: "sqlite", "postgres", "qdrant", or "http"

postgres_url = "postgresql://user:pass@localhost/openfang"
qdrant_url = "http://localhost:6334"