Skip to content

research_context.md

Latest commit

 

History

History
80 lines (57 loc) · 2.85 KB

research_context.md

File metadata and controls

80 lines (57 loc) · 2.85 KB

Research Context

This document describes the research project that led to sdr-memory.

Mission

Investigate whether a small model (trainable on consumer GPU hardware) can learn to encode experiences in compressed representations and reconstruct them by context -- similar to how human memory works.

Research Question

Can a neural network learn to RECONSTRUCT memories from compressed representations triggered by context, rather than simply searching a database?

Inspiration

  • DNA stores instructions, not data. Genes activate by context signals.
  • Human memory is associative, reconstructive, lossy-but-meaningful.
  • Current AI "memory" (RAG, vector DB, flat files) is primitive file search.

Hardware Constraints

  • NVIDIA T1000 8GB (FP32 only -- FP16 is 4.2x slower)
  • Intel i7-13700T, 31GB RAM
  • Ollama available for 7B models (inference + QLoRA fine-tune)

Conventions

  • Python 3.10+ via uv run python
  • PyTorch for training, HuggingFace for tokenizers/models
  • All experiments save JSON results to results/

Key Findings

Experiment 0.1 -- Associative Retrieval

Hopfield network sharpening over TF-IDF latent space provided modest improvements over raw cosine similarity baseline.

Experiment 0.2 -- Contrastive Learning

Dual-encoder MLP with InfoNCE loss learned effective query/key embeddings that outperformed TF-IDF on semantically similar but lexically different pairs.

Experiment 0.2 Semantic -- Transformer Embeddings

Pre-trained MiniLM embeddings + Hopfield sharpening achieved the best absolute retrieval metrics.

Experiment 0.2 Hybrid -- Fusion

Weighted fusion of lexical (TF-IDF) and semantic (MiniLM) scores, with alpha tuned on validation, often beat either method alone.

Experiment 0.5 -- Reconstruction

Multi-head classifiers recovered semantic attributes (entity, status, cause, action, urgency) from a 32D bottleneck well above chance level, even with corrupted input. This supports the thesis that memories can be meaningfully reconstructed from compact encodings.

Vector Duel -- Dense vs Sparse

Sparse 4096-bit SDR vectors (random projection + k-WTA + Hamming) achieved 92% top-1 vs 81% for dense 32D (PCA + cosine) on noisy query retrieval with 1600 memories.

Conclusion

The SDR approach was selected for the production daemon because it offers the best combination of:

  1. Zero external dependencies (no GPU, no embedding model)
  2. High retrieval accuracy on the target memory sizes
  3. Deterministic, interpretable behavior
  4. Sub-millisecond search latency

The research experiments are preserved in examples/research/ for reproducibility and further investigation.

Rules

  • Every hypothesis must be testable with current hardware
  • Every experiment must produce measurable results (JSON + plots)
  • Negative results are documented -- they are findings, not failures
  • No over-engineering: simplest approach first, iterate