Update MIRIX backend to use async native approach

.claudeignore

@@ -0,0 +1,10 @@
+.venv/
+build/
+*.egg-info/
+mirix.egg-info/
+.persist/
+__pycache__/
+*.pyc
+.pytest_cache/
+node_modules/
+dashboard/node_modules/

.cursorignore

@@ -0,0 +1,10 @@
+.venv/
+build/
+*.egg-info/
+mirix.egg-info/
+.persist/
+__pycache__/
+*.pyc
+.pytest_cache/
+node_modules/
+dashboard/node_modules/

.cursorrules

@@ -0,0 +1,187 @@
+# MIRIX Project Rules for Cursor AI
+
+## Project Overview
+MIRIX is a Multi-Agent Personal Assistant with an Advanced Memory System. It features a six-agent memory architecture (Core, Episodic, Semantic, Procedural, Resource, Knowledge Vault) with screen activity tracking and privacy-first design.
+
+## Tech Stack
+- **Backend**: Python 3.10+, FastAPI, PostgreSQL, Redis, Kafka
+- **Frontend**: React, TypeScript, Vite, TailwindCSS
+- **LLM Integration**: Multiple providers (OpenAI, Anthropic, Google, Azure, etc.)
+- **Observability**: LangFuse
+- **Testing**: pytest
+- **Linting**: Ruff, Pyright
+
+## Code Standards
+
+### Python Guidelines
+1. **Follow PEP 8** strictly
+   - Use 4 spaces for indentation (no tabs)
+   - Max line length: 79 characters for code, 72 for docstrings
+   - Use snake_case for variables/functions, CamelCase for classes, ALL_CAPS for constants
+   - Group imports: standard library, third-party, local application (blank lines between groups)
+
+2. **Type Hints**
+   - Always use type hints for function signatures
+   - Use `Optional[T]` for nullable types
+   - Use `Union` or `|` syntax for multiple types
+   - Example: `def create_user(user_id: str, name: Optional[str] = None) -> User:`
+
+3. **Documentation**
+   - Use triple-quoted docstrings for all modules, classes, and functions
+   - Include parameter descriptions, return types, and examples where helpful
+   - Document complex logic with inline comments explaining WHY, not WHAT
+
+4. **Error Handling**
+   - Use specific exception types (ValueError, TypeError, HTTPException, etc.)
+   - Never use bare `except:` clauses
+   - Log errors with appropriate context
+
+5. **Async/Await**
+   - Use async/await for I/O operations
+   - Prefix async functions with `async def`
+   - Always await async calls
+
+### Async Architecture (Critical)
+The codebase is fully async-native. Violating these rules will break the server.
+
+1. **All new manager methods in `mirix/services/` MUST be `async def`**
+2. **Never call `asyncio.run()` inside the server** — the event loop is already running
+3. **Never use sync DB drivers** — use `asyncpg` (not `pg8000`), `aiosqlite` (not `sqlite3`)
+4. **Never use sync Redis** — use `redis.asyncio` only, never `redis.Redis`
+5. **Never use `requests`** — use `httpx.AsyncClient` for all HTTP calls
+6. **Wrap unavoidable sync calls** with `asyncio.to_thread()` — only for these 5 approved touch-points:
+   - LangFuse SDK (no async version)
+   - Gmail OAuth (inherently blocking)
+   - SQLAlchemy DDL at startup
+   - Cleanup job entry point
+   - Pure CPU helpers with no I/O
+7. **Do NOT add new sync touch-points** — check `docs/Mirix_async_native_changes.md` first
+
+### Project-Specific Patterns
+
+#### 1. Memory System Architecture
+- **Meta Agent** orchestrates specialized memory agents
+- **Sub-agents** extract specific memory types (episodic, semantic, etc.)
+- **Memory extraction** happens via queue-worker architecture
+- Each memory type has dedicated ORM models, schemas, and managers
+
+#### 2. Database Patterns
+- **ORM Models**: Located in `mirix/orm/` using SQLAlchemy
+- **Schemas**: Located in `mirix/schemas/` using Pydantic
+- **Managers**: Located in `mirix/services/` for business logic
+- Always use `OrganizationMixin`, `UserMixin`, `AgentMixin` where applicable
+
+#### 3. Agent Execution Flow
+- `step()` method is the main agent execution loop (like LangChain's AgentExecutor)
+- `inner_step()` handles single LLM interactions with tool calls
+- `save_agent()` persists agent state to database
+- Steps are logged to the `steps` table for audit/analytics (write-only)
+
+#### 4. Message Flow
+- Queue messages (transient) != Database messages (persistent)
+- Database messages stored via `MessageManager.create()` after agent execution
+- Messages link to steps via `step_id` foreign key
+- Messages table used for conversation history and search
+
+#### 5. API Patterns
+- REST endpoints in `mirix/server/rest_api.py`
+- Client SDK in `mirix/client/remote_client.py`
+- Always authenticate with JWT or API key
+- Use `create_or_get_user()` to ensure user exists before operations
+
+#### 6. Settings Management
+- Environment variables defined in `mirix/settings.py`
+- Use Pydantic BaseSettings with `env` prefix `MIRIX_`
+- Provider API keys in `ModelSettings`
+- Tool configs in `ToolSettings`
+
+#### 7. Tool Execution
+- User-defined tools run in `ToolExecutionSandbox` for security
+- Tools defined in `mirix/functions/function_sets/`
+- Memory tools in `memory_tools.py`, base tools in `base.py`
+
+### Code Review Checklist
+Before suggesting changes, verify:
+- [ ] No duplicate code or configurations
+- [ ] Type hints are present and correct
+- [ ] Error handling is specific and informative
+- [ ] Docstrings follow PEP 257
+- [ ] No hardcoded secrets or API keys
+- [ ] Database operations use proper ORM patterns
+- [ ] Async functions are properly awaited
+- [ ] Tests are included for new functionality
+- [ ] Logging uses appropriate levels (debug, info, warning, error)
+
+### Common Pitfalls to Avoid
+1. **Do NOT** confuse queue messages with database messages
+2. **Do NOT** call `step_manager.get_step()` - steps are write-only audit logs
+3. **Do NOT** bypass `create_or_get_user()` - always ensure users exist first
+4. **Do NOT** create agents without proper `CreateAgent` schema objects
+5. **Do NOT** forget to persist agent state with `save_agent()`
+6. **Do NOT** use `message.step` relationship - it's never loaded in practice
+7. **Do NOT** add duplicate environment variables in settings.py
+
+### Testing Guidelines
+- Tests located in `tests/` directory
+- Use pytest fixtures defined in `conftest.py`
+- Mock external LLM calls in tests
+- Test user isolation, memory extraction, and API endpoints
+- Run tests with: `pytest tests/`
+
+### Documentation
+- Do NOT create documentation files unless explicitly requested
+- Do NOT create README files proactively
+- Update existing docs when making significant architectural changes
+- Keep code comments concise and focused on WHY
+
+### Git Workflow
+- Commit messages should be clear and descriptive
+- Run linters before committing: `ruff check .` and `pyright`
+- Resolve merge conflicts carefully (especially in `poetry.lock`)
+- Do NOT commit secrets or API keys
+
+### File Organization
+```
+mirix/
+├── agent/          # Agent implementations (MetaAgent, SubAgents)
+├── client/         # Client SDK (remote_client.py, client.py)
+├── database/       # Database connectors (PostgreSQL, Redis)
+├── functions/      # Tool definitions and function sets
+├── llm_api/        # LLM provider integrations
+├── orm/            # SQLAlchemy ORM models
+├── queue/          # Message queue (Kafka) implementation
+├── schemas/        # Pydantic schemas for validation
+├── server/         # FastAPI REST API (rest_api.py, server.py)
+├── services/       # Business logic managers (user, agent, memory)
+├── prompts/        # System prompts for agents
+└── settings.py     # Environment configuration
+```
+
+### When Analyzing Code
+1. Always trace from ORM models → Schemas → Managers → APIs
+2. Check both where code is defined AND where it's called
+3. Distinguish between operational flow and audit logging
+4. Verify parameter signatures match between callers and callees
+5. Understand the meta-agent → sub-agent orchestration pattern
+
+### Performance Considerations
+- Use bulk operations for database writes when possible
+- Leverage Redis caching for frequently accessed data
+- Monitor LLM token usage and costs
+- Use async patterns for I/O-bound operations
+- Consider queue backpressure for high-throughput scenarios
+
+### Security Best Practices
+- User-defined tools MUST run in `ToolExecutionSandbox`
+- Validate all user inputs with Pydantic schemas
+- Use parameterized queries (ORM handles this)
+- Sanitize log messages (no sensitive data)
+- Implement proper authentication and authorization
+
+## Response Style
+- Be concise and clear
+- Provide code examples when helpful
+- Reference specific files and line numbers
+- Explain complex logic flows step-by-step
+- Don't make changes unless explicitly requested
+- Ask clarifying questions when intent is ambiguous

.github/workflows/tests.yml

@@ -34,7 +34,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         pip install -e .
-        pip install pytest pytest-cov python-dotenv requests pyyaml
+        pip install pytest pytest-cov python-dotenv requests pyyaml pytest-asyncio
 
     - name: Verify workspace structure
       run: |

CLAUDE.md

@@ -0,0 +1,96 @@
+# MIRIX - Claude Code Instructions
+
+## What This Project Does
+MIRIX is an async-native multi-agent personal assistant with a six-component memory system
+(Core, Episodic, Semantic, Procedural, Resource, Knowledge Vault). It captures memories
+from conversations and screen activity, storing them in PostgreSQL with vector search via pgvector.
+
+## Key Architecture
+- **Entry point**: `mirix/server/rest_api.py` (FastAPI, port 8531)
+- **Orchestration**: `mirix/agent/meta_agent.py` → 6 sub-agents
+- **Data flow**: ORM (`mirix/orm/`) → Schemas (`mirix/schemas/`) → Managers (`mirix/services/`) → API (`mirix/server/`)
+- **Queue**: Kafka-backed (`mirix/queue/`) for async memory extraction
+- **All I/O is async** — never introduce sync blocking calls (see `docs/Mirix_async_native_changes.md`)
+
+## Local Development Setup
+
+### Prerequisites
+- Docker + Docker Compose
+- Python 3.10+
+- At least one LLM API key (OpenAI, Anthropic, or Google Gemini)
+
+### Start Infrastructure
+```bash
+cp docker/env.example .env   # then add your API keys to .env
+docker-compose up -d         # starts PostgreSQL (5432), Redis (6379), API (8531), Dashboard (5173)
+docker-compose ps            # verify all services healthy
+```
+
+### Run API Locally (without Docker)
+```bash
+pip install -e .
+export GEMINI_API_KEY=your-key   # or OPENAI_API_KEY / ANTHROPIC_API_KEY
+python scripts/start_server.py --port 8531
+```
+
+### Access Points
+- Dashboard: http://localhost:5173
+- API Swagger: http://localhost:8531/docs
+- API ReDoc: http://localhost:8531/redoc
+
+## Running Tests
+
+```bash
+# Fast unit tests — no running server needed (~20s)
+pytest tests/test_memory_server.py -v
+
+# All tests except integration
+pytest -m "not integration" -v
+
+# Integration tests — requires server on port 8899
+python scripts/start_server.py --port 8899          # Terminal 1
+pytest tests/test_memory_integration.py -v -m integration -s   # Terminal 2
+
+# Full suite
+pytest -v
+```
+
+**Required env var for tests**: `GEMINI_API_KEY`
+
+## Common Dev Tasks
+
+### Add a new API endpoint
+1. Add Pydantic request/response schemas to `mirix/schemas/`
+2. Add business logic method to the relevant manager in `mirix/services/`
+3. Add the route to `mirix/server/rest_api.py`
+4. Add the corresponding method to `mirix/client/remote_client.py`
+
+### Add a new memory type
+1. Create ORM model in `mirix/orm/`
+2. Create Pydantic schemas in `mirix/schemas/`
+3. Create manager in `mirix/services/`
+4. Create sub-agent in `mirix/agent/`
+5. Register in `mirix/agent/meta_agent.py`
+
+### Format & lint
+```bash
+make format   # ruff import sort + format
+make lint     # ruff check + pyright
+make check    # format + lint + test
+```
+
+## Async Rules (Critical)
+- **All new manager methods must be `async def`**
+- **Never use `asyncio.run()` inside the server** — the event loop is already running
+- Only 5 intentional sync touch-points exist (LangFuse, Gmail OAuth, SQLAlchemy DDL at startup, cleanup job entry, pure CPU helpers) — do not add more
+- Use `asyncio.to_thread()` to wrap any unavoidably sync third-party calls
+
+## Commit Convention
+Prefix commits with the Jira ticket: `[VEPEAGE-NNN] Description`
+
+## Do Not
+- Confuse queue messages (transient) with database messages (persistent)
+- Call `step_manager.get_step()` — steps are write-only audit logs
+- Skip `create_or_get_user()` — always ensure users exist first
+- Commit secrets or API keys
+- Create README or doc files unless explicitly requested

docker-compose.test.yml

@@ -83,6 +83,8 @@ services:
     ports:
       - "${SERVER_PORT:-8000}:8000"  # Expose on configurable port (default 8000)
     command: ["uvicorn", "mirix.server.rest_api:app", "--host", "0.0.0.0", "--port", "8000"]
+    env_file:
+      - .env  # Load API keys and other secrets from .env file
     environment:
       # Connect to test database and Redis
       - MIRIX_PG_HOST=test_db