Conversational AI Platform

A platform that allows configuring a user profile, agent profile, and supporting documents, and can then engage over audio in a video call with one or more human attendees.

System Architecture

The application consists of two main modules:

1. Agent Module - Handles real-time audio processing, conversation management, and AI interactions
2. Server Module - Manages API endpoints, database operations, and business logic

Installation and Setup

Prerequisites

• Python 3.9 or higher
• PostgreSQL
• Docker (optional)
• Node.js 18+ (for frontend)

Agent Setup

1. Navigate to the agent directory:

   cd finance-assistant/agent

2. Create a virtual environment and install dependencies:

   python3 -m venv .venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
   pip install -U pip
   pip install -r requirements.txt

3. Download required AI models:

   python3 download_files.py

4. Create .env file from template:

   cp env.template .env
   # Update the .env with your credentials

5. Start the agent:

   python3 src/assistant.py start

Server Setup

1. Navigate to the server directory:

   cd finance-assistant/server

2. Create a virtual environment and install dependencies:

   python3 -m venv .venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
   pip install -U pip
   pip install -r requirements.txt

3. Set up environment variables:

   cp env.template .env
   # Update the .env with your credentials

4. Apply database migrations:

   alembic upgrade head

5. Start the FastAPI server:

   uvicorn src.main:app --reload

Environment Variables

Both modules require the following environment variables:

# LiveKit Configuration
LIVEKIT_URL=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
LIVEKIT_WSS_URL=

# API Keys
DEEPGRAM_API_KEY=
OPENAI_API_KEY=
ELEVEN_API_KEY=


# Database Configuration
DB_USER=
DB_PASSWORD=
DB_HOST=
DB_PORT=
DB_NAME=

# Admin Configuration
ADMIN_USERNAME=
ADMIN_PASSWORD=
SECRET_KEY=

# Misc
FRONTEND_BASE_URL=
LITELLM_LOG=

Docker Deployment

Full Stack Deployment

1. Ensure Docker and Docker Compose are installed
2. Copy environment templates:

   cp env.template .env

3. Start all services:

   docker compose up --build -d

Database Only

To run only the PostgreSQL database:

docker compose up -d postgres

Database Management

Creating New Migrations

alembic revision --autogenerate -m "Description of changes"

Applying Migrations

alembic upgrade head

Rolling Back Migrations

alembic downgrade -1  # Roll back one migration
alembic downgrade base  # Roll back all migrations

Development Tools

Code Formatting

# Install development dependencies
pip install ruff

# Format code
ruff format .
ruff check --fix .

Linting

# Run linter
ruff check .

Testing

# Install test dependencies
pip install pytest pytest-asyncio

# Run tests
pytest

API Documentation

Once the server is running, access the API documentation at:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Playground Connection

To test the application:

1. Start both agent and server modules
2. Access the LiveKit Playground
3. Connect using your configured credentials

Troubleshooting

Common issues and solutions:

1. Database Connection Issues:

   • Verify PostgreSQL is running
   • Check database credentials in .env
   • Ensure database exists and is accessible

2. Model Download Failures:

   • Check internet connection
   • Verify disk space availability
   • Try running download_files.py with sudo if permission issues

3. Agent Connection Issues:

   • Verify LiveKit credentials
   • Check if required ports are open
   • Ensure WebSocket URL is accessible

Contributing

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details