🤖 Math Robot AI

Intelligent Mathematical Problem Solving Pipeline From whiteboard image → OCR → AI normalization → Wolfram evaluation → spoken result (Pepper robot)

---

📌 Overview

Math Robot AI is a distributed system that:

1. Captures a whiteboard image (Pepper robot or API upload)
2. Detects mathematical expressions
3. Converts them to LaTeX (Pix2Text OCR)
4. Cleans and normalizes LaTeX using LLM (Ollama – Qwen2.5 3B )
5. Converts to Wolfram syntax
6. Evaluates using Wolfram Kernel (via proxy)
7. Returns structured results
8. Generates HTML output
9. Speaks the result via Pepper robot

---

📦 Repository Structure

.
├── math-robot-api/         # Main FastAPI backend
│   ├── app/
│   │   ├── controllers/    # API endpoints
│   │   ├── services/       # Core business logic
│   │   ├── schemas/        # Pydantic models
│   │   ├── models/         # Internal domain models
│   │   ├── middlewares/    # Logging middleware
│   │   ├── config.py
│   │   └── main.py
│   ├── Dockerfile
│   └── requirements.txt
│
├── math-robot-client/      # Pepper robot client
│   ├── main.py
│   └── config.py
│
├── wolfram-proxy/          # Wolfram evaluation service
│   ├── main.py
│   └── requirments.txt
│
├── infrastructure/
│   ├── docker-compose.yml
│   ├── docker-compose-school.yml
│   └── example.env
│
└── yolo_data/
    └── best.pt             # YOLO model for problem detection

---

🚀 Quick Start

---

0️⃣ Pull AI model into Ollama

ollama list
ollama pull qwen2.5:3b

This downloads the Qwen2.5 3B model. Run this before first use or if the model is missing.

---

1️⃣ Clone repository

git clone <repository-url>
cd math-robot-api

---

2️⃣ Configure environment

cd infrastructure
cp example.env .env

Edit .env if needed.

---

3️⃣ Start Backend Services (Docker)

🧑‍💻 Normal development mode

docker-compose up -d

---

🎓 School mode (REQUIRED for school demo)

docker-compose -f docker-compose-school.yml up -d

School mode includes:

• Full pipeline services
• Preconfigured classroom setup

---

4️⃣ Start Wolfram Proxy (Required)

⚠ The Wolfram Proxy must be started manually in a separate terminal.

Open a new terminal:

cd wolfram-proxy

Create virtual environment

python3 -m venv venv

Activate virtual environment

Linux / macOS

source venv/bin/activate

Install dependencies

pip install -r requirements.txt

Start Wolfram Proxy

python main.py

If successful, you should see:

Running on http://0.0.0.0:8010

⚠ Make sure Wolfram Engine is installed and the path matches:

WolframLanguageSession("/usr/local/bin/WolframKernel")

---

⏳ First Startup Notice

First startup may take several minutes because:

• Pix2Text model initializes
• Ollama model (Qwen2.5 3B) loads
• YOLO weights are loaded
• Wolfram session initializes

---

🌐 Services

Service	Port	Description
math-robot-api	8000	Main FastAPI backend
wolfram-proxy	8010	Wolfram evaluation service

API docs available at:

http://localhost:8000/docs

---

🔐 Authentication

API uses Basic Authentication.

Default (example):

username: test
password: test

⚠ Change credentials in production.

Pepper client sends:

Authorization: Basic base64("test:test")

---

🧠 Processing Pipeline

The PipelineService orchestrates:

Step 1 — Whiteboard Processing

• YOLO model detects problem regions
• Extracts individual problem images

Step 2 — OCR

• Pix2Text converts image → LaTeX

Step 3 — LaTeX Filtering

• Ollama (Qwen2.5 3B)
• Fixes syntax
• Normalizes structure
• Converts to Wolfram syntax

Step 4 — Wolfram Evaluation

• Sends to wolfram-proxy
• Evaluates via Wolfram Kernel

Step 5 — Result Filtering

• LLM cleans output
• Removes unnecessary formatting

---

📄 HTML File Generator

After pipeline execution:

HtmlService.save_problem(...)

Generates:

• Structured HTML file
• Saved in public directory
• Accessible via:

/public/index.html

This allows:

• Viewing results on tablet
• Shows last solved problem
• Prints helpful info for debug

---

🤖 Pepper Robot Client

Located in:

math-robot-client/

What it does:

• Waits for head touch
• Captures camera image
• Sends image via multipart/form-data
• Receives result
• Speaks solution
• Displays HTML on tablet

---

🔬 Wolfram Proxy

Located in:

wolfram-proxy/

Lightweight Flask service that:

• Maintains persistent WolframLanguageSession
• Evaluates Wolfram code
• Exposes:

GET /eval?code=...
GET /health

Required for full pipeline functionality.

---

🛠 Technology Stack

Backend

AI & Machine Learning

Mathematical Processing

Infrastructure & Tools

---

🧪 Main Endpoint

POST /pipeline/{target_regions}

Parameters:

• target_regions — expected number of expressions (1–20)
• file — whiteboard image (multipart/form-data)

Returns:

{
  "total_problems": 1,
  "successful": 1,
  "failed": 0,
  "results": [
    {
      "problem_id": 1,
      "latex_raw": "...",
      "latex_filtered": "...",
      "result_wolfram": "...",
      "result_filtered": "...",
      "success": true
    }
  ],
  "processing_time": 3.42
}

---

⚠ Important Notes

• YOLO model file must exist:

yolo_data/best.pt

• Wolfram Kernel path must match:

WolframLanguageSession("/usr/local/bin/WolframKernel")

• For school demo → always use docker-compose-school.yml

---