refactor: Move non-data-intensive operations from Express backend to Supabase Edge Functions

[baderdean]

Context

The Express backend currently handles both data-intensive operations (mining, extracting, cleaning, signature extraction) and lightweight operations. Data-intensive operations need to stay in the Express backend due to:

• Heavy processing requirements
• Long-running tasks with Redis streams
• Worker queue management

However, lightweight operations that don't involve intensive processing should be moved to Supabase Edge Functions for better scalability, lower latency, and reduced infrastructure cost.

Current Architecture

Express Backend Routes

Data-Intensive Operations (Keep in Express):

Endpoint	Purpose
POST /api/imap/mine/email/:userId	Email mining (fetch + extract + clean + signature extraction)
POST /api/imap/mine/file/:userId	File mining with extraction
POST /api/imap/mine/pst/:userId	PST mining with signature extraction
GET /api/imap/mine/:type/:id/progress/	SSE progress streaming
GET /api/imap/mine/:userId/	Get task status (requires Redis pub/sub)
POST /api/imap/mine/:type/:userId/:id	Stop task (requires Redis stream cleanup)

Non-Data-Intensive Operations (Move to Edge Functions):

Endpoint	Purpose	Priority
POST /api/enrich/person	Single contact enrichment	High
POST /api/enrich/person/bulk	Bulk enrichment	High
POST /api/enrich/webhook/:id	Async enrichment webhook	High
GET /	Health check	Low
DELETE /api/auth/users	Delete user account	High
POST /api/imap/boxes	Fetch IMAP folder tree	High
POST /api/imap/mine/sources/imap	Create IMAP source with manual credentials	High
POST /api/imap/mine/sources/:provider	OAuth URL generation	High
GET /api/imap/mine/sources/:provider/callback	OAuth callback	High
POST /api/contacts/export/:exportType	Export contacts	Medium

Why Task Management Stays in Express

The task status and stop endpoints require Redis for:

• Pub/Sub: Real-time progress updates to clients via SSE
• Redis Streams: Message queues between fetch → extract → clean workers
• In-memory Map: Active tasks stored in memory for fast access

Moving these would require replacing Redis with database-based queues, which is complex and would lose real-time SSE progress updates.

Already in Edge Functions

The following edge functions already exist:

• email-campaigns - Email campaign management
• passive-mining - Passive mining operations
• fetch-mining-source - OAuth handlers for Google/Microsoft
• add-mining-source - Add mining source
• delete-mining-source - Delete mining source
• mail/* - Email notifications
• email-templates - Email template management
• imap - IMAP operations

Migration Plan

Phase 1: High Priority

1. Enrichment (/api/enrich/*)

   • External API calls + lightweight JSON processing
   • Can run efficiently in edge functions

2. OAuth Flow (/api/imap/mine/sources/:provider, /callback)

   • Currently partly in edge functions (fetch-mining-source)
   • Need to consolidate and remove Express dependencies

3. IMAP Boxes (POST /api/imap/boxes)

   • Fetch folder tree - simple DB + IMAP query
   • Can be handled by existing imap edge function

Phase 2: Medium Priority

4. Export Contacts (POST /api/contacts/export/:exportType)

   • Requires billing integration
   • Generate file and return download URL

5. Delete User (DELETE /api/auth/users)

   • Cleanup user data across tables

Phase 3: Low Priority

6. Health Check (GET /)
   • Simple ping endpoint
   • Could use Supabase's built-in health check

Requirements

• All edge functions must use the shared middleware for auth validation
• Billing integration needs to work from edge functions
• Proper error handling and logging (using shared logger)
• CORS handled via shared cors middleware

Benefits

1. Scalability: Edge functions scale automatically
2. Latency: Lower latency for users worldwide
3. Cost: Reduce Express backend load and costs
4. Maintenance: Smaller Express codebase to maintain

Todo

• Audit all Express routes to confirm classification
• Create new edge functions for missing endpoints
• Migrate high-priority endpoints (enrichment, OAuth, IMAP boxes)
• Update frontend to call edge functions directly
• Remove deprecated Express routes
• Test all migrated endpoints

/cc @ankaboot-source