feat: resilient background job retry & monitoring

[chengyixu]

Summary

Implements a production-ready background job execution framework for FinMind that adds resilience to reminder dispatch and other background tasks.

/claim #130

Features

Job Execution Engine:

• JobExecution model tracking every background job with full lifecycle metadata (status, retries, errors, timestamps)
• Configurable maximum retries per job (default: 3)
• JSON-encoded payload and result storage for audit trail

Exponential Backoff with Jitter:

• Retry 1: ~5 minutes (300s)
• Retry 2: ~15 minutes (900s)
• Retry 3: ~45 minutes (2700s)
• ±10% jitter prevents thundering-herd on mass retries

Circuit Breaker Pattern:

• Separate breakers for email and WhatsApp external services
• CLOSED → OPEN → HALF_OPEN state transitions
• Configurable failure threshold and reset timeout
• Prevents cascading failures when external services are down

Dead-Letter Handling:

• Jobs that exhaust all retries are marked DEAD with full error history
• Can be manually retried via API

REST API Endpoints:

Endpoint	Method	Description
/jobs/status	GET	Paginated job list with status/type filters
/jobs/stats	GET	Aggregate stats: success rate, avg duration, counts by status/type
/jobs/health	GET	Health check with circuit breaker states (no auth required)
/jobs/	GET	Single job detail
/jobs/retry/	POST	Manual retry of failed/dead jobs
/jobs/process-retries	POST	Process all pending retries (for cron/scheduler)

Integration:

• dispatch_reminder_with_retry() creates tracked jobs for reminder sends
• Pluggable handler registry for extending to new job types
• Wired into existing reminder dispatch pipeline

Test Coverage

30 tests covering:

• Exponential backoff calculation (2 tests)
• Circuit breaker state machine (5 tests)
• Job creation, execution, failure (4 tests)
• Retry scheduling and processing (2 tests)
• Manual retry logic (2 tests)
• Statistics computation (1 test)
• Reminder integration (1 test)
• All 6 API endpoints (11 tests)
• Model serialization (2 tests)

All 30 tests pass.

Files Changed

File	Change
packages/backend/app/models.py	Added JobExecution model and JobStatus enum
packages/backend/app/services/job_scheduler.py	New: job engine, backoff, circuit breaker, handler registry
packages/backend/app/routes/jobs.py	New: monitoring and management API endpoints
packages/backend/app/routes/init.py	Register jobs blueprint at /jobs
packages/backend/app/db/schema.sql	Added job_executions table with indexes
packages/backend/tests/test_jobs.py	30 comprehensive tests

Database Migration

CREATE TABLE IF NOT EXISTS job_executions (
  id SERIAL PRIMARY KEY,
  job_type VARCHAR(100) NOT NULL,
  status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
  payload TEXT,
  result TEXT,
  retry_count INT NOT NULL DEFAULT 0,
  max_retries INT NOT NULL DEFAULT 3,
  last_error TEXT,
  next_retry_at TIMESTAMP,
  started_at TIMESTAMP,
  completed_at TIMESTAMP,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  user_id INT REFERENCES users(id) ON DELETE SET NULL
);

No Breaking Changes

• All existing tests continue to pass
• No modifications to existing reminder, bill, or expense logic
• Backward compatible with existing database schema