Hermes

High-throughput real-time event processing engine. Target: ~10M events/min (~166k/sec).

Quick start

Local

# Build
go build -o hermes ./cmd/hermes

# Run without DB (events queued, batched, aggregated in-memory only)
./hermes

# Run with Postgres
export DB_DSN="postgres://user:pass@localhost/hermes?sslmode=disable"
./hermes

Docker (recommended)

# Hermes + Postgres (full stack)
docker compose up -d

# Hermes only (no DB; in-memory aggregation only)
docker build -t hermes . && docker run --rm -p 8080:8080 -e DB_DSN= hermes

• Hermes is at http://localhost:8080 (events, metrics, aggregates, pprof).
• Postgres is at localhost:5432 (user/pass/db: hermes/hermes/hermes). Override with env in docker-compose.yml or a .env file.
• Load test from the host: BASE_URL=http://localhost:8080 k6 run scripts/loadtest-k6.js

Endpoints

Path	Method	Description
/events	POST	Ingest event (JSON). Returns 202 Accepted or 503 when queue full.
/aggregates	GET	JSON snapshot of in-memory counts by event_type:minute_bucket.
/metrics	GET	Prometheus metrics (events/sec, queue length, batch size, flush latency, circuit breaker, etc.)
/debug/pprof/*	GET	CPU, heap, goroutine, block profiles

Event payload

{
  "id": "unique-id",
  "user_id": "user-123",
  "type": "song_played",
  "timestamp": 1700000000,
  "payload": {}
}

Valid type: trip_started, song_played, ad_clicked, location_update.

Configuration (env)

Variable	Default	Description
HTTP_ADDR	:8080	Listen address
QUEUE_SIZE	100000	Bounded queue capacity
WORKER_COUNT	NumCPU*4	Worker pool size
BATCH_SIZE	1000	Flush when buffer reaches this
FLUSH_INTERVAL	100ms	Max time before flush
DB_DSN	(empty)	Postgres DSN; if empty, DB writes are no-op
MAX_OPEN_CONNS	100	DB pool size
BACKPRESSURE_MODE	strict	strict (503 when full) or block
SHUTDOWN_TIMEOUT	30s	Graceful shutdown deadline

Load testing

# k6 (install: https://k6.io)
k6 run scripts/loadtest-k6.js
k6 run --vus 2000 --duration 120s scripts/loadtest-k6.js

# Vegeta (install: go install github.com/tsenart/vegeta@latest)
./scripts/loadtest-vegeta.sh
RATE=10000 DURATION=60s ./scripts/loadtest-vegeta.sh

Architecture (summary)

POST /events → Ingestion (validate, parse) → Bounded queue
                                                    ↓
Worker pool (fixed goroutines) ← pull events ← Queue
       ↓
Local buffer → flush on batch size or 100ms → DB bulk insert
       ↓
Sharded in-memory aggregator (event_type:minute_bucket) → optional flush

• No goroutine per request; fixed worker pool.
• Bounded memory: capped queue, sync.Pool for events, batch writes.
• Backpressure: 503 when queue full (strict mode).
• Graceful shutdown: drain queue, flush buffers, then exit.
• Circuit breaker: after 3 consecutive DB write failures, stop calling DB for 30s, then one retry (half-open); on success, close again. Batches dropped while open; hermes_db_circuit_open_rejects_total in Prometheus.

See ENGINEERING.md for the full RFC-style spec.