Skip to content

README.md

Latest commit

 

History

History
196 lines (138 loc) · 15.1 KB

README.md

File metadata and controls

196 lines (138 loc) · 15.1 KB

Queen MQ - Partitioned Message Queue backed by PostgreSQL

Unlimited ordered partitions that never block each other. Consumer groups, replay, transactional delivery — ACID-guaranteed.

License Node Python Go PHP C++ libuv libpq uWebSockets

📚 Complete Documentation • 🚀 Quick Start • ⚖️ Comparison

Queen Logo

Version 0.14.0 — the biggest release yet. New dynamic libqueen loop, rewritten push / pop / ack / stats stored procedures, maxPartitions on all clients, new frontend. Benchmarked on real hardware: 104k msg/s push (batch=100), 165k msg/s fan-out across 10 consumer groups, pop throughput +80–90% vs 0.12 under partition contention, server RSS under 52 MB at peak, zero message loss across 1.6 billion events. Full benchmark results →

Queen MQ is a partitioned message queue backed by PostgreSQL, built with uWebSockets, libuv, and libpq async API. It features unlimited FIFO partitions that process independently, consumer groups with replay, transactional delivery, tracing, and ACID-guaranteed durability — all in a single stateless binary alongside the Postgres you already run. Version 0.14.0 sustains 104k msg/s push and 165k msg/s fan-out on a 32-core host, with a broker RSS under 52 MB.

See examples/base.js for a complete (push, consume, transactionally ack and push to another queue) example. An experimental PostgreSQL extension version is also available at pg_qpubsub.

Why Queen?

Born at Smartness to power Smartchat, Queen solves a unique problem: unlimited FIFO partitions where slow processing in one partition doesn't block others.

Perfect for:

  • One ordered lane per entity, no preallocation — 10,000 partitions cost index rows, not 10,000 commit-log files. A partition is created on first push; a slow consumer on one partition never stalls another.
  • Transactional integration with PostgreSQL — BEGIN; INSERT order; queen.push(...); COMMIT; in a single PG transaction. The transactional-outbox pattern is built in.
  • Fan-out with fairness — consumer groups each get a full copy of every message; the adaptive engine keeps delivery fair across groups at sub-linear CPU cost.
  • 52 MB broker at 104k msg/s — no JVM, no Erlang, no cluster to operate. One Docker container plus your existing Postgres.
  • Replay and DLQ — rewind any consumer group to any timestamp; failed messages surface in a per-queue dead-letter queue automatically.
  • Zero message loss, verified — 1.6 billion events across the benchmark suite, zero lost, zero duplicates.

Quick Start

Create a Docker network and start PostgreSQL and Queen Server:

# Create a Docker network for Queen components
docker network create queen

# Start PostgreSQL
docker run --name qpg --network queen -e POSTGRES_PASSWORD=postgres -p 5433:5432 -d postgres

# Wait for PostgreSQL to start
sleep 2

# Start Queen Server
docker run -p 6632:6632 --network queen -e PG_HOST=qpg -e PG_PORT=5432 -e PG_PASSWORD=postgres -e NUM_WORKERS=2 -e DB_POOL_SIZE=5 -e SIDECAR_POOL_SIZE=30 smartnessai/queen-mq:0.14.0

Then in another terminal, use cURL (or the client libraries) to push and consume messages

Push message:

curl -X POST http://localhost:6632/api/v1/push \
  -H "Content-Type: application/json" \
  -d '{"items": [{"queue": "demo", "payload": {"hello": "world"}}]}'

that returns something like:

[{"message_id": "...", "status": "queued", ...}]

Consume message:

curl "http://localhost:6632/api/v1/pop/queue/demo?autoAck=true"

that returns something like:

{"messages": [{"data": {"hello": "world"}, ...}], "success": true}

Then go to the dashboard (http://localhost:6632) to see the messages and the status of the queue.

Documentation

📚 Complete Documentation

Getting Started

Client Libraries

Structure of the repository

The repository is structured as follows:

  • lib: C++ core queen library (libqueen), implementing libuv loops, sql schema and procedures
  • server: Queen MQ server, implementing the HTTP API that talks to the libqueen library
  • pg_qpubsub: PostgreSQL extension for using queen-mq semantics as a PostgreSQL extension
  • clients/client-js: JavaScript client library (browser and node.js)
  • clients/client-py: Python client library (python 3.8+)
  • clients/client-go: Go client library (go 1.21+)
  • clients/client-laravel: PHP / Laravel client library (php 8.1+)
  • clients/client-cpp: C++ client library (cpp 17)
  • proxy: Proxy server (authentication)
  • app: Vue.js dashboard (vue 3)
  • website: Documentation website (vitepress)
  • examples: JS client examples

Release History

JS clients from version 0.12.0 can be run inside a browser

Server Version Description Compatible Clients
0.14.0 Major release: new dynamic libqueen loop; rewritten push_messages_v3, pop_messages_v3, ack_messages_v2, and stats stored procedures; maxPartitions on all clients (JS, Python, Go, Laravel, C++); new frontend. Benchmarked on real hardware: 104k msg/s push (batch=100), 165k msg/s fan-out across 10 consumer groups, pop throughput +80–90% vs 0.12 under partition contention, 52 MB server RSS at peak, zero message loss across 1.6B events. See benchmarks → All ≥0.13.x clients work unchanged — upgrade clients to gain maxPartitions support
0.13.0 Major release: new libqueen with adaptive batch/concurrency/scheduling engine (S1 ~2x, S3 ~3x push throughput), new push_messages_v3 stored procedure, new Vue 3 dashboard, and server-stamped producerSub from the JWT on every message (closes #23) All ≥0.12.x work unchanged — 0.13.0 pop responses add a new producerSub field that older clients silently ignore. Upgrade to 0.13.0 clients only if you want typed access to producerSub (Go struct field, Python TypedDict hint)
0.12.19 Fix bug that on seek or cg delete do not deleted the watermark JS ≥0.7.4, Python ≥0.7.4
0.12.18 Improved charts and filters JS ≥0.7.4, Python ≥0.7.4
0.12.17 Improved stats JS ≥0.7.4, Python ≥0.7.4
0.12.13 Added watermark tracking for efficient wildcard POP discovery. x20 faster pop on high partition count queues JS ≥0.7.4, Python ≥0.7.4
0.12.12 Built-in database migration (pg_dump pg_restore, no temp file, selective table groups, row count validation)
0.12.10 Fixed JWKS fetch over HTTPS (cpp-httplib TLS support) JS ≥0.7.4, Python ≥0.7.4, 0.12.0 if needs to use
0.12.9 Fixed server crash (SIGSEGV) on lease renewal, added EdDSA/JWKS auth, fixed examples JS ≥0.7.4, Python ≥0.7.4, 0.12.0 if needs to use
0.12.8 Added single partition move to now to frontend JS ≥0.7.4, Python ≥0.7.4, 0.12.0 if needs to use
0.12.7 Optimized cg metadata creation for new consumer groups JS ≥0.7.4, Python ≥0.7.4, 0.12.0 if needs to use
0.12.6 Improved slow cg discovery when there are tons of partitions JS ≥0.7.4, Python ≥0.7.4, 0.12.0 if needs to use
0.12.5 Fixed cg lag calculation for "new" cg at first message JS ≥0.7.4, Python ≥0.7.4, 0.12.0 if needs to use
0.12.4 Fixed window buffer debounce behavior JS ≥0.7.4, Python ≥0.7.4, 0.12.0 if needs to use proxy auth
0.12.3 Added JWT authentication JS ≥0.7.4, Python ≥0.7.4, 0.12.0 if needs to use proxy auth
0.12.x New frontend and docs JS ≥0.7.4, Python ≥0.7.4, 0.12.0 if needs to use proxy auth
0.11.x Libqueen 0.11.0; added stats tables and optimized analytics procedures, added DB statement timeout and stats reconcile interval JS ≥0.7.4, Python ≥0.7.4
0.10.x Total rewrite of the engine with libuv and stored procedures, removed streaming engine JS ≥0.7.4, Python ≥0.7.4
0.8.0 Added Shared Cache with UDP sync for clustered deployment JS ≥0.7.4, Python ≥0.7.4
0.7.5 First stable release JS ≥0.7.4, Python ≥0.7.4

Full Release Notes →

Latest bug fixing and improvements

  • Server and clients 0.14.0: New dynamic libqueen loop. Full rewrite of the core scheduling engine — adaptive concurrency controller (TCP-Vegas-style) now drives push, pop, ack, and stats independently. Active DB connections stay at ~2.5 even with a pool of 50 under 104k msg/s peak load. Eliminates the PG deadlock mode that appeared under heavy fan-out at high partition counts on 0.12.
  • Server 0.14.0: Rewritten stored procedures. push_messages_v3, pop_messages_v3, ack_messages_v2, and stats procedures redesigned around the new loop. PG memory usage 30–70% lower for equivalent workloads vs 0.12. Pop throughput +80–90% under partition contention.
  • Clients 0.14.0: maxPartitions on all clients. JS, Python, Go, Laravel, and C++ clients expose maxPartitions on queue creation and configuration.
  • Server 0.14.0: New frontend. Redesigned dashboard for the new stats model.
  • Server 0.13.0: New libqueen with adaptive engine. Per-worker push/ack drain factored into three independently-tuned concerns — batching, concurrency, scheduling — glued by an event-driven orchestrator. Fixes two long-standing bottlenecks: per-commit overhead amortization on small-batch workloads, and the single-slot-per-drain cap on high-fanout workloads. Perf harness numbers: S1 ~6.2k → ~13k pg_ins/s, S3 ~4.7k → ~20k pg_ins/s, PG pinned instead of idle. Design notes in cdocs/LIBQUEEN_IMPROVEMENTS.md.
  • Server 0.13.0: New push stored procedure. queen.push_messages_v2 rewritten around a temp-table + batched-insert pipeline that feeds cleanly into the adaptive engine. HTTP contract (queued/duplicate/failed) unchanged.
  • Server 0.13.0: New Vue 3 dashboard. Reworked queues, analytics, DLQ management, and maintenance-mode views. Served by the same C++ acceptor at /.
  • Server 0.13.0: Added server-stamped producerSub to close the impersonation vector from GitHub issue #23. When JWT auth is enabled the server stamps the validated sub claim on every pushed message; clients cannot set this field and it is exposed on pop responses and admin message APIs. Schema migration is additive and metadata-only (no table rewrite), safe on tables with millions of rows.
  • Clients 0.13.0: All clients (JS, Python, Go, Laravel, C++) expose producerSub on popped messages; Go adds a typed Message.ProducerSub field.
  • Server 0.12.19: Fix bug where seek or cg delete did not delete the watermark.
  • Server 0.12.13: Added watermark tracking for efficient wildcard POP discovery — x20 faster pop on high partition count queues.
  • Server 0.12.12: Added built-in database migration — stream pg_dump | pg_restore directly from the dashboard, no temp file, selective table groups, row count validation.
  • Clients 0.12.2: Added custom headers option to JS, Python, and Go clients for API gateway authentication.
  • Server 0.12.9: Fixed server crash (SIGSEGV) on lease renewal; added native EdDSA and JWKS JWT authentication (auto-discovery via JWT_JWKS_URL).
  • Server 0.12.3: Added JWT authentication.

License

Queen MQ is released under the Apache 2.0 License.

Built with ❤️ by Smartness

Why "Queen"? Because years ago, when I first read "queue", I read it as "queen" in my mind.