[Feature] Resource usage tracking per organization and project

[Polliog]

Feature Description

Introduce a generic resource usage tracking system that records metering events (event type, quantity, timestamp, organization, project) for key platform operations. The system should be storage-agnostic, queryable for aggregations, and exposed via dashboard widgets so users can see what they consume.

Problem/Use Case

Today there is no first-class way to answer questions like "how many logs did this project ingest last month?" or "which organization is generating the most webhook deliveries?". Operators running self-hosted Logtide want this for capacity planning and cost attribution. Beyond the OSS use case, this also serves as the foundation that downstream consumers (e.g. a hosted service built on top of @logtide/backend) can extend for billing and quota enforcement, without having to patch the core.

Proposed Solution

A metering module exposing a single API:

metering.record({
  type: 'logs.ingested.bytes' | 'logs.ingested.events' | 'queries.executed' |
        'alerts.evaluated' | 'webhooks.delivered' | 'storage.snapshot',
  quantity: number,
  organizationId: string,
  projectId?: string,
  metadata?: Record<string, unknown>,
})

Events are written asynchronously to avoid impacting the request path. Aggregations (per-org, per-project, per-period) are exposed via a MeteringService and surfaced in a "Usage" dashboard section showing ingestion volume, storage occupancy, query count, and webhook deliveries over configurable time ranges.

Recording sites (initial set):

• ingestion/service.ts after reservoir.ingest() -> logs.ingested.bytes and logs.ingested.events
• Search/query handlers -> queries.executed
• Sigma rule evaluation worker -> alerts.evaluated
• Outbound webhook dispatcher -> webhooks.delivered
• Periodic job (hourly) -> storage.snapshot per project from reservoir's storage stats

Alternatives Considered

• Hardcoding usage queries against logs and logs_hourly_stats. Works for ingestion volume but doesn't generalize to non-log events (queries, webhooks, alerts), and ties usage reporting to the active storage adapter.
• Logging usage events as regular Logtide log events. Self-referential and skews the user's own log volume. Bad signal-to-noise and bad UX.
• Skipping this entirely until needed. Would require touching every recording site again later and would mean the hosted/cloud path has to fork the core. Better to land the primitive once.

Implementation Details (Optional)

Two storage options to evaluate:

1. TimescaleDB hypertable metering_events with continuous aggregates per (org, project, type, hour). Natural fit for time-series, retention, and aggregation. Adds a TimescaleDB-only feature, which is acceptable given it's already the default and other reservoir adapters can implement their own version.
2. Plain PostgreSQL table with manual cleanup job. Storage-engine-agnostic, simpler, but loses continuous aggregates.
   Leaning toward option 1 with a fallback rollup table for non-Timescale deployments.

Other notes:

• Recording must be non-blocking. Use a small in-process buffer flushed every N seconds or N events to a single batched insert. Loss tolerance is acceptable: this is metering, not audit.
• Schema should include a metadata JSONB column so new event types can carry context (e.g. query_duration_ms, webhook_url_hash) without migrations.
• Expose a read API GET /api/usage?orgId=...&from=...&to=...&groupBy=type|project|day that the dashboard widget consumes.
• No quota enforcement in this issue — just measurement.

Priority

• Critical - Blocking my usage of Logtide
• High - Would significantly improve my workflow
• Medium - Nice to have
• Low - Minor enhancement

Target Users

• Self-hosted operators who want visibility into per-project consumption
• Teams running Logtide for multiple internal customers and needing chargeback data
• Downstream platforms built on @logtide/backend that need usage data as input for billing or quota systems

Contribution

• I would like to work on implementing this feature