Skip to content

refactor: refactor expired trash items job to bullmq#1000

Merged
apsantiso merged 1 commit intomasterfrom
feat/refactor-job-to-bullmq
Mar 24, 2026
Merged

refactor: refactor expired trash items job to bullmq#1000
apsantiso merged 1 commit intomasterfrom
feat/refactor-job-to-bullmq

Conversation

@apsantiso
Copy link
Copy Markdown
Collaborator

@apsantiso apsantiso commented Mar 20, 2026

  • Jobs module is not initialized unless envar EXECUTE_JOBS = true (we can remove the check from all tasks)
  • Expired trash items cronjob refactored to bullMQ

How it works

The trash cleanup job was refactored from a plain NestJS cron task (with a Redis lock) to a BullMQ-backed queue.

The flow is split into two classes:

TrashCleanupScheduler — runs every 10 minutes via @Cron. It queries all tiers that have a TrashRetentionDays feature limit configured and enqueues one BullMQ job per tier into the trash-cleanup queue.

TrashCleanupProcessor — a @Processor (BullMQ worker) with concurrency: 2, meaning up to 2 tier jobs run in parallel at a time. For each job it:

  1. Fetches the retention limit for the tier and computes the cutoff date. (so if a job is retried, it has a cutoffDate updated)
  2. Skips if the cutoff is before the first deployment date (period of grace we're giving to all users).
  3. Creates a job_executions record via startJob.
  4. Deletes expired files and folders in batches of 100, looping until a partial batch is returned.
  5. Marks the execution as completed or failed accordingly.

Duplicate job prevention

When the scheduler enqueues each tier it sets a deterministic jobId:

jobId: trash-cleanup:tier:${tier.id}

BullMQ silently ignores queue.add calls for a jobId that already exists in the queue (waiting or active). This means if the cron fires again before the previous run finishes, no duplicate job is created for that tier.

Env variables

REDIS_JOBS_CONNECTION_STRING=redis://@:

This is the Redis instance BullMQ uses for the queue. It is separate from the existing REDIS_CONNECTION_STRING used for general caching.

@sonarqubecloud
Copy link
Copy Markdown

@apsantiso apsantiso changed the title feat: refactor expired trash items job to bullmq refactor: refactor expired trash items job to bullmq Mar 20, 2026
@apsantiso apsantiso requested a review from sg-gs March 20, 2026 11:54
@apsantiso apsantiso marked this pull request as ready for review March 20, 2026 11:55
@apsantiso apsantiso requested a review from jzunigax2 as a code owner March 20, 2026 11:55
Copy link
Copy Markdown
Member

@sg-gs sg-gs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Configuring this on Monday, to avoid surprises haha

@sg-gs
Copy link
Copy Markdown
Member

sg-gs commented Mar 23, 2026

REDIS_JOBS_CONNECTION_STRING environment variable has been added on production @apsantiso

@apsantiso apsantiso merged commit 746634a into master Mar 24, 2026
16 of 17 checks passed
@apsantiso apsantiso deleted the feat/refactor-job-to-bullmq branch March 24, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants