Background jobs and queues

Not everything belongs in the request

A request should finish fast, ideally in well under a second. Some work is slow or depends on other services that may be down. If you do it inside the request, the user waits, and if it fails, the whole request fails.

• Sending a welcome email or an invoice email
• Making a PDF ticket or a salary slip
• Resizing and compressing uploaded photos
• Building a monthly sales report
• Calling a slow partner API, like a courier tracking service

Queues, producers and workers

A queue is a list of jobs waiting to be done, in order. A job is a small message that says what to do, like { type: "welcome-email", userId: 42 }. The producer is the code that adds jobs (usually your API). The consumer, or worker, is a separate process that takes jobs from the queue and does them. You can run more workers when there is more work.

import { Queue, Worker } from 'bullmq';
const connection = { host: 'localhost', port: 6379 };

// Producer: inside the signup API
const emails = new Queue('emails', { connection });
await emails.add('welcome', { userId: 42 }, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000 },
});

// Worker: a separate process
new Worker('emails', async (job) => {
  await sendWelcomeEmail(job.data.userId);
}, { connection });

Retries and exponential backoff

Jobs fail for temporary reasons: the email service is busy, the network blinked. So workers retry. Exponential backoff means each wait is double the one before: 1 second, 2, 4, 8, and so on, up to a maximum. This gives the other service time to recover instead of hammering it. Many systems also add a little random "jitter" so thousands of jobs do not retry at the same moment.

Retries mean a job can run more than once. An idempotency key is a unique id for one piece of work, like "invoice-email:order-9001". Before doing the work, the worker checks if that key is already done. If it is, it skips. Then a retry never sends two invoices or charges a card twice.

A dead-letter queue (DLQ) is a separate queue for jobs that failed every attempt. They are kept there, not lost, so a developer can see the error, fix the cause and run them again.

Scheduled jobs

Some jobs run on a timetable, not because of a request: clean old sessions every night, send EMI reminders every morning, build a report every Monday. These are scheduled jobs, often written with cron syntax: five fields for minute, hour, day of month, month and day of week.

0 2 * * *      every day at 02:00
*/15 * * * *   every 15 minutes
0 9 * * 1      every Monday at 09:00

• BullMQ: a popular Node.js job queue that uses Redis.
• RabbitMQ: a message broker (a server whose job is passing messages between programs).
• Amazon SQS: a managed queue service on AWS, with built-in dead-letter queues.
• Celery: the common choice for background jobs in Python and Django apps.