Task Queue
Redis Task Queue
Build reliable task queues and job processing systems with Redis Lists and Streams. Handle background jobs, delayed tasks, and work distribution.
Real-World Business Scenario
An image processing service receives 10,000 upload requests per hour. Processing each image (resize, compress, watermark) takes 2-5 seconds. Handling this synchronously would timeout HTTP requests. A Redis-backed task queue decouples upload from processing: the API enqueues jobs instantly, and worker processes consume them at their own pace. Failed jobs can be retried without data loss.
Architecture Diagram
API Server (Producer)
↓ LPUSH job to queue
┌──────────────────────────────────────┐
│ Redis │
│ ┌──────────────────────────────────┐ │
│ │ LIST: queue:images │ │
│ │ [job3] [job2] [job1] │ │
│ ├──────────────────────────────────┤ │
│ │ LIST: queue:images:processing │ │
│ │ (jobs being worked on) │ │
│ ├──────────────────────────────────┤ │
│ │ LIST: queue:images:failed │ │
│ │ (dead letter queue) │ │
│ └──────────────────────────────────┘ │
└──────────────────────────────────────┘
↓ BRPOPLPUSH (atomic dequeue)
Worker 1 | Worker 2 | Worker 3
Key Commands Explained
Performance Analysis
LPUSH/BRPOP: O(1) operations. Redis can enqueue/dequeue 100K+ jobs per second on a single instance.
BRPOP blocks efficiently — no CPU wasted on polling. The connection idles until a job arrives.
Memory: 1K job payload × 100K queued jobs = ~100MB. Redis handles millions of queued jobs easily.
BRPOPLPUSH provides at-least-once delivery guarantee. Combined with idempotent workers, this ensures reliable processing.
Common Pitfalls
Using RPOP instead of BRPOP: RPOP in a tight loop wastes CPU and network. BRPOP blocks efficiently and reduces Redis load.
No processing list: If a worker crashes after RPOP, the job is lost forever. Always use BRPOPLPUSH to keep a backup in the processing list.
Unbounded queue growth: If producers outpace consumers, the queue grows until Redis runs out of memory. Monitor LLEN and set up alerts.
No dead letter queue: Silently dropping failed jobs makes debugging impossible. Always move failures to a separate list for inspection.
Best Practices
Use BRPOPLPUSH for reliable queue processing with crash recovery.
Implement exponential backoff for retries on failed jobs.
Monitor queue depth (LLEN) and set up auto-scaling triggers.
Add job metadata: timestamp, retry count, priority level.
Use separate queues for different priority levels (queue:high, queue:low).
Consider Redis Streams (XADD/XREADGROUP) for consumer groups and message acknowledgment in complex scenarios.
Runnable Demo
Redis Demo
Click "Step" or "Run All" to execute commands...
▌