Message Queues

A durable, intermediary buffer that decouples producers (who enqueue messages) from consumers (who dequeue and process them). Classic brokers (RabbitMQ, SQS) deliver-then-delete per message; log-based systems (Kafka) keep an ordered, replayable log that consumers read by offset. Both let work happen asynchronously and absorb load.

Synchronous coupling is fragile: if A calls B directly and B is slow or down, A blocks or fails. A queue breaks that link: A enqueues and continues, B processes at its own pace. Queues also absorb bursts (a traffic spike fills the queue instead of melting the consumer), enable async work (resize the image after the upload returns), and let you scale consumers independently of producers.

Producers publish messages to a queue or topic. The broker persists them (so they survive a crash). Consumers pull (or are pushed) messages, process them, and acknowledge. Unacked messages are redelivered (at-least-once delivery). Failed messages eventually land in a dead-letter queue. Log-based brokers like Kafka instead keep messages on disk by offset, letting multiple consumer groups read the same stream and replay history.

• Decouples producers and consumers, so either side can scale, deploy, or fail independently
• Absorbs traffic spikes: the queue buffers load the consumer couldn't take synchronously
• Enables async background work and natural retries via redelivery + dead-letter queues

• At-least-once delivery is the norm, so consumers must be idempotent or you'll double-process
• Adds a moving part with its own scaling, monitoring, and backpressure concerns
• Ordering and exactly-once are expensive and often only per-partition; global ordering is usually a fantasy

• WhatsApp →

  Messages to offline users are queued per-recipient and delivered when the device reconnects

• Notification System →

  Send requests are enqueued and workers fan them out to providers, retrying failures and dead-lettering the dead ones

• Apache Kafka →

  The log-based queue itself: partitioned, replicated, replayable, consumed by independent consumer groups

• Assume at-least-once delivery and make consumers idempotent. Exactly-once is either a lie or extremely expensive, so design for duplicates instead.
• A queue is not a fix for a permanently slow consumer; it just hides the backlog. If producers outpace consumers forever, the queue grows unbounded. Monitor queue depth and consumer lag.
• Ordering guarantees are usually only per-partition/per-key, not global. If you need order, partition by the key that must stay ordered (e.g. per conversation).
• Always wire up a dead-letter queue. Without one, a single poison message can block a partition or get retried forever.