Message Queues
RabbitMQ, AWS SQS/SNS; point-to-point, fan-out, delivery guarantees, DLQs
Message queues decouple producers from consumers, enabling asynchronous processing, load leveling, and fault tolerance. RabbitMQ implements AMQP with flexible exchange types (direct, fanout, topic, headers) and delivery acknowledgments. Amazon SQS is a managed queue with at-least-once delivery and optional FIFO ordering; SNS is a managed pub/sub service that fans out to SQS, HTTP, Lambda, and email. Dead Letter Queues (DLQs) capture messages that fail processing N times, preventing poison pills from blocking the queue indefinitely.
Key Points
- Point-to-point (queue): one producer, one consumer group — each message is delivered to exactly one consumer; work queue pattern for distributing tasks among workers.
- Fanout (exchange/topic): one message delivered to multiple consumers simultaneously — RabbitMQ fanout exchange, SNS topic → multiple SQS subscriptions, Kafka topic → multiple consumer groups.
- At-most-once delivery: fire-and-forget; message deleted from queue before processing — simple, no duplicates, but messages can be lost on consumer crash.
- At-least-once delivery: message retained in queue until consumer sends ACK; consumer must be idempotent because crashes before ACK cause redelivery — most queues use this model.
- Exactly-once delivery: requires coordination (Kafka transactions, SQS FIFO + deduplication) — highest overhead; pseudo-exactly-once is more practical (at-least-once + idempotent consumer).
- SQS visibility timeout: after a consumer receives a message, it becomes invisible to others for N seconds (default 30); if not deleted within that window, it reappears — set timeout ≥ max processing time.
- Dead Letter Queues: after maxReceiveCount failed deliveries, SQS moves the message to the DLQ; configure CloudWatch alarm on DLQ depth for alerting; DLQ messages retain original metadata for debugging.
- RabbitMQ message TTL and queue TTL: per-message TTL (expiration property) or per-queue TTL (x-message-ttl argument); expired messages go to DLX (Dead Letter Exchange) for inspection.
| Delivery Guarantee | Description | Consumer Requirement | Examples |
|---|---|---|---|
| At-most-once | Message sent once; may be lost on crash or failure | None (fire-and-forget) | UDP, SNS default, MQTT QoS 0 |
| At-least-once | Message delivered one or more times; retried until ACKed | Must be idempotent (handle duplicates) | SQS, RabbitMQ, Kafka (auto-commit off) |
| Exactly-once | Message delivered precisely once, no duplicates | Idempotency built into infra layer | Kafka transactions, SQS FIFO + dedup ID |
| Effectively-once | At-least-once + idempotent consumer logic | Idempotent consumer (practical approach) | Most production systems use this model |
Real-World Example
Amazon.com processes orders through SQS to decouple the order service from inventory, billing, and shipping; FIFO queues ensure per-order sequencing. Slack processes billions of real-time notifications through a custom message queue system built on Kafka, routing each notification type (push, email, in-app) to separate consumer fleets.