Message Queues vs Event Streams
Queues and logs look similar and behave completely differently. Which to reach for, how backpressure actually works, and what to do with messages that will never succeed.
Covers: Queue vs log semantics, decoupling, backpressure, dead-letter queues, ordering, fan-out, when not to use a queue
Adding a queue is the standard answer to “this is too slow” - and it is often right. But a queue is not a magic latency eraser: it converts a slow synchronous operation into a fast one plus a promise, and that promise introduces ordering questions, duplicate delivery, and a brand new failure mode called the queue is 4 million messages deep.
30-second answer
A queue holds work: a message is delivered to one consumer, acknowledged, and deleted. It is a job dispatcher, and it scales by adding competing consumers. A log holds an ordered, durable, replayable record: messages are appended and retained for a configured period, and each consumer group tracks its own offset independently. Many groups can read the same data without affecting one another, and a new consumer can replay history from the beginning. Choose a queue for task distribution, and a log when several independent consumers need the same events, or when replay matters.
| Queue (SQS, RabbitMQ) | Log (Kafka, Kinesis, Pulsar) | |
|---|---|---|
| Message lifetime | Deleted after acknowledgement | Retained for a configured period |
| Consumers | Compete for messages | Independent groups, each with its own offset |
| Replay | No | Yes - reset the offset |
| Ordering | Best effort, or FIFO with reduced throughput | Strict per partition |
| Fan-out | One queue per consumer, or a topic exchange | Native - add a consumer group |
| Throughput | High | Very high - sequential disk writes |
| Per-message operations | Delete, delay, change visibility | None - you can only advance the offset |
| Best for | Task distribution, background jobs | Event sourcing, analytics, multi-consumer streams |
What a queue actually buys you
- Latency shifting - return to the user in 50 ms instead of 4 s by doing the slow part later. The work still takes 4 s; the user just is not waiting for it.
- Load levelling - a 10× traffic spike becomes a deeper queue rather than a saturated database. The queue absorbs the burst and consumers drain at a steady rate.
- Decoupling - the producer does not know or care who consumes, or whether the consumer is currently deployed.
- Retry with isolation - a failing consumer retries without the user seeing anything, and without blocking the request path.
Backpressure: what happens when consumers cannot keep up
If the producer rate exceeds the consumer rate for any sustained period, queue depth grows without bound and end-to-end latency grows with it. Queueing theory again: the queue is stable only while utilisation ρ = λ/μ < 1, and there is no configuration that fixes ρ > 1.
| Response | Mechanism | Trade-off |
|---|---|---|
| Scale consumers | Autoscale on queue depth or consumer lag | Only works if the downstream can also take the load |
| Shed load | Reject or drop low-priority messages at the producer | Work is lost - must be acceptable for that message class |
| Rate-limit producers | Block or throttle upstream | Pushes the problem to the caller, which is often correct |
| Priority queues | Separate queues per priority class | Low priority can starve indefinitely |
| Bounded queue | Reject new messages when full | Fails fast and visibly rather than degrading silently |
Dead-letter queues
A message that fails repeatedly - malformed payload, a referenced record that was deleted, a bug - must not be retried forever. After N attempts it moves to a dead-letter queue, which preserves it for inspection while unblocking everything behind it.
- Always configure one. Without a DLQ, a single poison message can block an ordered partition indefinitely.
- Alert on non-zero depth. A DLQ nobody watches is a silent data-loss channel.
- Store the failure reason and the full original message, so the message can be understood and replayed later.
- Build a replay path - after fixing the bug you need a supported way to push messages back onto the main queue, ideally with a dry run.
Say these points
- Queue = who does this work; log = what happened, replayably, for many consumers.
- Queues shift latency and level load; they do not make work faster.
- Every mainstream broker is at-least-once - consumers must be idempotent.
- Backpressure is unavoidable when λ > μ; scale, shed, throttle or bound.
- Alert on oldest-message age, and always configure and watch a DLQ.
Avoid these mistakes
- Using a queue for work the user is synchronously waiting on anyway.
- No dead-letter queue, so one poison message blocks a partition.
- Alerting on queue depth instead of message age.
- Assuming a queue guarantees ordering with parallel consumers.
- No replay path, making the DLQ a place messages go to die.
Expect these follow-ups
- Your queue has 4 million messages and is growing. Walk me through the response.
- How do you guarantee per-user ordering with 50 parallel consumers?
- When is a queue the wrong tool entirely?
Showing 1 of 1 questions for message-queues-and-event-streams.
Check your understanding
3 questions · no sign-up, nothing stored