Message Queues — The Complete Guide for System Design

Everything you need to know about message queues, from zero to interview-ready.
Covers: What, Why, How, Patterns, Kafka, RabbitMQ, SQS, BullMQ, DLQ, Backpressure,
Idempotency, Outbox Pattern, Saga, Event-Driven Architecture, Monitoring, and Interview Playbook.

Table of Contents

1. Why Message Queues Exist — The Core Problem
2. What Is a Message Queue? — The Mental Model
3. Key Components — Producers, Consumers, Brokers
4. Two Fundamental Models — Point-to-Point vs Pub/Sub
5. How It Works Under the Hood
6. Delivery Guarantees — The Triangle of Trust
7. Ordering Guarantees
8. Dead Letter Queues (DLQ)
9. Backpressure — When Producers Outpace Consumers
10. Idempotency — The Non-Negotiable Pattern
11. Partitioning — Scaling Horizontal
12. Consumer Groups — Parallelism at Scale
13. The Big Three — Kafka vs RabbitMQ vs SQS
14. BullMQ — The Developer's Job Queue
15. The Dual Write Problem & Transactional Outbox
16. Saga Pattern — Distributed Transactions Without 2PC
17. Event-Driven Architecture (EDA)
18. Event Sourcing & CQRS
19. Anti-Pattern: Using Databases as Message Queues
20. When to Use a Queue — The 4 Signals
21. When NOT to Use a Queue
22. Monitoring & Observability
23. Interview Playbook — What to Say and When
24. Real-World Companies & Their Choices
25. Complete Comparison Table
26. Sources & Further Reading

1. Why Message Queues Exist — The Core Problem

The Problem: Synchronous Communication Breaks

Imagine you're building an e-commerce app. When a user places an order, your system needs to:

1. Save the order
2. Process payment
3. Update inventory
4. Send confirmation email
5. Notify the warehouse
6. Update analytics

In a synchronous world, your order service calls each of these one by one. Here's what happens:

User clicks "Buy"
    ↓
Order Service calls Payment Service (200ms)
    ↓
Order Service calls Inventory Service (300ms)  ← Inventory is slow today
    ↓
Order Service calls Email Service (100ms)
    ↓
Order Service calls Analytics Service (150ms)
    ↓
Response to user: ~750ms minimum

What goes wrong:

• Latency: User waits 750ms+ staring at a spinner
• Fragility: If Email Service crashes, the entire order fails
• Bottleneck: Your Order Service is stuck waiting, unable to handle other requests
• Cascading failure: If Payment Service is slow, everything behind it is blocked
• Traffic spikes: Flash sale → 10,000 orders/second → your servers can only handle 200/second → 9,800 orders fail

The Solution: Put a Queue in Between

Instead of calling services directly, your order service drops a message into a queue and returns immediately:

User clicks "Buy"
    ↓
Order Service saves order
    ↓
Order Service writes to Queue: "Order #456 needs processing"
    ↓
Response to user: "Order placed! We'll email you the details."
    ↓
(Background) Email Worker picks up message → sends email
(Background) Inventory Worker picks up message → updates stock
(Background) Analytics Worker picks up message → logs event

What this gives you:

• Low latency: User gets response in 50ms, not 750ms
• Isolation: If Email Service crashes, orders still process
• Buffer: Traffic spike? Queue absorbs it. Workers process at their own pace
• Scalability: Add more workers to handle more load, independently

Think of it like a restaurant:

• The waiter (producer) takes your order and puts it on the ticket rail (queue)
• The cook (consumer) grabs tickets when ready and makes food
• The waiter doesn't stand there waiting — they serve other tables
• If 50 orders come in at once, they queue up. The kitchen processes them at its own pace
• No order is lost. No customer is stuck waiting forever.

2. What Is a Message Queue? — The Mental Model

A message queue is a buffer that sits between a producer (the thing creating work) and a consumer (the thing doing work). It stores messages temporarily until a consumer picks them up.

The one-sentence definition:

A message queue is a system that stores messages sent by one service (producer) and delivers them to another service (consumer), allowing them to communicate without being online at the same time.

Key properties:

• Asynchronous: Producer doesn't wait for consumer to finish
• Decoupled: Producer doesn't know (or care) who consumes the message
• Buffered: Messages accumulate if consumers are slow
• Durable: Messages can survive crashes (depending on configuration)

The pizza shop analogy (from your notes):

You walk into a pizza shop. The person at the counter takes your order and says "Please sit down, we'll call you when it's ready." You don't stand at the kitchen watching them make pizza. You sit down, check your phone, do whatever. When the pizza is ready, they call your number. That's a message queue. The counter is the queue. The kitchen workers are consumers. You are the producer.

3. Key Components — Producers, Consumers, Brokers

Producer (Publisher / Sender)

The service that creates messages and sends them to the queue.

Order Service  ──(produces)──>  "Order #456 created"
Payment Service ──(produces)──> "Payment of $50 processed"

Consumer (Subscriber / Worker)

The service that reads messages from the queue and processes them.

Email Worker reads "Order #456 created" → sends confirmation email
Inventory Worker reads "Order #456 created" → decrements stock

Broker (Message Broker / Queue Server)

The middleman that manages the queue. It:

• Receives messages from producers
• Stores them durably (on disk, in memory, or both)
• Routes them to the right consumers
• Handles acknowledgements, retries, dead-lettering
• Manages subscriptions, partitions, consumer groups

Examples of brokers: RabbitMQ, Kafka, Amazon SQS, ActiveMQ, NATS, Redis (with BullMQ)

Queue / Topic

The logical channel where messages live. In some systems it's called a queue (RabbitMQ, SQS), in others a topic (Kafka). The difference:

• Queue: Each message is consumed by exactly one consumer (point-to-point)
• Topic: Each message can be consumed by multiple subscribers (pub/sub)

Message

The actual data being sent. A message typically contains:

• Payload: The actual data (JSON, protobuf, etc.)
• Metadata: Headers, timestamps, IDs, routing keys
• Idempotency key: A unique identifier to prevent duplicate processing

4. Two Fundamental Models — Point-to-Point vs Pub/Sub

This is the most important architectural decision. Everything else follows from it.

Point-to-Point (Work Queue)

Producer ──> [Queue] ──> Consumer A
                    ──> Consumer B (competing)
                    ──> Consumer C (competing)

How it works:

• Each message is delivered to exactly one consumer
• Multiple consumers compete for messages (competing consumers)
• Once a consumer ACKs the message, it's deleted from the queue
• Load is automatically distributed across consumers

When to use:

• Task distribution: "Process this image", "Send this email", "Resize this photo"
• Background jobs: Each job should be done once by one worker
• Work queues: You want parallelism but not duplication

Example:

Email Queue:
  [send-welcome]  → Worker A picks it up
  [send-receipt]  → Worker B picks it up
  [send-reset]    → Worker C picks it up

Each email is sent once. Workers divide the work.

Publish-Subscribe (Pub/Sub / Fan-Out)

Producer ──> [Topic] ──> Subscriber A (gets ALL messages)
                   ──> Subscriber B (gets ALL messages)
                   ──> Subscriber C (gets ALL messages)

How it works:

• Each message is delivered to all subscribers
• Each subscriber gets its own copy of the message
• Subscribers are independent — adding a new one doesn't affect the producer
• Messages can be retained for replay

When to use:

• Event broadcasting: "Order placed" → email, analytics, billing, notifications
• Multiple systems reacting to the same event
• Audit logs, real-time dashboards
• Any scenario where one event triggers multiple independent workflows

Example:

Order Event:
  "OrderPlaced" → Email Service (sends confirmation)
                 → Analytics Service (logs event)
                 → Inventory Service (updates stock)
                 → Fraud Detection (checks for abuse)

Every service gets the same event independently.

The Critical Distinction

Kafka's hybrid model:Kafka uses topics with partitions. Within a consumer group, each partition goes to one consumer (point-to-point). But multiple consumer groups can subscribe to the same topic independently (pub/sub). This means one Kafka topic serves as both a work queue and an event log.

5. How It Works Under the Hood

The Message Lifecycle

RabbitMQ (Traditional Broker):

1. Producer sends message to Exchange
2. Exchange routes message to Queue(s) based on bindings
3. Queue stores message (READY state)
4. Broker pushes message to Consumer
5. Consumer processes message
6. Consumer sends ACK back to broker
7. Broker deletes message from queue

States: READY → DELIVERED → UNACKED → (ACK) → DELETED

Kafka (Distributed Log):

1. Producer sends message to Topic
2. Kafka appends message to partition (based on partition key)
3. Message sits in the log permanently (until retention expires)
4. Consumer reads message from its current offset
5. Consumer processes message
6. Consumer commits offset (advances its position)
7. Message STILL EXISTS in the log (not deleted)

States: Always in the log. Consumer just moves its pointer (offset).

Acknowledgements (ACKs)

This is how queues know a message was successfully processed.

Scenario without ACKs:

1. Worker picks up message "Process payment for order #456"
2. Worker starts processing
3. Worker crashes halfway through
4. Message is lost forever. Payment never processed. Customer never charged.

Scenario with ACKs:

1. Worker picks up message
2. Worker processes message successfully
3. Worker sends ACK: "Done, you can delete this"
4. Broker removes message

If worker crashes before ACK:

1. Worker picks up message
2. Worker crashes before completing
3. No ACK received
4. Broker assumes message wasn't processed
5. Message is redelivered to another worker
6. Nothing is lost

SQS's Visibility Timeout:SQS takes a different approach. When a consumer picks up a message, it becomes invisible to other consumers for a configurable window (default 30 seconds). If the consumer processes it and deletes it within that window, all good. If it doesn't (consumer crashed), the message becomes visible again and another consumer can retry it.

Consumer picks up message
    ↓
Message becomes invisible for 30 seconds
    ↓
Consumer processes + deletes within 30s → Done
    ↓
Consumer crashes → 30s expires → Message becomes visible again

6. Delivery Guarantees — The Triangle of Trust

This is one of the most important concepts in message queues. When a consumer fails to process a message, you have two choices: lose it or deliver it again (risking duplicates). Delivery guarantees define which trade-off you make.

At-Most-Once (Fire and Forget)

Producer → Queue → Consumer → (message deleted immediately)

• Broker sends the message once and doesn't retry
• If consumer crashes, message is lost
• Fastest option
• Use only when losing messages is acceptable (analytics events, metrics, logging)

Example: "Track page views" — losing a few is fine

At-Least-Once (The Industry Standard)

Producer → Queue → Consumer → (waits for ACK)
    ↓
If ACK received → message deleted
If no ACK → redeliver to another consumer

• Every message is delivered at least once, possibly more
• If consumer fails, message is retried
• No data loss, but consumers might see duplicates
• Consumers must be idempotent (processing twice = same result as processing once)
• This is what almost every production system uses

Example: "Charge $50" — if processed twice, customer gets charged $100 (BAD). So make it idempotent: "Charge $50 for order #456" — check if order #456 already charged before processing.

Exactly-Once (The Holy Grail)

Producer → Queue → Consumer → (exactly one processing)

• Each message is processed exactly once
• Sounds perfect, but extremely hard to achieve in distributed systems
• Kafka supports it with transactions, but ONLY when:
  • Input is a Kafka topic
  • Processing happens within Kafka
  • Output is a Kafka topic
  • All within the same Kafka cluster
• The moment you write to a database, call an external API, or cross cluster boundaries → you're back to at-least-once

Reality check: Don't promise exactly-once in interviews unless you can explain the mechanism and defend it. At-least-once with idempotent consumers is almost always the right answer.

Summary Table

7. Ordering Guarantees

The Ordering Problem

Imagine banking transactions:

1. Deposit $100
2. Withdraw $50

If these arrive out of order at the consumer, the withdrawal might fail (insufficient funds) even though the deposit should have covered it. Ordering matters.

RabbitMQ Ordering

• Single consumer: Strict FIFO order guaranteed
• Multiple consumers: Order can break! Consumer A might finish message 1 before Consumer B finishes message 2, even though message 1 was picked up first
• To maintain order: use a single consumer (sacrifices throughput)

Kafka Ordering

• Order is guaranteed within a partition, not across partitions
• Messages with the same partition key always go to the same partition
• You control which messages go together via the partition key

Partition Key = User ID

User 123's events:
  Partition 0: [Deposit $100] → [Withdraw $50]  (ordered!)
  
User 999's events:
  Partition 1: [Deposit $200] → [Withdraw $100]  (ordered!)

User 123's transactions are ordered. User 999's are ordered. But there's no global ordering across users — and that's fine because they're independent.

Choosing a Partition Key

Two goals that can conflict:

1. Ordering: Messages for the same entity must go to the same partition
2. Even distribution: Keys should spread work evenly across partitions

Bad example: Partition by city

• New York: 10 million messages → one partition overloaded
• Boise: 100 messages → partition sitting idle
• This creates a hot partition

Good example: Partition by user ID, order ID, or ride ID

• Distributes evenly
• Maintains per-entity ordering

The Trade-off

The key that gives you ordering might not give you the best distribution. This is worth thinking through in interviews.

8. Dead Letter Queues (DLQ)

The Poison Message Problem

A poison message is a malformed or problematic message that crashes the consumer every single time, no matter how many retries.

Example: You upload a corrupted image. The consumer tries to resize it → crashes. Tries again → crashes. Again → crashes. Without DLQ, this one bad message blocks the entire queue forever.

What a DLQ Does

Main Queue → Consumer tries 5 times → All fail
    ↓
Message moves to Dead Letter Queue
    ↓
Main queue keeps processing other messages
    ↓
Engineer investigates DLQ later
    ↓
Fixes the bug, replays messages from DLQ

Why DLQ Is Essential

1. Prevents head-of-line blocking: One bad message doesn't stop the queue
2. Observability: You can see what's failing and why
3. Replay: After fixing a bug, you can replay failed messages
4. Debugging: DLQ messages carry headers showing the original error

DLQ Best Practices

• Monitor DLQ depth: If DLQ is growing, something is systematically wrong
• Set alerts: DLQ > 20 messages → PagerDuty alert
• Carry rich metadata: Original topic, partition, offset, exception, attempt count, stack trace
• Configure DLQ for every production queue: This is not optional
• Replay with caution: Make sure consumers are idempotent before replaying

Problems DLQ Creates (Often Ignored)

1. Silent data loss: Message lands in DLQ, nobody checks, user never gets their email
2. Replay complexity: Replaying a payment message might charge twice without idempotency
3. Wrong retry count: Too low → messages go to DLQ unnecessarily. Too high → system melts from retries

Rule of thumb:

• Transient failures (network timeout, DB overload): Retry 3-10 times
• Permanent failures (invalid schema, bad payload): Direct to DLQ, no retry

9. Backpressure — When Producers Outpace Consumers

The Problem

Queue is not magic. It's a buffer, not a solution to insufficient capacity.

Producer: 300 messages/second
Consumer: 200 messages/second
Queue growth: +100 messages every second

Eventually the queue fills up and things go wrong.

Solutions

1. Scale Consumers (Auto-scaling)

• Monitor queue depth
• When it grows, spin up more consumers
• Cloud providers support auto-scaling based on queue metrics

2. Apply Backpressure to Producers

• Slow producers down
• Return errors: "System overloaded, try again later"
• Reject messages when queue is full

3. Monitor and Alert

• Set alerts on queue depth
• Alert when growth rate exceeds threshold
• Be ready to intervene

Key insight for interviews:

"A queue isn't just magic — it's a buffer, not a solution to insufficient capacity. I'd implement auto-scaling based on queue depth, and if that's not enough, apply backpressure to producers by returning 429 errors."

10. Idempotency — The Non-Negotiable Pattern

What Is Idempotency?

An operation is idempotent if running it multiple times produces the same result as running it once.

Idempotent examples:

• SET user.photo = "photo5.jpg" → Running twice = same result ✅
• CHARGE order #456 $50 → Running twice = $100 charged ❌
• SET order.status = "CONFIRMED" → Running twice = same result ✅
• INCREMENT post.count BY 1 → Running twice = count increased by 2 ❌

Why It Matters for At-Least-Once Delivery

With at-least-once delivery, consumers WILL see duplicate messages. If your consumer isn't idempotent, duplicates cause:

• Double charges
• Double inventory deductions
• Duplicate notifications
• Corrupted analytics

How to Implement Idempotency

Pattern 1: Idempotency Key

Message contains: { "orderId": "456", "amount": 50, "idempotencyKey": "evt_789" }

Consumer:
1. Check if "evt_789" exists in processed_events table
2. If yes → skip (already processed)
3. If no → process payment, store "evt_789" in processed_events

Pattern 2: Natural Idempotency

Instead of: "Deduct $50 from account"
Use: "Set account balance to $900"

Running twice → balance is still $900 ✅

Pattern 3: Check Before Acting

Before charging: Check if order #456 already has a successful payment
If yes → return existing payment
If no → process new payment

Stripe's approach (gold standard):Every API call includes an idempotency key. If the same key is sent twice, Stripe returns the original result without processing again.

The Golden Rule

In distributed systems with at-least-once delivery, idempotent consumers are not optional — they are mandatory. Every message consumer must be designed to handle duplicates safely.

11. Partitioning — Scaling Horizontal

Why Partition?

A single queue can only handle so much. When you need more throughput, you partition — split the queue into multiple independent sub-queues.

Topic: "orders"

Partition 0: [order1, order4, order7, ...]
Partition 1: [order2, order5, order8, ...]
Partition 2: [order3, order6, order9, ...]

Different workers process different partitions in parallel. Throughput scales with the number of partitions.

How Partitioning Works

1. Producer sends message with a partition key (e.g., user_id)
2. Kafka hashes the key to determine which partition gets the message
3. All messages with the same key go to the same partition
4. Within a partition, order is preserved
5. Consumers are assigned to partitions

Partition Key Selection

The partition key is analogous to choosing a shard key in a database. It matters for:

Ordering: Messages with the same key always go to the same partition, preserving per-entity order.

Distribution: Keys should spread work evenly. If one key is way more popular than others, that partition becomes a bottleneck (hot partition).

Example:

Partition key: user_id
  User 123: [login, purchase, logout] → all in Partition 0 (ordered!)
  User 456: [login, purchase] → all in Partition 1 (ordered!)

Partition key: city (BAD!)
  New York: 10M events → one partition overloaded
  Boise: 100 events → idle partition

12. Consumer Groups — Parallelism at Scale

What Is a Consumer Group?

A consumer group is a pool of workers that divide partitions among themselves.

Topic: "orders" (6 partitions)
Consumer Group: "order-processors"

Consumer 1 → Partition 0, Partition 1
Consumer 2 → Partition 2, Partition 3
Consumer 3 → Partition 4, Partition 5

How It Works

1. Each partition is assigned to exactly one consumer within a group
2. Consumers within a group don't compete — they each have their own partitions
3. Multiple consumer groups can subscribe to the same topic independently
4. Each group maintains its own offset (position in the log)

Scaling Rules

• More consumers than partitions: Extra consumers sit idle (no partition to read from)
• More partitions than consumers: Some consumers read multiple partitions
• To scale: Add more partitions AND more consumers together

6 partitions, 3 consumers → each consumer handles 2 partitions
6 partitions, 6 consumers → each consumer handles 1 partition
6 partitions, 7 consumers → 7th consumer sits idle

Multiple Consumer Groups (Pub/Sub)

Topic: "order-events"

Consumer Group "analytics":
  Consumer A → Partition 0, 1
  Consumer B → Partition 2, 3

Consumer Group "notifications":
  Consumer C → Partition 0, 1
  Consumer D → Partition 2, 3

Consumer Group "billing":
  Consumer E → Partition 0, 1, 2, 3

Each group independently reads all events. Analytics crashes? No problem — it can replay when it comes back. Notifications is unaffected.

13. The Big Three — Kafka vs RabbitMQ vs SQS

Apache Kafka — The Distributed Event Log

What it is: A distributed streaming platform built as an append-only log.

Core mental model: "I want to store and stream events at massive scale."

How it works:

• Messages are appended to a topic (split into partitions)
• Messages are retained for a configurable period (default 7 days, configurable to forever)
• Consumers pull messages at their own pace
• Consumers track their own offset (position in the log)
• Multiple consumer groups can read the same stream independently

Key characteristics:

• Throughput: Millions of messages/second
• Latency: 5-50ms (batched)
• Ordering: Per partition
• Retention: Days/weeks/indefinitely (messages survive after consumption)
• Replay: Yes — reset offset and reprocess from any point
• DLQ: Not built in (you build it yourself with retry topics)
• Operations: Complex (partitions, consumer groups, replication)

Best for:

• Event streaming, log aggregation
• Multiple systems reading the same events
• Replay capability (debugging, backfill)
• High-throughput data pipelines
• Event sourcing

Real companies: Netflix (petabytes daily), Uber (real-time pricing), LinkedIn (invented Kafka), Twitter, Spotify

RabbitMQ — The Smart Message Broker

What it is: A traditional message broker implementing AMQP protocol.

Core mental model: "I have jobs/events and workers should process them reliably."

How it works:

• Producer sends message to an Exchange
• Exchange routes message to Queue(s) based on bindings
• Broker pushes messages to consumers
• Consumer ACKs → message is deleted
• Built-in retries, DLQ, routing, priority queues

Key characteristics:

• Throughput: 10K-100K messages/second
• Latency: Very low (sub-millisecond possible)
• Ordering: Per queue (can break with multiple consumers)
• Retention: Until consumed (deleted after ACK)
• Replay: No (messages are gone after consumption)
• DLQ: Built-in (Dead Letter Exchange)
• Operations: Simple (single binary, built-in UI)

Best for:

• Task queues, background jobs
• Complex routing (direct, topic, fanout, headers exchanges)
• Request-reply patterns
• Microservice communication
• When you need per-message ACK and retry logic

Real companies: Instagram (photo processing), Reddit (karma calculations), Robinhood

Amazon SQS — The Managed Queue

What it is: AWS's fully managed message queue service.

Core mental model: "I just need a queue with zero ops."

How it works:

• Create a queue, start using it
• Standard queue: at-least-once, best-effort ordering, very high throughput
• FIFO queue: strict ordering, exactly-once processing, lower throughput
• Visibility timeout prevents duplicate processing
• Fully serverless — no clusters to manage

Key characteristics:

• Throughput: Nearly unlimited (Standard), 3K+/sec (FIFO)
• Latency: Medium (network round-trip)
• Ordering: Best-effort (Standard), strict (FIFO)
• Retention: Up to 14 days
• Replay: No
• DLQ: Built-in
• Operations: None (fully managed)

Best for:

• Simple async processing on AWS
• Decoupling Lambda functions and microservices
• When you want zero operational overhead
• Serverless architectures

Limitations:

• AWS-only (vendor lock-in)
• No replay, no multi-consumer groups
• Higher latency than co-located RabbitMQ
• 256KB max message size

Head-to-Head Comparison

Decision Framework

Need high-throughput streaming with replay?
  → Kafka

Need complex routing and task queues?
  → RabbitMQ

Need fully managed, zero ops on AWS?
  → SQS

Need message ordering?
  → Kafka (per partition) or FIFO SQS

Building event-driven microservices?
  → Kafka

Simple task queues?
  → RabbitMQ or SQS

Interview recommendation: Default to Kafka for most system design answers. It's the most versatile. Use RabbitMQ when you explicitly need complex routing or simple task queues. Use SQS when the interviewer is okay with cloud-managed solutions.

14. BullMQ — The Developer's Job Queue

What Is BullMQ?

A Redis-based background job queue, extremely popular in Node.js ecosystems.

Core mental model: "I just need background jobs inside my app."

What It Does Well

• Background jobs: email, video processing, PDF generation
• Retries with exponential backoff
• Delayed jobs / scheduling
• Multiple workers processing jobs in parallel
• Simple, lightweight, no separate broker needed

What It Doesn't Do

• No exchange/routing layer (you manually route)
• No replayability (Kafka territory)
• No strong microservice communication
• No advanced pub/sub routing
• Not suitable for enterprise-scale messaging

Comparison with Others

BullMQ:  Like hiring one office assistant (simple, cheap, good for moderate scale)
RabbitMQ: Like a professional mail room (routes intelligently, retries, reliable)
Kafka:    Like an immutable company history ledger (every event recorded forever)

When to Use BullMQ

• Node.js/MERN stack apps
• Startup products, internal tools
• Background jobs: send email, generate invoice, cleanup tasks
• You already use Redis
• Simple, fast, easy

When You've Outgrown BullMQ

• Need true microservice async communication → RabbitMQ
• Need event replay → Kafka
• Need complex routing → RabbitMQ
• Need massive scale → Kafka

15. The Dual Write Problem & Transactional Outbox

The Dual Write Problem

This is one of the most important reliability problems in distributed systems.

The problem:

# Your code
await create_booking()           # DB write ✅
await rabbitmq.publish("booking-confirmed")  # Publish to queue ❌ (RabbitMQ is down)

Booking exists in DB, but event never published. Email never sent. Loyalty points never awarded. System is inconsistent.

Why DB transactions don't solve it:

await db.transaction():
    await create_booking()        # Inside DB transaction
    await rabbitmq.publish(...)   # Outside DB transaction!

The DB transaction can't control RabbitMQ. It can only rollback DB writes, not queue publishes.

The Solution: Transactional Outbox Pattern

Instead of publishing directly to the queue, write the event to an outbox table in the same database transaction as your business data.

BEGIN TRANSACTION;
 
-- Business data
INSERT INTO bookings (id, userId, status) VALUES (123, 55, 'CONFIRMED');
INSERT INTO payments (id, bookingId, amount, status) VALUES (999, 123, 4000, 'SUCCESS');
 
-- Event for later publishing (same transaction!)
INSERT INTO outbox_events (id, event_type, payload, status)
VALUES ('evt_789', 'BOOKING_CONFIRMED', '{"bookingId": 123}', 'PENDING');
 
COMMIT;

How it works:

1. Business data + event saved in same DB transaction (atomic)
2. Background worker polls outbox table for PENDING events
3. Worker publishes event to RabbitMQ/Kafka
4. Worker marks event as SENT
5. If RabbitMQ is down → event stays PENDING → retry later

Why this works:

• If DB transaction commits → event is saved → will eventually be published
• If DB transaction rolls back → event never saved → no ghost events
• Business state and event state never diverge
• If queue is down, events wait safely in the database

Outbox Table Schema:

Outbox Relay Strategies

Polling (simple):

SELECT * FROM outbox_events WHERE status = 'PENDING' LIMIT 100;
-- Publish to queue
-- Mark as SENT

• Simple but adds latency (polling interval)
• Adds database load

CDC (Change Data Capture) with Debezium:

• Reads database transaction log (PostgreSQL WAL)
• Publishes new outbox rows to Kafka automatically
• Near-real-time (sub-second latency)
• No polling overhead
• Recommended for production

Key Insight

The outbox pattern guarantees that business state and event state never diverge. If your booking exists, the event will eventually be published. If your booking doesn't exist, no event is created. This is the foundation of reliable event-driven systems.

16. Saga Pattern — Distributed Transactions Without 2PC

The Problem: Distributed Transactions

In microservices, a single business operation spans multiple services:

Book Hotel:
  1. Check availability (Hotel Service)
  2. Reserve room (Hotel Service)
  3. Charge payment (Payment Service)
  4. Send confirmation (Email Service)
  5. Update analytics (Analytics Service)

If step 3 (payment) succeeds but step 4 (email) fails, what happens? You charged the customer but didn't send confirmation. In a single database, you'd use a transaction (ACID). Across services, you can't.

Two-Phase Commit (2PC) — The Old Way

Coordinator:
  Phase 1 (Prepare): "Everyone ready to commit?"
    Hotel: "Yes"
    Payment: "Yes"
    Email: "Yes"
  Phase 2 (Commit): "OK, everyone commit now"

Problems:

• Blocking: All resources locked during prepare phase
• Single point of failure: If coordinator crashes, everyone is stuck
• Doesn't scale: Doesn't work across services with different databases

The Saga Pattern — The Modern Way

A Saga breaks a distributed transaction into a sequence of local transactions, each in a single service. If any step fails, compensating transactions undo the previous steps.

Saga Steps:
  1. Reserve Hotel → OK
  2. Charge Payment → OK
  3. Send Email → FAILS

Compensating Transactions (reverse order):
  2. Refund Payment
  1. Release Hotel Reservation

Each step has:

• Forward transaction: The actual operation
• Compensating transaction: The "undo" operation

Two Types of Sagas

Orchestration (Central Coordinator):

Orchestrator
  ↓ calls Hotel Service: "Reserve room"
  ↓ calls Payment Service: "Charge $200"
  ↓ calls Email Service: "Send confirmation"
  ↓ if any fails → orchestrator runs compensations

• Single point of control
• Easy to understand and debug
• Orchestrator knows the entire workflow
• Risk: orchestrator becomes a single point of failure

Choreography (Decentralized):

Hotel Service → publishes "RoomReserved" event
  ↓
Payment Service → listens, charges, publishes "PaymentCharged" event
  ↓
Email Service → listens, sends confirmation
  ↓
If any fails → each service handles its own compensation

• No central coordinator
• Services are fully decoupled
• Harder to understand the overall flow
• Risk: circular dependencies, hard to debug

Compensating Transactions — Key Rules

1. Idempotent: Retrying a compensation must not cause double-refunds
2. Order-independent: In choreography, compensations may run in any order
3. Handle partial state: Original operation may have partially completed
4. Irreversible actions: You can't unsend an email. Delay irreversible actions to the end of the saga.

The Pivot Point

Saga:
  Compensatable steps: Reserve Hotel (can cancel)
  Compensatable steps: Reserve Flight (can cancel)
  PIVOT: Charge Payment (irreversible - this is the point of no return)
  Retriable steps: Send Email (retry until success)
  Retriable steps: Update Loyalty Points (retry until success)

Once the pivot succeeds, the saga must run to completion with retries rather than compensations.

Saga vs 2PC

17. Event-Driven Architecture (EDA)

What Is EDA?

Instead of services calling each other directly, services emit events (facts about what happened) and other services react to them.

Request-Response (Traditional):
  Client → API Gateway → Order Service → Payment Service → Inventory Service
  (Tightly coupled, synchronous)

Event-Driven:
  Client → Order Service → publishes "OrderCreated" event to Event Bus
  Email Service → reacts to "OrderCreated" → sends email
  Inventory Service → reacts to "OrderCreated" → updates stock
  Analytics Service → reacts to "OrderCreated" → logs event
  (Loosely coupled, asynchronous)

Key Principles

1. Services don't call each other directly — they emit events
2. Services react to events independently — no coupling
3. Events are facts, not commands: "OrderPlaced" not "PleasePlaceOrder"
4. Adding a new service is easy — just subscribe to relevant events

Commands vs Events

Command: "PlaceOrder" → tells someone to do something
Event: "OrderPlaced" → states that something happened

Bad event names: PaymentStarted, BookingInitiated, RoomReservedMaybe Good event names: PaymentSucceeded, BookingConfirmed, BookingCancelled

Events represent facts that happened, not intentions.

Benefits of EDA

• Loose coupling: Services are independent
• Scalability: Each service scales independently
• Resilience: If one service is down, events wait in queue
• Auditability: Events are a record of everything that happened
• Extensibility: Add new consumers without changing producers

When to Use EDA

• Multiple systems need to react to the same event
• Services have different scaling requirements
• You need audit trails
• You want resilience (failures don't cascade)

When NOT to Use EDA

• Strong consistency required (financial transactions)
• Immediate response needed (user waiting for result)
• Simple system with few services

18. Event Sourcing & CQRS

Event Sourcing

Instead of storing current state in a database, store every state change as an event. Current state is derived by replaying events.

Traditional (State-based):
  DB: { "orderId": 123, "status": "shipped", "total": 50 }

Event Sourcing:
  Events:
    OrderCreated { orderId: 123, items: [...], total: 50 }
    PaymentReceived { orderId: 123, amount: 50 }
    OrderShipped { orderId: 123, trackingId: "XYZ" }
  
  Current state = replay all events for order 123

Benefits:

• Complete audit trail
• Can reconstruct state at any point in time
• Replay to debug issues
• Multiple views of the same data

Kafka is excellent for event sourcing because it retains messages and supports replay.

CQRS (Command Query Responsibility Segregation)

Separate the write model (commands) from the read model (queries).

Write side: Commands → Events → Event Store
Read side: Projections → Optimized Read Models

Example:
  Write: "PlaceOrder" → OrderCreated event → stored in event store
  Read: Projection updates read-optimized table for fast queries

When to use:

• Read and write patterns are very different
• Need different data models for reads vs writes
• High-scale systems where read optimization matters

Caution: CQRS adds significant complexity. Don't use it for simple CRUD applications.