Message Queues and Event-Driven Architecture

Introduction to Message Queues

Message queues enable asynchronous communication between services. Instead of direct calls, services communicate by sending messages through a queue.

Why Use Message Queues?

Decoupling

Services don't need to know about each other:

Without queue: Service A → Service B → Service C
With queue:    Service A → Queue ← Service B, Service C

Asynchronous Processing

Handle time-consuming tasks without blocking:

• Send email confirmations
• Process image uploads
• Generate reports

Load Leveling

Absorb traffic spikes without overwhelming downstream services.

Reliability

Messages persist even if consumers are temporarily unavailable.

Core Concepts

Producer

Creates and sends messages to the queue.

Consumer

Receives and processes messages from the queue.

Message

The data being transmitted. Usually includes:

• Payload (the actual data)
• Metadata (timestamp, ID, headers)

Queue/Topic

Where messages are stored until consumed.

Delivery Guarantees

At-Most-Once

Messages may be lost but never duplicated. Fastest but least reliable.

At-Least-Once

Messages are never lost but may be duplicated. Requires idempotent consumers.

Exactly-Once

Messages are delivered exactly once. Most complex, often achieved through idempotency + deduplication.

Message Queue Patterns

Point-to-Point

One producer, one consumer. Each message processed once.

Producer → Queue → Consumer

Use case: Task distribution (background jobs)

Publish-Subscribe

One producer, multiple consumers. Each consumer gets all messages.

Producer → Topic → Consumer 1
                → Consumer 2
                → Consumer 3

Use case: Event broadcasting (notifications)

Fan-Out

Route messages to multiple queues based on rules.

Request-Reply

Synchronous communication over async infrastructure.

RabbitMQ

RabbitMQ is a traditional message broker implementing AMQP protocol.

Key Features

• Exchanges: Route messages to queues (direct, topic, fanout)
• Queues: Store messages until consumed
• Bindings: Rules connecting exchanges to queues
• Acknowledgments: Confirm message processing

When to Use RabbitMQ

• Complex routing logic needed
• Message priority required
• Request-reply pattern
• Smaller scale (thousands of messages/sec)

Example Flow

Producer → Exchange → Binding → Queue → Consumer

Apache Kafka

Kafka is a distributed event streaming platform.

Key Features

• Topics: Categories for messages
• Partitions: Enable parallelism within topics
• Consumer Groups: Distribute load across consumers
• Log-based: Messages persisted to disk

When to Use Kafka

• High throughput (millions of messages/sec)
• Event sourcing / audit logs
• Stream processing
• Long-term message retention

Partitions and Consumer Groups

Topic (3 partitions)
├── Partition 0 → Consumer A (Group 1)
├── Partition 1 → Consumer B (Group 1)
└── Partition 2 → Consumer C (Group 1)

Each partition is consumed by one consumer in a group.

Kafka vs RabbitMQ

| Feature | RabbitMQ | Kafka | |---------|----------|-------| | Throughput | Moderate | Very High | | Retention | Until consumed | Configurable (days/weeks) | | Ordering | Per queue | Per partition | | Replay | No | Yes | | Protocol | AMQP | Custom | | Use Case | Task queues | Event streaming |

Event-Driven Architecture

What is EDA?

Systems communicate through events — facts about things that happened.

Event Types

Domain Events: Business facts

• OrderPlaced
• UserRegistered
• PaymentProcessed

Integration Events: Cross-service communication

• UserCreatedEvent (for other services)

Event Sourcing

Store all changes as events:

Account 123:
1. AccountOpened { balance: 0 }
2. MoneyDeposited { amount: 100 }
3. MoneyWithdrawn { amount: 30 }

Current state: balance = 70

Benefits:

• Complete audit trail
• Rebuild state at any point
• Debug by replaying events

CQRS (Command Query Responsibility Segregation)

Separate read and write models:

Commands → Write Model → Events → Read Model ← Queries

Benefits:

• Optimize reads and writes independently
• Scale reads and writes separately

Designing Event-Driven Systems

Event Schema

Define clear, versioned schemas:

{
  "eventType": "OrderPlaced",
  "eventVersion": "1.0",
  "timestamp": "2024-01-15T10:30:00Z",
  "data": {
    "orderId": "123",
    "customerId": "456",
    "total": 99.99
  }
}

Idempotency

Consumers must handle duplicate messages:

def process_order(event):
    if already_processed(event.id):
        return  # Skip duplicate
    
    # Process the order
    mark_processed(event.id)

Error Handling

Use dead letter queues for failed messages:

Main Queue → Consumer (fails) → Dead Letter Queue
                                      ↓
                              Manual investigation

Monitoring

Track key metrics:

• Queue depth
• Consumer lag
• Processing time
• Error rate

Best Practices

1. Design for idempotency: Messages may be delivered multiple times
2. Use schemas: Validate message structure
3. Plan for failure: Dead letter queues, retries
4. Monitor closely: Queue depth, lag, errors
5. Keep events immutable: Never modify published events
6. Version your events: Plan for schema evolution

Conclusion

Message queues and event-driven architecture enable scalable, resilient systems. Choose the right tool for your use case — RabbitMQ for traditional messaging, Kafka for high-throughput streaming — and design carefully for reliability.