What is the core difference between RabbitMQ and Kafka?

The message model. RabbitMQ uses a "queue model" — a message is consumed and deleted from the queue. Kafka uses a "log model" — messages are appended to partition logs, and consumers read by offset. Messages are not deleted after consumption and can be replayed. This determines their use cases: RabbitMQ is for task distribution (each message processed once), Kafka is for event streams (each message consumed by multiple consumers).

Can Redis Streams replace RabbitMQ or Kafka?

Depends on the scenario. Redis Streams is simpler — no need for additional message queue cluster deployment. Suitable for lightweight scenarios (task queues, simple pub/sub). But for message persistence, high throughput (millions/sec), message replay, and partition scaling, RabbitMQ or Kafka are more appropriate. Redis Streams is not a replacement — it is a lightweight choice for scenarios that do not need the heavier solutions.

Can messages be lost in a message queue?

Yes, depending on configuration. Messages can be lost at three points: ① Producer → Broker — enable Publisher Confirms to ensure message arrival; ② Broker storage — enable message persistence (disk writes) and mirrored queues/replicas; ③ Consumer consumption — use manual acknowledgment (Manual Ack), not auto-ack. With all three configured correctly, message loss probability is very low but not zero.

How do you determine Kafka partition count?

Partition count = max(target throughput / single partition throughput, consumer count). A single Kafka partition handles approximately 10-20 MB/s. If you need 100 MB/s throughput, you need at least 5-10 partitions. But do not set too many partitions blindly — more partitions increase ZooKeeper load and Leader election time. Recommended initial setting: 3-6 partitions, adjust based on actual load.

Message Queue Selection: RabbitMQ vs Kafka vs Redis Streams

Message queue selection is not about "picking the most popular one" — it is about "picking the one that fits your scenario." This article compares RabbitMQ, Kafka, and Redis Streams across message model, persistence, throughput, and consumption patterns — for backend architects and tech leads evaluating or selecting a message queue solution.

The Bottom Line: There Is No Best Message Queue — Only the Best Fit

Message queue selection is one of the most trend-driven technical decisions. Kafka is popular, so use Kafka. RabbitMQ is stable, so use RabbitMQ. Redis Streams is simple, so use Redis Streams. But the wrong choice leads to high operational costs down the road.

1. Core Comparison

Dimension	RabbitMQ	Kafka	Redis Streams
Message model	Queue (consume and delete)	Log (read by offset)	Stream (read by ID)
Throughput	Tens of thousands/sec	Millions/sec	Hundreds of thousands/sec
Persistence	Disk + optional memory	Disk (sequential writes)	Disk (RDB/AOF)
Consumption	Competing consumers	Consumer groups	Consumer groups
Message replay	Not supported	Supported (by offset)	Supported (by ID)
Ops complexity	Medium	High (needs ZooKeeper)	Low (no extra components)
Best for	Task distribution, RPC	Event streams, log aggregation	Lightweight queues, real-time data

2. Decision Tree

Need a message queue?
├── High throughput (millions/sec)?
│   ├── Yes → Kafka
│   └── No ↓
├── Need message replay/rewind?
│   ├── Yes → Kafka
│   └── No ↓
├── Need complex routing (Topic → Exchange → Queue)?
│   ├── Yes → RabbitMQ
│   └── No ↓
├── Already using Redis, no new components?
│   ├── Yes → Redis Streams
│   └── No → RabbitMQ (default recommendation)

3. Message Reliability Configuration

RabbitMQ Reliable Config

const channel = await connection.createConfirmChannel();
await channel.publish('exchange', 'routingKey', content, { persistent: true });
await channel.waitForConfirms();

channel.consume('queue', (msg) => {
  try {
    process(msg);
    channel.ack(msg);
  } catch (err) {
    channel.nack(msg, false, true);
  }
}, { noAck: false });

Kafka Reliable Config

const producer = new KafkaProducer({
  acks: 'all',
  retries: 3,
});

consumer.run({
  eachMessage: async ({ message }) => {
    await process(message);
    await consumer.commitOffsets([{ topic, partition, offset: message.offset + 1 }]);
  },
});

4. Scenario Recommendations

Scenario	Recommended	Reason
Task queue (one message, one consumer)	RabbitMQ	Queue model fits naturally
Event stream (one message, multiple consumers)	Kafka	Log model supports multiple consumers
Log aggregation	Kafka	High throughput + message persistence
Real-time notifications (WebSocket push)	Redis Streams	Low latency + simple deployment
Microservice async communication	RabbitMQ	Flexible routing + mature stability
Data pipeline (ETL)	Kafka	High throughput + message replay

Summary

Dimension	RabbitMQ	Kafka	Redis Streams
Learning curve	Medium	High	Low
Ops cost	Medium	High	Low
Throughput	Tens of thousands	Millions	Hundreds of thousands
Message persistence	Supported	Supported	Supported
Message replay	Not supported	Supported	Supported
Best for	Task distribution, microservices	Event streams, logs	Lightweight queues

Message queue selection is not about “picking the best one” — it is about “picking the least bad one.” Each solution has trade-offs. The key is understanding your scenario requirements for throughput, reliability, and message model, then choosing the one that is “least bad” across those dimensions.

Need message queue design or backend architecture consulting? Contact us — tell us about your communication scenario and scale, feasibility within 24 hours.