Apache Kafka vs RabbitMQ vs AWS SNS/SQS: Which message broker is best?

Message brokers is a broad category. While each of the three options we’re looking at moves data between components in a bigger system, deciding between them comes down to how they do it.

So we need to inspect the characteristics of each message broker according both to how they affect its ability to move data and also what it is like to work with. We’ll look at the following six factors and how they apply to Apache Kafka, RabbitMQ, and AWS SNS/SQS:

• Messaging patterns: How the message broker organizes and distributes messages, such as Publish/Subscribe or request-reply, is better suited  to some architectures and use cases than others.

• Message routing: Can the message broker route messages according to their content and other criteria?

• Scalability: Not every situation requires millions of messages per second. Ensure that the message broker can scale to the throughput levels you need, while bearing in mind how that affects the complexity of set-up and operation.

• Reliability: How does the message broker deal with faults? Does it persist messages or do they only pass through the system?

• Security: Is data encrypted? Does the message broker provide audit logs? How does it handle authentication and access control? Is it certified for industry standards such as HIPAA and PCI DSS?

• Cost and licensing: Is the message broker open source? What cloud or server resources will it require? Are there ongoing license fees? How much of your team’s time will it take to run it?

Before we get into the detail of the comparison, let’s touch on why we’re focusing on these three solutions. One reason is that they’re amongst the most common message broker solutions you’re likely to come across. Research shows that almost 50,000 companies globally use RabbitMQ, while more than 60,000 are using Apache Kafka. And although AWS SQS seems to have a smaller user base, at just over 24,000 companies, it plays an important role in the broader Amazon Web Services ecosystem.Such extensive deployment makes it easier to recruit experienced staff, integrate with your existing tech stack, and find support when you need it. But there is another consideration, too. Each of the message brokers we’re looking at approaches the problem from a different perspective. Apache Kafka is designed for throughput, whereas RabbitMQ focuses more on complex message routing. Like RabbitMQ, the two AWS solutions, SNS and SQS, focus more on routing than throughput and we are considering them here because of the importance of the AWS ecosystem.

Apache Kafka is a realtime streaming platform designed for building scalable, fault tolerant, distributed applications. It specializes in streaming data at very high throughput rates and is often used for operational data, such as logging and metrics.

With such a large install base, there are tools that connect Apache Kafka to almost any other system in operation. Although Apache Kafka focuses on high throughput, it can also process data in transit, such as triggering actions and transforming data. However, that flexibility and power come at a cost because Apache Kafka is harder to set-up and operate than some of the alternatives.

Unlike Apache Kafka’s focus on streaming data, RabbitMQ is an open source distributed message broker that supports several different messaging patterns, such as Publish/Subscribe and Producer/Consumer. Although it does boast high throughput, its focus is more on routing data intelligently according to message content, time of day, sender, and other criteria, which makes it a good fit for event driven architectures.

Like Apache Kafka, there is widespread support for RabbitMQ with a rich ecosystem of integrations.

Amazon Web Services (AWS) offers two products that fit under the message broker banner. AWS Simple Notification Service (SNS) is a high throughput messaging service that uses a Publish/Subscribe model to deliver messages between microservices or the components of a distribution application.

AWS SNS uses topics to create separate streams of notifications. It also allows for filtering, so that even if a consumer is subscribed to a particular topic, they don’t have to receive every message.

AWS Simple Queue Service (SQS) is a first in, first out (FIFO) queuing service that distributes messages to potentially large numbers of consumers. People tend to use SNS and SQS together, for example by publishing messages to a topic using SNS and then distributing those messages reliably to the appropriate destinations using SQS.

Unlike Apache Kafka and RabbitMQ, SNS and SQS are available only by subscribing to Amazon Web Services.

To help you understand which broker is best for your use case, here's a look at how they compare against the considerations highlights above.

How the broker organizes and distributes messages impacts almost everything else. Not only does it influence which use cases a message broker suits best but it also affects the design, scalability, performance, and flexibility of the rest of your application’s architecture.

So, what are the messaging patterns that Apache Kafka, RabbitMQ, and AWS SNS/SQS offer?

When it comes to routing, there are two broad options for a message broker:

• Smart router, dumb consumer: The broker decides where to route messages, meaning the microservice or application that receives the messages has less work to do.

• Dumb router, smart consumer: The broker sends messages to predetermined queues, but some other part of the application stack must decide which microservice or application reads which queue.

Broadly speaking, RabbitMQ adopts the smart router, dumb consumer model. RabbitMQ can use a message’s content, metadata, sender, time of day, and other criteria to decide how to route it.

Apache Kafka, on the other hand, is more geared towards the dumb router, smart consumer model. However, there are third-party tools that plug into Apache Kafka to enable different types of smart routing.

The combination of AWS SNS and SQS offers a mix of both approaches. As we saw earlier, a publisher can specify the AWS SNS topic for each message. Similar to RabbitMQ, SNS also allows for finer grained filtering, based on message metadata and the message body.

All three of the message brokers we’re considering scale well. But how much each one scales depends on the other capabilities it offers.

Whether a message broker adopts the smart router or dumb router approach has some impact here. By asking the message broker to do more work, the smart router approach is somewhat less scalable than the dumb router model.

Both RabbitMQ and Apache Kafka scale horizontally. In other words, you can create a cluster of RabbitMQ or Apache Kafka nodes. Each topic can be partitioned across multiple nodes, so adding more instances increases the cluster’s capacity.

Although some of the architectural decisions differ between the two, they are both relatively easy to scale. However, they differ significantly when it comes to how scaling impacts throughput. Typically, RabbitMQ deals in thousands of messages per second, whereas even a relatively small Apache Kafka cluster can process millions of messages per second.

As hosted services, scaling AWS SNS and SQS is less about how many instances you can spin up and more about throughput limits. AWS SQS, for example, quotes a maximum of 30,000 messages per second and only then if the messages are sent in batches.

Scalability and reliability go hand in hand. As horizontally scalable distributed systems, both Apache Kafka and RabbitMQ can survive an individual node going offline because others will pick up the slack.

In RabbitMQ’s case, individual messages are replicated across multiple nodes, which means that if one node fails, the messages are still available on the other nodes. Similarly, RabbitMQ can persist messages to disk, so that even if the entire cluster goes offline it is possible to recover.

Like RabbitMQ, Apache Kafka uses multiple nodes to improve availability. Typically, Apache Kafka replicates messages to three nodes but that replication factor is configurable. If a node becomes unavailable, Apache Kafka automatically reconfigures itself to spread the workload across the reduced cluster.

AWS SNS and SQS take a similar approach. They replicate messages across multiple availability zones (that is, separate data centers within an AWS region), so that the cluster can survive an outage in one zone. And like RabbitMQ and Apache Kafka, messages can be persisted to disk, allowing for recovery for a larger failure.

Other than trusting how each vendor/project tackles security vulnerabilities, message broker security is a matter of three factors:

• Authentication: Verifying the identity of publishers and consumers to grant access to the message broker.

• Authorization: Using access control lists (ACLs) to determine which topics and functionality are available to each publisher and consumer.

• Data encryption: Protecting data both while it is moving through the system (in-transit) and on disk (at rest).

As with many things, Apache Kafka’s authentication is configurable through the use of third-party add-ons. By default, Apache Kafka offers TLS or SASL authentication. TLS auth uses public-private key pairs to confirm that a client is allowed to access the Apache Kafka instance. SASL offers different levels of authentication, from username and password through to OAUTH 2.0.

Similarly, RabbitMQ offers TLS-based authentication using X.509 certificates. Alternatively, RabbitMQ can support username/password pairs for authentication.

Both Apache Kafka and RabbitMQ allow for the granting of fine grained permissions–such as read, write, configure, and admin–through ACLs.

As with many of the factors here, AWS SNS/SQS are a little different because they are part of the AWS ecosystem. As such, they primarily use Amazon’s IAM certificates for authentication and access control.

There is less of a difference, though, between the offerings when it comes to data encryption. By default, both Apache Kafka and RabbitMQ send data in the clear but can also encrypt data in-transit using the TLS. In the case of Apache Kafka, there are a number of third-party add-ons that enable other forms of in-transit encryption. Neither Apache Kafka nor RabbitMQ encrypts data at rest, instead recommending the use of third-party tools, such as an encrypted file system. AWS SNS and SQS rely on the AWS Key Management Service both to encrypt data at rest and in-transit.

Calculating the total cost of ownership (TCO) can be complex. Both RabbitMQ and Apache Kafka are open source projects that don't require licensing fees for usage. However, it's worth noting that Apache Kafka can present challenges when it comes to setup and operation that might mean that you need to hire dedicated staff or consultants to manage and administer deployment.

At smaller scales, RabbitMQ can be easier to run than Apache Kafka and it is known for its operational stabilityIf you choose a commercial edition of Apache Kafka or RabbitMQ, obviously that introduces licensing costs but you’ll also benefit from additional functionality, enhanced documentation, and support, which could in turn reduce costs elsewhere.

Again, the AWS products are the odd ones out here as they are not open source, being available only as hosted services. That brings together licensing, cloud, and operational costs into one bill.

Apache Kafka, RabbitMQ, and AWS SNS/SQS are all message brokers, but they each have their own strengths and weaknesses.

• Apache Kafka is known for its high throughput and ability to scale to very large throughputs, making it a good choice for applications that need to process large amounts of data in real time.

• RabbitMQ is less focused on sheer throughput and more on routing messages intelligently.

• The combination of AWS SNS and SQS is simpler to set up and run than either of the other two options, but there could be cost implications as your workloads scale.

So, which message broker is right for your project? The most significant factors to consider are:

• How well it can route messages according to their contents and metadata.

• Data volume and throughput.

• Scalability and fault tolerance.

• Security features.

• Costs and ownership model.

With that in mind, here are some suggestions for which message broker to choose:

• Very large volumes of data, flexible scalability, and basic routing needs: Apache Kafka.

• Large volumes of data and sophisticated routing: RabbitMQ.

• Large volumes of data and a fully hosted option: AWS SNS and SQS.