RocketMQ

Alibaba originally developed RocketMQ in 2012 as a distributed messaging engine for online e-commerce transaction processing. It was initially released under the name MetaQ and renamed RocketMQ when the 3.0 version of MetaQ was released. The evolution of RocketMQ can be split into three distinct generations, each with different modes.

The first generation was built around the Push mode to transport data and uses a relational database to store it. The second generation mainly relied on the Pull mode to transport data and used file systems to manage data storage. The third generation inherits the operations of the two previous generations. It adopts the Pull mode along with some Push operations to ensure high-performance optimization in terms of scenarios of concurrency and massive amounts of data.

In November 2016, RocketMQ was donated to the Apache Software Foundation (ASF) as an incubator project and successfully made it through ASF’s competitive evaluation process. In February 2017, the 4.0.0 version of Apache RocketMQ was released with the goal of starting a new generation of programming model in fields such as e-commerce, finance, big-data, and the internet of things (IoT). All RocketMQ’s implementations are licensed under Apache License 2.0 since entering the Apache Incubator in 2016. The latest Apache RocketMQ release tag is v4.1.1.

Today, RocketMQ is used by Alibaba Group, ANT FINANCIAL, HUAWEI, WeBank, China Minsheng Bank, Express, Tree Finance, and Beibei Group, among others.

RocketMQ is composed of four (4) main components, each with its own cluster. They include: name servers, brokers, produces, and consumers. Each component can be horizontally extended without a single point of failure.

A lightweight component that manages source data, including topic and routing information. It serves as the routing information and service discovery provider. Each NameServer node documents global routing information and supports fast storage expansion.

This component manages data storage by relying on the lightweight queue mechanism with which messages are grouped by topic. Broker Cluster provides two or three copies of data to analyze fault tolerance mechanisms. It supports the push and pull model. It also supports disaster recovery, rich metrics statistics, and alert mechanisms.

This component is responsible mainly for generating messages. Producers send messages to the Broker Cluster through multiple load balancing modes. This component supports distributed deployment. In addition, the sending processes support fast failure and low latency.

Consumers receive and consume messages from the broker cluster. This component also supports distributed deployment in the push and pull models. Consumer Cluster supports two message consumption models: clustering (cluster consumption) and broadcasting (message broadcast). Also supports a real-time message subscription mechanism that suits most consumer scenarios.

Figure 1: RocketMQ Architecture

Each broker registers with the NameServer cluster when it starts. Producer/Consumer clients look up topics to find the corresponding broker list before sending/consuming messages by establishing a connection with one of the nodes in the NameServer cluster (randomly selected).

Once the connection has been established, a producer cluster (which may contain multiple producer groups) can send messages of various topics to the broker cluster (broker master only) through multiple paradigms of sending: synchronous, asynchronous, and one-way. The broker receives the messages sent by the producer, stores them (along with related metadata, including consumer groups, consuming progress offsets, and topic/queue information), and prepares to handle pull requests from all consumers registered in the cluster.

A consumer cluster (which may contain multiple consumer groups) pulls messages from the broker cluster (Broker Master and Slave) to consume messages sent from the producers. Consumers can also subscribe to messages from the Broker Master or the Broker Slave.

Note the following terms: 

• Topic – a category in which producers deliver messages and consumers pull messages. RocketMQ provides two types of queues: read and write queue.

• Message – a combination of the data or information to be delivered to consumers. A message includes a topic, an optional tag, and extra key-value pairs.

• Message Queue – the partition of the topic into one or more sub-topics

• Tag – also referred to as sub-topic. Tags help facilitate the query system provided by RocketMQ.

• Producer Group – a group of producers that sends similar kinds of messages. 

• Consumer Group – a group of consumers that consumes similar kinds of messages.   

RocketMQ supports HTTP and RESTful APIs to let developers integrate Message Queue for Apache RocketMQ management tools into their own console. RocketMQ also provides developers with a web console tool that supports topic management, group management, message query, message trace display and query, resource report, and monitoring and alert management.

RocketMQ supports a variety of cross-language clients such as Java, C/C++, Go, Python, Node.js, and PHP. It also supports multiple pluggable transport protocols such as TCP, SSL, and AIO. The RocketMQ’s TCP-based SDK is more specialized, reliable, and stable than the HTTP-based access mode. The TCP version supports programming languages, including Java, C/C++, and .NET.

RocketMQ supports ordered and unordered message payloads up to 4 MB in any format. It also supports message size up to 64 KB for a transactional, scheduled, or delayed messages. RocketMQ retains messages for at most three days before they are deleted or discarded.

RocketMQ verifies each HTTP request for authorization by issuing an AccessKey pair that consists of an AccessKey ID and an AccessKey secret to a visitor through the Alibaba Cloud console. The AccessKey ID allows RocketMQ’s system to verify the identity of the user, while the AccessKey secret is used for encrypting and verifying the signature string on the server.

• Supports messaging patterns including publish/subscribe and request/reply

• Supports several client SDK such as Java, C/C++, Python, Go, Node.js, etc.

• Guarantees FIFO and strict ordering of messages

• Supports property filter expressions based on SQL92

• Supports Master-Slave model

• Allows multiple messaging protocols including JMS and OpenMessaging

• No interface such as JMS in the mq core

• No message priority

• RocketMQ you should know – Programmer Sought

• From Alibaba to Apache: RocketMQ’s Past, Present, and Future

• RocketMQ Reaches Top-Level Status at the Apache Software Foundation

• The first step in learning RocketMQ – Programmer Sought