ZeroMQ

ZeroMQ (aka ∅MQ, 0MQ, or ZMQ) was registered in May 2007 as a domain name and trademarks by iMatix CEO Pieter Hintjens. With Martin Sustrik as the architect and lead developer during the development process, this messaging queue took three years to develop as free software by a large community of contributors founded by iMatix. On July 8, 2019, iMatrix released its 0MQ version 4.3.2 stable, while the latest release (0MQ 4.3.3 stable) hit the market on September 7, 2020. Today, ZeroMQ is used as by AT&T, Cisco, Samsung, Facebook, Bitcoin, EA, Los Alamos Labs, NASA, Microsoft, and CERN, among others.

ZeroMQ is distributed in nature. It carries messages from its source and delivers directly to the destination in a peer-to-peer fashion without passing through any single point of failure. ZeroMQ comes with four (4) basic messaging patterns, each with its own pair of sockets and use cases. Each messaging pattern defines a specific network topology and determines the flow of sent messages. The four basic messaging patterns of the ZeroMQ system include:

• Request/Reply Pattern: Used for connecting a set of clients to a group of servers. This messaging pattern can be chosen to send a request and receive subsequent replies for each request sent.

• Publish/Subscribe Pattern: This is a data distribution pattern. This messaging pattern is used to push messages from a publisher (a single process) to a set of subscribers (multiple recipients).

• Push-Pull (Pipeline) Pattern: This is a parallel task distribution and data collection pattern chosen for distributing messages to multiple workers or to connected nodes.

• Exclusive Pair Pattern: This is used to connect two peers (sockets) in an exclusive pair

Note: The key to designing your ZeroMQ system is understanding its basic messaging patterns and sockets. It also helps knowing that the way each socket works is dependent on the type of socket chosen.

Before getting started with any ZeroMQ library functions, you'll need to create a context (the relevant set of data needed to complete a task). The caller must create a ZeroMQ context when initializing a process using zmq_init(3) and destroy the ZeroMQ context when the process is terminated using zmq_term(3). Generally, ZeroMQ contexts help manage any sockets created and the number of threads used behind the scenes (contexts are thread-safe). You may also share contexts among as many application threads as necessary without any extra locking required on the caller's part.

To get a grasp of how this messaging library works, you have to be familiar with the following five main core patterns (same as the four patterns discussed earlier, but with the request/reply split into synchronous and asynchronous):

The ZeroMQ REQ/REP sockets are for synchronous communication (i.e., these sockets can only talk to one peer at a time). In terms of the standard ZeroMQ request-response pattern, you can think of response sockets as the servers and the request socket as the client for appropriate binding and connection. Once the sockets are established, both the request and response sockets can send and receive calls. Since request and response sockets are synchronous, there are strict requirements on the sequencing of send and receive calls. Each transmission must follow this specific sequence: client send, server receive, server send, and client receive.

You cannot use ZeroMQ REQ/REP sockets for asynchronous transmission that involves many clients requesting work from many workers. A better solution is to set up a broker for clients and workers to connect to. This broker is responsible for passing messages back and forth. Since this broker needs to handle many simultaneous requests and responses, a different socket is needed. In this case, developers can create a router/dealer pattern to which the clients and workers can connect. Here, you can think of ROUTER-DEALER sockets as a non-blocking, asynchronous version of the REQ-REP sockets. In this context, the ROUTER socket acts like the REP socket, while the DEALER socket acts like the REQ socket.

ROUTER socket expects all received messages to have an identity frame attached to the message itself, as this allows the ROUTER to know the exact source of the incoming messages. When using a ROUTER to send a message, the first frame of the message is removed, which is then used to identify which client to send the response to. The DEALER sockets are created to “deal” messages out to any number of connected workers. Once a router and dealer socket is created, you can send out multiple requests and print the response, and whenever a socket receives a message, it sends it out to its counterparts directly.

ZeroMQ has a pattern that allows the distribution of a stream of data from a publisher to many clients willing to consume that stream. This pattern uses PUB and SUB sockets. First, you need to create a context and a publisher socket and bind to an endpoint (INPROC, IPC, or TCP). Then, create another context, a subscriber socket, and connect to an endpoint (INPROC, IPC, or TCP). Once the sockets are correctly set up, a PUB socket pushes messages out, which can be received by all the associated SUB sockets. You must set a subscription to subscribe to all events published by the server. A subscription helps perform a string match against the beginning of the message. If a match is found, the subscribing client receives the message, otherwise the client receives nothing.

ZeroMQ offers PUSH/PULL sockets that allow you to distribute messages to multiple workers. PUSH socket helps distribute sent messages to all its available PULL clients evenly, while the PULL socket helps to queue messages from all connected clients fairly. ZeroMQ PUSH-PULL sockets are used to distribute work to some workers and then to collect the results in a sink. The PUSH-PULL pattern is useful in situations where you are doing a large number of small tasks.

ZeroMQ has a pattern (PAIR sockets) for co-ordinating multi-threaded applications and bidirectional communication. The PAIR sockets can only connect to one peer at a time, and you can only use them for messaging Between Process Threads (but cannot be used over the network). To use this pattern, you first need to create a three-step process, create exclusive PAIR sockets, and connect them via the inproc transport, as shown in the figure below. The steps should be created in reverse order from which they'll be used (for appropriate connection).

ZeroMQ offers sockets that can use multiple different underlying transport mechanisms. ZeroMQ provides the following transport mechanisms, each suited to a particular purpose:

• Unicast transport using TCP: zmq_tcp(7)

• Multicast transport using PGM: zmq_pgm(7)

• Local inter-process communication transport: zmq_ipc(7)

• Local in-process (inter-thread) communication transport: zmq_inproc(7)

ZeroMQ also provides a mechanism for applications that allow developers to multiplex input/output events over a set containing both ZeroMQ sockets and standard sockets. This mechanism for input/output multiplexing is described in detail in zmq_poll(3).

The ZeroMQ library provides interfaces that support every modern language and platform. The ZeroMQ provides a C++ language binding documented separately in zmq_cpp(7), while other language bindings, including Python, Ruby, Java, etc., are made available by members of the ZeroMQ community.

The ZeroMQ library functions use the standard conventions found on POSIX systems to handle errors. In a case where there is a failure, a ZeroMQ library function is designed to return either a NULL value (if returning a pointer) or a negative value (if returning an integer) and store the actual error code in the errno variable. The zmq_sterror() function is described in detail in zmq_strerror(3).

ZeroMQ supports several different authentication methods, including NULL, PLAIN, and CURVE. CURVE is the ZeroMQ standard public/private key encryption — for both the client and the server, each with a public and a private key. CURVE is based on elliptic public key encryption. The client only needs to know the server's public key to connect to. Once the client-server connection is established, ZeroMQ encrypts the entire conversation to keep it secure from any third party. ZeroMQ also provides extra authentication for more security using a ZAP (ZeroMQ Authentication Protocol) server. This additional authentication defines how the servers connect to the handler and the messages they exchange.

• Asynchronous messaging library

• Features a lightweight message broker (Malamute)

• Carries messages across multiple transports (INPROC, IPC, TCP, TPIC, multicast)

• Smart messaging patterns like pub-sub, push-pull, and router-dealer

• High-speed asynchronous I/O engines

• Easier multithreading

• Unclear and inefficient threading model

• Difficult to understand and use abstractions for common developers and engineers

• ZeroMQ – Reviews, Pros & Cons | Companies using ZeroMQ

• Comparison of ActiveMQ, RabbitMQ, and ZeroMQ message-oriented middleware

• ZeroMQ Feature List – zeromq

• An Introduction to ∅MQ (ZeroMQ)

• A quick and dirty introduction to ZeroMQ