In this post, we take a look at 5 pub/sub examples that will help us understand how companies and organizations across all types of industries are leveraging this architectural design pattern to solve common engineering challenges and improve their architecture.
What Is Pub/Sub?
The publish/subscribe (pub/sub) pattern is a simple but powerful architectural design pattern which can benefit your messaging application design. Among other applications, you can use pub/sub messaging for distributing events, propagating logs to multiple locations, and seeding distributed caches.
The pub/sub pattern is a way of architecting message exchanges among entities such as services, for example. Instead of communicating directly with each other, in the pub/sub pattern services can communicate via a message broker. This approach decouples the concerns of publishers and subscribers: publishers can focus only on publishing, and subscribers can focus only on which publications they’re subscribed to. The resulting setup allows for asynchronous sending and receiving of messages by a broker dedicated to the task, which is one of several ways you can implement event-based systems.
More specifically, in pub/sub, publishers forward messages to a broker. The broker is the dedicated component in charge of organizing the messages into appropriate delivery categories, and then, depending on the way the broker is implemented, either pushing the messages out to subscribers or letting subscribers themselves retrieve messages they want down.
For example, the broker might use metadata provided by the message publisher such as a channel or topic designation to route the message to the correct recipient or recipients. Message recipients connect to the broker and subscribe to messages based on that same information, seeing only the messages that match their interests. Meanwhile, publishers have no awareness of how many subscribers receive their messages, so the size and scope of the subscriber base doesn't impact them.
As a result, the pub/sub pattern provides applications with increased scalability compared to other message exchange patterns. The only coupling between parties is the message contents and metadata, with (theoretically) no hard limits on the number of participants or message throughput.
The pub/sub pattern provides applications with increased scalability compared to other message exchange patterns.
Pub/sub’s loose coupling makes it a powerful solution for many common engineering challenges. Its scalability makes it a great solution for distributed architectures with a high number of clients. Let’s take a look at a few examples of how you can use pub/sub messaging to address said problems and improve your architecture.
Sending events to large numbers of recipients simultaneously is a common technical challenge. You need the system to rapidly deliver events to your clients but without negative side effects that materialize when the number of destinations increases massively. Depending on the design, clients might also not always be available. When they do come online after the fact, you need them to receive the messages they missed.
Pub/sub’s loose coupling means that publishers can send events without worrying about which clients are online. They don't even need to know when new clients connect to the system. Furthermore, assuming a robust implementation of a broker which can persist messages and has exactly-once delivery (or at-least-once delivery at a minimum), publishers can simply “fire and forget,” knowing that the message broker will deliver the messages when the client needs them.
Split uses Ably’s pub/sub to distribute feature flag events to tens of millions of client applications. Delivering feature flag changes within seconds is critical to Split’s customers' needs since breaking changes need to be reverted as quickly as possible. Or Split might need to coordinate a new feature with a marketing announcement. Either way, events need to be delivered on time, and reach all clients, regardless of whether said clients were online at the time or end up connecting later.
Split made this possible by shifting their system from a polling model to a responsive, event-based architecture. Client applications receive events in real time if they're online. If they're not, Ably updates their application state as soon as they reconnect to the network.
Caches are an important tool for improving system performance by having the latest message available to the subscribers stored locally/nearby and updating this copy only when changes occur. Depending on your application, you may need to distribute them across multiple systems, in order to, for example:
- Set up parallel systems to overcome memory or CPU constraints.
- Distribute caches across multiple locations to reduce network traffic and minimize load on central servers.
- Keep the state of multiple caches synchronized.
With pub/sub, you can seed caches asynchronously. As processes create data and events, the messaging system delivers events to caches. As such, rather than waiting for a nearby client to make a related request, the data is already in the cache. When new information supersedes data in the cache, a new event replaces it. This design is far more efficient in updating state than relying on polling semantics. Using an event stream to populate caches prevents the case where your clients are polling for information that isn’t up to date.
Seats.io is a seating reservation system. They draw their floor plan, assign seats to it, and then manage the live seat-booking events. As clients buy tickets, Ably’s pub/sub system updates the floor plan everywhere in real time and keeps the distributed cache system in sync. Applications thus cache the seating chart and can update it in real time. Clients never end up requesting a seat only to find out someone had bought it while they were still in the browsing/shopping phase.
Another critical tool for distributed systems is application logs. Logs are an inexpensive tool for reconstructing events, triggering alerts as events occur, and tracking critical transactions.
With the pub/sub pattern, you can send logs to many subscribed destinations simultaneously. You have the option of creating explicit logging channels or having message recipients log events in more than one destination. For example, one endpoint can subscribe to logs and send them on to a centralized system like Splunk or Elasticsearch. At the same time, other endpoints can send logs to screens or save them to a file for localized troubleshooting. Yet another use case is subscribing to transactional logs for compliance purposes.
Pub/sub is well suited for this and can thus add extra flexibility to one of your most important monitoring resources in a number of ways without writing new code or building expensive redundant systems.
Multiple Data Sources
Loose coupling between message senders and message receivers makes pub/sub systems naturally elastic. You can add new processes at any time, and you can remove or replace them as needed. Message senders and receivers can be located anywhere that a connection to the message broker is possible. Additionally, any system can be a sender, a receiver, or both.
Experity uses Ably’s pub/sub to collect data from multiple sources and process it with Kafka. Combining Kafka’s streaming capabilities with a last-mile delivery pub/sub system that matches its realtime guarantees displays data from diverse sources on a single dashboard for their users. Since Experity’s business logic isn’t coupled to the messaging system, devices can leave and join the network without additional configuration or network changes.
Burstability and Scale
A well-designed pub/sub system has the ability to send ordered, messages out to large numbers of subscribers suddenly and simultaneously as they come and go, connecting and reconnecting unpredictably, and keeping their state in sync. In terms of design, this is best handled by outsourcing the task to a designated broker component that is designed to handle busy, haphazard network behavior (and which ideally also handles data integrity). Offloading the responsibility of massive fanout bursts messages to a broker gives pub/sub systems a distinct advantage in terms of scale.
Wooclap is an education tool that allows students to participate in the classroom via a web-based app. Wooclap’s large numbers of simultaneous clients often encounter issues such as unreliable Wi-Fi or unpredictable cellular networks. The robust design of Ably’s pub/sub system recovers their connection when they rejoin the network, and is able to handle the often sudden, large numbers. With the added benefit of data integrity, this allows the participants to rejoin the classroom experiences seamlessly.
Pub/Sub Simplifies Building Networks
We've covered what pub/sub messaging is and some use cases of delivering data to and from client applications. You've seen the benefits of loose coupling between senders and receivers, adding flexibility while reducing complexity. Our pub/sub examples also demonstrated how delivering messages to multiple destinations at the same time can improve burstability and scale, synchronization of state, and monitoring.
Ably is an enterprise-grade pub/sub messaging platform. We make it easy to efficiently design, quickly ship, and seamlessly scale critical realtime functionality delivered directly to end-users.
We invite you to try our pub/sub APIs and see why developers from startups to industrial giants choose Ably to implement pub/sub in their systems.