The Ably Kafka Connector has a raft of new enhancements, and is now available in a full general availability (GA) release. Developers now have the option to use pattern-based mapping rules to enable streaming of data from many Kafka topics to many Ably channels - ideal for chat solutions, live sports updates, live streaming, broadcasting notifications and alerts. This new feature presents greater flexibility, whilst further easing the management of extending Kafka-based pipelines to end users, over the public internet.
How to extend Kafka pipelines to users over the public internet
In the realms of event streaming, Kafka has become the number one choice for many organizations. It’s capable of handling and processing vast amounts of critical, time-sensitive event data, but Kafka is not designed for distributing events between internal systems and consumers on the public internet. That’s a key differentiator: rich, live digital experiences are a must-have as far as users are concerned. How can you harness the power of Kafka to make it “Internet ready”?
For many engineering teams, the effort needed to build and maintain a public internet-facing messaging layer, capable of streaming realtime event data, takes time and focus away from their core engineering.
Ably delivers data at low latencies to millions of concurrently connected devices via a fault-tolerant, dynamically elastic, global edge network. There’s no complex infrastructure to provision, manage, or scale. And we provide guarantees and performance SLAs across performance, data integrity, security, and reliability.
Many of our customers, from Hubspot and Experity to Split and Genius Sports, use Ably as an Internet-facing extension of Kafka. We are the solution that helps them send mission-critical event data to consumers across network boundaries in realtime.
Integrating your Kafka deployment with Ably
Ably and Kafka are complementary services. Kafka is an internal event streaming platform. Ably is an edge messaging service. Ably not only matches the same guarantees, messaging semantics, and characteristics as Kafka, but augments and extends these for event streaming over the public internet.
Kafka works best with a low number of topics, and a limited and predictable number of producers and consumers. Think of it as a ‘big fat pipe’ designed to process high volumes of data.
Conversely, Ably can rapidly scale horizontally to an unlimited number of channels and millions of concurrent subscribers with no need to manually provision capacity. Think of it as millions of thin pipes.
The Ably Kafka Connector
The Ably Kafka Connector is a sink connector that allows you to publish messages from Kafka topics to Ably channels. Ably performs heavy lifting of fanning out data to subscribers.
The Kafka Connector supports two mapping options - static and pattern-based. Static mapping enables the connection of one or multiple Kafka topics to one Ably channel. Here you provide a string for a configuration (eg. channel name). This stays the same no matter what record the connector is consuming. For example if you set a channel="global", all Kafka records will be published through that channel. This is great if, for example, you are a sports site broadcasting updates to millions of subscribers to a single channel.
channel = basketball
message.name = news_update
However this is limiting if your use case requires many topics to connect and stream data to many channels. Take the sports broadcasting example again. Here the solution is streaming one piece of data to multiple subscribers, on the same channel. But if you have a multi-tenanted chat solution, for example, servicing multiple customers, with multiple users, you are typically having to stream lots of ‘user specific’ data to those users. That’s across multiple channels. A much more complex model to provision.
In response to this challenge, the full, general availability, release of the Ably Kafka Connector enables pattern-based mapping to map Kafka topics directly into any number of Ably channels.
What is pattern-based mapping?
Pattern-based mapping gives you the ability to map many Kafka topics to many Ably channels following set rules. It’s ideal for a chat application, for example, where certain data needs to go to specific users. Think of one Kafka topic that represents a particular client company - acting as a big letter box. All the different chat messages for employees are processed then the Ably Kafka Connector uses Ably as a postman who delivers messages to the correct recipients. Pattern-based mapping enables each message (Kafka record) to be interrogated to determine the recipient, and establish which channel to send it through.
Essentially pattern-based mapping provides customers the ability to tweak their configuration by interpolating across record key and record topic values. For example where channel name is configured as channel="channel_#{topic}", the message will be published to a channel according to the topic name. Please note that there are currently two configurations that support pattern-based mapping: channel and message.name.
channel = channel_#{topic}
message.name = message_#{key}
For an "events" topic, the channel would translate as "channel_events". For a “remove” key, the message.name will be configured as message_remove.
It is important to note that you can also use static mappings in addition to pattern-based mapping. For example you could define the channel as #{topic}channel but message.name as singlemessage. In this instance the channel will be mapped by pattern and the message name will be mapped statically.
channel = channel_#{topic}
Message.name = add
In this case, while the messages will go through channels depending on their topic, the message.name will be the same for all messages that goes through different channels.
Ably Kafka Connector in action
We have created a number of demos and 'How to' guides to show you not only how the Connector works but the use cases and additional advantages Ably offers. Have a play, get stuck in and tell us what you think!
- Build a realtime ticket booking solution with Kafka, FastAPI, and Ably
- Building a dependable realtime banking solution with Kafka and Ably
- How to build a realtime chat solution with Kafka and Ably
- How to build a realtime betting solution with Confluent and Ably
How to get started with Kafka and the Ably Connector
We have created a step by step tutorial on everything you need to do and know to get up and running with Kafka, Ably and the Ably Connector.
Alternatively sign-up for a free developer account now!
