Ably Kafka Connector

The Ably Kafka Connector provides a ready-made integration between Kafka and Ably. It allows for realtime event distribution from Kafka to web, mobile, and IoT clients, over Ably’s feature-rich, multi-protocol pub/sub channels. If instead you need to send data from Ably to Kafka, use a Kafka rule.

The Ably Kafka Connector is verified by Confluent as Gold, following the guidelines set forth by Confluent’s Verified Integrations Program.

You can use the Ably Kafka Connector to send data from one or more Kafka topics into Ably channels. The following diagram illustrates a typical deployment:

How does the Ably Kafka Connector work?

The Ably Kafka Connector is a sink connector built on top of Kafka Connect. It can be self-hosted or hosted with a third-party provider, the most common being the Confluent Platform. You can download it from either GitHub or Confluent Hub and install it into your Kafka Connect workers.

The Ably Kafka Connector provides a ready-made integration between Kafka and Ably via your Ably API key. Once installed, you can configure the connector with your Ably API key to enable data from one or more Kafka topics to be published into a single Ably channel or multiple Ably channels. Events are then distributed in realtime to web, mobile, and IoT clients over feature-rich, multi-protocol pub/sub Ably channels optimized for last-mile delivery.

Mapping options

The Ably Kafka Connector supports two mapping options:

• Static
• Pattern-based

Static mapping

Static mapping enables the connection of one or multiple Kafka topics to a single Ably channel. Here you provide a string for a configuration, such as for channel name. This stays the same no matter what record the connector is consuming. For example if you configure a channel with channel="global", all Kafka records will be published through that channel. This is useful if, for example, you are a sports site broadcasting updates to millions of subscribers through a single channel. The following configuration provides an example of this:

channel = basketball
message.name = news_update

Pattern-based mapping

Pattern-based mapping gives you the ability to map many Kafka topics to many Ably channels following set rules. Pattern-based mapping enables each message, a Kafka record, to be checked to determine the recipient, and then determine which channel to send it through. Pattern-based mapping provides customers with the ability to adjust their configuration by interpolating across record key and record topic values. For example, where the channel name is configured as channel="channel_#{topic}", the message will be published to a channel according to the topic name. There are two configurations that support pattern-based mapping: channel and message.name.

channel = channel_#{topic}
message.name = message_#{key}

For an events topic, the channel would translate as channel_events. For a remove key, the message.name would be configured as message_remove.

It is important to note that you can also use static mapping in conjunction with pattern-based mapping. For example, you could define the channel as channel_#{topic}, but message.name as single_message. In this instance, the channel will be mapped by pattern and the message name will be mapped statically, as shown in the following configuration:

channel = channel_#{topic}
message.name = single_message

Publishing messages with a schema

The Ably Kafka Connector supports messages that contain schema information by converting them to JSON before publishing them to Ably. To learn how to use schema registry and supported converters, see Using Kafka Connect with Schema Registry.

Install the connector

The Ably Kafka Connector is available to install from GitHub or Confluent Hub.

View detailed instructions for installation on GitHub.

Configure the connector

Configure the connector to set properties such as which Kafka topics to send data from, and which Ably channel should receive the data.

Note that the configuration method differs depending on whether you are running a single or distributed set of connect workers.

Tutorials

View the Ably Kafka Connector tutorial for an example of how to use the Ably Kafka Connector.

See also

• Firehose overview
• Kafka rules
• Ably Kafka Connector tutorial
• Ably Kafka Connector GitHub repo
• Confluent Hub