Ably Kafka Connector

The Ably Kafka Connector integrates Kafka with Ably to enable realtime event distribution from Kafka to web, mobile, and IoT clients via Ably's channels.

The connector is Confluent Gold verified, ensuring compliance with Confluent's Verified Integrations Program. You can use it to send data from one or more Kafka topics to one or more Ably channels.

Install

The Ably Kafka Connector is a sink connector built on top of Kafka Connect.

Install the Ably Kafka Connector from:

• GitHub to run within your own infrastructure.
• Confluent Hub to run on the Confluent Platform.

Once installed, configure it with your Ably API key to enable data from Kafka topics to be published into Ably channels.

Mapping

The Ably Kafka Connector supports two mapping methods:

• Static to assign messages to a fixed Ably channel.
• Pattern-based to dynamically assign messages based on interpolation of topic and record keys.

Static mapping

Static mapping assigns one or more Kafka topics to a single Ably channel. The channel name stays the same, regardless of the Kafka record.

For example, a sports website streaming live updates can set channel = basketball, ensuring all Kafka records, regardless of their Kafka topic, publish to the basketball channel.

The following example maps all Kafka topics to the basketball channel:

channel = basketball
message.name = news_update

Pattern-based mapping

Pattern-based mapping dynamically maps multiple Kafka topics to different Ably channels. It provides the ability to adjust configuration by interpolating across record key and record topic values. Each Kafka record determines the target channel. Additionally each message is published to the Ably channel corresponding to its Kafka topic. For example, setting channel = channel_#{topic} routes Kafka messages to a channel matching their topic name.

The following example maps Kafka topics to Ably channels based on the topic name:

channel = channel_#{topic}
message.name = message_#{key}

Static and pattern-based mapping

You can use static and pattern-based mapping in conjunction. For example, you can dynamically map the channel while keeping the message name static, ensuring messages are routed to topic-specific channels while maintaining a consistent message name. The following configuration maps Kafka topics to Ably channels based on the topic name, while keeping the message name static:

channel = channel_#{topic}
message.name = single_message

Publish messages with a schema

The Ably Kafka Connector supports messages that include schema information. It converts these messages to JSON before publishing them to Ably using the Kafka Connect Schema Registry and supported converters.

For example, if messages on the Kafka topic are serialized using Avro, and schemas are registered in a Schema Registry, configure the connector to convert Avro to JSON.

Set the following properties in your Kafka Connect configuration:

value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=https://<your-schema-registry-host>

This configuration ensures Kafka messages are correctly deserialized and transformed before reaching Ably.

Configure the Kafka connector

The Ably Kafka Connector sends Kafka messages to Ably channels in realtime. Configuration depends on the installation method used:

Connector configuration properties

You must configure these core properties to get the connector working.