Learn how to use the Ably Kafka Connector

Kafka is an incredibly powerful tool, allowing for distributed event streaming and stream processing. It excels when it comes to acting as a traditional backend data broker. Due to its impressive throughput, scalability, and reliability, Kafka is often used to engineer large-scale event-driven systems.

What is the Ably Kafka Connector

Whilst Kafka is an excellent choice for internal event streaming and stream processing, it’s not designed to distribute data to client devices.

In an example use-case, we can imagine that we have a tracked delivery service, and are feeding in the locations of all drivers into Kafka, with Kafka doing some heavy-lifting to appropriately store and process the raw GPS data.

Once the data has been stored and processed, we need a way to make the result available to users of our service, so they can see where our drivers are.

In order to allow for all these users to access the data, we need another service to handle this distribution of data. Ably is the perfect candidate for this, as not only can it provide low-latency, highly reliable data distribution to millions of users, but it provides many of the additional bells and whistles you’d expect to keep your service secure and functional, such as Token Authentication.

The Ably Kafka Connector – a sink connector built on top of Kafka Connect – makes it simple to map your Kafka topics to Ably channels, and handles all the complexity involved in getting the data out of Kafka and into Ably.

Step 1 – Create your Ably app and API key

To follow this tutorial, you will need an Ably account. Sign up for a free account if you don’t already have one.

Access to the Ably global messaging platform requires an API key for authentication. API keys exist within the context of an Ably application and each application can have multiple API keys so that you can assign different capabilities and manage access to channels and queues.

You can either create a new application for this tutorial, or use an existing one.

To create a new application and generate an API key:

1. Log in to your Ably account dashboard
2. Click the “Create New App” button
3. Give it a name and click “Create app”
4. Copy your private API key and store it somewhere. You will need it for this tutorial.

To use an existing application and API key:

1. Select an application from “Your apps” in the dashboard
2. In the API keys tab, choose an API key to use for this tutorial. The default “Root” API key has full access to capabilities and channels.
3. Copy the Root API key and store it somewhere. You will need it for this tutorial.
   
   
   

Step 2 – Setting up the Ably Kafka Connector

The Ably Kafka Connector is open-source and available on GitHub, as well as on Confluent as a Verified Gold Connector. In this tutorial we’ll be building it into a Docker Image, so you’ll need to install Docker first. Once you’ve installed it, check it’s working correctly by typing docker help into the terminal. If it is installed correctly, you should get a bunch of information on the usage of docker as a response.

Once you have Docker, we can download the Ably Kafka Connector from GitHub. This repository contains all of the crucial code required to create the Ably Kafka Connector. There are some notable files in here which you should be aware of:

• docker-compose.yml – This Docker Compose file defines not only the building of a Docker Image for the Ably Kafka Connector, but also sets up Zookeeper, Kafka, and the necessary network settings to allow for all 3 of these images to eventually interact with one another once constructed into Containers
• config/example-connector.properties – This contains a sample of the config settings you’ll need for the Connector to work. Notably within it, topics defines the Kafka Topics you want the Connector to apply to, and channel defines the Ably Channel you’ll want to publish in to.

For this tutorial, we’ll be largely keeping our config details the same as those within example-connector.properties. The Connector is expecting a config file to exist as config/docker-compose-connector.properties, so let’s copy the example file to that:

cp config/example-connector.properties config/docker-compose-connector.properties

The contents of this file should roughly be:

name = ably-channel-sink
topics = kafka-connect-ably-example
tasks.max = 1
connector.class = com.ably.kafka.connect.ChannelSinkConnector

channel = kafka-connect-ably-example
client.key = YOUR_ABLY_API_KEY
client.id = kafka-connect-ably-example

We can leave most of the config settings the same, such as the name, topic used, and Ably Channel it will use. We do however need to provide an Ably API key for one of your Ably Apps with permission to publish to Ably.

You can use the Ably API key you got in the first step, or find it again from the API key section of your Ably App.

Replace the value in client.key with your Ably API key.

Step 3 – Running the Ably Kafka Connector

Now you have the code for the Connector and have set up the config file as desired, we need to have a kafka instance running to apply the Connector to. As part of the connector code there is a file called docker-compose.yml, which allows for the creation of various Docker Containers which will hold the required components for a Kafka instance to operate, in addition to compiling the Connector code into a Docker Image and deploying it as a Container as well.

To do this, all you should need to do is run the following in the terminal whilst in the base of the project’s directory:

docker-compose up -d

After a while you should have a kafka instance running in addition to the connector, with all the appropriate ports available to each Container as defined within the docker-compose.yml file.

Step 4 – Using the Ably Kafka Connector

Now that we have the Ably Kafka Connector attached to our Kafka instance, we can test it out. Assuming you’ve not changed the config file outside of the insertion of an Ably API key, you should have the Connector watching the Kafka Topic kafka-connect-ably-example, and looking to send messages from this topic onwards to the Ably Channel kafka-connect-ably-example.

To test the Connector is working correctly, let’s send some messages to the Kafka instance we have running:

docker-compose exec -T kafka kafka-console-producer --topic kafka-connect-ably-example --broker-list kafka:9092 <<EOF
message 1
message 2
EOF

This will run the kafka-console-producer command on the kafka container, which allows for events to be published into a topic. We specify the topic to be the one which the Connector is listening to, kafka-connect-ably-example, and define the messages to be sent to be message 1 and message 2.

Once that’s done, the messages should have been added to the Kafka Topic, and the Connector should have sent these messages on to the Ably Channel kafka-connect-ably-example.

We can check if the messages were sent correctly by making a REST request to the channel to obtain the channel’s History, a feature Ably provides to check previously sent messages on a channel. To keep this simple we can do this from the terminal with the following command, making sure to replace MY_API_KEY with the same API key you used earlier in the config file:

curl https://rest.ably.io/channels/kafka-connect-ably-example/messages \
 -u "MY_API_KEY"

If it’s worked you should see the same messages you sent in the response. Note that by default Ably only persists messages for 2 minutes, so if the messages aren’t there try sending the messages again and make the History request within 2 minutes of that.

Step 5 – Sending Messages from One Topic to Multiple Channels

With the above, we’re able to send all messages which go into a Kafka Topic to an Ably Channel. Often though it can be far more powerful to have ways to define where messages in a Kafka Topic should go dependant on certain parameters. A common example would be that chat messages sent to a group chat should all be sent to a specific group of users.

The Ably Kafka Connector allows you to do this by defining the channel parameter in the config file with variables. For example, if we specify the channel in the config/docker-compose-connector.properties file to be:

...
channel = group:#{key}
...

This will result in the Ably Channel messages are sent to to correspond in name to the key of each message. If the key were set to my_group for example, the message would be sent to the channel group:my_group.

We can test this out by stopping the current instances, starting up a new instance of Kafka and the Connector using the docker-compose file as above, and then sending a message.

docker-compose down
docker-compose up -d
docker-compose exec -T kafka kafka-console-producer --topic kafka-connect-ably-example --broker-list kafka:9092 --property "parse.key=true" --property "key.separator=:" <<EOF
my_group:message 1
my_group:message 2
EOF

In the above code when producing data, we’re specifying how we’ll include a key in the messages via the properties key.separator and parse.key. The separator is defined to be a colon (:), so the messages both have a key of my_group.

We can check if the messages have been sent to the channel we expect (group:my_group) with a call to History as we did in the previous example:

curl https://rest.ably.io/channels/group:my_group/messages \
 -u "MY_API_KEY"

Note make sure to replace MY_API_KEY with your API key.

Conclusion

With the above running, we now have the ability to send messages from Kafka into an Ably Channel. From here, we can distribute these messages to as many users as desired. This can be incredibly powerful for many use-cases, such as vehicle tracking, ticket booking systems, chat apps, and generally any scenario where you’d be interested in distributing data to end users. Find out more on these use-cases and best practices in this article.