Apache Kafka integration

Kafka integrations enable you to automatically forward events that occur in Ably to Kafka topics.

Create a Kafka integration

To create a Kafka integration in your dashboard:

  1. Login and select the application you wish to integrate with Kafka.
  2. Click the Integrations tab.
  3. Click the New Integration Rule button.
  4. Choose Firehose.
  5. Choose Kafka.
  6. Configure the Kafka settings.
  7. Click Create.

You can also create a Kafka integration using the Control API.

Settings

The following settings are available when creating a Kafka integration:

SettingDescription
SourceSpecifies the event types being sent to Kafka.
Channel filterFilters the source channels based on a regular expression.
EncodingSpecifies the encoding format of messages. Either JSON or MsgPack.
EnvelopedCheckbox to set whether messages should be enveloped or not. Enveloped is the default.
Routing keySpecifies the routing key used to route messages to Kafka topics.
MechanismThe SASL/SCRAM mechanism used for Kafka connection.
UsernameThe username to connect to Kafka with.
PasswordThe password to connect to Kafka with.
BrokersList of Kafka broker endpoints in the format <host>:<port>.

Authentication mechanism

The available authentication mechanisms for Kafka are: