Outbound streaming overview

Ably's streaming integrations enable you to stream data that's published in the Ably platform to an external streaming or queueing service.

For example, any device that publishes messages on a channel can immediately stream those messages to Amazon Kinesis, so that you can process the data in realtime.

Each message is delivered once to your streaming or queueing server, making the system well-suited for asynchronous processing of realtime data published by Ably. For example, workers consuming data from your stream or queue can persist each message of a live chat to your own database, start publishing updates when a channel becomes active, or trigger an event if a device submits a location indicating it has reached its destination.

There are two ways to create an outbound streaming integration:

• Using the Ably dashboard.
• Using the Control API.

Create an outbound streaming integration using the dashboard

You can create outbound streaming integrations in your dashboard: for the following services:

• Amazon Kinesis
• Amazon SQS
• AMQP
• Apache Kafka
• Apache Pulsar

Create an outbound streaming integration using the Control API

To create a new integration for an Ably application, send a POST request to /apps/{app_id}/rules with the integration configuration in the request body.

Event types

You can configure webhooks to listen for the following event types:

Channel filter

Set a filter to restrict which channels an integration applies to using a regular expression.

The following examples demonstrate channel names that you can match against using regular expressions to control which channels a webhook rule applies to:

mychannel:public
public
public:events
public:events:conferences
public:news:americas
public:news:europe

Enveloped messages

Message enveloping adds structured metadata such as the publisher's clientId and the originating channel name, alongside the message payload.

This metadata is useful when processing events dynamically or when additional context about the message source is required. Enveloped messages are recommended for most use cases, as they provide a consistent format for all events.

Enveloped messages include the following headers:

Each enveloped message contains the following fields:

In addition, it will contain another field which will contain the actual message. This field is named according to the message type.

The following is an example of an enveloped message payload:

{
  "source": "channel.message",
  "appId": "aBCdEf",
  "channel": "channel-name",
  "site": "eu-central-1-A",
  "ruleId": "1-a2Bc",
  "messages": [{
    "id": "ABcDefgHIj:1:0",
    "connectionId": "ABcDefgHIj",
    "timestamp": 1123145678900,
    "data": "some message data",
    "name": "my message name"
  }]
}

Decode enveloped messages

Ably SDKs automatically decode messages into Message objects. Messages sent via an integration need to be decoded manually.

There are two methods available for decoding messages into Message objects:

• Message.fromEncodedArray() for an array of messages.
• Message.fromEncoded() for single messages.

There are also equivalent methods for decoding presence messages into PresenceMessage objects:

• PresenceMessage.fromEncodedArray() for an array of presence messages.
• PresenceMessage.fromEncoded() for single messages.

Decoding is essential because it reconstructs the original data payload using the encoding field, ensuring the correct data type is restored, whether it's a string, binary, or structured object. If the message was encrypted, passing your encryption key to the method allows the SDK to decrypt data automatically.

Ably strongly recommends decoding all messages received over integrations before processing them to avoid issues with unexpected data formats.

Non-enveloped messages

You can turn off enveloping if you only need the raw message payload or the external service follows a strict data structure. This results in smaller payloads and eliminates the need to parse additional metadata. However, it requires you to handle raw payload decoding manually.

Non-enveloped messages use a more straightforward format, delivering events directly without additional wrapping. Instead of a structured envelope, key properties are exposed through HTTP headers alongside the raw payload.

For example, if you publish a JSON message to the channel my_channel using the following cURL request:

curl -X POST https://rest.ably.io/channels/my_channel/messages \
          -u "{{API_KEY}}" \
          -H "Content-Type: application/json" \
          --data '{ "name": "publish", "data": "example" }'

The received message will include:

• x-ably-message-name: publish
• Payload: example

The following example demonstrates a non-enveloped presence event:

realtime = new Ably.Realtime({
  key: 'demokey:*****',
  clientId: 'bob'
});
channel = realtime.channels.get('some_channel');
await channel.presence.enter('some data');

The received presence message will include:

• x-ably-message-action: enter
• Payload: "some data"

Non-enveloped structure

Non-enveloped messages include headers that provide essential context about the payload, such as its source, format, and metadata. The following headers are included in all non-enveloped messages: