Firehose overview
Ably’s Firehose can stream your realtime data published within the Ably platform directly to another streaming or queueing service. For example, all messages published by any device on a channel could be immediately streamed to Amazon Kinesis allowing you to process this data in realtime.
Using configurable rules, you can stream various data-sources, including messages, presence events, occupancy, and channel lifecycle events.
Unlike channels, which follow a pub/sub pattern, where each message is delivered to any number of subscribers, Firehose operates such that each message is delivered once to your streaming or queueing server.
As each message is delivered once to your streaming or queueing server, this design is commonly used to process realtime data published by Ably asynchronously. For example, using workers consuming data from your stream or queue, you could persist each message of a live chat to your own database, start publishing updates once a channel becomes active, or trigger an event if a device has submitted a location that indicates that it has reached its destination.
Find out why Ably thinks streams and message queues help solve many of the challenges associated with consuming pub/sub data server-side in the article: Message queues — the right way to process and work with realtime data on your servers.
Note that if you want to consume realtime data from a queue, you should take a look at Ably Queues. They provide a simple and robust way to consume realtime data from your worker servers without having to worry about queueing infrastructure.
Data sources
Ably currently supports the following sources of data that can be streamed by Firehose:
- Messages – messages are streamed as soon as they are published on a channel.
- Presence – when clients enter, update their data, or leave channels, the presence event is streamed.
- Channel lifecycle – get notified when a channel is created (following the first client attaching to this channel) or discarded (when there are no more clients attached to the channel).
- Occupancy – when a channel is created (following the first client attaching to this channel) or discarded (when there are no more clients attached to the channel), the lifecycle event is streamed.
Firehose rules
You can create Firehose rules for the following supported services:
Integration skipping
Integrations can be skipped on a per-message basis by privileged users. This provides a greater degree of flexibility when publishing messages to a channel. It also prevents infinite-loops occurring, where a message published back to a channel by the receiving end of an integration is then forwarded back to itself.
Skipping integrations is especially useful in applications such as chat. For example, where a moderation function publishes a message instructing clients to edit or delete a given message, but does not want that message itself to be subject to moderation.
Skip an integration
Messages can be flagged to skip an integration by setting the skipRule field, contained in the privileged section of the message extras.
This field can be set to skip all integration rules:
const rest = new Ably.Rest('<loading API key, please wait>');
const channel = rest.channels.get('sky-top-mow');
await channel.publish({
name: 'event_name',
data: 'event_data',
extras: {
privileged: { skipRule: '*' }
}
});
Demo OnlyCopyCopied!It can also be set to skip only specific rules using the ruleId of an integration rule. This can be found in the integrations tab of your dashboard, from fetching a list of integration rules from the Control API or as part of the message envelope.
const rest = new Ably.Rest('<loading API key, please wait>');
const channel = rest.channels.get('sky-top-mow');
await channel.publish({
name: 'event_name',
data: 'event_data',
extras: {
privileged: { skipRule: ['rule_id_1'] }
}
});
Demo OnlyCopyCopied!AWS Authentication
For details of authenticating AWS services with Ably, see the AWS authentication documentation.