Retrieving historical data with Channel Rewind

The Ably Realtime service organizes the message traffic within applications into named channels. Channels are the “unit” of message distribution; clients attach to any number of channels to subscribe to messages, and every message published to a channel is broadcasted to all its subscribers.

Usually in applications that implement the Pub/Sub messaging pattern, a client is interested in receiving messages as they occur in real time. Ably’s API reflects this, in that an application can subscribe to a channel; the client attaches to the channel, and receives messages from that point forward.

However, in some applications it is useful to have access also to messages that occurred in the past. Ably supports a History API whereby, for channels that are persisted, it is possible to query explicitly for historical messages on a channel within any specified time bounds.

You can now persist the last realtime message on a channel for 365 days. Persist last message allows you to instantly retrieve realtime messages sent in the past, helping with long-term state synchronization. Read how to enable persist last message.

For some applications, the requirement for historical access is simply to be able to obtain the most recent message(s) on a channel. This can apply, for example, in a data streaming scenario where a data provider continuously streams data in realtime as it is generated but a new consumer just tuning in to get that data may have to sit idle until the next available update arrives, unless they have access to the previously published message.

It would be possible for an application to achieve that by subscribing to a channel and also using the history API. However, Ably now provides a much simpler way to subscribe to a channel from a specified point in the past; that is, the subscriber will be sent all historical messages on the channel from the specified point forwards, and will receive all subsequent messages in realtime.

Channel Rewind is simply a parameter you add when attaching to channels on Ably. In the present Ably libraries it is possible to specify the rewind param by qualifying the channel name. Later library releases will allow it to be specified explicitly in the API. Also, rewind being just another parameter on Ably’s existing channels offering, this new feature can be seamlessly used with any of the protocols that Ably currently supports.

In this tutorial we’ll look at implementing the rewind feature for a weather monitoring app and see examples using Ably’s realtime protocol, SSE and MQTT.

Step 1 – Create your Ably app and API key

To follow this tutorial, you will need an Ably account. Sign up for a free account if you don’t already have one.

Access to the Ably global messaging platform requires an API key for authentication. API keys exist within the context of an Ably application and each application can have multiple API keys so that you can assign different capabilities and manage access to channels and queues.

You can either create a new application for this tutorial, or use an existing one.

To create a new application and generate an API key:

  1. Log in to your Ably account dashboard
  2. Click the “Create New App” button
  3. Give it a name and click “Create app”
  4. Copy your private API key and store it somewhere. You will need it for this tutorial.

To use an existing application and API key:

  1. Select an application from “Your apps” in the dashboard
  2. In the API keys tab, choose an API key to use for this tutorial. The default “Root” API key has full access to capabilities and channels.
  3. Copy the Root API key and store it somewhere. You will need it for this tutorial.

    Copy API Key screenshot

Step 2 – Subscribe to weather data stream on the Ably Hub

As mentioned before, the Ably Hub is a marketplace for free, open or commercial data streams. The Weather Data product on it is open and free to consume. It’s also self-subscribe, meaning if you are interested in using it you can simply navigate to this product and click on the subscribe button and you’ll instantly have access to this data stream. We’ll make use of this data stream to see how channel rewind works.


Self subscribe option on Ably Hub

Step 3 – Display the data received from the product on a web page

The Weather Data product offers weather information for 11 global cities. We’ll add a drop down menu to let the user choose a city, on selection of which we’ll show the corresponding live weather information.

The information about the channels available within this product can be found in the documentation of the product.

Option 1: Consume rewound and new data using Ably’s client library SDKs

Create a new HTML file and add the following to include the drop down for various city options and a box to display the information:

<html>

<head>
    <title>Ably's rewind channel param tutorial</title>
    <script src="https://code.jquery.com/jquery-3.4.1.min.js"
        integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script>
    <script src="https://cdn.ably.com/lib/ably.min-1.js" crossorigin="anonymous"></script>
</head>

<body>
    <div>
        City:
        <select id="city-id">
            <option value="2643741">London, UK</option>
            <option value="1850147">Tokyo, Japan</option>
            <option value="1273294">Delhi, India</option>
            <option value="1796236">Shanghai, China</option>
            <option value="3448439">Sao Paulo, Brazil</option>
            <option value="5128581">New York, US</option>
            <option value="360630">Cairo, Egypt</option>
            <option value="2988507">Paris, France</option>
            <option value="2950158">Berlin, Germany</option>
            <option value="756135">Warsaw, Poland</option>
            <option value="2147714">Sydney, Australia</option>
        </select>
    </div>
    <br />
    <table style="width:100%; padding: 20px; height: 50px;" border="1">
        <tr>
            <td style="text-align: center;">
                <div class="container">
                    <h4 id="result-open-weather"></h4>
                </div>
            </td>
        </tr>
    </table>
</body>

</html>

Next, we’ll instantiate the Ably library and attach to the channels with the rewind param enabled. In the present Ably libraries it is possible to specify the rewind param by qualifying the channel name. Later library releases will allow it to be specified explicitly in the API. If the channel already has a qualifier like [meta]* or like in our case [product:ably-openweathermap/weather]*, we can simply append the existing qualifier name with ?rewind=<position> or ?rewind=<time>

Add the following after the closing <body> tag of the HTML file above to instantiate the Ably library and declare some variables which we’ll use later:

<script type="text/javascript">
  window.addEventListener("load", function () {
      var ably = new Ably.Realtime("<YOUR-API-KEY>"),
          $resultWeather = $('#result-open-weather'),
          channelWeather;
  });
</script>

Make sure to add in your own Ably API Key in the above code snippet.

We now need to retrieve the user selection from the drop down menu and use the correlated code to subscribe to that particular channel. We’ll add the rewind functionality here and set its value to 1, signifying the number of previously published messages we’d like to retrieve from this data stream, on attaching to a channel.

Add the following within your script tag:

subscribeOpenWeather($('select#city-id').val());
$('select#city-id').on('change', function () {
    subscribeOpenWeather($('select#city-id').val());
});

function subscribeOpenWeather(id) {
    // get the channel object with the rewind parameter set to 1
    channelWeather = ably.channels.get('[product:ably-openweathermap/weather?rewind=1]weather:' + id);
    let weatherDesc;
    // subscribe to this channel and update the text box when new updates arrive
    channelWeather.subscribe((msg) => {
        if ($('select#city-id').val() == msg.data.id) {
            weatherDesc = msg.data.weather[0].description;
            $resultWeather.text((msg.data.main.temp - 273.15).toFixed(2) + '°C with ' + msg.data.weather[0].description);
        }
    });
}

If you run this file in your browser now, you should see the realtime data coming through from the weather data stream. Well, even if a new update wasn’t published just when you run this app, you can still see the most recent update previously sent on this channel, thanks to the channel’s rewind param.

See the full source code on GitHub

Option 2: Consume rewound and new data using Ably’s SSE protocol adapter

Let us now look at consuming the same data stream with the rewind parameter, but only this time using Server-Sent Events (SSE) instead of Ably’s client library SDKs. Note that since we’ll use the SSE protocol, we don’t need to use any client SDK at all. We’ll make use of the EvenSource() API that comes with all major browsers by default. Replace the script in the previous step with the following:

<script>
    var $resultWeather = $('#result-open-weather'),
        oldTemperature = -50,
        channelWeather,
        eventSource;
    var apiKey = "<YOUR-API-KEY>";
    $('select#city-id').on('change', function () {
        eventSource.close();
        subscribeOpenWeather($('select#city-id').val());
    });
    subscribeOpenWeather($('select#city-id').val());
    function subscribeOpenWeather(id) {
        var channelWeather = "[product:ably-openweathermap/weather]weather:" + id;
        var URL = `https://realtime.ably.io/sse?v=1.2&key=${apiKey}&channels=${encodeURIComponent(channelWeather)}&rewind=1`;
        eventSource = new EventSource(URL);
        eventSource.onmessage = function (msg) {
            var message = JSON.parse(msg.data);
            var weatherData = JSON.parse(message.data);
            $resultWeather.text((weatherData.main.temp - 273.15).toFixed(2) + '°C with ' + weatherData.weather[0].description);
        }
        eventSource.onerror = function (error) {
            if (error.data) {
                $resultWeather.text(`Error: ${error.data}`);
            }
        }
    }
</script>

If you run this file in your browser now, you should see the realtime data coming through from the weather data stream via SSE. Notice that we haven’t used any client SDKs here. Also, we get the latest message that was previously published on that channel, even if a new update isn’t available at the time.

See the full source code on GitHub

Option 3: Consume rewound and new data using Ably’s MQTT protocol adapter

You can use any MQTT library to publish and receive messages to and from Ably. For this tutorial, we’ll make use of the MQTT NPM library, but feel free to use any that you like.

Create a new javascript file, say main.js. We’ll begin by requiring the MQTT library and setting the correct options for our client to use Ably:

var mqtt = require('mqtt')
var options = {
    keepalive: 30,
    username: '<FIRST-HALF-OF-YOUR-ABLY-API-KEY>',
    password: '<SECOND-HALF-OF-YOUR-ABLY-API-KEY>',
    port: 8883
};

Don’t forget to replace the placeholders in the above snippet with your own API key.

Next, we need to connect to Ably’s MQTT endpoint and subscribe to the channel that we are interested in, with the rewind parameter enabled of course:

var client = mqtt.connect('mqtts:mqtt.ably.io', options);
client.on('connect', () => {
    //subscribe to the channel streaming the weather data for London, UK
    client.subscribe('[product:ably-openweathermap/weather?rewind=2]weather:2643741');
});

When a message comes through as a callback to this subscribe method, we’ll simply log it to the console:

client.on('message', (topic, message) => {
    var msg = JSON.parse(message);
    console.log((msg.main.temp - 273.15).toFixed(2) + '°C with ' + msg.weather[0].description);
});

client.on('error', (topic, message) => {
    console.log(message.toString());
});

See the full source code on GitHub

Live demo

Conclusion

In this tutorial, we saw the usage of the rewind channel parameter in a real-world data streaming application. We saw three different ways to use it, i.e., using Ably’s client library SDKs, Ably’s SSE endpoint and Ably’s MQTT endpoint.

Further reading

- Channels documentation
- Ably Realtime SDK documentation
- Ably SSE documentation
- Ably MQTT documentation