Delta compression

The delta parameter enables delta compression. It is applied on the channel you are subscribing to, enabling delta mode.

Delta mode is a way for a client to subscribe to a channel so that message payloads sent contain only the difference (ie the delta) between the present message and the previous message on the channel.

This is useful for channels that carry messages representing a series of updates to a particular object or document with a significant degree of similarity between successive messages. The client can apply the delta to the previous message to obtain the full payload. Using delta mode can significantly reduce the encoded size of each message in the case that message payloads change by differences that are small relative to the size of the value. This reduction in size can reduce bandwidth costs, reduce transit latencies, and enable greater message throughput on a connection.

The delta mode implementation supports a single representation of a delta, VCDIFF.

Subscribing in delta mode is enabled for a given channel by specifying a delta channel parameter with the value vcdiff. This will cause delta messages to be generated by the server and sent to the client, and the library reconstitutes the original message payload. Messages on the channel are delivered to the subscriber’s listener in the same way as with a normal subscription.

Deltas apply to the principal payload of a Message published via Ably, which is the data member. Other elements of a message, such as clientId, name, or extras are unchanged by use of deltas and are not compressed.

Deltas are supported for realtime subscriptions only. Messages retrieved via history, and messages delivered to Ably Integrations, are not compressed.

Delta compression via vcdiff is supported for all payloads, whether string, binary, or JSON-encoded. The delta algorithm processes message payloads as opaque binaries and has no dependency on the structure of the payload – it does not process line-oriented diffs, for example. In principle, vcdiff deltas can be applied to encrypted message payloads, but in practice this provides no benefit because there is no similarity between successive encrypted payloads even on identical or near-identical plaintext message payloads.

Delta compression is a subscriber-specified option only – the publisher has no control over whether or not deltas are generated for any given message; the processing is performed if there is at least one subscriber on a channel that has requested a delta-mode subscription.

There is no constraint on how many publishers or subscribers there are. If there are multiple publishers, then deltas can still be generated, and they will be determined based on the order of messages in the channel in question.

Delta mode, when activated on a channel, is performed for all messages on a channel, and deltas are calculated strictly based on the message ordering in that channel. The effectiveness of delta mode is dependent on the level of similarity between successive payloads.

If a delta is generated and it results in a difference that is not appreciably smaller than the original message, or is even larger than the original message (which can happen if successive messages are completely different), then the delta will not be sent and clients will receive the original, unprocessed message. Therefore, the sequence of messages that will be delivered to a client for any given channel will be a combination of regular messages and delta-compressed messages.

A channel subscriber can experience a discontinuity in the sequence of messages it receives on a given channel for the following reasons:

• The connection can drop, and there will be a discontinuity if the client is unable to reconnect within the two-minute window it is allowed to preserve connection continuity.
• The outbound connection might have been rate-limited, which causes some messages to be dropped.
• There might have been some internal error in the Ably system which leads to the server being unable to preserve continuity on the channel.

In these cases, the service indicates the discontinuity to the client, together with the reason, and this is usually visible to the subscriber in a channel UPDATE event.

If a subscriber has a delta-mode subscription and the channel in question experiences a discontinuity, then a non-delta message will be delivered to the client as the first message after the discontinuity. This ensures that lost messages do not prevent the client from reconstituting messages from deltas.

The most common way to subscribe to Ably channels is via a realtime connection, using an Ably realtime library.

For many libraries this requires no change on the part of the caller except to specify the delta channel parameter when subscribing to the channel. In some libraries, the vcdiff delta decoding library is excluded from the default library distribution in order to avoid bloating the library. In these cases, it is also necessary to supply the delta decoder plugin when instancing the Ably library.

JavaScriptNode.jsJavaSwiftC# .NET
JavaScript

/* Make sure to include <script src="//cdn.ably.com/lib/vcdiff-decoder.min-1.js"></script> in your head */
const realtime = new Ably.Realtime({
  key: '<loading API key, please wait>',
  plugins: {
    vcdiff: vcdiffDecoder
  }
});
realtime.channels.get('sax-pan-boo', { 
  params: { 
    delta: 'vcdiff' 
  } 
}).subscribe(msg => console.log("Received message: ", msg));

API Key:
Demo Only
Copy
Copied!

If subscribing to a channel in delta mode using SSE or one of the protocol adaptors such as MQTT, then you will need to decode any received delta messages yourself. There are decoder libraries available to do this for several platforms; see the download section for details.

When subscribing without an Ably library, the channel delta parameter must be specified using a qualified channel name. In the case of SSE, it is also possible to specify channel parameters as regular query parameters on the connection URL.

Some transports provide raw message payloads – that is, the content of the data attribute of a Message – without the accompanying metadata. That means that the recipient of the message does not have access to the extras or encoding attributes of the message that would ordinarily be used to decode delta message payloads.

Examples of such transports are MQTT, and SSE in non-enveloped mode. In order to assist applications that use these transports, the vcdiff decoder libraries can check for the vcdiff header at the start of the message payload as an inexact method of determining whether or not the message is a regular message or a delta. Note that, in order to rely on that check, you need to know that that header will not be present in any valid (uncompressed) message in your app. No valid JSON value, for example, will match the vcdiff header check, so it is safe to perform this sniffing on JSON message payloads.

You can subscribe to messages in delta mode, using the SSE transport, as follows.

/* Make sure to include <script src="https://cdn.ably.com/lib/delta-codec.min-1.js"></script> in your head */
var key = '<loading API key, please wait>';
var channel = 'sax-pan-boo';
var baseUrl = 'https://realtime.ably.io/event-stream';
var urlParams = `?channels=${channel}&v=1.1&key=${key}&delta=vcdiff`;
var url = baseUrl + urlParams;
var eventSource = new EventSource(url);
var channelDecoder = new DeltaCodec.CheckedVcdiffDecoder();


eventSource.onmessage = function(event) {
  /* event.data is JSON-encoded Ably Message
     (see https://ably.com/docs/realtime/types#message) */
  var message = JSON.parse(event.data);
  var { id, extras } = message;
  var { data } = message;


  try {
    if (extras && extras.delta) {
      data = channelDecoder.applyBase64Delta(data, id, extras.delta.from).asUtf8String();
    } else {
      channelDecoder.setBase(data, id);
    }
  } catch(e) {
    /* Delta decoder error */
    console.log(e);
  }


  /* Process decoded data */
  console.log(data);
};

API Key:
Demo Only
Copy
Copied!

For more information on enveloped and unenveloped SSE, please see the SSE API

/* Make sure to include <script src="https://cdn.ably.com/lib/delta-codec.min-1.js"></script> in your head */
var DeltaCodec = require('@ably/delta-codec');


var key = '<loading API key, please wait>';
var channel = 'sample-app-sse';
var baseUrl = 'https://realtime.ably.io/event-stream';
var urlParams = `?channels=${channel}&v=1.1&key=${key}&delta=vcdiff&enveloped=false`;
var url = baseUrl + urlParams;
var eventSource = new EventSource(url);
var channelDecoder = new DeltaCodec.VcdiffDecoder();


eventSource.onmessage = function(event) {
    var data = event.data;


    try {
        if (DeltaCodec.VcdiffDecoder.isBase64Delta(data)) {
            data = channelDecoder.applyBase64Delta(data).asUtf8String();
        } else {
            channelDecoder.setBase(data);
        }
    } catch(e) {
        /* Delta decoder error */
        console.log(e);
    }


    /* Process decoded data */
    console.log(data);
};

API Key:
Demo Only
Copy
Copied!

var mqtt = require('mqtt');
var { VcdiffDecoder } = require('@ably/vcdiff-decoder');


var options = {
  keepalive: 30,
  username: '{{API_KEY_NAME}}', /* API key's name */
  password: '{{API_KEY_SECRET}}', /* API key's secret */
  port: 8883
};
var client = mqtt.connect('mqtts:mqtt.ably.io', options);
var channelName = 'sample-app-mqtt';
var channelDecoder = new VcdiffDecoder();


client.on('message', (_, payload) => {
  var data = payload;


  try {
    if (VcdiffDecoder.isDelta(data)) {
      data = channelDecoder.applyDelta(data).asUint8Array();
    } else {
      channelDecoder.setBase(data);
    }
  } catch(e) {
    /* Delta decoder error */
    console.log(e);
  }


  /* Process decoded data */
  console.log(data);
});


client.subscribe(`[?delta=vcdiff]${channelName}`);

Copy
Copied!

View the Channel Options API Reference.