Realtime Protocol Definition

The Ably Realtime protocol operates over a connection-oriented, framed, transport. It supports a single connection (and thus a single application) at a time, and enables traffic belonging to multiple channels to be sent over that single connection.

In the current Ably service the backend supports the Realtime protocol over a WebSocket transport and over a Comet transport. Other transports are possible and this document aims to define the protocol in a transport-independent manner.

The protocol supports both a text-based format using JSON and a binary format using MessagePack. All structures passed on the wire are defined in a JSON-like format. Other encodings may be considered in the future.

The unit of any protocol transmission is a ProtocolMessage key value Hash object, referred to in this description as a Protocol Message.

The WebSocket transport transmits Protocol Messages in a single WebSocket data frame. In the binary transport, these are sent as binary data frames, with the Protocol Message being encoded with MessagePack. In the text transport these are sent as text data frames, the text itself being the white-space-free JSON encoding of the Protocol Message. Empty fields may be encoded as null, or may be handled as undefined (and thus absent from the JSON encoding) and clients should handle both possibilities. Handling empty fields as undefined is clearly preferable, however, since the encoded text is shorter and the encode and decode overhead is minimized. Binary data payloads are handled in the same was as JSON encoded payloads are handled.

In the Comet protocol, all request and response bodies contain an array of one or more protocol messages. In the binary protocol this is standard MessagePack binary encoding of a MessageSet Hash object. In the text (JSON) protocol request and response bodies are simply JSON-encoded arrays containing the Protocol Message content, again either in binary or text encoding.

Separate sections provide more detailed information on the WebSocket and Comet transports.

Each Protocol Message has an action that indicates the nature of the message.

HEARTBEAT (0)

A heartbeat message is sent periodically by the service over a connection in order to keep the connection alive (anticipating that proxies or other gateways may close the connection if it is idle indefinitely) and, in the Comet case, to prevent HTTP request timeout errors. The heartbeat interval is a configurable property of the realtime service and is not selectable by the client.

Heartbeat messages are not exposed to the client app, and are silently consumed in the transport layer of the client library. However, client library unit tests will typically wish to check for the presence of a heartbeat to confirm that a connection is intact, and therefore client libraries should expose some means for tests to observe the occurrence of heartbeats.

No other message fields are populated in a heartbeat message.

Note that to disconnect a transport without destroying connection state, the client simply closes the transport without sending any protocol message.

CLOSED (8)

Passed by the service back to the client to signify that a connection has been in response to a CLOSE request from the client. The transport will be closed by the service immediately after, and all connection state for that connection will be disposed. No connection state recovery is possible.

ERROR (9)

Passed by the service to a client to signify an unrecoverable error condition. The scope of the error might be the connection, or a single channel. ERROR messages that apply to a channel have the channel field populated with the channel name; ERROR messages that apply to the connection have no channel field value. ERROR messages will have a value in the error field that provides information about the failure condition. Client libraries must make a transition (either of the connection or the relevant channel) to the failed state, and pass the contained error information to the client.

Note that non-fatal errors may occur in various contexts (notably connection state recovery) and CONNECTED and ATTACHED response messages may also have the error field populated with information about those non-fatal conditions. These are handled differently from the unrecoverable conditions indicated with an ERROR message.

Clients must silently ignore an ATTACHED response if the channel is not in the pending state.

DETACH (12)

Sent by a client to the service to request detachment to a channel. The request must include the channel name in the channel field. Detachment is acknowledged by the service, but the client library must place the channel object into the detached state immediately on sending the DETACH request.

DETACHED (13)

Sent by the service to a client to indicate successful detachment from a channel. The message contains the channel name in the channel field, and may also contain non-fatal error information in the error field. Clients must move the channel to the detached state if not already detached and, where present, pass the error information to the caller.

PRESENCE (14)

Indicates that the Protocol Message has a payload of one or more PresenceMessages associated with a single channel. PRESENCE messages may be sent in either direction. The channel associated with these presence updates is indicated in the channel field.

The serial number of the message must be included in the msgSerial field (in the client → service direction) or in the connectionSerial field (in the service → client direction). Further information on message serial numbering is given below.

The serial number of the message must be included in the msgSerial field (in the client → service direction) or in the connectionSerial field (in the service → client direction). Further information on message serial numbering is given below.

SYNC (16)

Currently reserved for us with presence member synchronization following a channel ATTACHED Protocol Message being received by the client. Once a channel becomes attached, the server will automatically send a list of all members present on the channel to the client. Every SYNC Protocol Message received will contain a channelSerial value and one or more PresenceMessages for each member currently present on the channel i.e. they are in the PRESENT state. The channelSerial serves two purposes, it provides a way for the client library to resume a SYNC should the transport be disconnected, and it also provides a means for the client library to know when the sync is complete. A channelSerial that is being synced will contian an ID with followed by a cursor after a colon such as “cf30e75054887:psl_7g:client:189″, however the final page of SYNC messages will have a serial with an empty cursor such as “cf30e75054887:”. A client can explicitly request a SYNC with an optional channelSerial; if no channelSerial is provided the server will send a complete set of members on the channel; if a channelSerial is provided, the server will resume the SYNC operation.

AUTH (17)

Sent by the client with a new token to reauthenticate the connection, with the connection either being closed due to incompatible token details being provided, or a CONNECTED message being sent back to the client confirming the authentication succeeded. The server can request that the client authenticates by sending an AUTH protocol message to the client, and the client must respond with a new token in an AUTH protocol message.

ProtocolMessages are populated with one or more of the following fields.

i32 action

Indicates the purpose of the message. See Actions above.

string id

Unique identifier for each protocol message

AuthDetails auth

Object used for providing authentication details

string channel

Present when protocol message applies to a single channel

string channelSerial

Contains a serial number for a message on the channel

i32 count

The count field is used for ACK and NACK actions. See message acknowledgement protocol.

string connectionId

Contains a public string connection ID. This field is populated in the first CONNECTED Protocol Messages from the service to the client. The connection ID is a public identifier used to uniquely identify each connected client.

string connectionKey

Contains a private string connection Key. Note that this field is soon to be deprecated; when ConnectionDetails#connectionKey is present, it should be considered the definitive connectionKey for the current connection

ConnectionDetails connectionDetails

provides details on the constraints or defaults for the connection such as max message size, client ID or connection state TTL

i64 connectionSerial

Contains a serial number for a message on the current connection, in MESSAGE and PRESENCE protocol messages sent from the service to the client. The connectionSerial is a zero-based, serially increasing number which, in combination with the connectionId, uniquely identifies an attempted delivery of a Protocol Message to the client. The client uses the connection serial to track the receipt of messages, and may specify the connectionSerial when performing connection state recovery.

Error error

Contains error information. See Error type description for details of the contained information. The error field is populated in an ERROR message and may also be populated to provide supplementary information (eg for non-fatal errors) in various other message types (CONNECTED, ATTACHED, DETACHED, ACK, NACK).

i32 flags

Currently used to flag properties in messages such as the presence sync state of an ATTACHED ProtocolMessage. See client library spec TR4i

i64 msgSerial

Contains a serial number for a message sent from the client to the service. The msgSerial is a zero-based, serially increasing number which, in combination with the connectionId, uniquely identifies the message across the system. The msgSerial is also used in the message acknowledgement protocol.

list<Message> messages

A ProtocolMessage with a MESSAGE action contains one or more messages belonging to a channel. The messages field of the ProtocolMessage contains a collection of messages.

list<Presence> presence

A ProtocolMessage with a PRESENCE action contains one or more presence updates belonging to a channel. The presence field of the ProtocolMessage contains a collection of presence messages.

i64 timestamp

An optional timestamp, applied by the service in messages sent to the client, to indicate the system time at which the message was sent. Note that this differs from the timestamp field of a Message or PresenceMessage which is an indication of the timestamp of receipt of that message by the system.

Currently there are no requirements for the client library to process or populate the timestamp.

The protocol relies on a number of structs embedded in Protocol Messages.

An object containing error information. This is used for unrecoverable error conditions (relating to connections, channels or individual messages) and also “informatively” for non-fatal errors in ATTACHED or CONNECTED responses.

Error contains the following fields.

i16 statusCode

An optional HTTP response code that most closely matches the nature of the error. This is typically also the actual HTTP response code if this error is reported as an HTTP response to a Comet request.

i16 code

An Ably-specific error code that indicates the specific error condition. The various codes are defined in errors.json. Implementations may use errors.json to provide descriptive error messages to clients.

string reason

An optional string message that indicates the error condition, and may contain specific identifying information (e.g. channel name or message serial, for example).

In transports that support JSON encoding, Strings and the JSON Object and Array types are represented as their natural JSON value in the enclosing type. For example, a Message with a string payload would be encoded in JSON (with white-space added here for clarity) as:

{
  "name": "my_event",
  "data": "my_string_payload"
}
Copy
Copied!

For string and binary payload types, an encoded member is optionally added to the enclosing type. The only supported encodings are “utf8″ for strings, “json” for JSON and “base64″ for binary. If the encoding member is omitted it defaults to “utf8″.

Therefore, the following encoded messages each have string type:

{
  "name": "my_event",
  "data": "my string payload"
}
{
  "name": "my_event",
  "data": "my string payload",
  "encoding": "utf8"
}
Copy
Copied!

The following encoded message has binary type:

{
  "name": "my_event",
  "data": "bXkgYmluYXJ5IHBheWxvYWQ=",
  "encoding": "base64"
}
Copy
Copied!

The base64 encoding used is RFC4648 and clients must accept and process values with or without linefeeds.

The following encoded message has JSON type:

{
  "name": "my_event",
  "data": "{\<a href="\">id\</a>"value\"}",
  "encoding": "json"
}
Copy
Copied!

This is an individual channel message.

The members are as follows.

string id

A globally unique identifier for this message

string name

The optional event name.

string clientId

The optional clientId of the client that sent the message.

Client libraries do not need to populate this field if the clientId is implicit (ie a clientId was specified when the library was initialized, and is therefore connection-wide. Also, this field will be empty if no clientId has been specified either on library initialization or when publishing the message.

Messages sent from the service to the client will contain a clientId if one is available.

This is an individual channel presence update, as defined in the Thrift TPresence struct.

The members are as follows.

string id

A globally unique identifier for this presence message

Action state

The presence action (ABSENT, PRESENT, ENTER, LEAVE, or UPDATE) encoded as the ordinal in the PresenceMessage Action enum.

string clientId

The clientId string.

string or binary data

The optional payload data for the members such as member status, binary is supported when using MessagePack.

string connectionId

An optional member connection identifier if required to disambiguate multiple connected and entered members having the same clientId. When there are multiple members with the same clientId entered to the channel there will be multiple corresponding ENTER events.

If the connectionId of an already-entered member changes (eg in the situation that a new connection inherits from another connection, the new (clientId, connectionId) combination is not considered to be a new member but it is indicated as an UPDATE of the already-entered member, with a change in the value of connectionId.

The Ably client API allows a caller to provide a success callback when publishing messages, or presence updates, to the Ably service. The callback is called, either with success or a failure code, once the Ably service has indicated whether or not it has processed the message successfully. The callback is not simply an indication that the message sent without error; it is confirmation that the service has processed the message sufficiently that its onward delivery to relevant attached clients is now guaranteed.

In the Comet transport, success or failure is indicated on a per-call basis with an ACK or NACK Message body in the HTTP response to the send API call.

In the WebSocket transport, the service indicates success or failure to the client with the message acknowledgement protocol. This is a simple series of ACK or NACK responses, each addressing a contiguous sequence (by msgSerial) of messages.

An ACK message contains a msgSerial and count value. Receipt of this message signifies that the messages whose serial numbers are:

{ msgSerial ... msgSerial + count - 1 }
Copy
Copied!

have been processed successfully.

Similarly, a NACK message contains a msgSerial and count value and usually also an error value. Receipt of this message signifies that the messages whose serial numbers are:

{ msgSerial ... msgSerial + count - 1 }
Copy
Copied!

have encountered processing failures. The client library must call the callback, if supplied, with the contained error value, or with an error value that indicates an internal error.

ACK responses are sent so as to be responsive to the client but also to avoid sending a response for every single published message; when the client is publishing at a high rate the ACK responses will be sent periodically with an interval of 500ms (say) so many hundreds of messages my be covered in a single ACK response.

NACK responses are sent as soon as an error has arisen, and in any event only cover multiple messages to the extent that those messages are subject to the same underlying failure (since the error information is provided once for the group of NACK‘d messages, not individually.

It is a protocol error if the system sends an ACK or NACK that skips past one or more msgSerial without there having been either and ACK or NACK; but a client in this situation should treat this case as implicitly NACKing the skipped messages.

It is also a protocol error if the system sends an ACK or NACK that covers a msgSerial that was covered by an earlier ACK or NACK; in such cases the client library must silently ignore the response insofar as it relates to msgSerials that were covered previously (whether the response is the same now or different).