Reconnection and recovery

The agent keeps responding

The agent publishes each response to the conversation's Ably channel, and it keeps publishing while a client is away. If a client disconnects, the agent keeps generating the response, and every other subscribed client keeps receiving it.

If the agent's HTTP endpoint passes the request's signal to the model call, closing that request stops the response.

Automatic reconnection

The Ably SDK on the client attempts to reconnect automatically after a connection drops. If the connection comes back within about two minutes, Ably attempts to resume the channel. If it succeeds, the client receives every message published while it was away, in order.

On reconnect the client also gets awareness and artifact state back from the channel. If the client uses presence, it receives the current presence set when the channel attaches again, and the SDK enters presence again for any member that the client had entered. If the client uses LiveObjects, it receives the current artifact state. If the agent publishes its status as ephemeral messages, the client gets the status again only when the agent publishes its next update, because Ably does not store ephemeral messages.

Detect lost continuity

Sometimes the SDK cannot resume the channel. The connection may have been down for longer than about two minutes, or Ably may have detached the channel.

When the SDK attaches the channel again without continuity, Ably cannot deliver every message that the client missed. The client has lost continuity with the conversation. It may be missing part of a response, or a whole prompt and its response, and it cannot tell which.

The channel's state changes tell the client when continuity is lost:

• A change to ATTACHED, or an UPDATE, with resumed set to false means that Ably did not resume the channel.
• A change to FAILED means that the channel does not attach again by itself.

While the connection is down, the channel can go SUSPENDED. The SDK still tries to resume the channel from SUSPENDED, so the client should wait for the next ATTACHED change before it recovers.

The first attach of a channel also has resumed: false, so the client should add its listener after subscribe resolves, or the first attach starts a recovery.

Recover the conversation

After a loss of continuity, the client recovers the conversation with the same steps as a page load. It loads the conversation from your database, and rewinds the channel to the serial associated to the last stored conversation message. You should store the serial in your database to allow the client to know the position for the rewind. Ably then delivers every message published after that serial, followed by live messages.

The client recovers in these steps:

1. Remove the state change listener, so that the detach and attach in the next steps do not start another recovery.
2. Unsubscribe the decoder and detach the channel, so that the channel attaches again with the stored serial, even when that serial has not changed.
3. Load the conversation from your database, and replace what the UI shows with it.
4. Set the stored serial as the channel's rewind param with channel.setOptions, so that the rewind starts after the messages that step 3 loaded.
5. Subscribe a new decoder, and add the listener again.

The client creates a new decoder, because the old decoder has already seen messages that the rewind delivers again. channel.detach() rejects on a FAILED channel, so the client should ignore that error, and the attach in step 5 attempts to attach the channel again. Detaching the channel also removes the client from presence, so a client that entered presence should enter it again after it subscribes.

This client code loads the conversation, subscribes and recovers after a loss of continuity. loadConversation reads your database, render replaces the messages in your UI and apply updates your UI with each decoded event. Recovery can fail if the connection is still down, so onRecoveryFailed is your code, which can show an error and let the user try again:

import { createClient } from '@ably/pubsub-device';
import { openaiDecoder } from '@ably/ai-transport/openai';

const client = createClient({ authUrl: '/auth' });
const stored = await loadConversation(channelName);
render(stored.messages);
const channel = client.channels.get(channelName, stored.serial ? { params: { rewind: stored.serial } } : undefined);
let decoder;

async function subscribe() {
  decoder = openaiDecoder();
  decoder.onEvent(apply);
  await channel.subscribe(decoder.receive);
  channel.on(['attached', 'update', 'failed'], onStateChange);
}

function onStateChange(change) {
  if (!change.resumed) recover().catch(onRecoveryFailed);
}

async function recover() {
  channel.off(onStateChange);
  channel.unsubscribe(decoder.receive);
  await channel.detach().catch(() => {});
  const stored = await loadConversation(channelName);
  render(stored.messages);
  if (stored.serial) await channel.setOptions({ params: { rewind: stored.serial } });
  await subscribe();
}

await subscribe();

When the agent stops mid-response

When the adaptor's pipe is cancelled or fails, the pipe call does not automatically publish an error or cancellation message to the channel. Subscribed clients will not know about the cancellation unless the agent publishes a message to the channel informing them of the cancel, stop, or error.

The agent should publish a terminal event whenever it stops a response, so that every subscribed client can close the response. With the Vercel AI SDK, the terminal event for a stopped response is the abort chunk. With OpenAI or a custom codec, your application decides which event the agent publishes to end a stopped response.

If the agent process stops mid-response, the agent cannot publish a terminal event. The messages it published so far stay on the channel, and subscribed clients do not receive a terminal event.

A client can detect that the agent stopped in these ways:

• If the agent sets a requestId header on each message of a response, no further messages arrive with that header.
• If the agent entered presence, Ably removes it from the presence set about 15 seconds after its connection drops, and a client subscribed to presence receives a leave event.

The client can also use a timeout to decide that a response has stopped.

Edge cases

• The channel keeps its rewind param until the client changes it. The recovery sets the newer serial with channel.setOptions, so the rewind does not deliver messages that your database already holds.
• A rewind delivers at most 1,000 messages after the serial, and Ably counts a message and its appends as one message. Past that limit, Ably delivers the most recent 1,000 messages and reports a non-fatal error when the channel attaches, so the conversation has a gap just after the stored serial. The client can fill the gap from channel history.
• Rewind and channel history reach back only as far as your retention period. A client that was away for longer can read the older messages from your database.
• If your database does not have a serial, the client cannot rewind. With the typical prompt path, the agent should store the prompt's serial before it streams the response, so the database has no serial only before the first prompt.
• Rewind needs only the subscribe capability, so a client without history can still recover the conversation. Reading channel history past the rewind limit needs history.

FAQ

How long can a client be offline before it must catch up?

If a connection drops for up to about two minutes, Ably attempts to resume it and deliver the messages published in that time. After a longer gap, the channel may attach with resumed: false, and the client must then recover the conversation from your database.

Does a short drop need any code?

No. The Ably SDK attempts to resume the connection, and if it succeeds, the client receives the messages published during the drop.

Why does my client show responses twice after reconnecting?

The channel still has the rewind param it got on page load, the client reused the old decoder, or the client added the stored messages to the UI without replacing what it showed. The recovery sets the newer serial, subscribes a new decoder and replaces what the UI shows.

What if the agent process dies mid-response?

The partial response stays on the channel, and the agent cannot publish a terminal event for it. A client can detect the stopped agent from the response's messages and, if the agent entered presence, from presence.

Related features

• History and replay: store the serial, rewind the channel to it and read older messages.
• Token streaming: stream a response to every subscribed client as the model writes it, and stop it.
• Multi-device and fan-out: participate in a single conversation from several devices or users.