Token streaming

The model's output streams to every subscribed client as the model produces it. The same response is also available as one message to any client that reconnects, reloads, or replays history.

An agent streams a response with one call:

// Agent code.
const { serial } = await transport.pipe(stream, { signal: req.signal });

pipe() reads the model's stream, encodes each event through the codec, publishes to the channel, and resolves with the serial of the last message it published once the stream ends.

How it works

The codec decides how each event is written to the channel. With the codecs that ship with the SDK, each streamed part of a response, such as a text stream or a tool call's input, is one Ably message that grows by appends:

1. The first delta of a stream publishes the message.
2. Each later delta appends its text to that message.
3. The event that ends the stream is published after the stream's appends, and closes the stream.

The events around a stream, such as the start of a response and its end, are plain publishes of their own.

A subscribed client receives each append as it arrives. A client that joins later, reloads, or replays history receives the message as it is now: one delivery with the stream's text so far, and then the remaining appends live. It does not replay each token to rebuild the response.

Implement token streaming

Agent

The agent creates a transport for the conversation's channel, calls the model, and pipes the output:

// Agent code. This runs on your own infrastructure, so it holds the API key.
import * as Ably from 'ably';
import OpenAI from 'openai';
import { createTransport } from '@ably/ai-transport';
import { openai } from '@ably/ai-transport/openai';

const ably = new Ably.Realtime({ key: process.env.ABLY_API_KEY });
const transport = createTransport({ client: ably, channelName, codec: openai });

const stream = await new OpenAI().responses.create({ model: 'gpt-5.5', input, stream: true }, { signal: req.signal });

try {
  const { serial } = await transport.pipe(stream, {
    signal: req.signal,
    headers: { requestId },
  });
  await saveResponse(channelName, serial);
} finally {
  await transport.close();
}

pipe() accepts a ReadableStream or any async iterable of your codec's events. For the Vercel AI SDK, toUIMessageStream produces the right events. The OpenAI codec takes the Responses API stream as it is, the AG-UI codec takes an AG-UI event stream as it is, and for any other provider, yield your own codec's events.

The headers go on every message the pipe writes, including each append, so a client can tell which prompt each response answers.

Client

Each append reaches the client as a delivery with the decoded event.

With the OpenAI codec, a client appends each response.output_text.delta to the text of the output item it belongs to, keyed by item_id:

// Client code.
const text = new Map();
transport.subscribe(({ event }) => {
  if (event?.type !== 'response.output_text.delta') return;
  text.set(event.item_id, (text.get(event.item_id) ?? '') + event.delta);
  render(event.item_id, text.get(event.item_id));
});

Each output item has its own item_id, so the deltas of several responses stay apart, as the quickstart does.

Repair a failed append

The pipe sends each append without waiting for it, so a response streams at the speed of the connection. If an append fails, for example on a transient network issue, the pipe carries on. When the stream ends, the pipe replaces the message with an update that carries the full content, before it sends the event that ends the stream. A live client reads the missing text from the update, and history holds the whole message.

If the update fails too, the pipe rejects with PipeFailed and the partial message stays on the channel.

Append rollup

LLM token streaming produces high-rate traffic. Some models emit over 150 distinct token events per second. Ably rolls up multiple appends into a single published message, so a single response stays within the message rate limit on a connection.

1. Your agent streams tokens to the channel at the model's output rate.
2. Ably publishes the first token immediately, then rolls up subsequent tokens within the rollup window.
3. Clients receive the same content, delivered in fewer discrete messages.

By default, Ably delivers a single response stream at 25 messages per second, or the model output rate, whichever is lower. Ably charges per published message rather than per streamed token.

Configure rollup behaviour

On the client, set the rollup window for a connection using the appendRollupWindow transport parameter:

const ably = new Ably.Realtime({
  authUrl: '/api/auth/token',
  transportParams: { appendRollupWindow: 100 },
});

Edge cases and unhappy paths

• A missing channel rule fails the first append with error 93002, and the response does not stream. This is the most common reason a working agent produces nothing on the client. Configure the channel rule once per app.
• A closed tab does not stop the response. The agent keeps publishing to the channel whether or not a client is subscribed. To stop the agent when the client closes its request, pass the request's signal to pipe().
• A network drop pauses delivery to that client. The agent keeps publishing. After a brief drop the client receives the missed appends. After a longer gap, the client subscribes again with a replay and receives the response so far in one delivery.
• A cancelled pipe waits for the appends it has sent, and repairs any that failed, before it rejects with OperationCancelled. The partial response stays on the channel with all the text the pipe wrote. The pipe does not publish the events that end the response. With the Vercel AI SDK codec, publish an abort chunk with send() if your clients wait for a finish chunk.
• If you set appendRollupWindow to 0ms for the model's full output rate, you are responsible for keeping the publish rate under your connection limit.
• A repaired append is invisible to subscribers, and the message content stays consistent. If you log channel operations, you see one update at the end of each stream that had a failed append.

FAQ

What happens to the response when the client tab closes?

The agent keeps streaming, and the channel keeps the response. When the user returns, the client replays the response so far and receives any further appends in realtime.

Does Ably charge per token?

No. Ably charges at the current message rates per published message rather than per token. The append rollup reduces the publish rate, because several tokens become one published message.

How do I stream more than one message per response?

A single pipe() call reads the whole stream. Text, reasoning and tool calls within one stream all go through the same call, and the codec gives each streamed part its own message. If your agent produces several separate streams for one prompt, call pipe() once for each stream with the same headers.

Why does my client see fewer tokens than the model emits?

The append rollup combines several appends into one delivery within the rollup window. The content is identical, delivered in fewer, larger updates. Set appendRollupWindow to 0ms to deliver each append on its own, subject to the connection rate limit.

Related features

• Reconnection and recovery: receive what you missed after a disconnection.
• History and replay: replay past responses from the channel.