# Token streaming
Tokens are streamed to subscribing clients in realtime, as the model generates them. The same response is available as a single aggregated message to clients connecting later. AI Transport streams tokens by appending to one durable message on the session.
Tokens stream from the model to every connected client as the LLM generates them. The same response is also available as a single coherent message to any client that reconnects, refreshes, or loads history.
A minimal server-side stream uses one call:
#### Javascript
```
const { reason } = await run.pipe(result.toUIMessageStream());
```
That single line reads the LLM stream, encodes tokens through the codec, publishes messages to the session, handles abort signals, and returns when the stream completes or is cancelled.
## How it works
The transport layer treats a streamed response as one logical message built incrementally by appending each token to a single Ably channel message. A realtime subscriber receives each appended token as it arrives. A client that joins later, refreshes, or reconnects sees the accumulated content of that message up to the latest append; it does not need to replay each token to rebuild the response.
A streamed message moves through three states:
| State | Meaning |
| --- | --- |
| `streaming` | Tokens are being appended. The message grows as tokens arrive. |
| `complete` | The stream completed normally. The message is final. |
| `cancelled` | The stream was cancelled. The partial message is preserved. |
The stream status is carried in the message header `status` under the `extras.ai.codec` tier. Clients check this to detect whether a message is still streaming.
## Implement token streaming
### Server
The server creates a turn, invokes the LLM, and streams the response:
#### Javascript
```
import { Invocation } from '@ably/ai-transport';
import { createAgentSession } from '@ably/ai-transport/vercel';
const invocation = Invocation.fromJSON(await req.json());
const session = createAgentSession({ client: ably, channelName: invocation.sessionName });
await session.connect();
const run = session.createRun(invocation, {}, { signal: req.signal });
// Rebuild the conversation from run.view before run.start(): draining pages in
// this run's triggering input (otherwise run.start() awaits it arriving live).
while (run.view.hasOlder()) {
await run.view.loadOlder();
}
const conversation = run.view.getMessages().map(({ message }) => message);
await run.start();
const result = streamText({
model: anthropic('claude-sonnet-4-20250514'),
messages: conversation,
abortSignal: run.abortSignal,
});
const { reason } = await run.pipe(result.toUIMessageStream());
await run.end({ reason });
await session.end();
```
`Run.pipe` accepts any `ReadableStream`. For Vercel AI SDK, `result.toUIMessageStream()` provides the right format. For other frameworks, produce a `ReadableStream` of your codec's event type.
### Client
With Vercel's `useChat`:
#### Javascript
```
const { chatTransport } = useChatTransport();
const { messages } = useChat({ transport: chatTransport });
```
With the generic hooks:
#### Javascript
```
const { messages } = useView();
// Each message updates in realtime as tokens are appended on the channel.
```
## Under the hood
The codec converts domain events to Ably operations:
- Start: create a new Ably message on the channel.
- Append: append content to the existing message (Ably message append operation).
- Close: append a terminal status (`complete` or `cancelled`) to the message.
If an append fails, for example due to a transient network issue, the encoder falls back to a full message update operation to recover. The accumulated response is never lost.
## Append rollup
LLM token streaming produces high-rate traffic. Some models emit over 150 distinct token events per second. AI Transport rolls up multiple appends into a single published message, so a single response does not hit the [message rate limit](https://ably.com/docs/platform/pricing/limits.md#connection) on a connection.
1. Your agent streams tokens to the channel at the model's output rate.
2. Ably publishes the first token immediately, then rolls up subsequent tokens within the rollup window.
3. Clients receive the same content, delivered in fewer discrete messages.
By default, Ably delivers a single response stream at 25 messages per second, or the model output rate, whichever is lower. Ably charges per published message rather than per streamed token.
### Configure rollup behaviour
On the client, set the rollup window for a connection using the `appendRollupWindow` [transport parameter](https://ably.com/docs/pub-sub/api/javascript/realtime/realtime-client.md#constructor-params):
| `appendRollupWindow` | Maximum message rate for a single response |
|---|---|
| 0ms | Model output rate |
| 20ms | 50 messages/s |
| 40ms (default) | 25 messages/s |
| 100ms | 10 messages/s |
| 500ms (maximum) | 2 messages/s |
#### Javascript
```
const ably = new Ably.Realtime({
authUrl: '/api/auth/token',
transportParams: { appendRollupWindow: 100 },
});
```
## Edge cases and unhappy paths
- A network drop during streaming pauses delivery to the affected client. The server keeps publishing. On reconnect, the client receives the accumulated content of the message up to the latest append rather than a replay of every token.
- A cancelled stream leaves the partial message on the session with status `cancelled`. Render it the same as a complete message; treat the absence of further tokens as the signal to stop animating.
- If `appendRollupWindow` is set to `0ms` to maximise model output rate, you become responsible for keeping the publish rate under your connection limit.
- An append fallback (full message update) is invisible to subscribers; the message content is consistent. If you log channel operations, you see periodic updates instead of appends.
- A turn that times out on the server before the stream finishes ends with run reason `'error'`. The partial message is closed with status `cancelled`.
## FAQ
### What happens to the stream when the client tab closes?
The agent keeps streaming. The session and its messages persist. When the user returns, the client loads the accumulated content of the message and receives any further tokens in realtime.
### Does Ably charge per token?
No. Ably charges at the current [message rates](https://ably.com/docs/platform/pricing.md) per published message rather than per token. The append rollup reduces the publish rate; multiple tokens become one published message.
### How do I stream more than one message per turn?
A single `run.pipe(stream)` consumes the LLM's output stream and appends each chunk as it arrives. Multiple assistant messages, tool calls, and follow-on text within one stream all flow through the same `pipe` call. If your framework produces several discrete streams in one turn (for example a planner emits a status line, then the responder streams the answer), call `run.pipe` once per stream; the run is the unit that groups them.
### Why does my client see fewer tokens than the model emits?
The append rollup compacts multiple tokens into single published messages within the rollup window. The content is identical; the delivery is fewer, larger updates. Set `appendRollupWindow` to `0ms` to disable rollup and deliver every model token as its own message, subject to the connection rate limit.
### What status do I see on a cancelled response?
The message keeps the content it had at the time of the cancel and its `status` header transitions to `cancelled`. Use this to distinguish a partial response from a complete one.
## Related features
- [Cancellation](https://ably.com/docs/ai-transport/features/cancellation.md): stop a stream mid-response.
- [Reconnection and recovery](https://ably.com/docs/ai-transport/features/reconnection-and-recovery.md): resume streams after disconnection.
- [History and replay](https://ably.com/docs/ai-transport/features/history.md): load past streamed responses from channel history.
## Related Topics
- [Agent presence](https://ably.com/docs/ai-transport/features/agent-presence.md): Show agent status in your AI application with Ably Presence. Display streaming, thinking, idle, and offline states in realtime.
- [Branching, edit, and regenerate](https://ably.com/docs/ai-transport/features/branching.md): Edit user messages, regenerate AI responses, and navigate branches with Ably AI Transport. The full history is preserved in the conversation tree.
- [Cancellation](https://ably.com/docs/ai-transport/features/cancellation.md): Cancel AI responses mid-stream with Ably AI Transport. Scoped cancel signals, server-side authorization, and graceful abort handling.
- [Chain of thought](https://ably.com/docs/ai-transport/features/chain-of-thought.md): Stream reasoning and thinking content alongside responses with Ably AI Transport. Display chain-of-thought in realtime.
- [Concurrent turns](https://ably.com/docs/ai-transport/features/concurrent-turns.md): Run multiple AI turns simultaneously with Ably AI Transport. Independent streams, scoped cancellation, and multi-agent support.
- [Database hydration](https://ably.com/docs/ai-transport/features/database-hydration.md): Hydrate an AI conversation from your own database with AI Transport and reconcile it with the live Ably channel, with no gap and no duplicate.
- [Double texting](https://ably.com/docs/ai-transport/features/double-texting.md): Handle users sending multiple messages while the AI is streaming with Ably AI Transport. Queue or run messages concurrently.
- [Durable execution](https://ably.com/docs/ai-transport/features/durable-execution.md): Run AI Transport agents inside a durable workflow engine. Adopt an in-flight run from a fresh process, retry a failed step under a stable stepId, and let the retry supersede the failed attempt on the channel.
- [History and replay](https://ably.com/docs/ai-transport/features/history.md): Load conversation history from Ably channels with AI Transport. Paginated history, gapless continuity, and scroll-back patterns.
- [Human-in-the-loop](https://ably.com/docs/ai-transport/features/human-in-the-loop.md): Add human approval gates to AI agent workflows with Ably AI Transport. Approve tool executions and provide input across devices.
- [Interruption and steering](https://ably.com/docs/ai-transport/features/interruption-and-steering.md): Let users change direction mid-response in Ably AI Transport. Three patterns: steer the active run with a follow-up prompt, cancel and re-prompt, or send alongside as a concurrent run.
- [LiveObjects state](https://ably.com/docs/ai-transport/features/liveobjects.md): Give an AI agent live awareness of what the user is doing, and the user live awareness of what the agent is doing, with shared state on the AI Transport session channel via Ably LiveObjects.
- [Multi-device sessions](https://ably.com/docs/ai-transport/features/multi-device.md): Share AI conversations across tabs, phones, and laptops with Ably AI Transport. All devices see the same session in realtime.
- [Optimistic updates](https://ably.com/docs/ai-transport/features/optimistic-updates.md): User messages appear instantly in Ably AI Transport. Optimistic insertion with automatic reconciliation when the server confirms.
- [Push notifications](https://ably.com/docs/ai-transport/features/push-notifications.md): Notify users when AI agents complete background tasks with Ably Push Notifications. Reach users even when they're offline.
- [Reconnection and recovery](https://ably.com/docs/ai-transport/features/reconnection-and-recovery.md): AI Transport streams survive connection drops automatically. Clients reconnect and resume from where they left off with no lost tokens.
- [Tool calling](https://ably.com/docs/ai-transport/features/tool-calling.md): Stream tool invocations and results through Ably AI Transport. Server-executed and client-executed tools with persistent state.
## Documentation Index
To discover additional Ably documentation:
1. Fetch [llms.txt](https://ably.com/llms.txt) for the canonical list of available pages.
2. Identify relevant URLs from that index.
3. Fetch target pages as needed.
Avoid using assumed or outdated documentation paths.