Resumable token streaming: why AI streams break on reconnect

Refresh the page, lose signal, switch tabs. Users expect the AI conversation to still be there when they return, with the answer picking up where it left off. Resumable token streaming makes that possible: a returning client catches up with the in-progress response instead of starting over.

Making a stream resumable is an architecture decision. Starting generation, storing the response, and delivering it need to happen in separate parts of your stack. A dropped connection then only interrupts delivery, and the response waits where the client can get it back.

This post covers why AI streams break on reconnect, what a resumable architecture needs, and the options for building one.

Key takeaways

• An AI response is lost on reload or disconnect when it exists only in the connection and the browser.
• A resumable architecture separates starting generation, storing the response, and delivering it, so a dropped connection only interrupts delivery.
• Resuming a token stream takes four things: generation that outlives the connection, retained output, a stable session ID, and ordered catch-up.
• Store each response as one growing message, so a returning client gets the answer so far in one request instead of replaying every token.
• Each failure has its own owner: the transport resumes brief drops, your app restores from history after a reload, and a workflow engine retries crashes.
• A framework helper can cover short answers that mostly need reload recovery. A managed transport fits when responses run long or have to reach every tab and device.

Why do AI token streams break?

AI token streams break when a connection closes, and the application cannot reconnect the client to the response already in progress. Continuity can fail in three places: at the connection, in state held only by the browser, and in a backend that cannot restore missed output.

A closed connection interrupts live delivery

If you're streaming an AI response over a standard HTTP connection and it drops, that request ends. HTTP does not resume a half-finished response on its own. A WebSocket connection also closes when it drops; either approach needs recovery logic to restore what the client missed.

In enterprise environments, the problem is worse. Proxies and load balancers close connections that look idle. An AWS Application Load Balancer, for example, closes them after 60 seconds by default unless the timeout is changed.

For timeout values by platform and the server-side ping fix, see WebSocket reconnection in AI agents.

Browser-only state disappears on reload

If the browser tab crashes or reloads, any part of the response held only in browser memory is lost. The application can bring it back only if a copy of the in-progress response was kept outside the browser. Otherwise, the user faces a hard reset.

A network blip does similar damage when no copy was kept: the browser keeps what it received, but the rest of the response is lost. The user may need to send the prompt again and wait, and you pay for another model call while the output already generated goes to waste.

A backend without retained state cannot restore missed output

Many backends treat streaming as fire-and-forget: tokens go out and aren't stored. When a client reconnects and asks what it missed, the server has no record to answer from.

Restoring missed output needs three things that HTTP streaming leaves to your application:

1. A way to identify the response the client was watching
2. Output that is retained somewhere other than the connection
3. An ordered way to send back only what the client is missing

SSE's Last-Event-ID header tells the server where a client stopped, but the server still has to supply all three.

Response identity, retained output, and ordered catch-up are harder to add after launch than to design in. All three touch storage, every client's reconnect logic, and how history is read back. The storage choice is the hardest to reverse: moving a live system from one record per token to one record per response is a migration project of its own. Switching from SSE to WebSockets doesn't remove the need for any of them, because a WebSocket reconnect restores the connection, not the output generated while it was down.

What does a resumable streaming architecture look like?

A closed connection, browser-only state, and a backend with nothing retained share one root cause. A basic HTTP stream ties three jobs to one connection (starting generation, keeping the only copy of the output, and delivering it to the user), so when the connection closes, all three can fail at once. A resumable architecture gives each job its own part of the stack, and losing the connection then only interrupts delivery.

The parts of a resumable streaming stack

Starting generation takes two parts, a short HTTP request and the agent, and a workflow engine sits alongside the agent to recover its work after a crash.

Job	Part of the stack	What it does
Starting generation	Short HTTP request	Starts the work and returns before any tokens are generated
Starting generation	Background worker (your agent)	Runs the model call outside the request, so a dropped connection doesn't stop it
Keeping the copy	Shared store	Holds each response as one growing message under a session ID, with a sequence number on every update
Delivering the output	Transport layer (session subscription)	Sends every update live to each tab and device on that session ID
Recovering from a crash	Workflow engine	Retries the agent's failed step, so the retry's output replaces the partial answer

How a response flows through a resumable architecture

With generation, storage, and delivery separated, every response moves through the same three steps:

1. The client sends the prompt with a session ID. The HTTP request returns as soon as the work has started.
2. The agent runs the model call and appends output to the response's message in the shared store.
3. Clients subscribe to the session, not the request, so every tab and device receives the same updates live.

Which part of your stack recovers the response?

The flow assumes nothing goes wrong. When something interrupts a response, a different part of your stack handles it, and only a crash stops generation unexpectedly:

What happens	What the user sees	What the architecture does	What handles it
Brief connection drop	The answer pauses, then continues	Keeps output on a channel or log that outlives the connection; the client asks for every update after its last sequence number	The transport layer, which resumes delivery from the last message the client received
Reload or longer absence	A blank or partial answer until it's restored	Keeps the response and conversation history long enough to fetch; the client loads the response so far, then continues live	The application, which fetches history and sets retention and access
User presses Stop	The answer stops	Sends the stop through the app's own request path to the agent, which ends the model call; other clients see that the response stopped	The application, which sends the stop signal
Agent crash	A partial answer that stops partway	Keeps the partial output and enough run state to retry; the retry's output replaces the partial response	The workflow engine, which retries the agent's failed step

Because each interruption needs its own recovery path, treating them as a single "reconnection" problem breaks in predictable ways. Reconnect logic on its own survives a network blip but loses the answer on reload, and re-running the model on every failure means paying for a second model call each time.

What does it take to resume a token stream after a disconnect?

Splitting the stack only helps if each part is built to survive a disconnect. That takes generation that keeps running when the client disconnects, plus the three things a backend usually lacks: retained output, a stable session ID, and ordered catch-up. HTTP streaming and raw WebSocket connections leave all four to your application.

1. Generation that keeps running when the client disconnects

The problem: If generation runs inside the HTTP request that streams the response, a dropped connection can stop generation too. The user then has to send the prompt again, and you pay for a second model call.

What you need: A model call that keeps running when the client disconnects. This is your application's responsibility, because a transport layer can deliver output but can't keep your code running.

How to build it: Start the model call outside the streaming request, for example in a background job, and publish its output to a store the client can read from. On serverless platforms, the function must also stay alive after the response is sent: in Next.js, after() keeps the function running up to the route's maximum duration.

In production: Closing the connection no longer stops generation, and that independence is what makes recovery possible. But closing the tab, or a Stop button that only closes the stream, now leaves the model running and still using tokens. To let users stop a response, send a separate stop signal that reaches the running model call.

2. Retained output that a reconnecting client can replay

The problem: If output only travels down the connection, anything generated while a client is disconnected is lost. A reconnecting client has nothing to catch up from, even if generation kept running.

What you need: A stored copy of the response that outlives the connection and supports a client returning in one of two ways:

• After a brief drop, the client picks up from the last update it received and gets everything published during the gap, in order.
• After a longer absence or a reload, the client reads the response from history instead, because the connection can no longer replay what it missed.

How to build it: Store the message, not the tokens: keep each response as a single message that grows as the model generates it. Whichever way a client returns, it then gets the response so far in one request and the rest live. The alternative approaches each cost more, as A token is not a message shows: a log of separate chunks makes every client rebuild the response, and rewriting a database row for every token adds a write per token.

In production: Retention is a tradeoff. Keep output too briefly and returning users find nothing, but keep it longer and you pay to store it.

3. A stable session ID that survives reconnects

The problem: Retained output only helps if a returning client can find it. After a reload or on a second device, the original connection is gone, and with it the only link to the response the client was watching.

What you need: An ID for each conversation that doesn't depend on any one connection.

How to build it: Create the ID when the conversation starts and store it with the retained output. Keep a copy on the client, for example in the URL, so a reloaded page, a second tab, or another device can present the same ID and rejoin the in-progress answer.

In production:

• The ID must resolve on any server. With more than one backend instance, a reconnect can land on a server that has never seen the session, so session state has to live in shared storage, not one instance's memory.
• Holding the ID mustn't be enough to join. If any request carrying a valid ID can reattach, a leaked ID exposes the conversation, so check that the user is allowed into that session before resuming.

4. Ordered catch-up without duplicate or missing tokens

The problem: A connection can drop after the client receives an update but before the server knows it arrived, so the server sends that update again. Other updates can arrive out of order or go missing. The user then sees repeated text or a silent gap, and nobody can tell whether the response is complete.

What you need: Catch-up that puts updates in order and skips ones the client already holds.

How to build it: Give each update a sequence number. On reconnect, the client sends the last number it processed. The server replays every later update in order, and the client discards any number it has already applied.

In production: Storing the response as one growing message makes ordered catch-up simpler. A returning client replaces what it shows with the response so far, then applies only the updates that follow. For how resume tokens and SSE's Last-Event-ID work, and what each costs to build, see resume tokens and Last-Event-IDs for LLM streaming.

Checklist: can your stack resume a token stream after a disconnect?

Each of the four requirements lives in a specific part of a resumable stack. Use the table to review an existing setup, or a design before it ships: a "no" in any row shows where a user could lose a response, watch it start over, or see repeated or missing text. The workflow engine sits outside the four requirements: it doesn't help a client reconnect, but it's what recovers the response when the agent itself crashes.

Requirement	Where it lives in the stack	Question to ask
Generation that keeps running	Short HTTP request and agent	Does the model call keep running when the client disconnects, and does your Stop button send its own signal rather than just closing the stream?
Retained output	Shared store	Is each response stored as one growing message that outlives the connection, and kept long enough for users to come back to it?
Stable session ID	Shared store and transport layer	Can a reloaded page, a second tab, or another device find the response by its session ID, on any server, and only if the user is allowed into that session?
Ordered catch-up	Transport layer	Does every update carry a sequence number, so a returning client gets missed output in order with nothing repeated or skipped?

How can you implement resumable token streaming?

There are four common ways to meet these requirements. They differ mainly in how much of the stack you build and run yourself.

Option	Fits when	What you get	Tradeoffs
Build it yourself	You have a team to own it long term, and your needs are narrow	Full control over generation, storage, and the reconnect protocol	You build and maintain all four requirements, plus auth and routing across servers. The edge cases are the hard part: duplicates, gaps, reconnect races, and expired buffers
Framework helper, such as the Vercel AI SDK's resumable streams	You're on the Vercel AI SDK and mostly need reload recovery	useChat reconnects to an active stream after a page reload, or when a backgrounded page becomes visible again after a network drop	You run Redis plus an active-stream store, auth, and endpoints to create, resume, and stop streams. Buffered streams expire, and stop() only closes the connection, so stopping generation depends on that stop endpoint
Chunk log or stream store, such as Redis Streams, NATS JetStream, or Electric Durable Streams	You already run a stream store, or other services need every chunk, not just the finished response	Durable, replayable output with positions a client can resume from	Clients rebuild each response from chunks, skip duplicates, and detect completion. Long responses mean replaying many entries before anything renders, and once entries expire, the response can't be rebuilt
Managed transport, such as Ably	Responses have to survive disconnects and reach every tab and device	Catch-up while a response is in progress, and the same updates on every tab and device	A new vendor dependency, with setup and usage-based costs. You still own generation, storage, access, and crash retries

Building it yourself leaves all the work with your team; a framework helper still leaves you the store, stop path, and auth; and a chunk log makes every client rebuild the response. For short answers that mostly need reload recovery, a framework helper may be enough; AI chat stream resumption covers when a Redis buffer will do.

Once responses run long or users switch tabs and devices, a managed transport such as Ably fits better. It takes the parts with the most edge cases off your team (catch-up while a response is in progress, and delivery to every client), while you keep generation, storage, access, and crash retries.

Where Ably fits, and what your application still owns

Ably provides the live delivery layer in a resumable stack, while your database stays the record of the conversation. Your agent publishes each response to an Ably channel instead of down an HTTP connection. A client that drops, or joins from another device, picks up the same response without the model running again.

Ably delivers each response as one growing message, following a principle Ably CEO and co-founder Matthew O'Riordan describes in an interview with The New Stack: a client that reconnects mid-stream should request the current state of the response, not replay every token. A client that joins mid-response gets the answer so far in one piece, then the rest live, instead of waiting while hundreds of tokens replay and your code stitches them back together.

Here's what Ably's realtime infrastructure handles for each of the four requirements and for an agent crash, and what your application still owns:

Requirement	Ably handles	Your application owns
Generation keeps running	The agent publishes to the channel, not to a connection, and keeps publishing whether or not a client is attached	Where the model call runs, and keeping it running
Retained output	Live delivery and recent catch-up while the response is in progress: tokens are appended to one growing message on the channel	Storing completed responses in your own database
Stable session ID	Fan-out: every client on the conversation's channel gets the same updates, so a late joiner sees the answer so far, then the rest live	The conversation ID, stored with your own record, and who can join
Ordered catch-up	Brief drops: the client reconnects and picks up missed updates in order, with no extra code, within the live recovery window (up to two minutes, depending on connection state and channel configuration)	Reloads and longer absences: load the conversation from your own store
Agent crash	The partial response stays visible to clients during the active turn; an agent on the conversation's presence list is removed about 15 seconds after a crash, so the interface can show it's offline	Retrying the model call, which Ably doesn't do: a durable workflow engine such as Temporal or Vercel Workflow retries the failed step, and reusing the step's ID replaces the failed attempt's output

Choosing Ably means taking on a new dependency: your backend needs an Ably auth endpoint, and the conversation's channel needs a configuration rule. If your answers are short and only ever reach one device, that may not be worth it.

For longer answers, or users on more than one device, you integrate live catch-up and multi-device delivery rather than build and maintain them. A user who drops mid-response gets their answer back instead of a restart.

To build a resumable AI chat with Ably, start with the quickstart for the Vercel AI SDK, which walks through the auth endpoint, channel rule, agent route, and chat client. The finished app streams each response to every open tab, and a reload reads the conversation back from your own store. To review your own architecture, talk to Ably's engineering team.

Frequently asked questions

What is resumable token streaming?

Resumable token streaming lets a client that disconnects mid-response reconnect and receive the rest of the same AI response, without the model running again. It depends on keeping the response outside the connection, plus a way for the client to find it and catch up in order.

Should generation keep running after the user closes the tab?

Usually, yes, at least for a while: the user may be reloading or switching device, and that shouldn't cost them their answer. But a model call that nobody returns to still uses tokens, so set a limit.

With Ably, each open tab or device can join the conversation's presence list, which shows who is connected. The agent can check that list and stop the model call when no user is left.

When a response stops partway, how do we tell what failed?

Compare what the client shows with what your store holds. If the store has more than the client, delivery failed and catch-up should recover it. If the store's copy also stopped, then generation failed, so check whether the agent is still alive before blaming the model provider.

With Ably, the agent can join the conversation's presence list, and Ably removes it about 15 seconds after a crash. Its partial response stays visible to clients during the active turn, still marked as streaming.