When a stream breaks mid-response, most teams reach for the same fix: a Redis buffer between the LLM backend and the client, a hand-rolled session ID, and enough deduplication logic to stop the worst of it. Nearly every production team arrives at this independently, which says more about the problem than about any one team's choices. It's also five separate systems you now have to keep running at once, indefinitely, for something that was never the product you set out to build.
Here's what that workaround actually involves, piece by piece, and what it costs to keep working.
What you end up building the first time a stream drops
Nobody starts from a blank page here. Every team ends up building roughly the same four pieces, usually under pressure from a support ticket that already landed.
If you're building an AI support agent, this is the failure mode your customer actually sees: the bot goes silent mid-troubleshooting, or repeats a step when it reconnects.
Mobile clients hop between WiFi and cellular constantly, long responses have a real shot at a transient failure somewhere past the 30-second mark, and multi-agent setups multiply the number of things that can go quiet mid-session. None of this is rare once you're past a demo.
Message identifiers. A sequential ID on every token or message as it's published, because "resume from where we left off" has no anchor without one.
Client-side tracking. The client remembers the last ID it actually rendered. In a browser, that's memory or local storage, and someone has to remember it needs to survive a tab close. On mobile, it has to survive the app being backgrounded, which is a separate set of edge cases nobody budgets for up front.
A reconnection handshake. On reconnect, the client sends the last ID it saw. The server has to look that ID up, work out what came after it, and decide how to hand it back.
Catch-up delivery that doesn't show its seams. Missed messages need to arrive in order, before live tokens resume, without a visible jump or a loading flicker that gives away that anything dropped.
Each of these four pieces is straightforward on its own. Owning all four indefinitely, across every client platform you ship to, is where it gets hard.
SSE gets you one of these four for free, and stops there
Server-Sent Events ships a Last-Event-ID header. When an SSE connection drops, the browser sends it automatically on reconnect, and the server can pick up from there.
event: token id: 150 data: {"content": "production"} event: token id: 151 data: {"content": " systems"} // On reconnection after receiving event 150, // browser automatically sends: GET /stream HTTP/1.1 Last-Event-ID: 150 // Server resumes from 151
That's the reconnection handshake, handled. Everything else on the four-piece list is still yours: SSE has no native history beyond what you build server-side, it's unidirectional and HTTP-only so live steering needs a separate channel, and on distributed infrastructure a reconnecting client can land on an instance that has never heard of the session. SSE solves one piece of a four-piece problem and looks, from the outside, like it solved the whole thing. For use cases that need bidirectional messaging, WebSockets vs SSE covers the tradeoffs in detail.
Move to WebSockets and you're back to building all four
WebSockets don't include resume semantics. When a socket closes, the connection is gone, and reconnecting creates a new one with no memory of the last.
So you build the full list yourself: session IDs generated at stream start and stored server-side, message IDs assigned sequentially, server logic to look up a session and replay history before switching to live, buffer management for sessions that haven't reconnected yet, and cleanup logic to expire stale sessions without cutting off a legitimate reconnect that's mid-flight.
Each piece is roughly a day of work on its own. The combination is where a month disappears, mostly into edge cases nobody designs for until they show up in production: a client reconnecting twice in ten seconds, a buffer that expired four minutes before the user came back, a cleanup job that fires while a real reconnect is still in progress.
Message ordering: two ways a stream lies to you
Duplicates happen when a connection drops after the client received a message but before the server got the acknowledgment. On reconnect, the server doesn't know whether to replay it, and without dedup logic, the client renders the same token twice.
Gaps happen when message 153 arrives after 150, and 151 and 152 never show up, or arrive so late the client already moved on. Without gap detection, the client silently renders an incomplete response and has no way to know it's incomplete.
Both need logic that survives page reloads, works across tabs, and has an answer for what to do when a missing message genuinely can't be recovered. Both are rare enough to pass every test you write and common enough to show up in the first week of production: mobile handoffs, flaky WiFi, a corporate proxy resetting an idle connection. The first time you see either is usually a support ticket, not a stack trace.
Reconnection and session resumption: the pinning tax
A single server instance can hold session state in memory and mostly work, right up until you run more than one instance for redundancy, which you will. Now a client that connected to instance A can reconnect to instance B, and instance B has never heard of that session.
Two ways out, and both cost something. Pin every reconnect back to the originating instance, which creates hotspots and defeats the point of running multiple instances in the first place. Or move session state into Redis or equivalent shared storage, which means network round-trips on every reconnect, cache invalidation logic, and a plan for what happens when the cache itself goes down.
Solve this and you've also taken on a second job: running and monitoring a Redis cluster, on top of the streaming feature you actually set out to build.
There's a knock-on cost here too. A 500-word response is roughly 625 tokens. Store each one as its own record and loading a single response means 625 reads; twenty exchanges is 12,500. At real user volume, history retrieval becomes the bottleneck, and it's slowest at exactly the moment a user is waiting for their conversation to reappear on a new device. Most teams end up appending tokens to one logical message instead of writing one record per token. It works, but it's a storage decision you need to get right early, because migrating a live system off token-per-record storage is its own project.
Silent failure: the one that doesn't look broken
Duplicates and gaps eventually get caught, because something visibly renders wrong. Silent failure doesn't announce itself. The stream is technically fine and tokens are technically arriving, but nothing in the system confirms whether delivery is actually complete or only slow. Without an explicit acknowledgment model, the only tool available is watching the UI and hoping: refreshing, polling, waiting to see if more text shows up before deciding something's wrong.
That's the honest state of a system with no delivery confirmation built in. It's usually the hardest of these failure modes to justify engineering time for, right up until support tickets say otherwise.
Cross-device continuity: where the connection model runs out of road
When session state lives with the connection, or in memory tied to it, a device switch loses everything. The phone doesn't know what the laptop already received. Every reconnect from a new device looks like a brand-new session, because architecturally, it is one.
Fixing this isn't a patch on the existing design. It means decoupling state from the connection entirely: the conversation lives in a channel or persistent store that any device can query, with devices subscribing and catching up rather than resuming a socket. That's a different architecture, and most teams only discover it after the first version is already live and a support ticket says "I switched devices and lost my conversation."
Add it up
Session management. Storage with efficient retrieval by ID range. Client-side deduplication. Gap detection. Distributed routing. Cache invalidation. Buffer expiry. Some way to know delivery actually happened. Monitoring to catch the failures you can't reproduce locally.
None of it is exotic, but all of it needs building, testing under real network conditions, and maintaining for as long as the product exists. Teams who've shipped this describe the same arc: the first version takes a week, the edge cases take a month, and full cross-device reliability still isn't settled six months later.
The managed alternative
Building this yourself is a reasonable choice if you have a stable team, the time to maintain it indefinitely, and no multi-device or distributed requirements today. It gets harder than the SSE documentation makes it look the moment any of those conditions change, and most teams building AI products hit at least one of them within the first year.
The alternative is transport infrastructure that treats resume, ordering, delivery confirmation, and multi-device state as platform concerns instead of application code your team owns.
Ably AI Transport provides resumable token streaming with automatic replay, append-based storage by default, and channel-based delivery with ordered, deduplicated messages, so the four-piece list above isn't something you build and rebuild every time a framework or a client platform changes.
Silent failure closes too, through a different mechanism: presence fires an explicit crash-detection event the moment an agent or connection drops, so the client never has to infer a failure from a stream that's gone quiet. There's no separate confidence-building step left to build by hand.
You keep your LLM, your prompts, and your application logic exactly as they are. What changes is where your engineering time goes: into session plumbing, or into the product you actually set out to ship. Both paths are defensible, but only one of them spends your engineering time on the product you're trying to ship.
Streaming responses between AI agents and clients? Ably AI Transport includes resumable token streaming, automatic replay, and channel-based delivery with guaranteed ordering. Docs go deeper.
