Raw WebSockets drop connections, lose track of canceled responses, and don't natively reach a second device. If your AI streaming feature has been in production for a while, you've probably already built a fix for at least one of these and found another one waiting.
Reconnection, cancellation, multi-device delivery, and crash detection are the four problems raw WebSockets leave for you to solve, and each is its own piece of infrastructure to build. Solve one, and the other three remain unsolved. Solve all four, and you're committed to maintaining all four for as long as the feature stays live.
What follows is a way to work out which of your own builds are holding up, which are starting to fail in ways that point to a real infrastructure gap, and what it takes to hand those over to infrastructure already built for this.
Key takeaways
- Raw WebSockets solve the protocol problem for AI streaming outright: a persistent, bidirectional connection with low per-frame overhead.
- Reconnection, cancellation, multi-device delivery, and crash detection are the four pieces of infrastructure raw WebSockets leave you to build for AI streaming, each with its own build and its own upkeep.
- The fixes teams build for these AI streaming capabilities tend to fail under specific, predictable production conditions, not at random.
- Whether to keep maintaining what you built yourself, or hand these capabilities to infrastructure already built for them, comes down to a calculation you can actually run.
What raw WebSockets actually solve for AI streaming
WebSockets give you a persistent connection that both sides can write to at any time. That single property solves the problem that trips up Server-Sent Events: a user canceling a response, an agent asking for tool-call confirmation, or a client sending a steering instruction mid-stream all need a channel the client can write to without opening a new HTTP request. SSE is one-way. WebSockets are not.
The overhead is also lower. A WebSocket frame carries a few bytes of framing information once the connection is open, compared to the repeated HTTP headers SSE sends with every chunk. At the token rates AI streaming produces, hundreds of small messages a minute per active session, that difference is measurable.
The protocol comparison itself isn't new, and it isn't the argument that matters here. Whether WebSockets or SSE is the right protocol for a given AI streaming scenario has already been covered in detail elsewhere.
The point worth being direct about is narrower: for anything beyond a single-turn, single-device exchange, the protocol choice was never the hard part. WebSockets get you a connection. They say nothing about what happens when that connection drops, gets canceled mid-response, or gets joined by a second device.
What raw WebSockets don't solve, and what solving it actually takes
A WebSocket connection is a bytestream between two endpoints, open for as long as the connection lasts and nothing more. It has no memory of what was already sent if the connection drops, no way to target a specific in-flight response if a client wants to stop it, no mechanism for reaching a second device, and no signal about whether the process generating tokens is still alive. Building AI streaming that holds up in production means adding all four of those things yourself, as systems that persist across the individual connections that come and go around them.
These four aren't independent problems, either. The storage that makes reconnection possible needs redesigning once multiple devices are involved, since it now has to catch up with more than one connection at a time. The correlation ID, the identifier that ties a cancellation or a crash signal to one specific run, is the same one both problems need. Build one of these in isolation, and you'll likely have to revisit it when you build the next.
Ably has written about this exact failure mode in production LLM streaming: the connection recovers, but the conversation doesn't, unless something else is keeping track of it.
Reconnection and resume: catching a client up after a drop
If you've been running this in production for a while, you've probably already built some version of resume: a sequence number or offset on each token, storage on the server that holds the stream, and reconnect logic that replays whatever a client missed. Doing this reliably means that storage has to be reachable from whichever server instance the client reconnects to, not only the one it started on, and the replay has to be idempotent, so the client never renders the same token twice.
Two conditions expose it reliably. A deploy that disconnects every client at once creates a reconnect storm that the store wasn't sized for. A mobile network handoff that drops the connection twice in quick succession catches the replay logic mid-catch-up, duplicating or dropping a chunk. Neither is rare in production traffic; both are routine.
Cancellation and mid-stream control: stopping the right response
Canceling AI streaming is harder than it looks. A cancel button is usually one of the first things a team builds, because users complain about its absence immediately. Doing it properly needs a way to target one specific in-flight generation by ID, not just close the connection, since closing it doesn't stop the backend job that's still generating tokens and burning compute. It also needs a decision about who is allowed to cancel what, since more than one client can be attached to the same conversation.
In production, three things go wrong: two cancels arrive in quick succession, a cancel that lands after the run has already ended gets applied to the next one instead, and a cancel is triggered from a second tab that the original build never accounted for having to authorize. None of these show up in a demo. They show up once real users are clicking stop under real network latency.
Multi-device and multi-tab delivery: reaching more than one screen
Multi-device is the one most teams don't build until a customer specifically asks for it, and then they build it narrowly: a WebSocket per tab, with a light broadcast between tabs if the team got that far. Making this work across genuinely separate devices is a different problem from syncing tabs in the same browser: it needs a registry that maps one logical conversation to multiple live connections, and a fan-out mechanism that pushes each token to all of them, not just the one that happens to still be open.
The gap shows up as soon as the second device turns out to be a phone, not a second tab in the same browser. The broadcast built for two tabs sharing one origin doesn't reach a genuinely separate client, and state that lived only in the first tab's memory disappears the moment someone opens a second device.
Crash and silent-failure detection: knowing when the agent is actually gone
The typical build here is a client-side timeout: if no token arrives for a set number of seconds, assume the agent is dead. A more reliable version needs the agent process itself to report its own state (thinking, generating, or gone), correlated with the specific run, so the client can tell a slow tool call apart from an actual crash instead of guessing from silence alone.
A normal thinking pause, the agent waiting on a slow tool call rather than having crashed, trips the same timeout as an actual crash. Loosen the timeout to avoid false positives, and it also catches a real crash too slowly. Teams end up choosing between users occasionally seeing a false "agent offline" message, or a real crash going undetected longer than it should.
Build, then manage, vs. buy
Doing this yourself is only the first step. It commits you to maintaining it for as long as the feature is live, so the decision is really build-then-manage versus buy. Whether you're still deciding whether to build in the first place, or you're already past that point and maintaining what you built, the same cost data applies to both.
What it costs to build, and keep maintaining, these capabilities yourself
Reconnection, cancellation, multi-device delivery, and crash detection each take real engineering time to build the first time. Maintaining all four as traffic and features grow is a second, ongoing cost that doesn't stop once the first version ships. Ably's State of Serverless WebSocket Infrastructure report surveyed more than 500 engineering leaders on the cost of building and operating production WebSocket infrastructure that has to reconnect reliably, guarantee delivery, and scale a stateful connection under real traffic. We found:
- Building basic WebSocket infrastructure in-house takes an average of 10.2 person-months.
- 69% of self-build WebSocket projects took more than three months, and 93% needed more than four engineers.
- 41% of teams building this in-house reported missed deadlines and extended timelines as a significant problem.
- Half of self-built WebSocket systems cost $100,000 to $200,000 a year to maintain.
- 65% had an outage or significant downtime from their self-built system in the 12 to 18 months before the survey.
If you haven't built this infrastructure yet, then it's easier to weigh the time required to solve the hard engineering challenges involved in building your own reliable realtime infrastructure against investing in your core product.
But what if you've built some of this yourself already?
That's a different calculation, and one made less often than it should be. The cost of what you've already built doesn't show up as a single line item the way a new build's cost estimate does. It shows up as:
- On-call load, every time a connection drop turns into a support ticket.
- The "one more feature" tax: every time product wants multi-device support or a proper cancel button, someone has to go back into code nobody has touched since launch and extend it correctly.
- Engineering time spent debugging a reconnect edge case that only happens on one carrier's mobile network, time your team could have spent on something users actually asked for.
If you want a number rather than a feeling, track these for a month:
- Engineering hours spent on tickets that trace back to a dropped or duplicated connection.
- The number of on-call pages tied to the realtime layer.
- Hours spent extending that layer for a new feature, such as a second device or a proper cancel button, instead of building the feature itself.
Multiply the total by the loaded engineering cost, then compare it to what a managed integration would take (typically a matter of weeks).
Fin, Intercom's AI customer service agent, is a case in point for what this trade-off looks like once AI is involved. The team behind it ran its own in-house realtime system, Nexus, for years, and it held up well for chat updates, presence, and typing indicators.
It reached its limitations once Fin AI Agent grew: a dropped connection started costing users a full AI response instead of a stalled typing indicator. And more importantly, the team realized that realtime infrastructure wasn't where they wanted to keep spending engineering time.
"We have a philosophy at Fin: run less software," said Colin Kennedy, Principal Product Engineer at Fin. "Realtime infrastructure isn't our core competency. We'd rather invest engineering time in building better AI agents and customer experiences than maintaining websocket infrastructure."
Moving to a managed layer took Fin from a system the team was still building and maintaining themselves to 99.9999% delivery reliability across workspaces, at more than 18 million peak concurrent connections.
If your team is already absorbing this cost, the comparison worth running isn't "build versus buy" against a hypothetical new project. It's the actual engineering time your team spends this quarter maintaining reconnection and delivery logic, set against what it would take to hand that specific, ongoing maintenance burden to something purpose-built for it.
Is this a bug, or a sign it's time to stop building your own realtime infrastructure?
Not every failure means you need to replace what you built. The distinction worth making is whether a given piece has failed once, under a genuinely new condition, or whether it keeps failing under conditions that are a normal part of running this in production.
- Something that has failed once, under a genuinely new condition, is probably a bug. Fix it, add the test case you were missing, move on.
- Something that keeps failing, under conditions like a routine deploy, a normal mobile handoff, or a second device your users already use, points to an infrastructure gap sitting underneath your implementation rather than a bug in it.
- If you're not sure which one you're looking at, ask whether the failure condition is something you could have reasonably anticipated when you first built this. Deploy-triggered reconnects and mobile handoffs are not unreasonable to anticipate. They're the same conditions already named: a deploy, a mobile handoff, a second device.
The point where it stops being worth maintaining this yourself isn't usually one catastrophic failure. It's the moment fixing the same handful of recurring conditions costs more engineering time each quarter than handing them to infrastructure already built for this would.
Colin Kennedy at Fin described the decision that led the team to replace Nexus this way: "Is maintaining this system helping you build a better product? If the answer is no, if you're spending time keeping the lights on instead of innovating, it's time to evaluate alternatives."
The test is simple: are the failures you're seeing new each time, or the same handful of conditions recurring? New each time points to bugs worth fixing. The same conditions recurring points to an infrastructure gap, one most teams have already reached without naming it.
How Ably fills the gap: reconnection, cancellation, multi-device delivery, and crash detection
Ably is a realtime infrastructure platform designed to solve reconnection, delivery guarantees, and stateful connection scaling for any use case, at the scale Fin and other production customers already run at. Ably AI Transport is the product built specifically for AI streaming on top of that platform: a drop-in layer that adds the four recurring capabilities to an existing WebSocket or SSE integration. Each is a specific, documented piece you can adopt on its own rather than as a single bundled abstraction.
For reconnection and resume, a dropped client reconnects and resumes from the exact point it left off. The agent keeps publishing to the conversation while the client is disconnected. On reconnect, the client loads everything that it missed with no lost tokens and no retry code to write. This is built into the transport layer, so your application code doesn't do anything extra to make it work, and a reconnect storm after a deploy is handled the same way as a single dropped connection.
Cancellation works as a signal on the channel, not a connection close. It targets a specific run by ID, the agent's abort signal fires, and the session stays open for the next turn. Authorizing who can cancel what is a hook you build once. You don't rebuild correlation logic for every new streaming feature or every new device that joins the session.
Multi-device delivery is native: every device subscribing to the same conversation sees the same messages in realtime, with no extra fan-out code to write. A user approving a tool call on their phone and watching the response continue on their laptop is the default behavior from day one, before any customer has to ask for it.
Crash and silent-failure detection relies on agent presence, a live status channel separate from the connection itself: it reports whether the agent is thinking, streaming, idle, or offline, and every connected client sees that status change in realtime. Combined with run-level lifecycle data, this replaces the timeout-guessing heartbeat that you'd otherwise have to write and tune yourself.
It's worth being direct about where AI Transport's boundary sits: it replaces the realtime transport layer, not your product logic. Deciding what to show users while reconnecting, or how to phrase a crash message, is still yours to build.
And adopting AI Transport doesn't mean abandoning WebSockets as the protocol. It's the same connection, with the four recurring problems closed by infrastructure built for exactly this, rather than infrastructure you keep extending one support ticket at a time.
FAQ
We've already built a Redis buffer for resumable AI streaming. What does a managed layer add on top of that?
A Redis buffer typically solves offset storage for a single failure case: a full page reload where the client reconnects from scratch. It tends to break down on deploy-triggered reconnects, mobile network handoffs that hit the reconnect window twice, and any scenario generating messages faster than the buffer drains. A managed transport adds correlation for cancellation, native multi-device fan-out, and presence-based crash detection, none of which a token buffer on its own provides, and removes the ongoing work of extending and debugging that buffer as new streaming features are added.
I've already built a fix for one of these problems. Is that reason enough to switch now?
Not on its own. One piece that's holding up under real production conditions doesn't need replacing. Watch for whether it starts failing more than once under ordinary conditions, like a routine deploy or a device switch, since that usually means the remaining three problems will follow the same pattern as your team's feature set grows.
When is SSE or raw WebSockets enough for AI streaming?
They're enough for a single-turn assistant with no mid-stream cancellation, no multi-device requirement, and no long-running background tasks. Once a product needs tool-call approval, steering, multi-device continuity, or crash detection, the protocol alone stops being the limiting factor and the missing capabilities have to be built or bought.
How much does it cost to build WebSocket reconnection and resume myself?
Building basic WebSocket infrastructure in-house takes an average of 10.2 person-months, and half of self-built systems cost $100,000 to $200,000 a year to maintain. There's no AI-streaming-specific figure, but reconnection and resume is typically the single largest piece of that build.
Does switching from SSE to WebSockets alone fix session continuity?
No. WebSockets solve the bidirectional signaling problem: cancellation and steering can travel over the same link used for tokens. Reconnecting still means a new connection, though, and without something tracking conversation state separately from it, a reconnect or device switch loses context regardless of which protocol you started with.
For more detail, see the docs on reconnection and recovery, cancellation, multi-device delivery, and agent presence.
