Backpressure in AI streaming: change how tokens move, keep each response one message

Backpressure in AI streaming can build anywhere from inference to the client, but delivery is where handling it can change the response itself. The three usual approaches there (buffering, batching, and rewriting a stored record) can change how the response is stored, or what storing it costs. Anyone who joins mid-answer then gets fragments to stitch together, or your database pays for a full write at every flush.

Instead, your backpressure strategy should change only how often tokens move, never how many messages a response becomes. This post tests the three approaches on whether each response stays one message, and introduces an alternative: message appends. For storage patterns in general, see A token is not a message; this post is about what happens under load.

Key takeaways

• Under load, change only how often tokens move, never how many messages a response becomes.
• Backpressure builds at four points: inference, worker queues, delivery, and clients. Delivery is where the usual fixes also change how the response is stored.
• Buffering, batching, and rewrites each have a cost under load: queued tokens add latency, stored chunks or batches split the response into records, and every flush becomes a full write.
• Test a backpressure strategy in two ways. Can a viewer who can't keep up with the model receive updates at their own pace, without affecting anyone else? And can someone joining mid-answer see the whole response so far?
• Ably message appends keep each response as one growing message, however delivery changes. Your application still controls how much work it accepts and when to cancel it.

How do you handle backpressure in AI streaming at the delivery layer?

Backpressure is how a system copes when a producer outpaces a consumer: the consumer pushes back, so something waits, slows down, or is dropped. In AI streaming, the producer is the model, and the consumers are every tab, device, and agent watching the response.

Pressure can build at four points between a user's prompt and their screen, and each is handled differently:

Where	What builds up	Common responses
Inference	More requests than your model capacity	Admission control, concurrency limits, load shedding
Worker queues	Requests waiting for a model slot	Bounded queues, timeouts, cancellation
Delivery	Tokens arriving faster than clients take them	Buffering, batching, record rewrites, message appends
Clients	Updates arriving faster than a screen renders them	Render throttling, dropping intermediate updates

Inference and worker queues are capacity decisions your application makes, and client rendering is usually a UI choice. At the delivery layer, teams have mostly relied on three approaches: buffering chunks, batching them, or rewriting one stored record. Handling backpressure well at this layer means changing how often tokens reach each viewer, without changing how many messages a response becomes.

Each works up to a point. The difference shows when the model outpaces a viewer: what each one does to the response, and what it costs.

Two tests for choosing how to deliver AI responses under load

1. The slow-consumer test

What it checks: live delivery. Can a viewer who can't keep up with the model receive updates at their own pace, without affecting anyone else? 

Why it matters: Mobile networks, background tabs, and dashboards that follow many conversations all receive updates more slowly than the model produces them. If one viewer's backlog grows without limit, memory and latency grow with it. Changing what's stored to cope makes every viewer pay for one slow connection.

How to run it: Stream a response to several clients and throttle one, for example with network throttling in browser developer tools. Watch that client's lag, your server's queue depth, and what the other clients and your store end up with. The slow viewer's lag should stay bounded, and everything else should match a run without it.

2. The late-joiner test

What it checks: catch-up. Does someone who joins mid-answer see the whole response so far? 

Why it matters: Users reload pages and switch devices, and supervisors or human agents join live conversations. If a late joiner sees a blank screen or a broken response, the user waits for the model to finish, or asks again and pays for a second model call.

How to run it: Start a response that streams for 20 seconds or more. Partway through, reload the page, open the conversation on a second device, and attach a separate observer client, including straight after a simulated network drop. Each should show the response so far almost immediately, then continue live, with text that matches the original exactly.

How buffering, batching, and rewrites perform on the two tests

Take a model that emits 120 token events a second over a 20-second response, and a viewer on a slow connection that can take only 40. The surplus tokens have to go somewhere. Here's where each approach puts them, and how each performs on the two tests:

	Buffering	Batching	Record rewrites
Usually chosen when	Bursts are short and the queue can be bounded	Fewer publishes are worth a short delay	Clients need the full response so far
Where the extra tokens go	Into a queue that grows by 80 events a second: 1,600 by the end of the response	Held until the batch window closes, then sent as one batch: about 500 per response at 40ms	Into the stored record: each flush rewrites the whole response so far
Slow-consumer test	Passes only if the queue is bounded; unbounded, the last token arrives 40 seconds late	Partly: fewer deliveries, but lag can still build	Partly: the stored view lags the live response
Late-joiner test	Fails: with nothing stored there's nothing to fetch, and with stored chunks the client rebuilds the response	Fails once batches are stored: the window decides how many records one response becomes	Passes: one usable record
Implications	Memory grows with the queue: about 160KB per slow stream at 100 bytes an event, or 160MB across 1,000. You need a policy for when it fills, and stored chunks mean 2,400 writes per response.	Up to 40ms of added latency, about 20ms on average. Stored batches mean about 500 writes per response, and every read fetches and joins them.	About 2.5MB written to store a 20-second, 10KB response at 40ms flushes, or 105KB at one-second flushes. Database cost and rate limits track flush rate, and retries need sequencing so older text can't overwrite newer.

Each approach moves the extra tokens somewhere that costs you: memory and latency, a growing record count, or a write bill. Only rewrites keep one usable response, and they pay for it at every flush.

Message appends: an alternative way to deliver AI responses under load

Buffering, batching, and record rewrites leave you choosing between a fragmented response and a full write at every flush. Ably message appends remove that tradeoff: your agent creates one message for the response and appends each new piece of text to it. Ably handles ordering, rollup, fan-out, and recovery, so the response stays one message however delivery changes.

Unlike a chunk log, clients get one growing message. Unlike a rewrite, your agent sends only the new text, not the whole response.

How Ably handles one streamed AI response under load

A streamed AI response needs two things from Ably: live delivery while the model generates, and catch-up whenever a viewer joins or reconnects. For live delivery, your agent creates one message and appends to it, and Ably orders, combines, and fans out every append. For catch-up, Ably gives each viewer the response so far as one message.

Here's how that plays out in the five moments a response goes through under load, from the first token to the final record:

Moment	What happens	Why it matters under load
The response starts	Your agent publishes one message for the response, with a stable ID, and Ably assigns it a serial: a unique identifier that every later append points to	Appends, retries, and reconnects all target the same response instead of creating a new one
The model streams new tokens	Your agent appends each new token to the response's message over one realtime connection. Ably applies appends in order, combines those that arrive within the rollup window, and sends the result to every viewer.	Text stays in order, and each viewer gets at most 25 updates a second for each response, however fast the model runs
A viewer joins mid-answer	Ably sends the joining viewer the response so far as one full update, then live appends. History and getMessage() also return the response so far as one message.	The viewer never has to reassemble the response
A viewer drops	If the viewer reconnects within Ably's recovery window, typically two minutes, missed updates arrive in order. After a longer gap, the viewer fetches the response so far as one message.	A dropped connection costs a pause, not the response
The turn ends	Your application saves the completed response to its own database	Your database holds the long-term record; Ably handles the live turn

Taken together, these moments mean message appends pass both tests, and the response stays one message throughout:

• Slow consumer: gets at most 25 updates a second for each response. If its connection still exceeds its outbound rate limit, Ably signals a continuity loss (error 80020), and the viewer fetches the response so far.
• Late joiner: gets the response so far as one full update, then live appends.

Handling backpressure outside the delivery layer

Message appends handle backpressure at the delivery layer. To handle it end to end in AI streaming, your application still manages the other three points where it builds:

1. Inference: limit concurrent model calls, reject work when capacity runs out, shed load, and cancel model calls through your own request path.
2. Worker queues: cap queue size, and expire requests that wait too long.
3. Clients: render updates by applying full updates and appends to each viewer's copy, and decide what happens when a viewer falls too far behind: catch up, refetch, or disconnect.

Your application also owns anything your agent does outside the stream: idempotency keys for side effects such as tool calls and payments, and saving completed responses to your database.

Buffering, batching, rewrites, and appends compared

Side by side, buffering and batching cope with load by splitting the response into records, and record rewrites keep it whole by paying for a full write at every flush. Of the four approaches, only message appends change how often tokens move without changing how many messages a response becomes.

To stream AI responses with message appends, start with the API, then the token streaming pattern for the agent side. Publish over a realtime connection so appends stay in order, and enable the channel rule for message annotations, updates, deletes, and appends first. For recovery behavior, see reconnection and recovery, and read how Ably engineered message appends.

Here's the full comparison of one AI response under load (assuming chunks and batches are stored for buffering and batching):

Comparison point	Buffering	Batching	Record rewrites	Ably message appends
Where a backlog builds	In the queue, which grows whenever the model produces tokens faster than viewers take them	Behind slow viewers: sends are fewer and larger, but a backlog can still build	In your service, where tokens wait between flushes; you still need to cap how much can wait	In your agent, which still needs a cap on queued work; Ably's rollup limits how often each viewer is sent updates
How the response is stored	A separate record for every chunk	A separate record for every batch	One record, rewritten as the response grows	One message that grows with each append; your database holds the final copy
Write cost	One write for every chunk	One write for every batch	The whole response so far, written again at every flush	Each append sends only the new text
Making retries safe	Give each chunk an ID so repeated chunks can be dropped	Give each batch an ID so repeated batches can be dropped	Order writes so an older version can't overwrite a newer one	Publish with a stable ID so Ably discards a repeated create, and check the current message before retrying an append
What a viewer does to show the response	Fetches every chunk and joins them in order	Fetches every batch and joins them in order	Reads the record	Replaces its copy on a full update, and adds the text on an append
When a viewer falls behind	Its queue keeps growing, or chunks are dropped, depending on the policy you set	It gets updates less often, but can still fall further behind	The record doesn't help; you decide how often a slow viewer rereads it	Rollup limits how often it's sent updates; if it still falls behind, it fetches the response so far
After a viewer reconnects	It fetches the chunks again and rebuilds the response	It fetches the batches again and rebuilds the response	It reloads the record, then follows new writes	Missed updates arrive in order, or it fetches the response so far
One usable response in storage?	No: it has to be rebuilt from chunks	No: it has to be rebuilt from batches	Yes, at the cost of a full write every flush	Yes: one message while the response streams, and your database holds the final copy

FAQs

Doesn't TCP flow control already handle backpressure in AI streaming?

Only one connection at a time. SSE, WebSockets, and HTTP streaming all run over TCP, or QUIC with HTTP/3, and their flow control stops a sender from overwhelming a single receiver, so a slow viewer's data piles up in the server's buffer for that connection instead. Flow control doesn't decide what happens to that backlog, help a viewer who joins mid-response, or coordinate one response across many viewers, so your delivery layer still has to.

Does backpressure matter if only one person watches each AI response?

Yes, though it shows up differently. A single viewer on a congested mobile network or in a background tab can still fall behind the model, and a page refresh or reconnect turns them into a late joiner of their own response. More viewers multiply the cost, but one slow or returning viewer is enough to expose how your delivery layer stores the response.

How can you tell when an AI streaming system is under backpressure?

Watch the gap between how fast the model generates and how fast each viewer receives. Warning signs are per-viewer lag that grows during a response, queue depth or memory that keeps rising on your servers, and final tokens arriving well after the model finishes. On Ably, a connection that exceeds its outbound rate limit receives a continuity-loss signal (error 80020), which your client can log and alert on.

Should one slow viewer throttle the model in AI streaming?

Usually not. When several people watch the same response, slowing generation for one lagging viewer delays everyone else and keeps the model call running longer. Let that viewer catch up from the current state of the response instead, and save throttling the model for real capacity limits, such as too many concurrent requests.