AblyDocs

Going to production

Before you ship AI Transport in production, walk through this checklist. It covers the operational concerns that come up across every feature.

Work through each item before traffic moves to production.

Pricing

AI Transport runs on Ably's usage-based billing. Ably meters published and delivered messages, channel time, and connection time. The exact rates depend on your package; contact Ably for enterprise pricing.

Two AI Transport patterns affect cost predictably:

  • Token streaming uses Ably's append rollup to compact many token events into fewer published messages. The default rollup window is 40 ms (about 25 messages per second per stream), and you can tune it.
  • Channel history retention determines how long a client can rebuild the session from Ably alone. A longer retention window means Ably stores more; pair it with an external store for long-lived conversations.

AI Transport pricing explains the costs of usage of the AI Transport product, stating how each operation contributes to your billable message total and discusses the levers that can be used to manage the cost. For a worked example, see the AI chatbot pricing example.

Limits and quotas

The platform limits page covers the hard limits in detail. The ones that matter most for AI Transport:

  • Connection inbound message rate. A single stream that exceeds the per-connection rate triggers rate-limiting. Configure the append rollup window to control the publish rate.
  • Channel message rate. Concurrent turns and multi-agent setups share the channel's rate budget.
  • Message size. Large tool outputs or attachments approach the per-message size limit. Stream large results or persist them externally and reference the URL.
  • Channel history retention window. Configure your retention window through the channel rule settings. Plan for the longest session you expect to hydrate from Ably alone.

Monitoring and observability

Watch three signals:

  • Channel publish rate per session, against the connection's inbound rate limit.
  • Run lifecycle outcomes (complete, cancelled, error). A rising failure rate is your first signal that something is wrong with model orchestration, auth, or capacity.
  • Server-side LLM latency and error rate, from your model provider. The Ably side is decoupled from the LLM; track them separately.

Send these signals to your existing dashboards through Ably's integrations. Webhooks, Kafka, and other sinks expose channel activity. Ably also offers Stats API for usage data.

Auth hardening

The defaults are safe for development. For production:

  • Scope channel capabilities to the specific conversation channel (or a wildcard covering them) the user needs access to. publish, subscribe, and history only on channels the user owns.
  • Set clientId in the token rather than on the client. The Ably service verifies tokens; client-set IDs are unverified and spoofable.
  • Use short token lifetimes and authUrl (or authCallback) so the SDK refreshes tokens automatically. One-hour tokens are a reasonable default.
  • Implement token revocation for forced sign-out.
  • For cancel authorisation, set the onCancel hook on every turn. Reject cancel requests from clients that do not own the turn unless you specifically want admin-style global cancellation.

Set up authentication covers the token endpoint, capability scoping, and the token lifecycle.

Data shipping and retention windows

Where long-term conversation history lives depends on how much of AI Transport you adopted. With streaming, your own database already stores the conversation and your retention window only has to cover the live-recovery window. With a durable session, the channel stores the conversation for the period your retention window is configured for, and your store covers anything beyond it.

  • The channel history retention window is configured per channel rule. Pick a value that matches your live-recovery window rather than your total retention window. An external store covers anything longer.
  • With a durable session, persist each completed run's messages on run.end() and seed from that store on load. Database hydration covers where the two meet.
  • For analytics or search, stream channel activity to your data warehouse through webhooks or Kafka. This is one-way; AI Transport reads from the channel rather than the warehouse.

Compliance

Ably is SOC 2 Type II certified and HIPAA compliant.

For the application-side:

  • Message content, including tool inputs and outputs, is visible to every subscriber whose capability allows it.
  • For region-pinned data residency, scope capabilities to channels hosted in the right region of Ably's edge network.

Deployment notes

A few patterns that come up:

  • Serverless agents are fine. Each turn creates a server transport, processes the turn, and tears down. The session lives on the channel rather than in the agent process.
  • after() (Next.js) or equivalent post-response continuation lets the HTTP route return before the model finishes streaming. Without it, the time available for streaming is limited by the request timeout.
  • Multiple agent regions are supported. Ably's infrastructure keeps state present in every region; an agent in one region publishes and clients in any region see the same state with low latency from their nearest edge.

Durable execution

If an agent turn spans multiple stages (inference, tools, follow-up inference) or exceeds a serverless function's runtime budget, run the agent inside a workflow engine such as Temporal or Vercel WDK. Each stage becomes its own retryable activity. AI Transport's AgentSession.adoptRun plus AgentRun.createStep({ stepId }) let a retry of the same stage supersede the failed attempt's output on the session rather than append beside it.

When a turn exhausts its retries, your failure path needs to adopt the run and publish run.end({ reason: 'error' }). Otherwise the run stays open on the session and every observer's UI stays on streaming.

Conversation state

These apply once AI Transport holds the conversation rather than your database.

  • Your channel retention window now bounds how far back a new client can rebuild the conversation from Ably alone. Set it against the longest conversation a user can return to, and add database hydration past that point.
  • Your agent has to be the only writer to the store you hydrate from. A second writer puts messages between your stored history and the live tail that neither side accounts for.
  • Point one paginator at a given session.view. A second one paging past the same join point reintroduces the duplicate messages that the join point exists to drop.
  • Branch-aware hydration is not available, so a conversation seeded from a store is linear. Plan for that if your interface offers edit or regenerate on old messages.

When adopting AI Transport in production, also consider

The cross-cutting items to walk through:

Read next