Chain of thought

Modern AI applications stream chain-of-thought reasoning from thinking models alongside their output. Rather than immediately generating output, thinking models work through problems step-by-step, evaluating different approaches and refining their reasoning before and during output generation. Exposing this reasoning provides transparency into how the model arrived at its output, enabling richer user experiences and deeper insights into model behavior.

What is chain-of-thought?

Chain-of-thought is the model's internal reasoning process as it works through a problem. Modern thinking models output this reasoning as a stream of messages that show how they evaluate options, consider trade-offs, and plan their approach while generating output.

A single response from the model may consist of multiple output messages interleaved with reasoning messages, which are all associated with the same model response.

As an application developer, you decide what reasoning to surface and to whom. You may choose to expose all reasoning, filter or summarize it (for example, via a separate model call), or keep internal thinking entirely private.

Surfacing chain-of-thought reasoning provides:

• Trust and transparency: Users can see how the AI reached its conclusions, building confidence in the output
• Better user experience: Displaying reasoning in realtime provides feedback that the model is making progress during longer operations
• Enhanced steerability: Users can intervene and redirect the model based on its reasoning so far, guiding it toward better outcomes

Streaming patterns

As an application developer, you decide how to surface chain-of-thought reasoning to end users. Ably's pub/sub model is flexible and can accommodate any messaging pattern you choose. Below are a few common patterns used in modern AI applications, each showing both agent-side publishing and client-side subscription. Choose the approach that fits your use case, or create your own variation.

Inline pattern

In the inline pattern, agents publish reasoning messages to the same channel as model output messages.

By publishing all content to a single channel, the inline pattern:

• Simplifies channel management by consolidating all conversation content in one place
• Maintains relative order of reasoning and model output messages as the model generates them
• Supports retrieving reasoning and response messages together from history

Publish

Publish both reasoning and model output messages to a single channel.

In the example below, the responseId is included in the message extras to allow subscribers to correlate all messages belonging to the same response. The message name allows the client distinguish between the different message types:

const channel = realtime.channels.get('job-map-new');

// Example: stream returns events like:
// { type: 'reasoning', text: "Let's break down the problem: 27 * 12", responseId: 'resp_abc123' }
// { type: 'reasoning', text: 'First, I can split this into 27 * 10 + 27 * 2', responseId: 'resp_abc123' }
// { type: 'reasoning', text: 'That gives us 270 + 54 = 324', responseId: 'resp_abc123' }
// { type: 'message', text: '27 * 12 = 324', responseId: 'resp_abc123' }

for await (const event of stream) {
  if (event.type === 'reasoning') {
    // Publish reasoning messages
    await channel.publish({
      name: 'reasoning',
      data: event.text,
      extras: {
        headers: {
          responseId: event.responseId
        }
      }
    });
  } else if (event.type === 'message') {
    // Publish model output messages
    await channel.publish({
      name: 'message',
      data: event.text,
      extras: {
        headers: {
          responseId: event.responseId
        }
      }
    });
  }
}

Subscribe

Subscribe to both reasoning and model output messages on the same channel.

In the example below, the responseId from the message extras is used to group reasoning and model output messages belonging to the same response. The message name allows the client distinguish between the different message types:

const channel = realtime.channels.get('job-map-new');

// Track responses by ID, each containing reasoning messages and final response
const responses = new Map();

// Subscribe to all events on the channel
await channel.subscribe((message) => {
  const responseId = message.extras?.headers?.responseId;

  if (!responseId) {
    console.warn('Message missing responseId');
    return;
  }

  // Initialize response object if needed
  if (!responses.has(responseId)) {
    responses.set(responseId, {
      reasoning: [],
      message: ''
    });
  }

  const response = responses.get(responseId);

  // Handle each message type
  switch (message.name) {
    case 'message':
      response.message = message.data;
      break;
    case 'reasoning':
      response.reasoning.push(message.data);
      break;
  }

  // Display the reasoning and response for this turn
  console.log(`Response ${responseId}:`, response);
});

Threading pattern

In the threading pattern, agents publish reasoning messages to a separate channel from model output messages. The reasoning channel name is communicated to clients, allowing them to discover where to obtain reasoning content on demand.

By separating reasoning into its own channel, the threading pattern:

• Keeps the main channel clean and focused on final responses without reasoning output cluttering the conversation history
• Reduces bandwidth usage by delivering reasoning messages only when users choose to view them
• Works well for long reasoning threads, where not all the detail needs to be immediately surfaced to the user, but is helpful to see on demand

Publish

Publish model output messages to the main conversation channel and reasoning messages to a dedicated reasoning channel. The reasoning channel name includes the response ID, creating a unique reasoning channel per response.

In the example below, the agent sends a start control message on the main channel at the beginning of each response, which includes the response ID in the message extras. Clients can derive the reasoning channel name from the response ID, allowing them to discover and subscribe to the stream of reasoning messages on demand:

const conversationChannel = realtime.channels.get('job-map-new');

// Example: stream returns events like:
// { type: 'start', responseId: 'resp_abc123' }
// { type: 'reasoning', text: "Let's break down the problem: 27 * 12", responseId: 'resp_abc123' }
// { type: 'reasoning', text: 'First, I can split this into 27 * 10 + 27 * 2', responseId: 'resp_abc123' }
// { type: 'reasoning', text: 'That gives us 270 + 54 = 324', responseId: 'resp_abc123' }
// { type: 'message', text: '27 * 12 = 324', responseId: 'resp_abc123' }

for await (const event of stream) {
  if (event.type === 'start') {
    // Publish response start control message
    await conversationChannel.publish({
      name: 'start',
      extras: {
        headers: {
          responseId: event.responseId
        }
      }
    });
  } else if (event.type === 'reasoning') {
    // Publish reasoning to separate reasoning channel
    const reasoningChannel = realtime.channels.get(`job-map-new:${event.responseId}`);

    await reasoningChannel.publish({
      name: 'reasoning',
      data: event.text
    });
  } else if (event.type === 'message') {
    // Publish model output to main channel
    await conversationChannel.publish({
      name: 'message',
      data: event.text,
      extras: {
        headers: {
          responseId: event.responseId
        }
      }
    });
  }
}

Subscribe

Subscribe to the main conversation channel to receive control messages and model output. Subscribe to the reasoning channel on demand, for example in response to a click event.

In the example below, responseId from the message extras is used to derive the reasoning channel name, allowing clients to subscribe to the reasoning channel on demand to retrieve the reasoning associated with a response:

const conversationChannel = realtime.channels.get('job-map-new');

// Track responses by ID
const responses = new Map();

// Subscribe to all messages on the main channel
await conversationChannel.subscribe((message) => {
  const responseId = message.extras?.headers?.responseId;

  if (!responseId) {
    console.warn('Message missing responseId');
    return;
  }

  // Handle response start control message
  if (message.name === 'start') {
    responses.set(responseId, '');
  }

  // Handle model output message
  if (message.name === 'message') {
    responses.set(responseId, message.data);
  }
});

// Subscribe to reasoning on demand (e.g., when user clicks to view reasoning)
async function onClickViewReasoning(responseId) {
  // Derive reasoning channel name from responseId and
  // use rewind to retrieve historical reasoning
  const reasoningChannel = realtime.channels.get(`job-map-new:${responseId}`, {
    params: { rewind: '2m' }
  });

  // Subscribe to reasoning messages
  await reasoningChannel.subscribe((message) => {
    console.log(`[Reasoning]: ${message.data}`);
  });
}