Tool calls

Open in

Modern AI models can invoke tools (also called functions) to perform specific tasks like retrieving data, performing calculations, or triggering actions. Streaming tool call information to users provides visibility into what the AI is doing, creates opportunities for rich generative UI experiences, and builds trust through transparency.

What are tool calls?

Tool calls occur when an AI model decides to invoke a specific function or tool to accomplish a task. Rather than only returning text, the model can request to execute tools you've defined, such as fetching weather data, searching a database, or performing calculations.

A tool call consists of:

Tool name: The identifier of the tool being invoked
Tool input: Parameters passed to the tool, often structured as JSON
Tool output: The result returned after execution

As an application developer, you decide how to surface tool calls to users. You may choose to display all tool calls, selectively surface specific tools or inputs/outputs, or keep tool calls entirely private.

Surfacing tool calls supports:

Trust and transparency: Users see what actions the AI is taking, building confidence in the agent
Human-in-the-loop workflows: Expose tool calls resolved by humans where users can review and approve tool execution before it happens
Generative UI: Build dynamic, contextual UI components based on the structured tool data

Publish tool calls

Publish tool call and model output messages to the channel.

In the example below, the responseId is included in the message extras to allow subscribers to correlate all messages belonging to the same response. The message name allows the client to distinguish between the different message types:

JavaScript

const channel = realtime.channels.get('ear-aim-red');

// Example: stream returns events like:
// { type: 'tool_call', name: 'get_weather', args: '{"location":"San Francisco"}', toolCallId: 'tool_123', responseId: 'resp_abc123' }
// { type: 'tool_result', name: 'get_weather', result: '{"temp":72,"conditions":"sunny"}', toolCallId: 'tool_123', responseId: 'resp_abc123' }
// { type: 'message', text: 'The weather in San Francisco is 72°F and sunny.', responseId: 'resp_abc123' }

for await (const event of stream) {
  if (event.type === 'tool_call') {
    // Publish tool call arguments
    await channel.publish({
      name: 'tool_call',
      data: {
        name: event.name,
        args: event.args
      },
      extras: {
        headers: {
          responseId: event.responseId,
          toolCallId: event.toolCallId
        }
      }
    });
  } else if (event.type === 'tool_result') {
    // Publish tool call results
    await channel.publish({
      name: 'tool_result',
      data: {
        name: event.name,
        result: event.result
      },
      extras: {
        headers: {
          responseId: event.responseId,
          toolCallId: event.toolCallId
        }
      }
    });
  } else if (event.type === 'message') {
    // Publish model output messages
    await channel.publish({
      name: 'message',
      data: event.text,
      extras: {
        headers: {
          responseId: event.responseId
        }
      }
    });
  }
}

Subscribe to tool call and model output messages on the channel.

In the example below, the responseId from the message extras is used to group tool calls and model output messages belonging to the same response. The message name allows the client to distinguish between the different message types:

JavaScript

const channel = realtime.channels.get('ear-aim-red');

// Track responses by ID, each containing tool calls and final response
const responses = new Map();

// Subscribe to all events on the channel
await channel.subscribe((message) => {
  const responseId = message.extras?.headers?.responseId;

  if (!responseId) {
    console.warn('Message missing responseId');
    return;
  }

  // Initialize response object if needed
  if (!responses.has(responseId)) {
    responses.set(responseId, {
      toolCalls: new Map(),
      message: ''
    });
  }

  const response = responses.get(responseId);

  // Handle each message type
  switch (message.name) {
    case 'message':
      response.message = message.data;
      break;
    case 'tool_call':
      const toolCallId = message.extras?.headers?.toolCallId;
      response.toolCalls.set(toolCallId, {
        name: message.data.name,
        args: message.data.args
      });
      break;
    case 'tool_result':
      const resultToolCallId = message.extras?.headers?.toolCallId;
      const toolCall = response.toolCalls.get(resultToolCallId);
      if (toolCall) {
        toolCall.result = message.data.result;
      }
      break;
  }

  // Display the tool calls and response for this turn
  console.log(`Response ${responseId}:`, response);
});

Generative UI

Tool calls provide structured data that can form the basis of generative UI - dynamically creating UI components based on the tool being invoked, its parameters, and the results returned. Rather than just displaying raw tool call information, you can render rich, contextual components that provide a better user experience.

For example, when a weather tool is invoked, instead of showing raw JSON like { location: 'San Francisco', temp: 72, conditions: 'sunny' }, you can render a weather card component with icons, formatted temperature, and visual indicators:

JavaScript

const channel = realtime.channels.get('ear-aim-red');

await channel.subscribe((message) => {
  // Render component when tool is invoked
  if (message.name === 'tool_call' && message.data.name === 'get_weather') {
    const args = JSON.parse(message.data.args);
    renderWeatherCard({ location: args.location, loading: true });
  }

  // Update component with results
  if (message.name === 'tool_result' && message.data.name === 'get_weather') {
    const result = JSON.parse(message.data.result);
    renderWeatherCard(result);
  }
});

Client-side tools

Some tools need to be executed directly on the client device rather than on the server, allowing agents to dynamically access information available on the end user's device as needed. These include tools that access device capabilities such as GPS location, camera, SMS, local files, or other native functionality.

Client-side tool calls follow a request-response pattern over Ably channels:

The agent publishes a tool call request to the channel.
The client receives and executes the tool using device APIs.
The client publishes the result back to the channel.
The agent receives the result and continues processing.

The client subscribes to tool call requests, executes the tool using device APIs, and publishes the result back to the channel. The toolCallId enables correlation between tool call requests and results:

JavaScript

const channel = realtime.channels.get('ear-aim-red');

await channel.subscribe('tool_call', async (message) => {
  const { name, args } = message.data;
  const { responseId, toolCallId } = message.extras?.headers || {};

  if (name === 'get_location') {
    const result = await getGeolocationPosition();
    await channel.publish({
      name: 'tool_result',
      data: {
        name: name,
        result: {
          lat: result.coords.latitude,
          lng: result.coords.longitude
        }
      },
      extras: {
        headers: {
          responseId: responseId,
          toolCallId: toolCallId
        }
      }
    });
  }
});

The agent subscribes to tool results to continue processing. The toolCallId correlates the result back to the original request:

JavaScript

const pendingToolCalls = new Map();

await channel.subscribe('tool_result', (message) => {
  const { toolCallId, result } = message.data;
  const pending = pendingToolCalls.get(toolCallId);

  if (!pending) return;

  // Pass result back to the AI model to continue the conversation
  processResult(pending.responseId, toolCallId, result);

  pendingToolCalls.delete(toolCallId);
});

Progress updates

Some tool calls take significant time to complete, such as processing large files, performing complex calculations, or executing multi-step operations. For long-running tools, streaming progress updates to users provides visibility into execution status and improves the user experience by showing that work is actively happening.

You can deliver progress updates using two approaches:

Messages: Best for discrete status updates and milestone events
LiveObjects: Best for continuous numeric progress and shared state synchronization

Progress updates via messages

Publish progress messages to the channel as the tool executes, using the toolCallId to correlate progress updates with the specific tool call:

JavaScript

const channel = realtime.channels.get('ear-aim-red');

// Publish initial tool call
await channel.publish({
  name: 'tool_call',
  data: {
    name: 'process_document',
    args: { documentId: 'doc_123', pages: 100 }
  },
  extras: {
    headers: {
      responseId: 'resp_abc123',
      toolCallId: 'tool_456'
    }
  }
});

// Publish progress updates as tool executes
await channel.publish({
  name: 'tool_progress',
  data: {
    name: 'process_document',
    status: 'Processing page 25 of 100',
    percentComplete: 25
  },
  extras: {
    headers: {
      responseId: 'resp_abc123',
      toolCallId: 'tool_456'
    }
  }
});

// Continue publishing progress as work progresses
await channel.publish({
  name: 'tool_progress',
  data: {
    name: 'process_document',
    status: 'Processing page 75 of 100',
    percentComplete: 75
  },
  extras: {
    headers: {
      responseId: 'resp_abc123',
      toolCallId: 'tool_456'
    }
  }
});

// Publish final result
await channel.publish({
  name: 'tool_result',
  data: {
    name: 'process_document',
    result: { processedPages: 100, summary: 'Document processed successfully' }
  },
  extras: {
    headers: {
      responseId: 'resp_abc123',
      toolCallId: 'tool_456'
    }
  }
});

Subscribe to progress updates on the client by listening for the tool_progress message type:

JavaScript

const channel = realtime.channels.get('ear-aim-red');

// Track tool execution progress
const toolProgress = new Map();

await channel.subscribe((message) => {
  const { responseId, toolCallId } = message.extras?.headers || {};

  switch (message.name) {
    case 'tool_call':
      toolProgress.set(toolCallId, {
        name: message.data.name,
        status: 'Starting...',
        percentComplete: 0
      });
      renderProgressBar(toolCallId, 0);
      break;

    case 'tool_progress':
      const progress = toolProgress.get(toolCallId);
      if (progress) {
        progress.status = message.data.status;
        progress.percentComplete = message.data.percentComplete;
        renderProgressBar(toolCallId, message.data.percentComplete);
      }
      break;

    case 'tool_result':
      toolProgress.delete(toolCallId);
      renderCompleted(toolCallId, message.data.result);
      break;
  }
});

Message-based progress is useful for:

Step-by-step status descriptions
Milestone notifications
Workflow stages with distinct phases
Audit trails requiring discrete event records

Progress updates via LiveObjects

Use LiveObjects for state-based progress tracking. LiveObjects provides a shared data layer where progress state is automatically synchronized across all subscribed clients, making it ideal for continuous progress tracking.

Use LiveCounter for numeric progress values like completion percentages or item counts. Use LiveMap to track complex progress state with multiple fields.

First, import and initialize the LiveObjects plugin:

JavaScript

import * as Ably from 'ably';
import { LiveObjects, LiveMap, LiveCounter } from 'ably/liveobjects';

// Initialize client with LiveObjects plugin
const realtime = new Ably.Realtime({
  key: 'demokey:*****',
  plugins: { LiveObjects }
});

// Get channel with LiveObjects capabilities
const channel = realtime.channels.get('ear-aim-red', {
  modes: ['OBJECT_SUBSCRIBE', 'OBJECT_PUBLISH']
});

// Get the channel's LiveObjects root
const root = await channel.object.get();

API key:

DEMO ONLY

Create a LiveMap to track tool progress:

JavaScript

// Create a LiveMap to track tool progress
await root.set('tool_456_progress', LiveMap.create({
  status: 'starting',
  itemsProcessed: LiveCounter.create(0),
  totalItems: 100,
  currentItem: ''
}));

// Update progress as tool executes
const progress = root.get('tool_456_progress');

await progress.set('status', 'processing');
await progress.set('currentItem', 'item_25');
await progress.get('itemsProcessed').increment(25);

// Continue updating as work progresses
await progress.set('currentItem', 'item_75');
await progress.get('itemsProcessed').increment(50);

// Final increment to reach 100%
await progress.set('currentItem', 'item_100');
await progress.get('itemsProcessed').increment(25);

// Mark complete
await progress.set('status', 'completed');

Subscribe to LiveObjects updates on the client to render realtime progress:

JavaScript

import * as Ably from 'ably';
import { LiveObjects } from 'ably/liveobjects';

// Initialize client with LiveObjects plugin
const realtime = new Ably.Realtime({
  key: 'demokey:*****',
  plugins: { LiveObjects }
});

// Get channel with LiveObjects capabilities
const channel = realtime.channels.get('ear-aim-red', {
  modes: ['OBJECT_SUBSCRIBE', 'OBJECT_PUBLISH']
});

// Get the channel's LiveObjects root
const root = await channel.object.get();

// Subscribe to progress updates
const progress = root.get('tool_456_progress');
progress.subscribe(() => {
  const status = progress.get('status').value();
  const itemsProcessed = progress.get('itemsProcessed').value();
  const totalItems = progress.get('totalItems').value();
  const percentComplete = Math.round((itemsProcessed / totalItems) * 100);

  renderProgressBar('tool_456', percentComplete, status);
});

API key:

DEMO ONLY

LiveObjects-based progress is useful for:

Continuous progress bars with frequent updates
Distributed tool execution across multiple workers
Complex progress state with multiple fields
Scenarios where multiple agents or processes contribute to the same progress counter

Choosing the right approach

Choose messages when:

Progress updates are infrequent (every few seconds or at specific milestones)
You need a complete audit trail of all progress events
Progress information is descriptive text rather than numeric
Each update represents a distinct event or stage transition

Choose LiveObjects when:

Progress updates are frequent (multiple times per second)
You're tracking numeric progress like percentages or counts
Multiple processes or workers contribute to the same progress counter
You want to minimize message overhead for high-frequency updates

You can combine both approaches for comprehensive progress tracking. Use LiveObjects for high-frequency numeric progress and messages for important milestone notifications:

JavaScript

// Update numeric progress continuously via LiveObjects
await progress.get('itemsProcessed').increment(1);

// Publish milestone messages at key points
if (itemsProcessed === totalItems / 2) {
  await channel.publish({
    name: 'tool_progress',
    data: {
      name: 'process_document',
      status: 'Halfway complete - 50 of 100 items processed'
    },
    extras: {
      headers: {
        responseId: 'resp_abc123',
        toolCallId: 'tool_456'
      }
    }
  });
}

Human-in-the-loop workflows

Tool calls resolved by humans are one approach to implementing human-in-the-loop workflows. When an agent encounters a tool call that needs human resolution, it publishes the tool call to the channel and waits for the human to publish the result back over the channel.

For example, a tool that modifies data, performs financial transactions, or accesses sensitive resources might require explicit user approval before execution. The tool call information is surfaced to the user, who can then approve or reject the action.

Tool calls

What are tool calls?

Publish tool calls

Subscribe to tool calls

Generative UI

Client-side tools

Progress updates

Progress updates via messages

Progress updates via LiveObjects

Choosing the right approach

Human-in-the-loop workflows