# Tool Calling & Emulation

> Source: https://parallelworks.com/docs/ai/ai-providers/tool-calling

# Tool Calling & Emulation

Agentic clients such as [`pw code`](/docs/ai/code/) work by calling tools: the model asks to read a file, run a command, or search the workspace, and the client executes the request and returns the result. Many model endpoints implement tool calling directly, but some do not.

Every model reachable through the ACTIVATE AI gateway is advertised with one of three tool-calling modes. Models whose endpoint has no tool-calling API can still be used for agentic work through **emulated tool calling**, which the gateway handles on their behalf.

## The Three Modes

| Mode | Meaning |
| --- | --- |
| `native` | The provider's endpoint implements tool calling. The gateway passes tool definitions and tool calls straight through. |
| `emulated` | The provider's endpoint has no tool-calling API. The gateway carries the tool contract in the prompt and reconstructs tool calls from the model's text. |
| `none` | Tool calling is unavailable for this provider. Requests that carry tool definitions are rejected. |

The mode is a property of the provider integration, not of the individual model. Platform administrators set it when creating or editing a provider, and it is required when registering a custom OpenAI-compatible integration. Built-in integrations ship with the correct mode already set; GenAI.mil, for example, is registered as `emulated`.

### Seeing the Mode

The mode is reported wherever models are listed:

```bash
# The "Tool calling" column shows yes, emulated, or no
pw ai models ls
```

The models endpoint returns it as `tool_calling_mode` on each entry:

```bash
GET /api/openai/v1/models
```

Inside an interactive `pw code` session, the status line below the composer shows `emulated tools` when the active model is running in emulated mode, and `tools unavailable` when the model's mode is `none`.

## Why Emulation Exists

Some endpoints, including agent-wrapper services and a number of self-hosted inference servers, accept OpenAI-compatible chat requests but do not support the `tools` parameter. Without emulation, those models would be limited to chat and could not be used with `pw code`.

Emulation makes tool calling a property of the gateway rather than of the provider. Clients send the same OpenAI-shaped request with a `tools` array and receive the same `tool_calls` in the response, so no client-side configuration changes.

## Architecture

All three modes follow one path from the client to the model. The gateway resolves the provider, reads its tool-calling mode, and applies the matching translation layer. Only the emulation layer rewrites the request; native passes it through and `none` rejects it.

or any OpenAI-compatible client"]

    subgraph gateway["ACTIVATE AI gateway"]
        resolve["Reads the provider's<br/>tool calling mode"]
        native["Native<br/>pass tools through"]
        emulate["Emulated<br/>tool contract in the prompt"]
        none["None<br/>tool calls rejected"]
    end

    subgraph providers["Model endpoints"]
        withtools["Endpoint with a tools API"]
        notools["Endpoint without a tools API"]
    end

    client -->|"request with tools"| resolve
    resolve --> native
    resolve --> emulate
    resolve --> none
    native <--> withtools
    emulate <-->|"plain chat"| notools
    emulate -.->|"tool calls"| client
    native -.->|"tool calls"| client

    classDef gatewayNode fill:#dbeafe,stroke:#3b82f6,stroke-width:2px,color:#1e3a5f
    classDef emulateNode fill:#fef3c7,stroke:#f59e0b,stroke-width:2px,color:#78350f
    classDef providerNode fill:#d1fae5,stroke:#10b981,stroke-width:2px,color:#064e3b
    classDef clientNode fill:#ede9fe,stroke:#8b5cf6,stroke-width:2px,color:#4c1d95
    class resolve,native,none gatewayNode
    class emulate emulateNode
    class withtools,notools providerNode
    class client clientNode
    style gateway stroke:#3b82f6,stroke-width:2px
    style providers stroke:#10b981,stroke-width:2px
`} />

The client uses the same contract in every mode. It sends `tools`, receives `tool_calls`, and runs them locally in your workspace. The model does not execute anything itself in either mode.

## How Emulation Works

When a request with tool definitions is routed to an emulated provider, the gateway translates in both directions. A single tool round trip looks like this.

<MermaidDiagram chart={`
%%{init: {"theme": "neutral", "themeVariables": {"actorBkg": "#ede9fe", "actorBorder": "#8b5cf6", "actorTextColor": "#4c1d95", "noteBkgColor": "#fef3c7", "noteBorderColor": "#f59e0b", "noteTextColor": "#78350f"}} }%%
sequenceDiagram
    participant C as pw code
    participant G as AI gateway
    participant M as Model endpoint

    C->>G: request with tool definitions
    Note over G: Move the tool contract into the prompt<br/>Add a per-request nonce
    G->>M: plain chat request
    M-->>G: JSON envelope as text
    Note over G: Check the framing values<br/>Validate the arguments
    G-->>C: standard tool calls
    Note over C: Run the tool in the workspace
    C->>G: tool result
    Note over G: Fence the result as untrusted data
    G->>M: chat request with the result
    M-->>G: plain text answer
    G-->>C: assistant message
`} />

### 1. The Tool Contract Moves Into the Prompt

The gateway strips `tools`, `tool_choice`, and `parallel_tool_calls` from the upstream request and appends an instruction block to the conversation describing:

- The full JSON Schema of every tool the client defined.
- The exact shape of a tool-call **envelope** the model must emit.
- The tool policy for this request, derived from `tool_choice` (optional, required, or one named function) and from whether parallel calls are allowed.

The block is appended to the **user turn** rather than sent as a system message. Some agent-wrapper endpoints apply their own system prompt and discard the one supplied by the caller, while still honoring user content, so instructions sent as a system message would not reach the model on those providers.

### 2. The Model Replies With an Envelope

To call a tool, the model responds with a single JSON object and nothing else:

```json
{
  "protocol": "parallelworks.tool-call.v1",
  "toolset": "sha256:9f2c…",
  "nonce": "4b81f0c39ad2",
  "calls": [
    { "name": "Bash", "arguments": { "command": "ls -la", "description": "list files" } }
  ]
}
```

The gateway checks three framing values before it honors an envelope:

- **`protocol`** identifies the envelope format.
- **`toolset`** is a digest of the tool definitions sent with this request, so an envelope written against a different set of tools is not accepted.
- **`nonce`** is generated for each request and must be echoed back. Content that existed before the request cannot contain the current nonce, so an envelope taken from earlier conversation content or from tool output is not accepted as a live call.

### 3. The Gateway Validates and Converts

The gateway locates the envelope in the model's output, allowing for code fences and surrounding prose, then checks it:

- Protocol, toolset, and nonce must match this request.
- The named tool must be one the client defined.
- The arguments must validate against that tool's JSON Schema.
- The call count must match the request's parallel-call policy.

A valid envelope becomes ordinary OpenAI `tool_calls` with generated call IDs and a `tool_calls` finish reason, so the client receives a standard tool-calling response.

Output that is not an envelope attempt is returned unchanged as a plain text answer. Ordinary replies, including replies that contain JSON, are not affected.

### 4. Near Misses Are Repaired, Then Retried Once

Models trained on other tool formats produce a small set of predictable variations. The gateway repairs these locally rather than making another upstream call:

- Alternate field names for the call list (`tool_calls`, `toolCalls`) and for arguments (`parameters`, `args`, `input`).
- A call wrapped in a `function` / `function_call` object.
- A single call emitted bare, or a call list emitted as one object instead of an array.
- Arguments emitted as a JSON string instead of an object.
- Extra top-level fields, duplicate keys, and reordered or reserialized JSON.

Repairs are re-validated against the same rules, and a repaired envelope that still fails validation is not accepted. Nonce repair is declined when the envelope appears to have been copied out of tool output.

If validation still fails, the gateway makes one corrective retry. It replays the rejected output, states the reason it was rejected, and supplies a skeleton envelope with the correct protocol, toolset, and nonce already filled in, leaving the model only the call to write. If the retry also fails, the request returns an error rather than passing malformed text back as an answer.

### 5. Tool Results Go Back as Fenced, Untrusted Data

On the next turn, the client's `tool` messages are rendered as text the model can read. Each result is wrapped in `[TOOL_OUTPUT <nonce>]` markers carrying the same per-request nonce and labeled as untrusted data that must not be followed as instructions. Since the nonce is new for each request, earlier output cannot reproduce the markers to close the fence early or imitate gateway framing. Images in tool results are passed through as images.

## Behavior to Expect

**Streaming.** A streaming request to an emulated model is still streamed from the provider, but the gateway needs the complete response before it can tell whether the output is an envelope or an answer. The response is buffered and then sent to the client as a stream, so tokens do not appear incrementally as they do with a native provider.

**Prompt caching.** The tool contract sits in the user turn and carries a per-request nonce, so the prompt prefix differs on every request. Emulated requests do not benefit from provider-side prompt-prefix caching.

**Token cost.** Each request repeats the JSON Schema of every tool in the prompt, which accounts for a noticeable share of input tokens when the toolset is large. A corrective retry adds the cost of a second call for that turn. Usage from both calls is metered.

**The Responses API.** Emulation is available over Chat Completions only. A request to `/api/openai/v1/responses` for an emulated provider is rejected with `tool_call_emulation_requires_chat_completions`, and the models list advertises emulated integrations as Chat Completions.

**Reliability.** Emulation depends on the model following instructions and copying the framing values accurately. Models that follow instructions well use tools consistently; smaller models, and models behind an agent wrapper, call tools less consistently than they would through a native API.

## Limits

Requests that exceed these bounds are rejected before they are sent upstream.

| Limit | Value |
| --- | --- |
| Tools per request | 128 |
| Calls per envelope | 32 |
| Encoded tool definitions | 2 MB |
| Envelope size | 2 MB |
| JSON nesting depth (tools and envelope) | 64 |
| Buffered response size | 16 MB |

Tool names must be 1 to 64 characters of letters, numbers, underscores, or hyphens, and must be unique within a request. Only function tools are supported. Parameter schemas that expand to a very large number of branches through `$ref` and `anyOf` are rejected before compilation.

## Troubleshooting

**The status line says `emulated tools` unexpectedly.** The active model's provider is registered as `emulated`. Switch to a model from a `native` provider with `/model`, or ask an administrator whether the provider's mode is correct for its endpoint.

**The model answers in prose instead of using a tool.** This is the most common emulation failure. Stating the action directly ("read `main.go` and tell me what it does") usually helps, as does selecting a larger model from the same provider.

**"This model does not support tool calling."** The provider is registered as `none`. Run `pw code --no-tools` for a chat-only session, or choose a different model.

**Calls fail repeatedly on one specific tool.** Emulated arguments are validated against the tool's JSON Schema as written. A large or deeply nested schema is harder for a model to satisfy from a prompt than through a native API, so simplifying the schema usually resolves it.

## Related Documentation

- [Custom OpenAI-Compatible](/docs/ai/ai-providers/tool-calling/custom-openai-compatible) — Register an endpoint and set its tool-calling mode
- [pw code](/docs/ai/code/) — The AI coding agent that uses tool calling
- [Models & Allocations](/docs/ai/code/models) — Discovering and switching models
- [AI Keys](/docs/ai/ai-keys/) — Programmatic access through the gateway