Skip to main content
Home

Speculative Tool Calling for Faster Agents

· 4 min read · By Dishant Miyani

Table of Contents

Speculative programmatic tool calling is compelling because it attacks agent latency at the harness layer, where a lot of real systems actually waste time.

Once an agent uses code as its action space, the old JSON tool-call rhythm starts to look too rigid. The model generates a full chunk of code, the harness waits, the REPL runs, and only then does the expensive sub-agent or search call finally start.

sPTC changes that shape. While the model is still streaming code, the harness reads partial REPL output, finds tool calls whose inputs are already knowable, and starts those calls early.

Sources #

Overview #

The clean idea is a shadow REPL.

The real REPL keeps the durable state of the run. The shadow REPL is a deep-copied namespace with hooked tools, so it can safely execute enough partial code to discover likely calls without mutating the real run.

If it sees llm_query("summarize this doc"), it does not need to wait for the full generated cell. It can launch the sub-call, register the future in a promise store, and move on.

Later, when the real REPL executes the final code and reaches the same call, the hooked real tool checks the store first. If the future is already done, the result returns immediately.

The Harness Owns the Async Work #

That is the part I like most: this is not asking the model to become better at async programming. It moves the optimization into the harness.

Two independent sub-agent calls can be overlapped even when the generated program wrote them as boring blocking calls. In that sense, sPTC is almost a tiny JIT compiler for agent code. It looks at the program while it is forming, detects work that can start now, and hides the slow part behind token generation or other REPL work.

This is the exact kind of infrastructure trick that compounds as agent programs get longer.

Latency Shape #

The benchmark result in the post is modest, around 1 to 1.2x on the reported RLM setup, but I do not read that as underwhelming.

Latency work starts with small honest wins, then gets larger when the pattern finds the right workload. Local inference is often memory-bound during decoding, so overlapping sub-calls can raise useful work per unit time.

Cloud or router setups get a different win: less idle wall-clock time while the root model is still thinking or while the REPL is about to hit a slow call. Either way, the point is the same. Agent harnesses have hidden parallelism, and programmatic tool calling exposes more of it than JSON slots ever did.

The Safety Boundary #

The safety boundary is the real engineering detail.

Literal inputs are easy. Variables derived from pure functions are usually fine. Calls that depend on file I/O, random external state, or unsafe functions should block speculation.

Identical non-deterministic calls also need occurrence tracking. Otherwise one speculative result can accidentally satisfy multiple real invocations that were meant to be separate.

This is why the promise store cannot just be a dumb cache keyed by function name. It needs enough identity to distinguish inputs, call occurrence, determinism, and the namespace state that made the speculation valid.

Why It Feels Inevitable #

If agents keep moving toward code-mode harnesses, sPTC feels inevitable.

The rough implementation already shows the shape, but the exciting version is language-agnostic, policy-aware, and aggressive without being reckless.

Imagine coding agents where searches, sub-agent reviews, eval probes, retrieval calls, and small verification jobs start as soon as their inputs become knowable, then collapse back into the real execution path when needed.

Less waiting, same program semantics, better machine utilization. Yeah, I am bullish on that.