Large language models are increasingly capable of writing and executing code, suggesting a natural paradigm for tool-augmented agents: expressing tool use as programs rather than isolated calls. Programmatic tool calling (PTC) captures this idea through loops, batching, abstraction, and structured computation, but existing PTC approaches are largely stateless, limiting their ability to reuse intermediate results and adapt over time. This is problematic for real-world tasks, where tool environments and problem-solving processes are inherently stateful. We therefore present I-PTC, an interactive framework that executes stateful PTC: models issue code snippets that call tools, maintain reusable state, and evolve the tool interface through API corrections. To evaluate the effectiveness of I-PTC and the general PTC setting, we introduce PTC-Bench, a benchmark of long-tail, production-style user requests whose difficulty comes from large initial environments, many dependent API interactions, extended planning, and recovery across follow-ups and transient failures. Experiments on PTC-Bench show that I-PTC improves performance on complex multi-step tasks with large tool sets while substantially reducing token consumption over conventional tool-calling baselines and the original PTC baseline, highlighting stateful programmatic tool calling as a promising direction for scalable tool-augmented agents.
Figures & tables
Figure 1: Pass rate vs. trajectory tokens for ten models on PTC-Bench
Figure 2: Comparison of traditional tool calling, programmatic tool calling (PTC), and interactive programmatic tool calling (I-PTC). Traditional tool calling executes tools through sequential agent-environment interactions, while PTC lets the agent generate executable code that orchestrates tool use inside a Python environment. I-PTC further extends PTC with a stateful execution environment, enabling reusable variables, intermediate-result inspection, execution feedback, tool inspection, and evolving API documentation across multiple rounds.
Figure 3: Self-evolving API documentation. After each task batch, the framework collects code snippets, API errors, tracebacks, and original tool descriptions. An updating agent converts these failure traces into concise API addenda, such as usage notes or corrections, which are appended to future tool descriptions so later agents avoid repeated mistakes.
Property
Value
Property
Value
Mean entities
244
Median state size
42 KB
Mean API calls
87.7
Median API calls
51
p90 API calls
202
≥ 50 API calls
51%
≥ 100 API calls
38%
Table 1: Complexity statistics for the PTC-Bench main subset.
Task Type
Method
Pass ↑
API ↓
Raw ↓
Traj. ↓
GPT-5.4 mini
Multi-turn
PTC
69.0%
180.9 (26.9)
83K
24K
I-PTC
77.1%
90.6 ( 4.1 )
51K
9K
Error-recovery
PTC
61.3%
92.6
70K
24K
I-PTC
72.9%
74.6
38K
7K
Main
PTC
50.0%
69.4
66K
23K
Table 3: Breakdown by task type. We compare PTC and I-PTC to isolate the effect of stateful programmatic execution. Metrics are averaged over three runs. API calls report the mean number of underlying simulated backend invocations per task. For multi-turn tasks, parentheses give API calls in the second turn (Q2) alone.
Figure 5: Generalization to existing tool-calling benchmarks. We evaluate GPT-5.4 mini and Gemini 3.1 Flash-Lite with traditional tool calling, standard PTC , and I-PTC on PTC-Bench , MCP-Universe subset and MCP-Bench subset.
Table 4: Execution overhead. Med. cell is the median execution time of I-PTC .
Subset
Total
Reuse
Rate
Main
279.3
158.7
56.8%
Multi-turn
195.0
116.3
59.7%
Error-recovery
238.3
160.0
67.1%
Overall
712.7
435.0
61.0%
Table 6: Cross-cell state reuse (GPT-5.4, three runs). Cells that fail when rerun in a fresh namespace count as state-reusing.
Figure 6: Effect of self-evolving API documentation. (a) Ablation over documentation-update batch size. (b) API errors increase as the learning batch size grows on the 100 main queries; batch size 100 corresponds to no effective learning during the run.
Model
Hosted interp.
Provider-native
I-PTC
GPT-5.4
17.6±3.6
76.2
80.0
GPT-5.4 mini
9.6±1.1
74.2
68.0
GPT-5.4 nano
6.7±0.0
48.3
46.4
Gemini 3.1 Pro
51.7±1.2
75.4
82.9
Gemini 3.1 Flash-Lite
39.4±3.2
60.8 †
72.8
Gemini 2.5 Pro
N/A ∗
46.2
60.0
Table 8: Provider-native code execution (pass rate, %). Provider-native: Codex code-only mode (GPT) and our interpreter on Gemini CLI tail tool calls (Gemini), single runs; Anthropic’s built-in code execution tool (Claude), from the runs of Table 2 . Hosted interpreters: mean ± std over three runs. I-PTC : Table 2 .
Runtime
Raw
Traj.
Wall
GPT-5.4 mini, Codex
332K
22.4K
55.5 s
GPT-5.4 mini, I-PTC
44K
7.6K
15.5 s
Gemini 3.1 Pro, Gemini CLI
174K
22.9K
211.2 s
Gemini 3.1 Pro, I-PTC
61K
7.9K
53.7 s
Claude Opus 4.7, built-in
246K
36.2K
–
Claude Opus 4.7, I-PTC
85K
7.6K
–
Table 9: Cost of provider-native code execution vs. I-PTC (per task). I-PTC rows for GPT-5.4 mini and Gemini 3.1 Pro are matched single runs with the documentation learner disabled (73.6% and 82.5% pass). Claude rows come from the runs of Table 2 , which did not record wall-clock time.
Model
Active params
Tool-only
PTC
Raw I-PTC
Qwen3.5-0.8B
0.8B
5.3±1.0
4.6±0.4
4.4±1.0
Qwen3.5-2B
2B
10.3±1.0
8.9±1.2
8.7±1.7
Qwen3.5-4B
4B
26.9±1.7
32.2±0.9
32.8±3.0
Qwen3.5-9B
9B
32.1±1.0
43.5±1.7
46.8±1.0
Qwen3.5-27B
27B
62.6±2.1
79.4±0.2
78.6±1.3
Qwen3.5-35B-A3B
3B
36.7±1.8
66.2±1.7
67.9±0.8
Table 10: Qwen3.5 model-scale study (pass rate, %, all 240 PTC-Bench tasks). Raw I-PTC uses persistent cells without inspection or documentation updates.
Model
Tool-only
PTC
Raw I-PTC
GPT-5.4
30.8±1.4
83.0±7.4
67.1±5.2
GPT-5.4 mini
45.0±1.8
135.1±32.3
81.6±7.3
Gemini 3.1 Flash-Lite
26.2±1.5
114.9±8.1
84.8±7.2
Table 11: Mean backend API calls per task on all 240 PTC-Bench tasks.
Large language models (LLMs) are increasingly expected to orchestrate diverse tools over long-horizon tasks. Existing agents typically use Direct Tool Calling (DTC), where the model invokes tools sequentially, and tool responses accumulate in the model context, which can lead to context rot and orchestration breakdown. In contrast, Programmatic Tool Calling (PTC) invokes tools through executable code and stores intermediate results in runtime variables rather than appending them to the context. Prior work has explored PTC mainly as an inference-time optimization, but its benefits for agentic training remain unclear, and suitable verifiable training tasks are scarce. In this work, we introduce ProgAgent, a full PTC training stack, comprising a pipeline for synthesizing verifiable long-horizon tasks, a lightweight PTC runtime, and a two-stage training recipe. Using only 215 tasks instantiated at four environment scales, ProgAgent yields higher pass rates than the evaluated baselines of comparable size across four benchmarks. On LOCA-bench, ProgAgent-14B achieves a 30.7% pass rate at a 64k-token environment state, where these baselines score zero. Cross-harness experiments show that PTC-trained models outperform DTC-trained models under both evaluation interfaces, suggesting that the benefits of PTC training extend beyond the interface.
Hao Li, Keqing He, Longtao Zheng +4
Hong Kong University of Science and Technology · Beijing University of Posts and Telecommunications · Nanyang Technological University
Tool use transforms LLMs into agents that act beyond their training data, and for code-capable models, programmatic tool calling extends this further by replacing rigid JSON calls with scripts that chain and parallelize naturally. However, a systematic evaluation of tools as code on an established benchmark across current and prior model generations under real-world task conditions has not been conducted. In this work, we empirically compare programmatic tool calling (PTC) to native JSON tool calling across 14 language models on BFCL v4. In the programmatic tool calling paradigm, tools are exposed as typed Python stubs that the model invokes through code, with execution and results handled in a single agent turn. Programmatic tool calling matches or exceeds native JSON tool calling in 11 of 14 models on BFCL v4, with the GPT-5.6 family achieving a 10.6% improvement over the JSON tool calling baseline. Further, it matches or outperforms baseline in 13 of 14 models under parallel fan-out, and holds stable under context rot conditions where baseline degrades 2.3% on average. Our results demonstrate that programmatic tool calling is a viable and robust alternative to JSON tool calling, with performance tracking model capability across release generations.
Ishan Patel, Sahil Sen, Elias Lumer +1
Commercial Technology and Innovation Office PricewaterhouseCoopers, U.S.A
Tool-augmented LLM agents commonly rely on step-wise atomic tool calls, where each invocation, observation, and value transfer is exposed in the main reasoning trace. This creates an execution-granularity mismatch: locally deterministic tool workflows are unfolded into repeated model-visible decisions, consuming context and forcing the model to manage low-level dataflow in the trace. We introduce HyperTool, a unified executable MCP-style tool interface that changes the model-visible unit of tool execution. A model invokes HyperTool with a code block that can call existing tools through their original schemas, manipulate returned values, and pass intermediate results locally, folding deterministic tool subroutines into a single outer call. To train models to use this interface, we synthesize HyperTool-format trajectories from cross-tool compositional tasks and verify them in real MCP environments. On MCP-Universe, HyperTool improves average accuracy from 15.69% to 35.29% on Qwen3-32B and from 9.93% to 33.33% on Qwen3-8B, and surpass GPT-OSS and Kimi-k2.5 on average accuracy, showing that our HyperTool can substantially improve multi-step tool use.
Yaxin Du, Yifan Zhou, Yujie Ge +7
Shanghai Jiao Tong University · IQuest Research · Beijing University of Aeronautics and Astronautics