Large language models (LLMs) are increasingly expected to orchestrate diverse tools over long-horizon tasks. Existing agents typically use Direct Tool Calling (DTC), where the model invokes tools sequentially, and tool responses accumulate in the model context, which can lead to context rot and orchestration breakdown. In contrast, Programmatic Tool Calling (PTC) invokes tools through executable code and stores intermediate results in runtime variables rather than appending them to the context. Prior work has explored PTC mainly as an inference-time optimization, but its benefits for agentic training remain unclear, and suitable verifiable training tasks are scarce. In this work, we introduce ProgAgent, a full PTC training stack, comprising a pipeline for synthesizing verifiable long-horizon tasks, a lightweight PTC runtime, and a two-stage training recipe. Using only 215 tasks instantiated at four environment scales, ProgAgent yields higher pass rates than the evaluated baselines of comparable size across four benchmarks. On LOCA-bench, ProgAgent-14B achieves a 30.7% pass rate at a 64k-token environment state, where these baselines score zero. Cross-harness experiments show that PTC-trained models outperform DTC-trained models under both evaluation interfaces, suggesting that the benefits of PTC training extend beyond the interface.
Figures & tables
Figure 1: Overview of ProgAgent. (a) Task synthesis. We construct verifiable tasks, scale their environment states from v0 to v3 while keeping instructions, task logic, and tool schemas fixed, and apply independent auditing and difficulty filtering. The scaling plot shows DeepSeek V4 Pro’s declining DTC pass@8 as environment size increases. (b) PTC runtime. A persistent IPython kernel executes tool-calling programs, retains intermediate results in variables across turns, and returns printed outputs to the agent. (c) Two-stage training. Cold-start SFT on teacher trajectories is followed by RL with binary rewards from Python task verifiers.
Figure 2: Direct tool calling (DTC) versus programmatic tool calling (PTC). DTC accumulates raw observations in the context, whereas PTC orchestrates tools via code and selectively prints outputs.
Table 2: Main results. Mean pass rates (%) on four benchmarks. Best and second-best scores within each 8B/14B comparison are bolded and underlined. Evaluation protocols and sources of reported scores are detailed in § 4.1 and Appendix C.4 .
Table 3: Training paradigms across benchmarks. Final pass rates (%) with the corresponding base model’s scores under the same harness shown in parentheses. Each trained model is evaluated under its training harness. The best score for each benchmark within each backbone is bolded.
Figure 3: Training paradigms on LOCA-bench. Each bar decomposes the final pass rate into the untrained backbone’s score and the gains from SFT and RL. Labels above the bars show rounded final pass rates; exact values are reported in Table 7 . Left: Results for the 8B and 14B backbones, averaged over four environment-state sizes. Right: Results for the 14B backbone at each environment-state size, showing PTC’s robustness as the environment grows.
Figure 4: RL training dynamics at 14B. Episode reward (left), tool-output tokens entering context (middle), and tool calls per episode (right) under DTC, Hybrid, and PTC.
Scale
Database Size (KB)
Workspace Files
pass@8
v0
99
12.7
0.95
v1
173
21.9
0.88
v2
335
31.1
0.65
v3
711
40.3
0.42
Table 4: Environment state and task difficulty. We present database size (KB), mean workspace file count, and DeepSeek V4 Pro’s DTC pass@8 across the four environment scales.
Model
Loops
Conditional execution
Error handling
8B
82.6
65.6
8.2
14B
82.1
64.0
9.5
Table 5: Programming structures used during RL. We present the percentage of generated code blocks using loops and conditional execution, and the percentage of executions using error handling.
Model
Eval
LOCA-bench
MCP-Atlas
Base
DTC-trained
PTC-trained
Base
DTC-trained
PTC-trained
8B
DTC
0.3
4.0
6.0
7.6
19.0
30.6
PTC
0.0
10.7
24.3
3.6
11.8
38.6
14B
DTC
0.3
9.0
13.9
12.4
29.7
41.6
PTC
0.0
20.7
38.3
3.8
29.0
45.6
Table 6: Cross-harness evaluation on LOCA-bench and MCP-Atlas. Pass rates (%) under DTC and PTC evaluation for untrained backbones (Base) and models trained with the DTC or PTC SFT+RL recipe. LOCA-bench scores are averaged over four environment-state sizes.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Distribution of 215 tasks across 23 domains.
Figure 6: Distribution of environment tool counts across the 215 tasks, excluding external harness tools. Each task is counted once.
Figure 7: Environment scaling under Hybrid and PTC. DeepSeek V4 Pro’s pass@8 on synthesized tasks at scales v0 – v3 ; the DTC counterpart appears in Figure 1 .
Model
Setting
8k
16k
32k
64k
Avg.
SFT
+RL
SFT
+RL
SFT
+RL
SFT
+RL
SFT
+RL
Untrained base models
8B
DTC
1.3
0.0
0.0
0.0
0.3
Hybrid
1.3
0.0
0.0
0.0
0.3
PTC
0.0
0.0
0.0
0.0
0.0
14B
DTC
1.3
0.0
0.0
0.0
0.3
Appendix
Table 7: Ablations on LOCA-bench. Pass rates (%) across four environment-state sizes. SFT and +RL denote checkpoints after supervised fine-tuning and subsequent reinforcement learning. For our trained models, Setting denotes the training paradigm; evaluation uses the same harness unless otherwise specified. Rows labeled “memory tool” use the DTC harness with the LOCA-bench memory tool. Values spanning the SFT and +RL columns report a single evaluated checkpoint.
Model
Environment
Tool calls
Tool-output tokens
Context limit (%)
DTC-like calls (%)
8B
Filesystem
50.5
21,086
16.7
6.21
Postgres
68.3
22,404
36.9
18.00
Notion
78.7
39,259
69.6
10.26
Playwright
122.1
37,551
64.0
34.71
14B
Filesystem
34.9
15,925
8.8
10.10
Postgres
58.0
22,200
8.3
14.70
Appendix
Table 8: Interaction costs of PTC on MCPMark. Tool calls and tool-output tokens are averaged per trajectory. Context limit reports the fraction of trajectories reaching the context limit. The “DTC-like calls” column reports the percentage of PTC calls containing a single explicit tool invocation without control flow, error handling, or selective printing; any printed tool return is reproduced in full without processing.
Figure 8: RL training dynamics at 8B. Episode reward (left), tool-output tokens entering context (middle), and tool calls per episode (right) under DTC, Hybrid, and PTC, with the same conventions as Figure 4 .
Tool use transforms LLMs into agents that act beyond their training data, and for code-capable models, programmatic tool calling extends this further by replacing rigid JSON calls with scripts that chain and parallelize naturally. However, a systematic evaluation of tools as code on an established benchmark across current and prior model generations under real-world task conditions has not been conducted. In this work, we empirically compare programmatic tool calling (PTC) to native JSON tool calling across 14 language models on BFCL v4. In the programmatic tool calling paradigm, tools are exposed as typed Python stubs that the model invokes through code, with execution and results handled in a single agent turn. Programmatic tool calling matches or exceeds native JSON tool calling in 11 of 14 models on BFCL v4, with the GPT-5.6 family achieving a 10.6% improvement over the JSON tool calling baseline. Further, it matches or outperforms baseline in 13 of 14 models under parallel fan-out, and holds stable under context rot conditions where baseline degrades 2.3% on average. Our results demonstrate that programmatic tool calling is a viable and robust alternative to JSON tool calling, with performance tracking model capability across release generations.
Ishan Patel, Sahil Sen, Elias Lumer +1
Commercial Technology and Innovation Office PricewaterhouseCoopers, U.S.A
Tool-augmented LLM agents commonly rely on step-wise atomic tool calls, where each invocation, observation, and value transfer is exposed in the main reasoning trace. This creates an execution-granularity mismatch: locally deterministic tool workflows are unfolded into repeated model-visible decisions, consuming context and forcing the model to manage low-level dataflow in the trace. We introduce HyperTool, a unified executable MCP-style tool interface that changes the model-visible unit of tool execution. A model invokes HyperTool with a code block that can call existing tools through their original schemas, manipulate returned values, and pass intermediate results locally, folding deterministic tool subroutines into a single outer call. To train models to use this interface, we synthesize HyperTool-format trajectories from cross-tool compositional tasks and verify them in real MCP environments. On MCP-Universe, HyperTool improves average accuracy from 15.69% to 35.29% on Qwen3-32B and from 9.93% to 33.33% on Qwen3-8B, and surpass GPT-OSS and Kimi-k2.5 on average accuracy, showing that our HyperTool can substantially improve multi-step tool use.
Yaxin Du, Yifan Zhou, Yujie Ge +7
Shanghai Jiao Tong University · IQuest Research · Beijing University of Aeronautics and Astronautics
Every major LLM agent framework gives the LLM the role of orchestrator; the model decides what to do next, when to call tools, and when to stop. We argue that token explosion, control-flow hallucination, and unreliable completion are not implementation bugs but architectural consequences of assigning the deterministic work of looping, branching, and sequencing to a probabilistic system. A better prompt or a stronger model cannot guarantee the reliability of the LLM agent. We therefore propose Agentic Programming, in which the program governs all control flow, and the LLM is itself part of it, an adaptive component we call LLM-as-Code and invoke only where a task calls for reasoning or generation. Within each call the model keeps full flexibility, but it cannot alter the program's execution path. With control in the program, the LLM's context is built from the execution history's call tree and forms a directed acyclic graph (DAG). Each call's context length is then determined by its call depth rather than by accumulation over steps. A case study of computer-use agents shows that the design is practical, not just a theoretical stance, substantially improving the stability of long visual operation sequences.