Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls

Zeyu Zhang; Guohao Li; Zhenchang Xing; Alexandros Apostolopoulos; Yu Lin Lee; Liang Zheng

cs.SEFeb 22, 2026

Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls

Authors: Zeyu Zhang, Guohao Li, Zhenchang Xing, Alexandros Apostolopoulos, Yu Lin Lee, Liang Zheng

Organizations: Australian National University · CSIRO’s Data61 · Aipotheosis Labs

Abstract

The ability to use tools is fundamental for large language model (LLM) agents. Given a task, existing systems use LLMs to plan and generate tool calls, which are executed by real-world tools to complete the task. However, tool calls are prone to errors because they are generated primarily from the intrinsic capabilities of LLMs. Moreover, while it is useful to let LLMs iteratively refine the tool-call sequence using execution results from real tools, this process can be expensive and may cause unsafe side effects. To improve LLM tool calls and address issues caused by using real tools for refinement, we introduce Gecko, a stateful simulation environment that provides informative feedback for refining LLM tool calls before real execution. Specifically, Gecko combines rules and LLMs to check the validity of tool names and arguments, synthesize schema-conforming and state-consistent responses, and judge task completion against the user objective. These three types of feedback allow LLMs to refine their tool calls in simulation, forming a simple yet effective test-time scaling method named GATS. On BFCLv3 and $τ^2$ -bench, GATS consistently improves the performance of various LLMs.

Figures & tables

Figure 1: GATS uses Gecko’s feedback from simulated tool executions to iteratively refine tool calls, consistently improving the performance of different LLMs on BFCLv3 and

\tau^{2}

-bench.

Figure 2: Overview of Gecko simulation and real-tool inference. Given a user task, a planning LLM first explores in Gecko. For each attempt, Gecko validates tool calls, simulates tool responses, updates an estimated task state, and judges the attempt fails or completes the task by comparing the estimated state with the user objective. In this example, Gecko marks the first attempt as failed because the estimated state shows that temp/ is beside document/ rather than under it and marks the second attempt as completed. The successful plan is then prepended to the original user message as in-context guidance, and the real-tool agent solves the task using real tools.

Detection rate	GPT-4o	GPT-4.1-nano	Qwen-2.5-7B
True positive	100%	100%	100%
Syntactic errors	100%	100%	100%
Semantic errors	71%	69%	63%

Table 1: Accuracy of argument validation on BFCL-Live-Simple. ‘True positive’ is the rate where correct arguments are determined as correct. We also show percentage of syntactic errors and semantic errors being detected. As helper LLMs, GPT-4o, GPT-4.1-nano and Qwen-2.5-7B are evaluated.

Figure 3: Examples of feedback provided by Gecko through (a) syntactic argument validation, (b) semantic argument validation, and (c) synthetic responses, task state, and task feedback. The validation feedback (orange), synthetic response (yellow), and task state and feedback (blue) will then be fed to the planning LLM during test-time scaling.

	True positive	True negative
Real tools, LLM judge	96.7%	87.5%
Simulated tools, LLM judge	91.3%	87.5%

Table 2: True positive and true negative rates of the LLM judge for task success/failure judgments. We compare judgments based on trajectories generated with real tools and simulated tools.

Model	Overall Acc	Non-live single turn				Live single turn			Multi-turn
Model	Overall Acc	simple	parallel	multiple	irrelevance	simple	multiple	irrelevance	base
State-of-the-art reference models
ToolACE-2-8B	73.12	88.00	92.50	92.50	95.41	70.93	79.01	84.80	49.00
watt-tool-70B	79.27	98.25	85.50	94.00	84.16	86.04	83.47	68.48	68.00
xLAM-2-70b	80.96	94.75	92.00	94.50	83.33	77.13	71.13	74.48	77.50
Baseline models and our proposed method

Table 3: Method comparison on BFCLv3. We select eight most important metrics from BFCL website. Overall accuracy is the average of average ‘Non-live single turn’, average ‘Live single turn’, and ‘Multi-turn’. GATS consistently improves various planning LLMs.

Model	$\tau^{2}$ -retail	$\tau^{2}$ -airline	Overall
State-of-the-art reference models
Claude Opus 4	81.8%	60.0%	70.9%
Claude Sonnet 4	75.0%	55.5%	65.3%
Kimi-K2-Instruct	70.6%	56.5%	63.6%
Baseline models and w/ GATS
GPT-4o	62.9%	45.5%	54.2%

Table 4: Method comparison on

\tau^{2}

-bench. We report success rate under

\tau

-retail and

\tau

-airline subsets and average accuracy (Overall).

Method	Perf.	Tokens (k)	Avg. T.C.
GPT-5-mini	57.0%	80.3	6.70
w/ Reflexion	60.0%	102.9	6.62
w/ Merge-to-one	53.5%	111.2	5.88
w/ Self-Refine	59.0%	259.6	23.88
w/ Best-of-N	58.5%	365.8	13.78
w/ Majority Voting	59.0%	291.7	6.62

Table 5: Comparison between GATS and existing test-time scaling methods. We report performance, average token usage, and average number of real benchmark tool calls per task. Simulated tool calls inside Gecko are excluded from the tool-call count. GATS achieves the best performance in both settings while keeping real-tool overhead reasonable.

Panel of Figure 4 — Figure 4: (a) Ablation study of Gecko on BFCL-Multi-Turn-Base. The full system (72.0%) is compared against variants with one component removed: argument validation (70.5%), task state estimation (68.0%), and task feedback (61.5%); (b) LLM replacement study on Gecko evaluated on BFCL ‘Multi-turn base’. We use GPT-4o as planning LLM. For each component, the bar on the left is the original performance 72.0%. Under LLM replacement, e.g. , replacing GPT-4.1-nano with GPT-4o for argument validation.

Figure 5: Test-time scaling behaviors. We evaluate GATS on the BFCL-Non-Live-Simple, using GPT-4o as the planning LLM. Retry times is the maximum number of feedback-based refinement steps allowed in GATS (Section 4 ). We report (a) accuracy (%), (b) average latency (s) and (c) average cost ($) per user task versus maximum retry times.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$\tau^{2}$ -retail	$\tau^{2}$ -airline	Overall
GPT-5-mini	73.5%	57.0%	65.3%
+GATS (all mock)	75.2%	59.5%	67.4%
+GATS (hybrid)	78.5%	65.0%	71.8%

Appendix

Table 6: All-mock versus hybrid GATS on

\tau^{2}

-bench with GPT-5-mini.

Setting	Error Type	Count
Multi-turn	Incomplete plans	16/28
Multi-turn	Wrong tool selection	9/28
Multi-turn	Wrong arguments	12/28
Multi-turn	Excessive trial-and-error	1/28
Single-turn	Missing/extra function calls	10/19
Single-turn	Incorrect argument content	7/19

Appendix

Table 7: Error types corrected by GATS in cases where the baseline fails but GATS succeeds. Categories are not mutually exclusive.

LLM Tool Use

Large language model tool use studies how models select, call, and coordinate external tools to extend their reasoning and take actions.

See all 295 papers

LLM Agents Already Know When to Call Tools -- Even Without Reasoning
May 10, 2026Chung-En Sun, Linbo Liu, Ge Yan +2Tool-Augmented Language Model Agents AI Agent Benchmarks
University of California, San Diego
Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives
Sep 1, 2026Haibo Jin, Suijin Wang, Xucheng Yu +2Tool-Augmented Language Model Agents LLM Agent Orchestration
School of Information Sciences University of Illinois at Urbana-Champaign Champaign, IL 61820 · Independent Researcher Starc Institute
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley

May 10, 2026J/K move · Enter open · S save

LLM Agents Already Know When to Call Tools -- Even Without Reasoning

Chung-En Sun, Linbo Liu, Ge Yan, Zimo Wang+1

University of California, San Diego

Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Tool Use Tool Selection Tool Selection for Language Model Agents LLM Agent Efficiency

Open paper AskPDF arXiv

Tool-augmented LLM agents tend to call tools indiscriminately, even when the model can answer directly. Each unnecessary call wastes API fees and latency, yet no existing benchmark systematically studies when a tool call is actually needed. We propose When2Tool, a benchmark of 18 environments (15 single-hop, 3 multi-hop) spanning three categories of tool necessity -- computational scale, knowledge boundaries, and execution reliability -- each with controlled difficulty levels that create a clear decision boundary between tool-necessary and tool-unnecessary tasks. We evaluate two families of training-free baselines: Prompt-only (varying the prompt to discourage unnecessary calls) and Reason-then-Act (requiring the model to reason about tool necessity before acting). Both provide limited control: Prompt-only suppresses necessary calls alongside unnecessary ones, and Reason-then-Act still incurs a disproportionate accuracy cost on hard tasks. To understand why these baselines fail, we probe the models' hidden states and find that tool necessity is linearly decodable from the pre-generation representation with AUROC 0.89--0.96 across six models, substantially exceeding the model's own verbalized reasoning. This reveals that models already know when tools are needed, but fail to act on this knowledge during generation. Building on this finding, we propose Probe&Prefill, which uses a lightweight linear probe to read the hidden-state signal and prefills the model's response with a steering sentence. Across all models tested, Probe&Prefill reduces tool calls by 48% with only 1.7% accuracy loss, while the best baseline at comparable accuracy only reduces 6% of tool calls, or achieves a similar tool call reduction but incurs a 5

\times

higher accuracy loss. Our code is available at https://github.com/Trustworthy-ML-Lab/when2tool

Abstract

LLM Tool Use

Explore similar work

LLM Agents Already Know When to Call Tools -- Even Without Reasoning

Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents