In repository-level question answering (QA), an agent explores a code repository with tools and should answer a question once enough information is gathered. Nevertheless, we find that LLM agents often continue to invoke tools even after the gathered information is already sufficient to answer. We call this phenomenon over-exploration. Moreover, even though existing methods improve task performance through supervised fine-tuning (SFT) on expert trajectories, SFT also propagates the over-exploration in the trajectories to the agent. To mitigate over-exploration, we propose Just-Enough Exploration Tuning (JET), a fine-tuning method that trains the agent to answer once the gathered information is sufficient. JET truncates each training trajectory at the step where the gathered information becomes sufficient, inserts a rationale that explains why the agent can answer at that step, and fine-tunes the agent with a higher loss weight on the rationale. Experiments on two repository-level QA benchmarks with two open-weight 8B models demonstrate that JET reduces tool calls by 11 to 33% on self-generated trajectories while largely preserving the score. On expert trajectories, JET makes 18 to 28% fewer tool calls than SFT while retaining most of the score gain of SFT. In both cases, JET also lowers the inference cost.
Figures & tables
Figure 1: An example of over-exploration on SWE-QA-Pro (Qwen3-8B). The answer forced after the third tool call (green) scores higher than the final answer after 26 more calls (red).
Figure 2: Forced-answer score (top) and percentage of trajectories still exploring (bottom) at each step, for (a) agents on the SWE-QA-Pro benchmark and (b) expert trajectories in the SFT training data. In (b), the agents answer from the context of the expert. The score reaches a plateau while the exploration continues. GPT-6 Luna scores the answers.
Method
Free-form
Trained
Sufficiency
Budget Tracker
✓
✗
✗
Autorater
✓
✗
✓
OTC, ET-Agent,
✗
✓
✗
DORA, Tool-Light
AutoSearch, DAS
✗
✓
✓
JET (ours)
✓
✓
✓
Table 1: Comparison with existing methods. Free-form : supports free-form answers rated on a scale. Trained : trains the agent, without extra inference cost. Sufficiency : targets the step at which the gathered information is sufficient.
Figure 3: Overview of JET. (1) The agent answers K times after every step, and the earliest step with the highest mean score is the golden step i⋆ . (2) We truncate the trajectory at i⋆ and append a rationale from a rationale generator, followed by the golden step answer. (3) We fine-tune the agent with the halting weight λ on the rationale.
Training data
Method
SWE-QA-Pro
SWE-QA
Score ↑
Calls ↓
Cost ()\downarrow$
Score ↑
Calls ↓
Cost ()\downarrow$
Qwen3-8B
Self
Base
30.88
18.10
19.2
36.50
19.57
23.0
Budget Tracker
30.29
26.12 + 44%
28.2
34.88
25.54 + 31%
27.8
Autorater
30.70
15.23 − 16%
29.0
35.61
16.18 − 17%
33.2
Trunc. SFT
31.22
13.37 − 26%
10.3
36.91
13.54 − 31%
11.7
Table 2: Main results on SWE-QA-Pro and SWE-QA. Cost is in US dollars per 1,000 questions. Purple shading marks an improvement and red shading a degradation, with darker shading for a larger difference. Shading and percentages are relative to Base in the Self setting and to SFT in the other two settings. Trunc. SFT denotes Truncated SFT (§ 5.2 ).
Figure 4: Percentage of questions on which the agent reaches the step limit. The numbers above the bars are the question counts. JET is compared with Base in the Self setting and with SFT in the Expert setting.
Truncation step
Score ↑
Calls ↓
Qwen3-8B
Base
30.88
18.10
Truncation
n/2
29.71
15.90 − 12%
2n/3
30.91
20.94 + 16%
i⋆ ( JET )
30.84
12.19 − 33%
Ministral-3-8B
Table 3: Ablation of the golden step on SWE-QA-Pro in the Self setting. The rows n/2 and 2n/3 truncate each trajectory at a fixed ratio of the number of steps n . Shading and percentages are relative to Base.
Figure 5: Tool calls and score of Ministral in the Self setting on the validation set for each halting weight λ .
Figure 6: Case study with Qwen3 in the Expert setting on a question from SWE-QA-Pro. Both agents read the relevant code within 11 tool calls. SFT continues to invoke tools until the step limit, whereas JET answers after step 11. Scores range from 5 to 50. We summarize the tool calls and shorten the responses, marked by […].
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Data synthesis
Intermediate answers per step K
3
Temperature of intermediate answers
0.6
Maximum tokens of a rationale
800
Fine-tuning
Halting weight λ
5
Appendix
Table 4: Hyperparameters shared by all runs. LoRA is applied to the linear layers of the language model.
Figure 7: Prompts of JET. (a) Forced-answer instruction pforce , shown with the first line for golden step labeling and the autorater. (b) Prompt of the rationale generator for the rationale r .
Figure 8: Prompts of the baselines. (a) Prompt of Budget Tracker, adapted from Liu et al. (2026b) . (b) Prompt of the autorater, adapted from Joren et al. (2025) .
Training data
Method
SWE-QA-Pro
SWE-QA
Corr.
Comp.
Rel.
Clar.
Reas.
Score
Corr.
Comp.
Rel.
Clar.
Reas.
Score
Qwen3-8B
Self
Base
4.77
3.78
8.14
8.77
5.42
30.88
6.25
4.40
8.98
11.13
5.74
36.50
Budget Tracker
4.47
3.38
8.59
8.73
5.11
30.29
6.02
3.92
8.87
10.79
5.29
34.88
Autorater
4.81
3.63
8.23
8.78
5.25
30.70
6.19
4.09
8.96
10.90
5.46
35.61
Trunc. SFT
4.92
3.63
8.62
8.77
5.28
31.22
6.48
4.23
9.37
11.20
5.63
36.91
Appendix
Table 5: Mean score of each judge criterion. Each criterion ranges from 1 to 10 on SWE-QA-Pro and from 1 to 20 on SWE-QA, and Score is the sum of the five criteria, as reported in Table 2 .
Tool-augmented LLM agents tend to call tools indiscriminately, even when the model can answer directly. Each unnecessary call wastes API fees and latency, yet no existing benchmark systematically studies when a tool call is actually needed. We propose When2Tool, a benchmark of 18 environments (15 single-hop, 3 multi-hop) spanning three categories of tool necessity -- computational scale, knowledge boundaries, and execution reliability -- each with controlled difficulty levels that create a clear decision boundary between tool-necessary and tool-unnecessary tasks. We evaluate two families of training-free baselines: Prompt-only (varying the prompt to discourage unnecessary calls) and Reason-then-Act (requiring the model to reason about tool necessity before acting). Both provide limited control: Prompt-only suppresses necessary calls alongside unnecessary ones, and Reason-then-Act still incurs a disproportionate accuracy cost on hard tasks. To understand why these baselines fail, we probe the models' hidden states and find that tool necessity is linearly decodable from the pre-generation representation with AUROC 0.89--0.96 across six models, substantially exceeding the model's own verbalized reasoning. This reveals that models already know when tools are needed, but fail to act on this knowledge during generation. Building on this finding, we propose Probe&Prefill, which uses a lightweight linear probe to read the hidden-state signal and prefills the model's response with a steering sentence. Across all models tested, Probe&Prefill reduces tool calls by 38% with only 1.7% accuracy loss, while the best baseline at comparable accuracy only reduces 6% of tool calls, or achieves a similar tool call reduction but incurs a 5× higher accuracy loss. Our code is available at https://github.com/Trustworthy-ML-Lab/when2tool
Research Agents enable models to gather information from the web using tools to answer user queries, requiring them to dynamically interleave internal reasoning with tool use. While such capabilities can in principle be learned via reinforcement learning with verifiable rewards (RLVR), we observe that agents often exhibit poor exploration behaviors, including premature termination and biased tool usage. As a result, RLVR alone yields limited improvements. We propose SynPlanResearch-R1, a framework that synthesizes tool-use trajectories that encourage deeper exploration to shape exploration during cold-start supervised fine-tuning, providing a strong initialization for subsequent RL. Across seven multi-hop and open-web benchmarks, \framework improves performance by up to 6.0% on Qwen3-8B and 5.8% on Qwen3-4B backbones respectively compared to SOTA baselines. Further analyses of tool-use patterns and training dynamics compared to baselines shed light on the factors underlying these gains. Our code is publicly available at https://github.com/HansiZeng/syn-plan-research.
Hansi Zeng, Zoey Li, Yifan Gao +7
University of Massachusetts Amherst · Amazon · University of Montreal +2
Large language models are increasingly deployed as tool-augmented agents to acquire information beyond parametric knowledge. While recent work has improved long-horizon tool-use reasoning, most approaches focus on tasks with a single correct answer. In contrast, many real-world queries require discovering a comprehensive set of valid answers, a setting known as Multi-Answer QA. This setting raises two challenges: fine-grained credit assignment over long search trajectories and reward alignment for sustained exploration beyond easy high-frequency entities. We propose SPADER, a reinforcement learning framework for long-horizon tool use in Multi-Answer QA. SPADER includes Step-wise Peer Advantage (SPA), a critic-free step-level credit assignment mechanism that aligns parallel trajectories by decision step and estimates advantages from peer returns. It also includes a diversity-aware exploration reward that promotes long-tail entity discovery by upweighting rare findings and downweighting redundant ones. Experiments on QAMPARI, Mintaka, WebQSP, and QUEST show that SPADER generally improves recall and overall F1 over prompting-based agents, outcome-supervised RL methods, and recent step-level supervision approaches. Our code and model weights are available at https://github.com/KhanCold/spader.
Qiming Shi, Zhaolu Kang, Yunfan Zhou +2
State Key Lab of CAD&CG, Zhejiang University · School of Software and Microelectronics, Peking University · School of Software Technology, Zhejiang University