In repository-level question answering (QA), an agent explores a code repository with tools and should answer a question once enough information is gathered. Nevertheless, we find that LLM agents often continue to invoke tools even after the gathered information is already sufficient to answer. We call this phenomenon over-exploration. Moreover, even though existing methods improve task performance through supervised fine-tuning (SFT) on expert trajectories, SFT also propagates the over-exploration in the trajectories to the agent. To mitigate over-exploration, we propose Just-Enough Exploration Tuning (JET), a fine-tuning method that trains the agent to answer once the gathered information is sufficient. JET truncates each training trajectory at the step where the gathered information becomes sufficient, inserts a rationale that explains why the agent can answer at that step, and fine-tunes the agent with a higher loss weight on the rationale. Experiments on two repository-level QA benchmarks with two open-weight 8B models demonstrate that JET reduces tool calls by 11 to 33% on self-generated trajectories while largely preserving the score. On expert trajectories, JET makes 18 to 28% fewer tool calls than SFT while retaining most of the score gain of SFT. In both cases, JET also lowers the inference cost.
Figures & tables
Figure 1: An example of over-exploration on SWE-QA-Pro (Qwen3-8B). The answer forced after the third tool call (green) scores higher than the final answer after 26 more calls (red).
Figure 2: Forced-answer score (top) and percentage of trajectories still exploring (bottom) at each step, for (a) agents on the SWE-QA-Pro benchmark and (b) expert trajectories in the SFT training data. In (b), the agents answer from the context of the expert. The score reaches a plateau while the exploration continues. GPT-6 Luna scores the answers.
Method
Free-form
Trained
Sufficiency
Budget Tracker
✓
✗
✗
Autorater
✓
✗
✓
OTC, ET-Agent,
✗
✓
✗
DORA, Tool-Light
AutoSearch, DAS
✗
✓
✓
JET (ours)
✓
✓
✓
Table 1: Comparison with existing methods. Free-form : supports free-form answers rated on a scale. Trained : trains the agent, without extra inference cost. Sufficiency : targets the step at which the gathered information is sufficient.
Figure 3: Overview of JET. (1) The agent answers K times after every step, and the earliest step with the highest mean score is the golden step i⋆ . (2) We truncate the trajectory at i⋆ and append a rationale from a rationale generator, followed by the golden step answer. (3) We fine-tune the agent with the halting weight λ on the rationale.
Training data
Method
SWE-QA-Pro
SWE-QA
Score ↑
Calls ↓
Cost ()\downarrow$
Score ↑
Calls ↓
Cost ()\downarrow$
Qwen3-8B
Self
Base
30.88
18.10
19.2
36.50
19.57
23.0
Budget Tracker
30.29
26.12 + 44%
28.2
34.88
25.54 + 31%
27.8
Autorater
30.70
15.23 − 16%
29.0
35.61
16.18 − 17%
33.2
Trunc. SFT
31.22
13.37 − 26%
10.3
36.91
13.54 − 31%
11.7
Table 2: Main results on SWE-QA-Pro and SWE-QA. Cost is in US dollars per 1,000 questions. Purple shading marks an improvement and red shading a degradation, with darker shading for a larger difference. Shading and percentages are relative to Base in the Self setting and to SFT in the other two settings. Trunc. SFT denotes Truncated SFT (§ 5.2 ).
Figure 4: Percentage of questions on which the agent reaches the step limit. The numbers above the bars are the question counts. JET is compared with Base in the Self setting and with SFT in the Expert setting.
Truncation step
Score ↑
Calls ↓
Qwen3-8B
Base
30.88
18.10
Truncation
n/2
29.71
15.90 − 12%
2n/3
30.91
20.94 + 16%
i⋆ ( JET )
30.84
12.19 − 33%
Ministral-3-8B
Table 3: Ablation of the golden step on SWE-QA-Pro in the Self setting. The rows n/2 and 2n/3 truncate each trajectory at a fixed ratio of the number of steps n . Shading and percentages are relative to Base.
Figure 5: Tool calls and score of Ministral in the Self setting on the validation set for each halting weight λ .
Figure 6: Case study with Qwen3 in the Expert setting on a question from SWE-QA-Pro. Both agents read the relevant code within 11 tool calls. SFT continues to invoke tools until the step limit, whereas JET answers after step 11. Scores range from 5 to 50. We summarize the tool calls and shorten the responses, marked by […].
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Data synthesis
Intermediate answers per step K
3
Temperature of intermediate answers
0.6
Maximum tokens of a rationale
800
Fine-tuning
Halting weight λ
5
Appendix
Table 4: Hyperparameters shared by all runs. LoRA is applied to the linear layers of the language model.
Figure 7: Prompts of JET. (a) Forced-answer instruction pforce , shown with the first line for golden step labeling and the autorater. (b) Prompt of the rationale generator for the rationale r .
Figure 8: Prompts of the baselines. (a) Prompt of Budget Tracker, adapted from Liu et al. (2026b) . (b) Prompt of the autorater, adapted from Joren et al. (2025) .
Training data
Method
SWE-QA-Pro
SWE-QA
Corr.
Comp.
Rel.
Clar.
Reas.
Score
Corr.
Comp.
Rel.
Clar.
Reas.
Score
Qwen3-8B
Self
Base
4.77
3.78
8.14
8.77
5.42
30.88
6.25
4.40
8.98
11.13
5.74
36.50
Budget Tracker
4.47
3.38
8.59
8.73
5.11
30.29
6.02
3.92
8.87
10.79
5.29
34.88
Autorater
4.81
3.63
8.23
8.78
5.25
30.70
6.19
4.09
8.96
10.90
5.46
35.61
Trunc. SFT
4.92
3.63
8.62
8.77
5.28
31.22
6.48
4.23
9.37
11.20
5.63
36.91
Appendix
Table 5: Mean score of each judge criterion. Each criterion ranges from 1 to 10 on SWE-QA-Pro and from 1 to 20 on SWE-QA, and Score is the sum of the five criteria, as reported in Table 2 .
State Key Lab of CAD&CG, Zhejiang University · School of Software and Microelectronics, Peking University · School of Software Technology, Zhejiang University