Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification

Yingfeng Luo; Shaowei Wei; Daixin Wang; Dingyang Lin; Kaiyan Chang; Weiqiao Shan; Tong Zheng; Zhiqiang Zhang

cs.CLSep 30, 2026

Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification

Authors: Yingfeng Luo, Shaowei Wei, Daixin Wang, Dingyang Lin, Kaiyan Chang, Weiqiao Shan, Tong Zheng, Zhiqiang Zhang, +2 more

Organizations: Northeastern University · Inclusion AI, Ant Group · University of Maryland, College Park

Abstract

Terminal agents rely on self-verification to assess and correct their solutions as they solve tasks through interaction with command-line environments. Yet how trustworthy such self-verification is remains poorly understood. To investigate this question systematically, we introduce a diagnostic framework that identifies the first complete solution in each trajectory, determines whether it is objectively correct, and uses this ground truth to quantify the agent's subsequent verification and recovery behavior. Applying it to ten terminal agents on TerminalBench2.1, we find that verification is nearly universal after a complete candidate is formed, yet only 61.43% of incorrect candidates are detected and only 49.36% of detected errors are successfully repaired. These results show that the main weakness in self-verification lies not in initiating verification, but in detecting and repairing errors. Motivated by these findings, we propose Student-Conditioned Verification Distillation (SCVD), which lets the student first produce a candidate solution and distills a stronger teacher's subsequent verification and recovery from the same interaction context. Across three Qwen3.5 backbones, SCVD improves \textsc{Pass@1} on TerminalBench2.1 by 9.74--16.85 percentage points over the corresponding base models and by 4.49--8.61 points over the standard full-trajectory distillation, while avoiding the pronounced out-of-distribution degradation of full-trajectory distillation on SWE-bench Verified.

Figures & tables

Figure 1: Overview of solution-level self-verification and SCVD. (a) After forming a candidate solution, the agent performs task-relevant checks, repairs detected errors, and rechecks the result before termination. (b) SCVD uses a teacher to continue from the student-generated candidate state to verify and revise the solution. After filtering successful trajectories and removing the temporary verification scaffold, the SFT loss is applied only to the teacher continuation.

Candidate correctness	$z=\text{error}$	$z=\text{no-error}$	$z=\text{unresolved}$
Incorrect ( $y_{c}=0$ )	Detected error	Missed error	No usable outcome
Correct ( $y_{c}=1$ )	False alarm	Correct pass	No usable outcome

Table 1: Verification outcomes and diagnostic metrics for solution-level self-verification. Here,

y_{c}

and

y_{f}

denote the correctness of the candidate and final solutions, respectively;

z

denotes the verification outcome; and

I_{c}

and

I_{v}

indicate the presence of a complete candidate and whether verification is attempted.

Figure 2: Self-verification diagnostics across ten terminal agents on TerminalBench2.1. Metric definitions are given in Table 1 . (a) Diagnostic results across models; gray bars show the range across models, vertical ticks mark the mean for each metric, and the rightmost numbers report corresponding mean values. (b) Improvement from initial candidates to final solutions. (c) Relationship between repair success rate and final task accuracy across models.

	TerminalBench2.1			SWE-bench Verified (OOD)
Model	Pass@1 $\uparrow$	Pass@3 $\uparrow$	$\Delta$	Pass@1 $\uparrow$	$\Delta$
GPT-5.5	78.65 $\pm$ 1.59	88.76	–	–	–
Opus-4.8	77.53 $\pm$ 2.43	88.76	–	–	–
DeepSeek-V4-Flash	77.15 $\pm$ 1.91	87.64	–	81.33 $\pm$ 0.50	–
GLM-5.2 (Teacher)	78.65 $\pm$ 1.84	86.52	–	83.33 $\pm$ 2.49	–
Qwen3.5-9B	26.59 $\pm$ 2.31	37.08	–	62.00 $\pm$ 0.28	–

Table 2: Main results on TerminalBench2.1 and the out-of-distribution SWE-bench Verified benchmark. Pass@1 is the mean

\pm

standard deviation over three runs, and

\Delta

denotes the absolute Pass@1 difference from Base.

Figure 3: Decomposition of final accuracy. Blue denotes initially correct candidates that remain correct, green denotes successful transitions from an incorrect candidate to a correct final solution, and red denotes harmful transitions from a correct candidate to an incorrect final solution.

Prefix generator	Pass@1 $\uparrow$
Qwen3.5-9B	33.33 $\pm$ 1.40
Qwen3.5-27B	31.46 $\pm$ 4.21
Qwen3.5-35B-A3B	31.83 $\pm$ 5.05
GLM-5.2 (teacher)	29.21 $\pm$ 2.42

Table 3: Effect of prefix source on Qwen3.5-9B.

Figure 4: Performance versus inference cost on TerminalBench2.1. Costs are averaged over common-success tasks within each backbone.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Candidate formed	Replay succeeded
Qwen3.5-9B	241/267 (90.26%)	233/240 (97.08%)
Qwen3.5-27B	263/267 (98.50%)	254/261 (97.32%)
Qwen3.5-35B-A3B	261/267 (97.75%)	250/259 (96.53%)
Qwen3.5-122B-A10B	261/267 (97.75%)	254/260 (97.69%)
Qwen3.6-27B	226/267 (84.64%)	220/225 (97.78%)
Qwen3.6-35B-A3B	256/267 (95.88%)	245/250 (98.00%)

Appendix

Table 4: Coverage of candidate formation and candidate-state replay on TerminalBench2.1. Entries report count/denominator (percentage), pooled across three independent runs. One Opus-4.8 trajectory was unavailable.

Model	Initial Acc.	Final Acc.	VTR	VOR	ESP	EDR	CAR	VPC	RSR
Qwen3.5-9B	12.02	26.60	99.59	99.58	98.69	69.75	92.59	29.75	23.65
Qwen3.5-27B	21.27	46.44	99.25	99.21	95.64	66.00	88.87	42.15	32.70
Qwen3.5-35B-A3B	19.93	38.20	99.23	99.18	95.97	71.53	87.41	43.85	25.35
Qwen3.5-122B-A10B	24.81	47.94	99.62	100.00	95.12	62.84	90.48	44.71	36.11
Qwen3.6-27B	32.01	56.93	99.59	99.54	92.91	63.44	90.43	54.07	54.95
Qwen3.6-35B-A3B	25.33	45.32	98.05	98.78	94.84	70.57	86.75	50.56	30.62

Appendix

Table 5: Per-model self-verification diagnostics on TerminalBench2.1. All values are percentages averaged over three independent runs. Initial Acc. (ICA) and Final Acc. denote accuracy at candidate formation and at the end of the trajectory, respectively; the remaining metrics are defined in Section 2.2 .

	TerminalBench2.1			SWE-bench Verified (OOD)
Model	Run 1	Run 2	Run 3	Run 1	Run 2	Run 3
GPT-5.5	80.90	77.53	77.53	–	–	–
Opus-4.8	80.90	76.40	75.28	–	–	–
DeepSeek-V4-Flash	75.28	76.40	79.78	81.20	80.80	82.00
GLM-5.2 (Teacher)	80.90	78.65	76.40	86.00	84.00	80.00
Qwen3.5-9B	23.60	26.97	29.21	61.60	62.20	62.20

Appendix

Table 6: Per-run Pass@1 results for the main experiments. Each column reports an independent evaluation run, and all values are percentages. Table 2 reports the corresponding aggregate statistics.

Backbone	Method	$N$	Pass@1	Turns	Tool calls	Input tok.	Generated tok.	Total tok.
9B	Base	27	$26.59\pm 2.31$	31.47	58.23	667.75	14.90	682.65
9B	FTD	27	$27.72\pm 3.71$	19.67	48.75	539.36	54.20	593.56
9B	SCVD	27	$36.33\pm 2.12$	20.28	48.27	468.79	61.01	529.80
27B	Base	43	$46.44\pm 0.53$	23.91	42.41	528.22	16.24	544.47
27B	FTD	43	$55.06\pm 4.00$	14.85	32.71	309.10	35.86	344.96
27B	SCVD	43	$63.30\pm 2.12$	17.93	38.67	396.61	42.15	438.77

Appendix

Table 7: Detailed performance and inference-cost statistics on TerminalBench2.1. Pass@1 is evaluated over all 89 tasks, whereas inference costs are measured on common-success tasks within each backbone.

Hyperparameter	Value
Epochs	3
Optimizer	AdamW
Learning rate	$1\times 10^{-5}$
LR scheduler	Cosine
Warmup ratio	0.03
Maximum sequence length	32,768

Appendix

Table 8: Supervised fine-tuning hyperparameters.

Explore similar work

Self-Trained Verification for Training- and Test-Time Self-Improvement
May 28, 2026Chen Henry Wu, Aditi RaghunathanRecursive Self-Improvement
Carnegie Mellon University
SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution
Jun 29, 2026Aojie Yuan, Yi Nian, Haiyue Zhang +2Facts
University of Southern California, Los Angeles, USA · University of Michigan, Ann Arbor, USA
Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents
Jul 27, 2026Diandian Guo, Cong Cao, Fangfang Yuan +3Self-Improving Agents Boltzmann Policies
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

May 28, 2026, cs.LGJ/K move · Enter open · S save

Self-Trained Verification for Training- and Test-Time Self-Improvement

Chen Henry Wu, Aditi Raghunathan

Carnegie Mellon University

Recursive Self-Improvement

Open paper AskPDF arXiv

Self-improvement at scale has been a longstanding goal for reasoning models, and there are two natural places to do it: at test time, through verification-refinement (V-R) loops; and at training time, through self-training methods. Both are gated by the same bottleneck: the verifier. V-R loops stall when verifier scores inflate while accuracy stagnates, and when feedback is too generic to act on; self-training fails similarly when bad self-generated data are added to training. Better verification would unlock both, but the capability we want to train, i.e., catching self-generated errors, lacks training signal. To address this challenge, we propose self-trained verification (STV). Our key observation is that, while a model cannot catch these errors alone, it can when shown the reference solution. We turn this asymmetry into a supervision target and train the verifier to imitate a more informed version of itself. At test time, STV substantially improves V-R loops on hard problems, while alternatives (e.g., SFT, RL on verifier scores, and even meta-verifiers) do not. STV roughly doubles accuracy on hard math and lifts it 14x on scientific reasoning tasks (1.5% to 21%). At training time, we additionally train the generator using RL with STV verifier's feedback inside the V-R loop - a procedure we call verifier-in-the-loop training (ViL). Starting from an RL-converged generator, ViL yields a further 33% gain in pass@1. More notably, the generator's standalone pass@1, with no verifier at test time, climbs 30% relative past where standard RL had converged. Hence, the next frontier in reasoning on hard problems may lie in how we train for and with verification. Website: https://ar-forum.github.io/stv-webpage

Abstract

Explore similar work

Self-Trained Verification for Training- and Test-Time Self-Improvement

SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution

Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents