When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs

Jason Liu

cs.SEAug 22, 2026

When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs

Authors: Jason Liu

Organizations: University of California San Diego

Abstract

Tool-using agents must decide when to stop. Existing systems already gate terminal success, certify execution traces, or enforce runtime polici es, but do not test this particular receipt-, scope-, and closed-replay design at the COMPLETE boundary across controlled termination faults. W e instantiate and evaluate Evidence-Carrying Termination (ECT): an agent may return COMPLETE only when a typed certificate binds every required answer claim to valid, in-scope trace evidence and a deterministic replay reconstructs the claimed value. A locked static study crosses 48 ful ly synthetic tasks in six tool-use families with clean execution and eight faults. ECT produced 0/288 unsafe completions versus 252/288 for the inspected termination-critic core (difference -87.50 pp, 95% task-cluster interval [-87.50, -87.50] pp). A fresh, prespecified and frozen 576- trajectory study then compares ECT with the critic core, its faithful controller, and a full-trace LLM critic. On 22 primary held-out task clus ters, ECT produced 0/66 premature unsupported terminations versus 40/66 for the controller (difference -60.61 pp, 95% interval [-78.79, -40.91] pp), while supported completion was 97/132 versus 92/132 (difference 3.79 pp, interval [0.00, 9.09] pp), satisfying a -10-point noninferiority margin. ECT executed successful recovery in 18/66 trajectories, of which 17 subsequently completed with support; all three closed-loop gates p assed. ECT certifies support in a recorded trace under declared assumptions, not external truth, safety, or alignment.

Figures & tables

Figure 1: ECT authorizes complete only when every verifier check passes. It certifies trace support and replayability under a trusted contract and evidence adapter, not external truth.

ID	Controlled intervention	Gold
C0	All required claims have valid in-scope support	Complete
F1	Completion signal without required claims	Continue
F2	Relevant receipt replaced by another entity	Continue
F3	Nested tool error beneath transport success	Continue
F4	Entity or date scope changed	Continue
F5	One required answer slot omitted	Continue

Table 1: The clean condition and eight faults applied to every base task. “Gold” is the permitted terminal decision at that snapshot.

Policy	U/288	C/36	S/288	F1
Completion	252	0	0	.222
Self-check	252	0	0	.222
Heuristic	252	0	0	.222
Critic core	252	0	36	.222
Full-trace critic	56	18	15	.595
Postcond. oracle	144	0	0	.500

Table 2: Held-out static evaluation. U: unsafe complete ; C: clean non-complete; S: stuck . Rows abbreviate the seven policies defined above. Counts are primary.

Check removed	Target fault	Effect/36
Claim coverage	Partial coverage	+36 U
Task/scope	Scope mismatch	+12 U
Outcome/validation	Nested error	+0 U
Ledger/reference	Forged reference	+0 U
Value replay	Synth. corruption	+36 U
Exact transform	No matched fault	–

Table 3: Prespecified single-check ablations. Residual checks and unmatched cases are retained. Effects are ablation-minus-ECT counts: U denotes unsafe complete and S denotes stuck . A zero count does not establish that a check is unnecessary.

Policy	U/66	S/132	FC/66	R/66	Turns	Calls
Current critic	47	84	0	3	1.02	0.02
Controller	40	92	0	12	1.09	0.09
Full-trace	18	64	21	19	2.34	0.14
ECT	0	97	0	18	1.99	0.14

Table 4: V2 primary-scope results. U is premature unsupported termination on incomplete checkpoints; S is supported completion over both conditions; FC is clean false continuation; R is successful recovery on incomplete checkpoints. Turns and Calls are per-trajectory means.

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 148 papers

Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions
Oct 5, 2026Chubin Zhang, Zhenglin Wan, Xingrui Yu +4LLM Agent Evaluation Tool-Using Agents
Nanyang Technological University, Singapore · National University of Singapore, Singapore · CFAR, Agency for Science, Technology and Research, Singapore +3
AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agents
Jul 1, 2026Laïla Elkoussy, Julien PerezLLM Agent Evaluation Formal Verification
LRE, EPITA · LRE · Bpifrance
AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Sep 27, 2026Tianzhuo Yang, Zirui Mi, Yantao Huang +4LLM Agent Evaluation LLM Agents
Peking University · Beijing Academy of Artificial Intelligence

Oct 5, 2026J/K move · Enter open · S save

Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions

Chubin Zhang, Zhenglin Wan, Xingrui Yu, Jingxuan Wu+3

Nanyang Technological University, Singapore · National University of Singapore, Singapore · CFAR, Agency for Science, Technology and Research, Singapore+3

LLM Agent Evaluation Tool-Using Agents LLM Agent Harnesses LLM Agent Reliability

Open paper AskPDF arXiv

An agent whose tool keeps returning nothing useful should stop relying on it. In a retrieval environment with controlled source failures, we separate how agents judge results from what they do. We compare stopping at the same step after longer and shorter runs of results the agent judged useless; this contrast is zero for clock- or deadline-driven stopping. Where we record their judgments, the seven agents we test call a failing source's results useless 97-100% of the time, yet most of them rarely stop on that judgment. Prompt cues change when they stop but not what they stop on. Permission to answer from memory and a reasoning mode can bring early stops regardless of evidence, a stated budget moves the 7-8B models' stops to the deadline, and a stopping rule or call cost in the prompt is followed at most partly. Stopping follows the evidence only when the harness enforces an integration step that makes the agent answer after five consecutive results it judged useless. This step raises failing-source success for every model, keeps the stopping point fixed when the budget doubles, and needs no extra judgment call when the agent states its judgments. A pre-registered replication on 300 fresh questions confirms the dissociation and the rule's effect.

Abstract

LLM Agent Verification

Explore similar work

Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions

AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agents

AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents