TwinCheck: Evidence-Grounded Negative-Twin Verification for Stateful Tool Agents

Jiaxuan Dai; Tianyi Huang

cs.AISep 22, 2026

TwinCheck: Evidence-Grounded Negative-Twin Verification for Stateful Tool Agents

Authors: Jiaxuan Dai, Tianyi Huang

Organizations: Phillips Exeter Academy · Ryquo

Abstract

A single locally plausible tool call can derail an otherwise successful agent trajectory. Suspicion alone does not justify intervention, because the replacement itself can introduce the very failure verification is meant to prevent. We introduce TwinCheck, an inference-time verification policy that considers replacement only when the trace satisfies an evidence condition tied to a trace-local failure hypothesis. It constructs a trace-grounded counterfactual alternative, a negative twin, and replaces the agent's proposal only if the twin passes structural checks and the pairwise verifier prefers it in both candidate orders. For paired evaluation, exact replay holds the agent's parsed responses and actions fixed until the first accepted replacement, separating intervention effects from resampling. In the primary analysis of 159 multi-turn BFCL V4 tasks with complete exact-replay pairs, the complete policy raises task success for GPT-5.6 Sol from 45.3% to 58.5% (95% task-bootstrap CI [8.2, 18.8]), with no observed success-to-failure regressions. Together, these findings recast execution-boundary repair as a constrained comparison, making the counterfactual action itself the object of verification.

Figures & tables

Figure 1: Counterbalanced verification and conservative switching. For an actor–twin pair that passes the evidence and structural checks, TwinCheck evaluates the same candidates in both A/B orientations; reversal tests order sensitivity rather than supplying an independent vote. Green circles encode the 0–4 rubric scores. The twin is executed only when both judgments prefer it, mark the actor’s proposal as fatal and the twin as nonfatal, and meet the confidence and quality-margin requirements. Otherwise, the actor’s proposal is preserved. Actor/twin annotations are expository, and the verifier receives only A/B labels. Candidate descriptions and scores are schematic; task indices are omitted.

Test set	$n$	Actor (%)	TwinCheck (%)	$\Delta$	95% CI	Rescue / harm
Primary analysis	159	45.3	58.5	+13.2	[8.2, 18.8]	21 / 0
Completed-primary sensitivity	160	45.0	58.1	+13.1	[8.1, 18.8]	21 / 0
Conditional extension	160	48.1	58.1	+10.0	[5.6, 15.0]	16 / 0
Pooled	320	46.6	58.1	+11.6	[8.1, 15.3]	37 / 0

Table 1: Exact-replay task success. Differences are percentage points; intervals are 95% CIs from category-stratified paired bootstrap resampling of tasks.

Category	Actor (%)	TwinCheck (%)	$\Delta$	95% CI
Multi-turn base	55.0	65.0	+10.0	[3.8, 16.2]
Missing function	52.5	65.0	+12.5	[6.2, 20.0]
Missing parameter	36.2	45.0	+8.8	[3.8, 15.0]
Long context	42.5	57.5	+15.0	[7.5, 23.8]

Table 2: Pooled results by BFCL category (80 tasks each). Intervals use the same paired task-bootstrap procedure and condition on extension launch.

Figure 2: Failure ranking and frozen-gate selectivity on the completed-primary sensitivity (Primary), the conditional extension (Replication), and their pooled set. (a) ROC and (b) precision–recall curves. Pooled ROC-AUC is 0.759 (numeric-suffix-cluster 95% CI [0.686, 0.824]) and average precision is 0.829; the dotted line marks the 0.534 pooled failure prevalence. (c) The switch gate selects actor failures (green) and no actor successes (red) in these sets.

Method	Success (%)	$\Delta$	95% CI	Accepted	Rescue / harm
Actor	45.0	–	–	–	–
Selective self-revision	46.2	+1.2	[0.0, 3.1]	9	2 / 0
Selective absolute verifier	46.9	+1.9	[0.0, 4.4]	4	3 / 0
TwinCheck (full protocol)	58.1	+13.1	[8.1, 18.8]	39	21 / 0

Table 3: Trigger-matched controls on the completed-primary sensitivity. Intervals are 95% CIs from category-stratified paired bootstrap resampling of tasks.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Additional exact-replay outcomes. (a) Task-success gain over the actor for the completed-primary sensitivity (Primary) and conditional extension (Replication), separated by BFCL category. (b) Pooled transitions across 320 tasks.

Figure 4: Schematic lexical inverse-action construction. Following a successful lock_account call, the actor proposes unlock_account on the same account without a user-requested reversal. The fixed-table inverse match authorizes deletion of that call. Because the trace contains a successful operation and no unresolved later failure, the resulting deterministic twin is the fixed completion response. The phrase “semantic inverse” in the diagram denotes this prespecified lexical relation, not open-ended semantic inference.

Figure 5: Schematic repeat-after-error construction adapted from a retail interaction pattern. The first modification is attempted but fails, after which the actor repeats the same tool with identical arguments. Because no later successful current-turn tool record intervenes, the certificate authorizes structural change and the generator proposes an evidence-gathering call instead of reissuing the failed write. The queried product identifier represents a value established in trace context omitted for space; without such grounding, structural validation would reject the candidate.

Figure 6: Schematic unique-stale-argument construction adapted from a retail interaction pattern. An earlier lookup supplies card_4332 , but the latest user message uniquely requests the card ending in 8107 . Context elided from the diagram supplies the corresponding current identifier, card_8107 . The actor carries forward the stale identifier, whereas the deterministic twin replaces only payment_method_id and preserves the remainder of the proposed call.

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 148 papers

When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents
Sep 29, 2026Zifu Tao, Changqing YinAI Agent Reliability Tool-Use Evaluation
Tongji University
Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents
Apr 1, 2026Alibek Kaliyev, Artem MaryanskyyAI Agent Reliability Tool-Use Evaluation
The University of Texas at Austin · Austin, Texas, USA · Uber +1
AgentProofBench: Benchmarking Reusable Verifiers for Web Agents
Oct 25, 2026Jingjie Ning, Xueqi Li, Yi Lu +1LLM Agent Verification Web Agent Benchmarks

Sep 29, 2026J/K move · Enter open · S save

When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents

Zifu Tao, Changqing Yin

Tongji University

AI Agent Reliability Tool-Use Evaluation AI Agent Benchmarks Tool-Using Agents LLM Agent Reliability

Open paper AskPDF arXiv

Tool-augmented data agents rely on tool outputs for analytical decisions. Yet successful execution can return plausible but incorrect evidence, requiring agents to decide whether to trust or verify it. Understanding this failure requires examining both the evidence obtained through checking and the answer ultimately adopted. We introduce ToxicBench to measure checking and adoption under numerical, label, schema, and retrieval errors, pairing clean and poisoned observations over fixed source data. In the 118-task GPT evaluation across three adapters, poisoning lowers task success by 26 to 39 percentage points. Ordinary retries help under one-shot poisoning, whereas repeated poisoning reveals wrong-answer adoption after checking. Controls on three public tables isolate how supplied evidence affects recovery. After freezing the scorer, we compare its judgments with human annotations on 200 trajectories, finding 96% task-success agreement. Human judgments support retry gains over Base and confirm adoption after checking on audited tasks. We release trajectories, versioned scoring, and reference and delivery audits. These findings highlight evidence availability and answer selection as complementary dimensions of agent reliability.

Abstract

LLM Agent Verification

Explore similar work

When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents

Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents

AgentProofBench: Benchmarking Reusable Verifiers for Web Agents