cs.AISep 22, 2026

TwinCheck: Evidence-Grounded Negative-Twin Verification for Stateful Tool Agents

Authors: Jiaxuan Dai, Tianyi Huang

Organizations: Phillips Exeter Academy · Ryquo

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A single locally plausible tool call can derail an otherwise successful agent trajectory. Suspicion alone does not justify intervention, because the replacement itself can introduce the very failure verification is meant to prevent. We introduce TwinCheck, an inference-time verification policy that considers replacement only when the trace satisfies an evidence condition tied to a trace-local failure hypothesis. It constructs a trace-grounded counterfactual alternative, a negative twin, and replaces the agent's proposal only if the twin passes structural checks and the pairwise verifier prefers it in both candidate orders. For paired evaluation, exact replay holds the agent's parsed responses and actions fixed until the first accepted replacement, separating intervention effects from resampling. In the primary analysis of 159 multi-turn BFCL V4 tasks with complete exact-replay pairs, the complete policy raises task success for GPT-5.6 Sol from 45.3% to 58.5% (95% task-bootstrap CI [8.2, 18.8]), with no observed success-to-failure regressions. Together, these findings recast execution-boundary repair as a constrained comparison, making the counterfactual action itself the object of verification.

Figures & tables

Figure 1: Counterbalanced verification and conservative switching. For an actor–twin pair that passes the evidence and structural checks, TwinCheck evaluates the same candidates in both A/B orientations; reversal tests order sensitivity rather than supplying an independent vote. Green circles encode the 0–4 rubric scores. The twin is executed only when both judgments prefer it, mark the actor’s proposal as fatal and the twin as nonfatal, and meet the confidence and quality-margin requirements. Otherwise, the actor’s proposal is preserved. Actor/twin annotations are expository, and the verifier receives only A/B labels. Candidate descriptions and scores are schematic; task indices are omitted.
Test setnnActor (%)TwinCheck (%)Δ\Delta95% CIRescue / harm
Primary analysis15945.358.5+13.2[8.2, 18.8]21 / 0
Completed-primary sensitivity16045.058.1+13.1[8.1, 18.8]21 / 0
Conditional extension16048.158.1+10.0[5.6, 15.0]16 / 0
Pooled32046.658.1+11.6[8.1, 15.3]37 / 0
Table 1: Exact-replay task success. Differences are percentage points; intervals are 95% CIs from category-stratified paired bootstrap resampling of tasks.
CategoryActor (%)TwinCheck (%)Δ\Delta95% CI
Multi-turn base55.065.0+10.0[3.8, 16.2]
Missing function52.565.0+12.5[6.2, 20.0]
Missing parameter36.245.0+8.8[3.8, 15.0]
Long context42.557.5+15.0[7.5, 23.8]
Table 2: Pooled results by BFCL category (80 tasks each). Intervals use the same paired task-bootstrap procedure and condition on extension launch.
Figure 2: Failure ranking and frozen-gate selectivity on the completed-primary sensitivity (Primary), the conditional extension (Replication), and their pooled set. (a) ROC and (b) precision–recall curves. Pooled ROC-AUC is 0.759 (numeric-suffix-cluster 95% CI [0.686, 0.824]) and average precision is 0.829; the dotted line marks the 0.534 pooled failure prevalence. (c) The switch gate selects actor failures (green) and no actor successes (red) in these sets.
MethodSuccess (%)Δ\Delta95% CIAcceptedRescue / harm
Actor45.0––––
Selective self-revision46.2+1.2[0.0, 3.1]92 / 0
Selective absolute verifier46.9+1.9[0.0, 4.4]43 / 0
TwinCheck (full protocol)58.1+13.1[8.1, 18.8]3921 / 0
Table 3: Trigger-matched controls on the completed-primary sensitivity. Intervals are 95% CIs from category-stratified paired bootstrap resampling of tasks.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Additional exact-replay outcomes. (a) Task-success gain over the actor for the completed-primary sensitivity (Primary) and conditional extension (Replication), separated by BFCL category. (b) Pooled transitions across 320 tasks.
Figure 4: Schematic lexical inverse-action construction. Following a successful lock_account call, the actor proposes unlock_account on the same account without a user-requested reversal. The fixed-table inverse match authorizes deletion of that call. Because the trace contains a successful operation and no unresolved later failure, the resulting deterministic twin is the fixed completion response. The phrase “semantic inverse” in the diagram denotes this prespecified lexical relation, not open-ended semantic inference.
Figure 5: Schematic repeat-after-error construction adapted from a retail interaction pattern. The first modification is attempted but fails, after which the actor repeats the same tool with identical arguments. Because no later successful current-turn tool record intervenes, the certificate authorizes structural change and the generator proposes an evidence-gathering call instead of reissuing the failed write. The queried product identifier represents a value established in trace context omitted for space; without such grounding, structural validation would reject the candidate.
Figure 6: Schematic unique-stale-argument construction adapted from a retail interaction pattern. An earlier lookup supplies card_4332 , but the latest user message uniquely requests the card ending in 8107 . Context elided from the diagram supplies the corresponding current identifier, card_8107 . The actor carries forward the stale identifier, whereas the deterministic twin replaces only payment_method_id and preserves the remainder of the proposed call.
More in this topic

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 148 papers

Explore similar work

CardsList
Sep 29, 2026

When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents

Tool-augmented data agents rely on tool outputs for analytical decisions. Yet successful execution can return plausible but incorrect evidence, requiring agents to decide whether to trust or verify it. Understanding this failure requires examining both the evidence obtained through checking and the answer ultimately adopted. We introduce ToxicBench to measure checking and adoption under numerical, label, schema, and retrieval errors, pairing clean and poisoned observations over fixed source data. In the 118-task GPT evaluation across three adapters, poisoning lowers task success by 26 to 39 percentage points. Ordinary retries help under one-shot poisoning, whereas repeated poisoning reveals wrong-answer adoption after checking. Controls on three public tables isolate how supplied evidence affects recovery. After freezing the scorer, we compare its judgments with human annotations on 200 trajectories, finding 96% task-success agreement. Human judgments support retry gains over Base and confirm adoption after checking on audited tasks. We release trajectories, versioned scoring, and reference and delivery audits. These findings highlight evidence availability and answer selection as complementary dimensions of agent reliability.

Zifu Tao, Changqing Yin

Tongji University

AI Agent ReliabilityTool-Use EvaluationAI Agent BenchmarksTool-Using AgentsLLM Agent Reliability
AskPDF
Apr 1, 2026

Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents

Agents that synthesize their own tools ship a second artifact alongside each answer: a software library that future tasks reuse, compose, and depend on. Task completion (TC) certifies the answer; it does not certify the library. On a Claude Haiku 4.5 pilot, we patch the harness to preserve per-tool source and replay every synthesised tool against a held-out conformance suite. Across 222 preserved tools and three protocols, 96.8% record per-tool correctness C=0.00: two protocols silent-rot at 100%, one at 91.7%. Hand-written reference implementations score C=1.00 on all 16 capability suites. The synthesised tools execute cleanly and the in-session verifier raises no flag; on held-out inputs they simply return wrong answers. This is the verification-vs.-conformance gap a pass-rate benchmark cannot see. EvolveTool-Bench makes that gap measurable. Sessions are structured into seed, gap, variant, composition, regression, and adversarial roles; runs emit per-tool manifests, verified-subset TC, correct-vs.-incorrect reuse, and audit traces for post-deployment monitoring. In the Haiku pilot over five protocols (3 seeds, 8 sessions, 51 verified decisions per pass), TC alone does not separate the protocols; the audit layer does. One pre-specified contrast shows the adapted decision-split synthesis protocol underperforming one-shot synthesis (-7.1 pp, 95% CI [-13.9,-0.1]; not significant after BH correction) -- a hypothesis-generating signal, not a confirmed effect. We release the harness, per-tool source preservation, conformance replay, and reproducibility manifests. An audit schema of this shape -- verifier coverage, reuse decomposition, held-out conformance -- is the minimum a deployed tool-evolving agent should emit between releases.

Alibek Kaliyev, Artem Maryanskyy

The University of Texas at Austin · Austin, Texas, USA · Uber +1

AI Agent ReliabilityTool-Use EvaluationAutomated Software TestingLLM-Based Program SynthesisLLM Agent VerificationAI Agent Benchmarks+1
AskPDF
Oct 25, 2026

AgentProofBench: Benchmarking Reusable Verifiers for Web Agents

Reusable Web-agent verification requires a rule to follow changing requirements and support subsequent execution. We introduce AgentProofBench, which connects paired tests of a frozen acceptance rule to execution from identical saved openings. The resource spans 97 tasks, 79 families, and four Web services. Experiments with DeepSeek Flash/Pro establish two linked measurements. Across 48 families, complete-panel and decision-replay tests retain 82/96 Flash and 79/96 Pro source programs; paired reuse-rebuild BA intervals are [-5.47, 1.57] and [-10.16, 0.00] percentage points. Across 22 execution families, deployment-state panel correctness separates program assignments whose two saved openings are both delivered correctly: 32/33 assignments with error-free panels, versus 1/11 with panel errors. All reference-defined program-gate regressions and withheld deliveries occur in the latter group. Neutral continuation attains the highest correct delivery under the tested generic feedback, providing a concrete baseline for measuring the value added by a gate. Released instructions, states, programs, reference rules, and trajectories connect each test failure to its execution consequences. AgentProofBench makes reusable verification assessable at both the requirement and policy levels.

Jingjie Ning, Xueqi Li, Yi Lu +1

LLM Agent VerificationWeb Agent BenchmarksAutomated Agent Evaluation
AskPDF