Trajectory-Derived Confidence for Reliable, Resource-Aware Clinical Text-to-SQL Agents

Mincheol Daniel Song; Joshua Ward; Jake Jung; Guang Cheng

cs.CLOct 2, 2026

Trajectory-Derived Confidence for Reliable, Resource-Aware Clinical Text-to-SQL Agents

Authors: Mincheol Daniel Song, Joshua Ward, Jake Jung, Guang Cheng

Organizations: University of California, Los Angeles

Abstract

LLM agents for clinical text-to-SQL applications reason autonomously over multiple steps but cannot assess whether their own reasoning or outputs can be trusted. In high leverage applications such as healthcare, this presents a critical risk where system mistakes can be costly. These reliability failures are also resource failures: an incorrect reasoning trajectory spends computation budget on outputs that must be discarded. We introduce Sentinel, a trajectory-derived, classifier-based confidence layer that analyzes an agent's reasoning, code and database outputs to decide at three points whether to stop: refusing unanswerable questions before the agent runs, halting doomed trajectories mid-run, and withholding untrustworthy answers at delivery. Here, utilizing Chow's rule, we optimize decisions under the EHRSQL shared task's Reliability Score, which penalizes incorrect answers given a utility weighting, and find on the benchmark EHRSQL that Sentinel raises this score from +0.08 to +0.24 when mistakes have a low utility weighting, well above the +0.03 earned by refusing every question, with delivered-answer accuracy rising from 54% to 69% as coverage falls from 84% to 55%. At higher stakes the agents we test rarely answer reliably enough to deliver, and Sentinel detects this on its own, abstaining to that same +0.03 where the unmonitored agent scores -3.02. The same stopping decisions cut computation: at low stakes, where the system still answers, Sentinel eliminates 13-28% of agent steps at little or no reliability cost.

Figures & tables

Figure 1: Diagram of failure modes Q1: on an unanswerable question the vanilla ReAct agent queries an unrelated admission field twice and commits it as an answer. Q2: on a question scoped to the current encounter, it pools every past admission and delivers a wrong value as fact. Bottom panel: the answerability head (purple) refuses Q1 before the agent runs; the correctness head (blue) withholds Q2’s wrong answer at delivery; each panel’s Compute row prices the steps spent or spared. Gauges show actual out-of-fold monitor estimates against thresholds

\theta

and

\tau

Figure 2: The Sentinel pipeline. A frozen ReAct agent appends each reasoning/SQL/result step to a serialized transcript. Two ModernBERT heads score every prefix, including

t{=}0

, so an unanswerable question can be refused before the agent runs a single step. Answerability drives the exit gate (

\theta

); calibrated correctness drives early stopping (

s_{\lambda}

) and Chow’s rule at delivery (

\tau

	Vanilla	Monitor AUROC		Vanilla $\mathrm{RS}(\lambda)$			Full System $\mathrm{RS}(\lambda)$
Agent	Acc.	Corr.	Ans.	$1$	$4$	$9$	$1$	$4$	$9$
GPT-4o-mini	$48\%$	$0.787$	$0.923$	$-0.02$	$-1.34$	$-3.53$	$+0.21^{\dagger}$	$+0.08$	$+0.04$
GPT-4o	$54\%$	$0.762$	$0.981$	$+0.08$	$-1.08$	$-3.02$	$+0.24^{\dagger}$	$+0.02$	$+0.03$
Llama-3.1-8B	$31\%$	$0.707$	$0.903$	$-0.23$	$-1.61$	$-3.91$	$+0.06$	$+0.03$	$+0.03$
Qwen2.5-Coder-7B	$22\%$	$0.920$	$0.959$	$-0.46$	$-2.43$	$-5.72$	$+0.10^{\dagger}$	$+0.03$	$+0.03$
Phi-4-mini	$8.7\%$	$0.921$	$0.987$	$-0.46$	$-2.03$	$-4.64$	$+0.04^{\dagger}$	$+0.03$	$+0.03$

Table 1: Cross-agent reliability. Vanilla acc. = accuracy of the answers the raw agent delivers on the held-out pool (answers to unanswerable questions count as wrong); monitor quality = correctness / answerability head AUROC on the same pool. Always abstain refuses every question and scores the unanswerable fraction at any

\lambda

\dagger

: gain over always abstaining whose 95% bootstrap interval excludes zero (Appendix F ). Per-agent early- exit results are in Table 8 .

System	Steps Saved	Gen. Tokens Saved	Total Time Saved
	( $\lambda{=}1/4/9$ )	( $\lambda{=}1/4/9$ )	( $\lambda{=}1/4/9$ )
+ exit gate / + Chow delivery	$1\%$	$1\%$	$3$ min
+ early exit	$\mathbf{13\%\,/\,42\%\,/\,42\%}$	$16\%\,/\,47\%\,/\,47\%$	$29$ min / $1.5$ h / $1.5$ h

Table 2: Resource savings ( GPT-4o ). Agent steps, generated tokens, and wall-clock time never spent. The monitor’s two ModernBERT-base passes per step are not subtracted. Token accounting in Appendix G .

Figure 3: Behavior of the confidence layer. (a) The full system as a single withhold classifier, with dots marking Chow’s rule at

\tau_{\lambda}=0.5,0.8,0.9

. (b) AUROC of each head scored on the prefix at step

t

; later points cover only the longer runs still going. (c) Reliability diagram of the correctness head before and after the isotonic map fit on the calibration pool.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	$\lambda{=}1$		$\lambda{=}4$		$\lambda{=}9$
System	Cov.	Acc.	Cov.	Acc.	Cov.	Acc.
vanilla agent	$0.84$	$0.48$	$0.84$	$0.48$	$0.84$	$0.48$
+ Chow delivery	$0.38$	$0.73$	$0.17$	$0.86$	$0.03$	$0.95$
+ early stopping	$0.37$	$0.73$	$0.09$	$0.86$	$0.01$	$0.89$

Appendix

Table 3: Operating characteristics ( GPT-4o-mini , dev_test): the fraction of questions each system answers (cov.) and the accuracy of the answers it delivers (acc.), at each stakes level. The GPT-4o equivalents appear in the main text (Section 4.2 ).

Agent	$\lambda{=}1$	$\lambda{=}4$	$\lambda{=}9$
GPT-4o-mini	$+0.177\ [+0.088,+0.270]$	$+0.048\ [-0.017,+0.130]$	$+0.015\ [-0.014,+0.040]$
GPT-4o	$+0.209\ [+0.113,+0.306]$	$-0.010\ [-0.058,+0.034]$	$+0.001\ [+0.000,+0.004]$
Llama-3.1-8B	$+0.029\ [-0.016,+0.081]$	$+0.000\ [+0.000,+0.002]$	$+0.000\ [+0.000,+0.001]$
Qwen2.5-Coder-7B	$+0.075\ [+0.022,+0.135]$	$+0.001\ [+0.000,+0.003]$	$+0.001\ [+0.000,+0.003]$
Phi-4-mini	$+0.010\ [+0.002,+0.021]$	$+0.000\ [+0.000,+0.000]$	$+0.000\ [+0.000,+0.000]$

Appendix

Table 4: Full-system

\mathrm{RS}(\lambda)

minus the always-abstain score (dev_test), with 95% template-cluster bootstrap intervals.

	Correctness		Answerability
Agent	$t{=}0$	Delivery	$t{=}0$	Delivery
GPT-4o-mini	$0.699$	$0.787$	$0.910$	$0.923$
GPT-4o	$0.705$	$0.762$	$0.988$	$0.981$
Llama-3.1-8B	$0.663$	$0.707$	$0.962$	$0.903$
Qwen2.5-Coder-7B	$0.742$	$0.920$	$0.961$	$0.959$
Phi-4-mini	$0.782$	$0.921$	$0.968$	$0.987$

Appendix

Table 5: Held-out AUROC of each head on the question alone (

t{=}0

) and at delivery.

Agent	lp_mean	lp_min	ent_mean	ent_max	empty_rate	error_rate	n_steps	failed_refs	bad_joins	ungrounded_frac	Survivors
GPT-4o-mini	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	✓	$\times$	$\times$	1
GPT-4o	✓	✓	✓	✓	✓	$\times$	✓	$\times$	✓	$\times$	7
Llama-3.1-8B	$\times$	$\times$	$\times$	$\times$	✓	✓	✓	✓	✓	$\times$	5
Qwen2.5-Coder-7B	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	0
Phi-4-mini	$\times$	$\times$	$\times$	$\times$	$\times$	✓	$\times$	✓	$\times$	$\times$	2

Appendix

Table 6: Which features individually survive the screen, per agent (train pool, out-of-fold; ✓ = incremental

\Delta

AUROC over BERT-alone with 95% cluster-bootstrap CI excluding zero; not a held-out result). Survivor counts are highest for the two agents the head reads worst, though five agents cannot establish a trend; only aggregate rulings are agent-general.

System	$\mathrm{RS}(1)$	$\mathrm{RS}(4)$	$\mathrm{RS}(9)$	Steps Saved ( $\lambda{=}1/4/9$ )
vanilla agent	$-0.02$	$-1.34$	$-3.53$	—
+ exit gate	$-0.01$	$-1.29$	$-3.43$	$2\%$
+ Chow delivery	$+0.21$	$+0.08$	$+0.04$	$2\%$
+ early stopping	$+0.20$	$+0.06$	$+0.03$	$28\%\,/\,64\%\,/\,70\%$

Appendix

Table 7: System ladder for GPT-4o-mini (dev_test, 2,100 questions, evaluated once): each decision component’s reliability and compute contribution.

Figure 4: The Figure 3 panels for GPT-4o-mini (dev_test; withhold-classifier AUC

0.80

Figure 5: The Figure 3 panels for Qwen2.5-Coder-7B (dev_test; withhold-classifier AUC

0.92

Figure 6: The Figure 3 panels for Llama-3.1-8B (dev_test; withhold-classifier AUC

0.72

Figure 7: The Figure 3 panels for Phi-4-mini (dev_test; withhold-classifier AUC

0.93

	Early-Stop $\mathrm{RS}(\lambda)$			Steps Saved
Agent	$1$	$4$	$9$	$1$	$4$	$9$
GPT-4o-mini	$+0.20$	$+0.06$	$+0.03$	$28\%$	$64\%$	$70\%$
GPT-4o	$+0.24$	$+0.02$	$+0.03$	$13\%$	$42\%$	$42\%$
Llama-3.1-8B	$+0.06$	$+0.03$	$+0.03$	$14\%$	$53\%$	$53\%$
Qwen2.5-Coder-7B	$+0.10$	$+0.03$	$+0.03$	$22\%$	$62\%$	$62\%$
Phi-4-mini	$+0.04$	$+0.03$	$+0.03$	$15\%$	$32\%$	$32\%$

Appendix

Table 8: Early-stop mode per agent (dev_test):

\mathrm{RS}(\lambda)

with sequential stopping, and the fraction of agent steps never executed.

Agent	Fabrication	Correct Refusal	Typical Answerable
GPT-4o-mini	$3.29$	$2.12$	$2.86$
GPT-4o	$3.35$	$2.12$	$3.01$
Llama-3.1-8B	$5.26$	$4.46$	$4.97$
Qwen2.5-Coder-7B	$2.25$	$1.68$	$1.96$
Phi-4-mini	$1.75$	$1.70$	$3.36$

Appendix

Table 9: Step economics of unanswerable questions per agent (train pool, mean steps): fabricating an answer vs. correctly refusing (an explicit abstention; runs that hit the 8-step cap are excluded) vs. a typical answerable question. Four of five agents work harder on the fabrication than on the correct refusal; Phi-4-mini is the exception, fabricating as fast as it refuses.

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 826 papers

Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents
Oct 6, 2026Brendan King, Farima Fatahi Bayat, Jean-Flavien Bussotti +2Probability Calibration Confidence Estimation in Language Models
University of California, Santa Cruz · Megagon Labs
SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing
Jul 15, 2026Tianyu Chen, Chujia Hu, Wenjie WangLanguage Model Safety Evaluation LLM Guardrails
ShanghaiTech University, Shanghai, China
Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline
Aug 10, 2026Morris LeeLLM Answer Verification LLM Agent Evaluation
Independent Researcher

Oct 6, 2026J/K move · Enter open · S save

Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents

Brendan King, Farima Fatahi Bayat, Jean-Flavien Bussotti, Pouya Pezeshkpour+1

University of California, Santa Cruz · Megagon Labs

Probability Calibration Confidence Estimation in Language Models LLM Agent Evaluation LLM Agent Reliability

Open paper AskPDF arXiv

When using an LLM agent in a consequential domain, making an informed decision about whether to trust its output or intervene requires calibrated confidence in the agent's success. Confidence estimation for agents is difficult because evidence about success is distributed across heterogeneous, interdependent steps of an agent's trajectory. Practical agentic deployments introduce further challenges: frontier LLMs often provide limited access to internal signals, agent roll-outs are costly, and training data may be unavailable or quickly become outdated. To address these challenges, we introduce Confidence Reasoning Graphs (CRGs), an inference-time framework that estimates the probability an agent accomplished its task from a single trajectory, without privileged model access or training data. Rather than compressing an execution into a single holistic judgment, a CRG begins with the claim that the agent accomplished its task, decomposes it into contextualized sub-claims grounded in trajectory evidence, estimates confidence for each terminal claim, and finally aggregates these into an overall confidence estimate. Across three agentic benchmarks, three backbone models, and three agent frameworks, CRGs yield better-calibrated confidence and stronger risk-aware decision making than verbalized, sampling-based, and white-box surrogate baselines. We further find that calibration error alone can be misleading: a white-box surrogate baseline appears well calibrated while providing near-chance discrimination. Ablations attribute CRG's improvements to claim-level confidence estimation and aggregation rather than graph construction alone. Finally, a CRG exposes the claims and trajectory evidence underlying each confidence estimate, enabling it to be audited at decision time.

Abstract

LLM Agent Reliability

Explore similar work

Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents

SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing

Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline