Verification Turns Capability Gaps into Routing Decisions: Small Language Models for Clinical Documentation

Nina Fatehi; Meysam Ghaffari; Animesh Agarwal; Anushka Swarup; Carlos Morato

cs.AIOct 13, 2026

Verification Turns Capability Gaps into Routing Decisions: Small Language Models for Clinical Documentation

Authors: Nina Fatehi, Meysam Ghaffari, Animesh Agarwal, Anushka Swarup, Carlos Morato

Organizations: Optum

Abstract

Clinical note generation from doctor--patient dialogue is high-volume and hallucination-intolerant: a fabricated laboratory value can create patient-safety risk, not a quality regression. We ask whether an agentic verification architecture, rather than generator scale, can make a small language model viable here. Holding a four-evaluator verification stack and refinement policy fixed and swapping only the generator between a 4B-parameter model with a QLoRA adapter and a frontier baseline, the small model reaches $0.984$ claim-level factual entailment against $0.992$ and comes within $3.3$ points on completeness, generating entirely on-device. Its deficit is confined to inter-section structural fidelity---arrangement rather than clinical comprehension, and the axis on which it also falls furthest below the corpus's reference notes. Because that deficit is an explicit verifier decision rather than a silent quality difference, it becomes a routing policy: run the small model by default, escalate what the verifier rejects. Measured end to end, that it achieved a similiar verified-rate acceptance to the frontier-only acceptance in this test set ( $99.6\%$ vs.\ $98.4\%$ over $250$ encounters) while routing $62$ , documenting three quarters without a frontier generation call. Ablating the loop separates architecture from scale: unrefined, the small model is accepted on $26.8\%$ of encounters and the frontier model on $74.8\%$ , so verification does the most work where the generator is weakest. We also contribute section-scoped refinement, which preserves unflagged sections by construction rather than instruction, and find the baseline is not uniformly stronger---its provider refused two trauma-related transcripts the on-device model handled and certified, one facet of a provider dependency a served adapter does not carry.

Figures & tables

Figure 1: The pipeline: only the generator differs between conditions; evaluators, rubrics, and the refinement policy are shared verbatim.

	GPT-4.1	Gemma-3-4B + QLoRA		Reference
Metric	( $N{=}3$ )	( $N{=}3$ )	( $N{=}5$ )	notes
	$n{=}250$	$n{=}250$	$n{=}250$	$n{=}250$
Factual accuracy, clinical (mean)	0.992	0.984	0.986	0.989
Completeness (mean)	98.5%	95.2%	95.7%	82.5%
Structural fidelity, SFS (mean)	0.843	0.778	0.777	0.873
Critical hallucinations (total)	1	13	6	10

Table 1: Generator substitution under a fixed verification stack, full test split. Both SLM columns come from the same generations, so

N{=}3

vs.

N{=}5

isolates the round budget. All columns cover the same

250

encounters; GPT-4.1 produced notes for only

248

—its provider refused two transcripts (below)—so those two count as not accepted, while its quality means are over the

248

that exist. Latency is per round; per-note latency and generator token cost are resolved in Appendix C .

Gemma-3-4B + QLoRA (on-device)
	Accepted	ROUGE-1	ROUGE-L	BLEU
$N{=}1$ (no refinement)	26.8%	0.709	0.536	41.8
$N{=}2$	66.0%	0.695	0.511	36.7
$N{=}3$	75.2%	0.686	0.503	33.4
$N{=}4$	79.2%	0.688	0.504	34.5
$N{=}5$	83.2%	0.686	0.502	32.3

Table 2: Ablating the refinement loop for both generators over the same

250

encounters. Budgets are derived by truncating one run per generator, so within a generator only the cutoff varies;

N{=}1

is a single generation pass scored once by the gate. The frontier run had a ceiling of

N{=}3

, the SLM run

N{=}5

. Reference agreement is against the corpus note, which no evaluator consumes; the SLM’s higher values there follow from fine-tuning on this corpus and are not evidence of better writing.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Raising the round budget buys autonomy and safety on identical generations: autonomy rises $26.8\rightarrow 83.2\%$ while critical hallucinations fall $26\rightarrow 6$ ( $N{=}2\rightarrow 3$ is the one non-monotonic step), with gains not yet plateaued at $N{=}5$ . Every budget is derived from one $N{=}5$ run by truncation, so only the cutoff varies.

Condition	rnds	calls	in	out	calls	in	out	in	out
		Fresh call			Refinement call			Per note
GPT-4.1 $N{=}3$	1.28	248	1,415	547	70	2,658	603	2,166	717
Gemma $N{=}3$	2.07	250	1,450	344	281	2,744	466	4,534	867
Gemma $N{=}5$	2.53	250	1,450	347	382	2,810	517	5,743	1,138

Appendix

Table 3: Generator token cost by round type, means over the test split. “Fresh” is the round-

0

generation from the transcript; “refinement” is every subsequent round. Per-note columns sum over all of a note’s rounds. Each row is counted with its own model’s tokenizer, so counts are comparable down a column but not across the two generators as a single unit.

Metric	Kept ( $n{=}188$ )	Escalated ( $n{=}62$ )	Reference ( $n{=}250$ )
Factual accuracy, clinical	0.987	0.975	0.989
Completeness	96.5%	91.2%	82.5%
Structural fidelity, SFS	0.828	0.627	0.873
Critical hallucinations (total)	0	13	10
ROUGE-1 F1 vs. reference	0.700	0.646	—
ROUGE-L F1 vs. reference	0.518	0.456	—

Appendix

Table 4: What the verifier separates. Rows 1–4 are the axes the gate itself scores; rows 5–6 are lexical agreement with the corpus reference note, a signal no in-loop evaluator optimizes. ROUGE is undefined for the reference column, which is the comparison target. Fine-tuning on this corpus’s reference distribution inflates the SLM’s absolute ROUGE, but affects both kept and escalated notes identically, so the kept–escalated contrast—not the absolute value—is the claim.

	Frontier only	SLM + escalation
Encounters routed to the frontier model	250	62
Frontier generation requests	318	90
generation cost	$2.50	$0.73
Rounds executed	318	608
Judge calls	4,246	9,055
Judge input tokens	3.62 M	5.45 M

Appendix

Table 5: Cost over the

250

-encounter test split at public list rates: GPT-4.1 at

2/

8 per million input/output tokens, hosted judge at

3/

15. Generation is billed only in the frontier condition, since SLM generation is on-device; verification is billed in both. Requests exceed routed encounters because a note may be refined over several rounds: the

62

escalated encounters take the frontier model

1.45

rounds each on average, hence

90

generation requests.

Medical Report Generation

Medical report generation turns medical images, clinical records, or conversations into structured, clinically useful reports.

See all 82 papers

Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering
Sep 14, 2026Jiashuo Zhang, Yuling Chen, Yvonne Commodore-Mensah +1LLM Grounding LLM Answer Verification
Department of Computer Science Johns Hopkins University · School of Nursing Johns Hopkins University
Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution
Mar 5, 2026Qiao Jin, Yin Fang, Lauren He +12LLM Answer Verification Healthcare
Division of Intramural Research, National Library of Medicine, National Institutes of Health · Department of Computer Science, University of Virginia · Center for Cancer Research, National Cancer Institute, National Institutes of Health +3
VeriSim: A Configurable Framework for Stress-Testing Medical AI Under Patient Communication Noise
Apr 12, 2026Sina Mansouri, Mohit Marvania, Vibhavari Ashok Shihorkar +5Healthcare LLM Reliability
Department of Computer Science, George Mason University · Department of Health Administration and Policy, College of Public Health, George Mason University · Department of Science, George Mason University

Sep 14, 2026J/K move · Enter open · S save

Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering

Jiashuo Zhang, Yuling Chen, Yvonne Commodore-Mensah, Michael Oberst

Department of Computer Science Johns Hopkins University · School of Nursing Johns Hopkins University

LLM Grounding LLM Answer Verification Citation Verification Clinical QA Clinical Language Model Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) have been widely adopted for clinical question answering (QA). Current systems can attach citations to their answers, but these often point to broad texts, leaving time-pressed clinicians unable to verify them efficiently. An alternative is to ensure that responses are verifiable by construction: providing fine-grained verbatim quotes from reference material that substantiate claims, so users can verify an answer without opening other documents. In this paper, we evaluate the ability of current models to perform this task end-to-end: from providing citations for every factual claim, to producing verbatim quotes, to ensuring that those quotes fully substantiate the claims. To do so, we build a standardized harness over four clinical practice guidelines and evaluate twelve LLMs on 222 synthetic clinical questions, measuring each of these stages separately. We find that most models can attach verbatim quotes to over 90% of their claims from prompting alone, apart from some lightweight models such as claude-haiku-4.5. Yet these quotes often fail to substantiate every detail of the claims they accompany. For instance, claude-opus-5 produces verbatim quotes for 98.0% of its claims, but fully substantiates only 37.1%. Our work provides insights into the current capability gap of LLMs in building verifiable clinical QA systems, along with artifacts for future research.

Abstract

Medical Report Generation

Explore similar work

Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

VeriSim: A Configurable Framework for Stress-Testing Medical AI Under Patient Communication Noise