ChestPheNoT: Deployable, Auditable Label-Status-Evidence Extraction from Radiology Reports

Kai Yu; Chenyu Zhu; Zaifu Zhan; Meijia Song; Min Zeng; Xiaoyi Chen; Mingquan Lin; Rui Zhang

cs.CLSep 9, 2026

ChestPheNoT: Deployable, Auditable Label-Status-Evidence Extraction from Radiology Reports

Authors: Kai Yu, Chenyu Zhu, Zaifu Zhan, Meijia Song, Min Zeng, Xiaoyi Chen, Mingquan Lin, Rui Zhang

Organizations: University of California Davis Davis, USA

Abstract

Structured phenotype extraction from radiology reports supports cohort construction, quality auditing, and clinical analytics, but practical deployment requires local inference and auditable predictions, while expert annotations remain scarce. Conventional labelers provide structured findings and assertion states but no supporting evidence, while API-hosted large language models may be unsuitable when clinical text cannot leave institutional infrastructure. We present CHESTPHENOT, a compact 0.5-3B language model that jointly extracts finding labels, three-class status (present/absent/uncertain), and verbatim supporting evidence spans. CHESTPHENOT is trained using hybrid CheXbert+72B silver supervision followed by supervised fine-tuning and lightweight GRPO refinement. Across three human-annotated gold sets spanning in-distribution, cross-taxonomy, and cross-institution evaluation, the 3B model remains below its CheXbert silver teacher in distribution but is competitive under distribution shift, significantly surpassing CheXbert on cross-institution detection (+2.0 F1). Task-specific training also enables the 3B model to match or exceed substantially larger prompted models on most detection and status comparisons. For evidence-grounded extraction, over 99% of final evidence spans are locatable in the source report, and the 3B model achieves 47.5 auditable-F1, outperforming Qwen2.5-7B one-shot prompting by 7.6 points and approaching Qwen2.5-72B. These results demonstrate that locally deployable models can provide competitive and directly auditable radiology-report extraction without relying on external inference APIs. Code and the full extraction/judge prompts will be made available at https://github.com/yukkai/ChestPheNoT.

Figures & tables

Fig. 1: Overview of ChestPheNoT . Hybrid silver construction: the CheXbert labeler supplies finding labels and status (present/absent/uncertain) while a local 72B LLM proposes candidate evidence spans, with exact-match filtering retaining only verbatim report text, yielding {label, status, evidence} silver annotations. Training: a small Qwen2.5 model ( $0.5$ / $1.5$ / $3$ B) is supervised-fine-tuned on the silver data and refined with GRPO. Inference: the model runs on-premise with no external API call, producing structured label/status/evidence output.

	Present F1			Status F1
Model	M-685	M-350	N-200	M-685	M-350	N-200
ChestPheNoT
0.5B	82.7	79.9	78.1	88.0	67.0	54.6
1.5B	84.0	79.5	81.1	89.3	66.2	53.8
3B	84.2	81.3	84.6	89.8	67.5	58.7
Established Labelers (no evidence output):

TABLE I: Present-only detection F1 and three-class status macro-F1 (%) on mimic-685, mimic-350, and nih-200 (column headers M-685/M-350/N-200). ChestPheNoT rows are SFT

+

GRPO. Bold : best per column. ⋆ our silver label/status teacher. § same base model as a trained row.

Fig. 2: Paired

\Delta

(ChestPheNoT-3B

-

comparator) on present-only micro-F1 and status macro-F1, with

95\%

paired report-level bootstrap CIs;

\Delta>0

favors our model. Shaded band:

\pm 2

F1, for scale.

*

p<0.05

Fig. 3: Per-status precision–recall over gold-annotated label cells (present/absent/uncertain support

=1465/660/316

on M-685,

1015/941/419

on M-350,

439/354/280

on N-200). Dotted: iso-F1 contours. *our silver label/status teacher.

	Present F1			Status F1
Model	M-685	M-350	N-200	M-685	M-350	N-200
0.5B SFT	83.0	79.1	75.2	87.6	65.5	51.0
$+$ GRPO	82.7 ( $-$ 0.3)	79.9 ( $+$ 0.8)	78.1 ( $+$ 2.9)	88.0 ( $+$ 0.4)	67.0 ( $+$ 1.5)	54.6 ( $+$ 3.6)
1.5B SFT	83.7	79.5	79.6	88.5	66.2	55.1
$+$ GRPO	84.0 ( $+$ 0.3)	79.5 ( $0.0$ )	81.1 ( $+$ 1.5)	89.3 ( $+$ 0.8)	66.2 ( $0.0$ )	53.8 ( $-$ 1.3)
3B SFT	84.2	79.8	82.3	90.2	66.5	57.2

TABLE II: SFT vs. SFT

+

GRPO: present-only micro-F1 and status macro-F1 (%). The

+

GRPO rows equal the ChestPheNoT rows of Table I . Parentheses: change from SFT, positive in bold .

System	Rec	Prec	F1
Present (gold cells $=439$ ):
ChestPheNoT -3B	81.1	84.2	82.6
Qwen2.5-3B	38.7	75.9	51.3
Qwen2.5-7B	62.0	78.4	69.2
Qwen2.5-72B	77.7	84.6	81.0
Absent (gold cells $=354$ ):

TABLE III: Auditable-F1 on nih-200 (%). Rec : recall over gold cells; Prec : precision over the system’s own predictions. Bold

=

best in the deployable (

\leq

8B) tier; gray

=

ceiling reference.

Clinical Information Extraction

Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.

See all 106 papers

Vision-language models for chest radiography do not always need the image
Jun 16, 2026Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams +4VLM Evaluation Medical VQA
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Department of Diagnostic and Interventional Radiology, TUM University Clinic, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany · Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany +1
Ran Score: a LLM-based Evaluation Score for Radiology Report Generation
Mar 24, 2026Ran Zhang, Yucong Lin, Zhaoli Su +13Radiology Report Evaluation Radiology Report Generation
School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China · Zhengzhou Research Institute, Beijing Institute of Technology, Zhengzhou 450003, China · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China +7
Standardizing Longitudinal Radiology Report Evaluation via Large Language Model Annotation
Jan 23, 2026Xinyi Wang, Grazziela Figueredo, Ruizhe Li +1Medical Report Generation LLM-Assisted Annotation
School of Computer Science, The University of Nottingham, Nottingham, NG7 2RD, United Kingdom. · School of Medicine, The University of Nottingham, Nottingham, NG7 2RD, United Kingdom.

Jun 16, 2026J/K move · Enter open · S save

Vision-language models for chest radiography do not always need the image

Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams, Tri-Thien Nguyen+3

Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Department of Diagnostic and Interventional Radiology, TUM University Clinic, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany · Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany+1

VLM Evaluation Medical VQA Shortcut Learning

Open paper AskPDF arXiv

Vision-language models that answer questions about chest radiographs are evaluated by their accuracy on labels derived from radiology reports. High benchmark accuracy is often interpreted as evidence that the model uses the image. A model that answers from the finding named in the question can score as well as a model that uses the radiograph. Keeping the question fixed, we audit eight open-weight systems by swapping in another patient's radiograph with the same or the opposite label, occluding the radiologist-marked region or an equal region elsewhere, and removing the radiograph or replacing it with noise or a photograph. On 2,548 yes-or-no questions from MIMIC-CXR, one multimodal model answers Yes regardless of the image, another multimodal model changes its answers without following the label, and four systems use the image but keep about half of their correct answers when the radiograph is swapped for an opposite-label radiograph. A medical model that receives only the question text scores 55.3% on the pooled questions, higher than two multimodal systems. It scores 91.8% where every finding is present, and answering Yes to every question scores 100% there. Where the image is necessary, the best multimodal system exceeds this model by 10.4% in balanced accuracy. The categories are unchanged on CheXpert. Confidence is not higher when a correct answer depends on the marked region. In a reader study with three radiologists, the two radiologists who read a balanced set of 200 cases score 86.0% and 82.0%, and the systems score 50.0% to 73.0%. Accuracy does not establish image use, but an intervention on the image can test it.

Abstract

Clinical Information Extraction

Explore similar work

Vision-language models for chest radiography do not always need the image

Ran Score: a LLM-based Evaluation Score for Radiology Report Generation

Standardizing Longitudinal Radiology Report Evaluation via Large Language Model Annotation