Autonomous Structuring of Radiology Reports Across Modalities at Archive Scale Using an Open-Weight Large Language Model
Authors: Friedrich Puttkammer, Fabian Drexel, Marlene Fritzsche, Era Stambollxhiu, Miriam Kumpf, Lena Schmitzer, Lea Schumann, Lina Xu, +14 more
Organizations: Department of Radiology, Charité – Berlin University of Medicine, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany · Department of Diagnostic and Interventional Radiology, School of Medicine and Health, TUM Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany · Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany · Department of Medicine, Surgery, and Dentistry, University of Salerno, Salerno, Italy · Lab of Medical Physics & Digital Innovation, School of Medicine, Aristotle University of Thessaloniki, Thessaloniki, Greece · Department of Computer Science and Media, Berlin University of Applied Sciences and Technology, Berlin, Germany · Department of Neuroradiology, Charité – Berlin University of Medicine, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany · Berlin Institute of Health (BIH), BIH Biomedical Innovation Academy, Berlin, Germany · Chair for Computer Aided Medical Procedures & Augmented Reality, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany · Institute for Artificial Intelligence in Medicine (IKIM), University Hospital Essen (AöR), Essen, Germany · Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany · National Center for Tumor Diseases (NCT), NCT West, a partnership between DKFZ and University Hospital Essen, Essen, Germany
Purpose: To develop and evaluate an open-weight large language model (LLM) pipeline that converts an entire archive of free-text radiology reports into structured reports without human oversight. Materials and Methods: In this retrospective study, a pipeline with 150 hierarchically organized templates was developed at one center and tested at a second center on reports from 2010 to 2025. The open-weight model gpt-oss-120B selects the template in three constrained-decoding steps and fills it on one local graphics processing unit. Template selection was scored against expert labels on 914 randomly sampled reports of five modalities, structuring quality on 920 radiography and CT reports corrected field by field by five residents. The pipeline then processed the complete archive of the second center. Proportions are reported with Wilson 95% confidence intervals (CIs). Results: An optimal template set was selected for 74.4% of reports (680 of 914; 95% CI: 71.5%, 77.1%) and an appropriate set for 82.3% (752 of 914; 95% CI: 79.7%, 84.6%), 87.7% for single-region and 54.1% for multi-region reports. Macro semantic textual similarity between output and corrected reference was 0.95 for radiography and 0.97 for CT, residents left 88.7% of 24,638 fields unchanged, and unsupported content was flagged in 1.0% and 1.5% of reports. Of 2,186,982 archive reports, 96.5% received structured output, 2,401,544 structured reports, at 1,258 reports per hour on one graphics processing unit. Conclusion: An open-weight LLM pipeline structured a complete multimodality report archive without human oversight with high content fidelity. Multi-region reports remained the main source of template errors.
Figures & tables
Figure 1: Creation of the two test sets from the TUM archive. A, selection test set. Of a random sample of the archive, reports without a report identifier were excluded before labeling, reports combining two modalities and non-structurable reports (report stubs, examination without a template) were labeled but not scored, the non-structurable reports served as the rejection test. B, structuring test set. Of a stratified sample of the pipeline output with 9 to 25 reports per template, the residents rated the selected template, reports rated inappropriate or unclear were not corrected, the unclear ratings were adjudicated by two board-certified radiologists in consensus for the template rating. Numbers in parentheses are radiography, CT.
Figure 2: Pipeline for autonomous structuring of free-text radiology reports. In the template selection stage the large language model answers three questions in constrained-decoding steps, imaging technique (modality), body region (examination type) and diagnosis or question (template), each narrowing the options of the next. Reports describing several examinations continue with multi-label selection of examination types and one template per examination (teal). Report stubs and examinations without a template are rejected (dashed grey). The template database supplies the options and the template files (dashed teal). The selected templates are composed from base and derived template and filled with the report content in the structuring stage, one call per template.
Modality
Templates (base, derived, independent)
Examination types
Selection test reports (single, multi)
Reader study reviewed (appropriate, corrected)
Radiography
25
(6,
17,
6
602
(568,
622
(544,
CT
45
(10,
30,
10
217
(110,
448
(417,
MRI
43
(6,
24,
11
59
(52,
—
Ultrasound
30
(0,
0,
6
26
(26,
—
Interventional radiology
7
(2,
4,
4
10
(10,
—
Total
150
(24,
75,
37
914
(766,
1,070
(961,
Table 1: Template library and test datasets
Group
n
Structured, % (95% CI)
Optimal, % (95% CI)
Appropriate, % (95% CI)
Optimal given structured, %
All
914
95.1 (93.5, 96.3)
74.4 (71.5, 77.1)
82.3 (79.7, 84.6)
78.3
Single-region
766
95.2 (93.4, 96.5)
83.0 (80.2, 85.5)
87.7 (85.2, 89.9)
87.2
Multi-region
148
94.6 (89.7, 97.2)
29.7 (23.0, 37.5)
54.1 (46.0, 61.9)
31.4
Radiography
602
97.0 (95.3, 98.1)
80.9 (77.6, 83.8)
85.9 (82.9, 88.4)
83.4
CT
217
97.7 (94.7, 99.0)
60.4 (53.7, 66.6)
77.4 (71.4, 82.5)
61.8
MRI
59
81.4 (69.6, 89.3)
69.5 (56.9, 79.7)
78.0 (65.9, 86.6)
85.4
Table 2: Template selection on the 914 reports of the selection test set
Figure 3: Template selection on 914 eligible external test reports. A, outcome of template selection per report as percentage of eligible reports, for all reports, by number of described regions and by modality, with report counts inside the segments and n at the bar end. B, level of the hierarchy at which the first error occurred, as percentage of the group, with report counts above the bars. IR, interventional radiology.
Modality
Reports
Templates
STS (95% CI)
BLEU, all fields (95% CI)
BLEU, content fields (95% CI)
Identical reports, n (%)
Radiography
513
25
0.95 (0.94, 0.95)
84.0 (83.3, 84.8)
74.2 (72.9, 75.4)
39 (7.6)
CT
407
45
0.97 (0.97, 0.98)
91.9 (91.2, 92.6)
83.9 (82.5, 85.3)
104 (25.6)
Table 3: Structuring quality against the resident-corrected reference
Figure 4: Structuring quality against the resident-corrected reference for 513 radiography and 407 CT reports. A, STS per report (points) and macro mean with 95% CI (open circle with line) by examination type, report counts in grey. B, macro BLEU over all fields (filled) and over fields with content in output or reference (open) with 95% CI. C, share of the 24,638 scored fields the radiologist left unchanged (correct) or edited, with field counts inside the bars. STS, semantic textual similarity. BLEU, bilingual evaluation understudy.
Figure 5: Template ratings, hallucination flags and error types in the reader study. A, rating of the selected template for 622 radiography and 448 CT reports after adjudication of the unclear ratings, with report counts inside the segments. B, STS of the 513 and 407 corrected reports as box plots (median, interquartile range, 1.5 IQR whiskers, outliers as points) with the reports flagged for unsupported content marked as triangles. C, primary error type in the 50 lowest-STS reports per modality, with report counts inside the segments where they fit (hallucination 1 and unclassified 2 in radiography, unclassified 2 in CT).
Figure 6: Structuring of the complete TUM report archive. A, flow from source rows to structured reports. B, structured reports per template, area proportional to count, color by modality, templates labeled where the tile holds the name, with counts in thousands where space allows (all templates in Table S6 ). C, number of templates per source report by modality. IR, interventional radiology.
Modality
Reports with structured output
Structured reports
Templates in use
Largest template (share of structured reports)
Radiography
1,330,875
1,354,491
25
chest radiograph (36.5%)
CT
518,815
780,857
45
CT head (13.6%)
MRI
165,659
169,097
38
MRI breast (17.9%)
Ultrasound
90,407
91,325
27
breast (47.2%)
Interventional radiology
5,774
5,774
8
image-guided interventions (82.4%)
Total
2,111,530
2,401,544
142
—
Table 4: Structured archive by modality
Modality
Examination type
Reports
Templates
STS (95% CI)
BLEU, all fields (95% CI)
BLEU, content fields (95% CI)
Identical reports, n (%)
Radiography
All
513
25
0.95 (0.94, 0.95)
84.0 (83.3, 84.8)
74.2 (72.9, 75.4)
39 (7.6)
Radiography
Abdomen
23
1
0.95 (0.93, 0.97)
83.1 (78.0, 87.8)
79.5 (73.4, 84.8)
2 (8.7)
Radiography
Chest
43
2
0.95 (0.94, 0.96)
86.0 (83.2, 88.7)
68.9 (64.0, 74.0)
2 (4.7)
Radiography
Fluoroscopy
114
5
0.96 (0.96, 0.97)
88.9 (87.5, 90.3)
79.3 (76.3, 82.0)
18 (15.8)
Radiography
Head
91
4
0.97 (0.96, 0.97)
87.6 (86.0, 89.1)
75.2 (72.2, 78.2)
10 (11.0)
Radiography
Mammography
15
1
0.94 (0.93, 0.96)
83.7 (79.3, 88.0)
79.4 (74.1, 84.5)
0 (0.0)
Table S1: Structuring quality by examination type
Modality
Labeled template
Test reports
Structured
Optimal
Appropriate
Optimal, %
Radiography
Chest radiograph
215
214
209
209
97.2
Radiography
Extremity radiograph, trauma
120
118
109
115
90.8
Radiography
Extremity radiograph, postoperative
93
89
67
68
72.0
Radiography
Extremity radiograph
29
28
11
23
37.9
Radiography
Pelvis radiograph
22
22
21
22
95.5
Radiography
Extremity radiograph, osteoarthritis
21
21
13
14
61.9
Table S2: Template selection per labeled template
Modality
Labeled template
Selected template(s)
Reports
Radiography
Extremity radiograph, postoperative
Pelvis radiograph
10
Radiography
Extremity radiograph, postoperative
Pelvis radiograph, trauma
6
Radiography
Extremity radiograph, osteoarthritis
Pelvis radiograph
5
Radiography
Chest radiograph
US lung
4
Radiography
Extremity radiograph
Pelvis radiograph
2
Radiography
Extremity radiograph, osteoarthritis
MRI knee
2
Table S3: Template confusions among single-region reports
Reports
n
Optimal, pipeline
Optimal, modality supplied
Gains
Losses
Difference, pp (95% CI)
P
All
914
678
697
51
32
+2.1 (0.1, 4.0)
.05
Single-region
766
634
654
41
21
+2.6 (0.6, 4.6)
.02
Multi-region
148
44
43
10
11
-0.7 (-6.7, 5.4)
> .99
Table S4: Ablation, labeled modality supplied to the pipeline
Reasoning effort
n
Optimal, low effort
Optimal, at setting
Gains
Losses
Difference, pp (95% CI)
P
Medium
111
45
47
8
6
+1.8 (-4.8, 8.4)
.79
High
111
45
41
7
11
-3.6 (-11.1, 3.9)
.48
Table S5: Ablation, reasoning effort
Modality
Template
Structured reports
Share within modality, %
Radiography
Chest radiograph
493,844
36.5
Radiography
Extremity radiograph, trauma
227,864
16.8
Radiography
Extremity radiograph, postoperative
183,504
13.5
Radiography
Pelvis radiograph
83,596
6.2
Radiography
Extremity radiograph, osteoarthritis
56,869
4.2
Radiography
Pelvis radiograph, trauma
53,686
4.0
Table S6: Templates in use in the structured archive
Figure S1: Template index of the CT branch, the largest of the five modality branches (template counts per modality in Table 1 , templates in archive use in Table S6 ). Ten examination types with 45 templates, five of which are listed under two or three examination types and appear under each. Base templates (navy outline) carry derived templates (light teal outline) that add or replace sections at run time, independent templates (black outline) have no base. Templates derived from a base template of another examination type are attached with dashed connectors. Three of the 31 combination types are shown as examples.
Figure S2: Hierarchical template selection as a tree. The modality step chooses one of 5 modalities, which restricts the examination-type step to the examination types of that modality (10 for CT), which restricts the template step to the templates of that examination type (6 for CT chest). Teal branches show the multi-label pathways for reports describing several examinations and for CT combination types. Dashed grey exits show the two reserved rejection responses.
Figure S3: Template composition at run time with the YAML files of the base template CT chest standard and the derived template CT chest mass. The production templates are German, keys and values are translated one to one for display and the placeholder hints of the production files are shown as empty strings. The pipeline updates the base template recursively with the derived template, a leaf value in the derived template replaces the base value (sand), sections present only in the derived template are added (teal), all other sections remain from the base (black). The composed template becomes the Pydantic model for constrained decoding.
Item
Topic
Reported
Location in manuscript
Notes
1
Identification as AI study
Yes
Title, Abstract
Title and abstract name the open-weight large language model pipeline.
2
Structured summary
Yes
Abstract
Retrospective design, data source (two centers), 914 and 920 test reports, five modalities, archive of 2,186,982 reports, metrics with 95% CIs.
3
Scientific and clinical background
Yes
Introduction, paragraphs 1 to 3
—
4
Study objectives and hypotheses
Yes
Introduction, last paragraph
Objective stated, no hypothesis (retrospective study), not registered.
5
Prospective or retrospective
Yes
Methods, Study Design
Retrospective.
6
Study goal and AI task
Yes
Introduction, last paragraph; Methods, Pipeline
Hierarchical classification of the template and zero-shot structuring of report text, intended as autonomous post hoc structuring of archived reports without human oversight.