Automatic Fine-grained Segmentation-assisted Report Generation

Frederic Jonske; Constantin Seibold; Osman Alperen Koras; Fin Bahnsen; Marie Bauer; Amin Dada; Hamza Kalisch; Anton Schily

cs.CVJul 22, 2025

Automatic Fine-grained Segmentation-assisted Report Generation

Authors: Frederic Jonske, Constantin Seibold, Osman Alperen Koras, Fin Bahnsen, Marie Bauer, Amin Dada, Hamza Kalisch, Anton Schily,

Organizations: Institute for AI in Medicine, University Medicine Essen Girardetstraße 2, 45131 Essen, Germany

Abstract

Reliable end-to-end clinical report generation has been a longstanding goal of medical ML research. The end goal for this process is to alleviate radiologists' workloads and provide second opinions to clinicians or patients. Thus, a necessary prerequisite for report generation models is a strong general performance and some type of innate grounding capability, to convince clinicians or patients of the veracity of the generated reports. In this paper, we present ASaRG (Automatic Segmentation-assisted Report Generation), an extension of the popular LLaVA architecture that aims to tackle both of these problems. ASaRG proposes to fuse intermediate features and fine-grained segmentation maps created by specialist radiological models into LLaVA's multi-modal projection layer via simple concatenation. With a small number of added parameters, our approach achieves a +0.89% performance gain ( $p=0.012$ ) in CE F1 score compared to the LLaVA baseline when using only intermediate features, and +2.77% performance gain ( $p<0.001$ ) when adding a combination of intermediate features and fine-grained segmentation maps. Compared with COMG and ORID, two other report generation methods that utilize segmentations, the performance gain amounts to 6.98% and 6.28% in F1 score, respectively. ASaRG is not mutually exclusive with other changes made to the LLaVA architecture, potentially allowing our method to be combined with other advances in the field. Finally, the use of an arbitrary number of segmentations as part of the input demonstrably allows tracing elements of the report to the corresponding segmentation maps and verifying the groundedness of assessments. Our code will be made publicly available at a later date.

Figures & tables

Figure 1: The ASaRG architecture - Model elements of ASaRG are highlighted in green and different input modalities are highlighted in blue. Plus symbols denote concatenation operations. Italics in any component denote that the component is part of original LLaVA architecture.

Method	BLEU-1	BLEU-4	METEOR	ROUGEL	CIDEr-D	CE (Pr)	CE (Rc)	CE (F1)
LLaVA (Baseline)	0.2126 ± 0.0012	0.0539 ± 0.0014	0.1668 ± 0.0014	0.1937 ± 0.0015	0.2573 ± 0.0082	0.4665 ± 0.0029	0.3179 ± 0.0056	0.3781 ± 0.0046
+Features, L. Mixing	0.2056 ± 0.0058	0.0504 ± 0.0028	0.1602 ± 0.0017	0.1881 ± 0.0024	0.2442 ± 0.0203	0.4375 ± 0.0117	0.2793 ± 0.0178	0.3409 ± 0.0168
+Features, Addition, $\alpha_{init}=0$	0.2134 ± 0.0010	0.0547 ± 0.0006	0.1661 ± 0.0006	0.1935 ± 0.0004	0.2583 ± 0.0052	0.4631 ± 0.0034	0.3117 ± 0.0012	0.3726 ± 0.0015
+Features, Addition, $\alpha_{init}=1$	0.2151 ± 0.0010	0.0541 ± 0.0007	0.1640 ± 0.0006	0.1925 ± 0.0011	0.2623 ± 0.0037	0.4656 ± 0.0021	0.3251 ± 0.0026	0.3828 ± 0.0020
+Features, Concat.	0.2195 ± 0.0019	0.0569 ± 0.0007	0.1664 ± 0.0007	0.1955 ± 0.0007	0.2798 ± 0.0064	0.4691 ± 0.0049	0.3291 ± 0.0013	0.3868 ± 0.0016

Table 1: Intermediate Feature Fusion

Method	BLEU-1	BLEU-4	METEOR	ROUGEL	CIDEr-D	CE (Pr)	CE (Rc)	CE (F1)
LLaVA, two-stage	0.2103 ± 0.0010	0.0518 ± 0.0010	0.1672 ± 0.0007	0.1910 ± 0.0006	0.2441 ± 0.0033	0.4762 ± 0.0054	0.3262 ± 0.0030	0.3872 ± 0.0037
LLaVA, fully ft’d	0.2278 ± 0.0013	0.0561 ± 0.0010	0.1622 ± 0.0009	0.1928 ± 0.0012	0.2902 ± 0.0089	0.4803 ± 0.0013	0.3508 ± 0.0008	0.4055 ± 0.0009
+Features, two-stage	0.2154 ± 0.0003	0.0544 ± 0.0002	0.1677 ± 0.0006	0.1917 ± 0.0004	0.2516 ± 0.0070	0.4752 ± 0.0031	0.3396 ± 0.0037	0.3961 ± 0.0036
+Features, +SegMaps, two-stage	0.2301 ± 0.0007	0.0607 ± 0.0003	0.1711 ± 0.0008	0.2009 ± 0.0008	0.2901 ± 0.0042	0.4903 ± 0.0035	0.3596 ± 0.0036	0.4149 ± 0.0034
+Features, +SegMaps, two-stage, SC-only	0.2298 ± 0.0012	0.0612 ± 0.0008	0.1716 ± 0.0010	0.2011 ± 0.0010	0.2944 ± 0.0104	0.4905 ± 0.0019	0.3594 ± 0.0019	0.4148 ± 0.0015
COMG † [ 10 ]	0.363	0.124	0.128	0.290		0.424	0.291	0.345

Table 2: Experiments with (fine-grained) Segmentation Maps

Method	BLEU-1	BLEU-4	METEOR	ROUGEL	CIDEr-D	CE (Pr)	CE (Rc)	CE (F1)
LLaVA, fully ft’d	0.2328 ± 0.0012	0.0530 ± 0.0005	0.1527 ± 0.0006	0.1841 ± 0.0004	0.2788 ± 0.0031	0.4544 ± 0.0026	0.3646 ± 0.0026	0.4046 ± 0.0025
+Features, two-stage	0.2414 ± 0.0005	0.0604 ± 0.0005	0.1594 ± 0.0007	0.1937 ± 0.0005	0.2982 ± 0.0021	0.4759 ± 0.0008	0.3802 ± 0.0020	0.4227 ± 0.0015
+Features, +SegMaps, two-stage	0.2518 ± 0.0015	0.0671 ± 0.0014	0.1654 ± 0.0010	0.2027 ± 0.0009	0.3337 ± 0.0111	0.4832 ± 0.0032	0.3981 ± 0.0026	0.4365 ± 0.0022

Table 3: Experiments with longer training time

Figure 2: Grounding with segmentation maps - This graphic depicts two examples from the MIMIC-CXR test set with ground truths and generated reports. The colors highlight parts of the generated report and their corresponding segmentation maps. Correspondences are derived via NER Tagging and limited to a few example classes and reports are truncated for readability.

Method	Num Workers	Training Time	max. GPU (VRAM) Overhead	max. RAM Overhead
LLaVA, fully FT’d (2 Epochs)	$4$	$\sim 14h$	$4x\sim 62.5GB$	$\sim 25.0GB$
ASaRG, two-stage, +F (1+1 Epochs)	$4$	$\sim 7.2h\newline \sim 5.2h$	$4\sim 62.6GB\newline 441.0-45.5GB$	$\sim 25.4GB$
ASaRG, +F (2 Ep.)	$4$	$\sim 14.5h$	$4*\sim 62.6GB$	$\sim 25.4GB$
ASaRG, two-stage, +F+S (1+1 Ep.)	$2$	$\sim 7.2h\newline \sim 14.5h$	$4\sim 62.6GB\newline 4\sim 54.3GB$	$\sim 523.0GB\newline \sim 431.5-460.0GB^{1}$
ASaRG, two-stage, +F+SCoS (1+1 Ep.)	$2$	$\sim 7.2h\newline \sim 14.7h^{2}$	$4*\sim 62.6GB\newline 4x\sim 47.0GB$	$\sim 523.0GB\newline \sim 431.5-460.0GB^{1}$

Table 4: Compute overhead comparison for tested settings +F denotes added image features, +S denotes added segmentation maps, +SCoS denotes added superclass-only segmentation maps. Peak RAM usage refers to training with MIMIC-CXR plus associated segmentation maps and will vary wildly depending on dataset size. Note that Linux will often autonomously cache data from the disk, which means startup or execution are sometimes randomly faster depending on the available system RAM which Linux can cache into and cache contents. 1 Peak RAM usage is significantly higher than consistent RAM usage. This peak occurs at the start of training, when the segmentation maps are cached, and is only experienced briefly. The second number denotes consistent RAM usage. 2 Note that when superclasses are aggregated as preprocessing, instead of during training, their RAM impact reduces by a factor of 10. Doing so, although not done during our experiments, allows efficiently training with at least 4 workers and no gradient accumulation again, improving the training time from 7+14.5 hours back to 7+7.5 hours. In this scenario, peak and consistent RAM usage would both decrease by a factor slightly lower than 10, as the segmentation maps make up the bulk of RAM usage. Alternatively, when crossing a break point of just above 1TB RAM, 4 workers can be used without extra preprocessing or other changes.

Source	ASaRG	LLaVA	Ground Truth
Avg. “Human-ness”	0.14	0.48	0.85

Table 5: Style assessment - This table shows average style rating across 100 samples, on a Likert scale from -2 (almost certainly AI-generated) to 2 (almost certainly human-made). Uncertainty on these averages is

\sigma\approx 0.1

if estimated as a counting error.

Panel of Figure 3 — Figure 3: User queries - The user queries $u_{i}^{F}\in P_{F}$ for prompting a VLM for radiology findings on an image $v_{i}$ , each paired with the findings $r_{i}^{F}$ as the response to form a single-turn chat.

Figure 4: Per-class performance - This series of plots shows the per-class performance of different variants of ASaRG that use additional intermediate features, as well as the accompanying baselines.

Figure 5: Per-class performance - This series of plots shows the per-class performance of different variants of ASaRG that use additional intermediate features and (fine-grained) segmentation maps, as well as the accompanying baselines.

Figure 6: Failure Mode: Prior Hallucinations - ASaRG and LLaVA, across variants, have a tendency to refer to prior reports. This behavior is likely learned because a significant fraction of ground truth reports reference prior reports. Whether part of MIMIC-CXR or not, such prior reports are not available in the style of one-report-one-image report generation ASaRG is trained on, and information presented in that context is typically also hallucinated.

Figure 7: Failure Mode: Length Hallucinations - ASaRG and LLaVA, across variants, report sizes or distances in centimeters. This typically happens in the presence of clearly visible ET tubes, but is observed for nodules and other objects as well. Generally, these numbers are hallucinated, likely even in examples where they are correct (such an example is depicted here), as ASaRG lacks a reference against which to measure lengths, even if it can make educated guesses based on learned human anatomy.

Figure 8: Failure Mode: Valid Reports - In some cases, ASaRG presents valid reports, which are different from the ground truth and then mischaracterized by metrics, proving that even CE metrics are not infallible. One such example is depicted here.

Figure 9: Failure Mode: Overconfidence - ASaRG and LLaVA are particularly prone to presenting findings as overlay certain. An example is again depicted here, where clinicians may prefer to be less explicit about hard facts, and instead frame findings as potential evidence. Note that neither overconfidence in generated reports or moderation on behalf of radiologists is absolute, and plentiful counterexamples exist.

Figure 10: Failure Mode: Under-reporting - ASaRG and LLaVA systematically under-report pathologies. This can result in false negatives, which are common, and partially false reports, where only one or several, but not all pathologies are correctly identified. This figure exhibits and example of an otherwise correct report missing a key diagnosis, atelectasis. Note that in some cases, MIMIC-CXR labels aren’t absolute truths and clinicians may disagree about diagnoses, meaning not every report missing one of several pathologies is necessarily false.

Figure 11: Failure Mode: Missing Views - Missing views, similar to missing prior reports, represent a systemic issue with training in a one-to-one fashion. In this showcased example, a lateral view is referenced as a source in the ground truth, but said lateral view is not available to the model to draw conclusions from. Training a model with multiple views or subsequent images in one data point has recently been shown as an effective mitigation strategy [ 3 ]

#	Class name	#	Class name
0	”spine”	36	”posterior 12th rib right”
1	”cervical spine”	37	”posterior 12th rib left”
2	”thoracic spine”	38	”anterior 11th rib right”
3	”lumbar spine”	39	”posterior 11th rib right”
4	”vertebrae C1”	40	”anterior 11th rib left”
5	”vertebrae C2”	41	”posterior 11th rib left”

Table 6: The ASaRG Segmentation Classes (1/3)

#	Class name	#	Class name
72	”anterior 3rd rib left”	108	”stomach”
73	”posterior 3rd rib left”	109	”small bowel”
74	”anterior 2nd rib right”	110	”duodenum”
75	”posterior 2nd rib right”	111	”liver”
76	”anterior 2nd rib left”	112	”pancreas”
77	”posterior 2nd rib left”	113	”kidney left”

Table 7: The ASaRG Segmentation Classes (2/3)

#	Class name	#	Class name
144	”right upper zone lung	180	”Consolidation”
145	”right mid zone lung	181	”Diffuse Nodule”
146	”right lung base”	182	”Effusion”
147	”right apical zone lung”	183	”Emphysema”
148	”left upper zone lung”	184	”Fibrosis”
149	”left mid zone lung”	185	”Fracture”

Table 8: The ASaRG Segmentation Classes (3/3)

Radiology Report Generation

Radiology report generation uses medical images and clinical context to produce structured, clinically accurate findings and impressions.

See all 159 papers

U-VLM: Hierarchical Vision Language Modeling for Report Generation
Feb 28, 2026Pengcheng Shi, Minghui Zhang, Kehan Song +3Radiology Report Generation Vision-Language Models
Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
CCS: Clinical Consensus Selection for Radiology Report Generation
May 28, 2026Xi Zhang, Yingshu Li, Zaiqiao Meng +2Radiology Report Generation Radiology VLMs
School of Computing Science, University of Glasgow · School of Electrical and Computer Engineering, University of Sydney · Language Technology Lab, University of Cambridge
CogRad: A Cognitively-Inspired Multi-Agent Framework for Radiology Report Generation
Jul 4, 2026Saif Ur Rehman Khan, Hasaan Maqsood, Sebastian Vollmer +2Radiology Report Generation Medical VLMs
Department of Computer Science, Rhineland-Palatinate Technical University of Kaiserslautern-Landau, Kaiserslautern, 67663, Germany. · 2German Research Center for Artificial Intelligence, Kaiserslautern, 67663, Germany. · 3Intelligentx GmbH (intelligentx.com), Kaiserslautern, Germany. +1

Feb 28, 2026J/K move · Enter open · S save

U-VLM: Hierarchical Vision Language Modeling for Report Generation

Pengcheng Shi, Minghui Zhang, Kehan Song, Jiaqi Liu+2

Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China

Radiology Report Generation Vision-Language Models CT Report Generation 3D VLMs

Open paper AskPDF arXiv

Automated radiology report generation is key for reducing radiologist workload and improving diagnostic consistency, yet generating accurate reports for 3D medical imaging remains challenging. Existing vision-language models face two limitations: they do not leverage segmentation-pretrained encoders, and they inject visual features only at the input layer of language models, losing multi-scale information. We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture: (1) progressive training from segmentation to classification to report generation, and (2) multi-layer visual injection that routes U-Net encoder features to corresponding language model layers. Each training stage can leverage different datasets without unified annotations. U-VLM achieves state-of-the-art performance on CT-RATE (F1: 0.414 vs 0.258, BLEU-mean: 0.349 vs 0.305) and AbdomenAtlas 3.0 (F1: 0.624 vs 0.518 for segmentation-based detection) using only a 0.1B decoder trained from scratch, demonstrating that well-designed vision encoder pretraining outweighs the benefits of 7B+ pre-trained language models. Ablation studies show that progressive pretraining significantly improves F1, while multi-layer injection improves BLEU-mean. Code is available at https://github.com/yinghemedical/U-VLM.

Abstract

Radiology Report Generation

Explore similar work

U-VLM: Hierarchical Vision Language Modeling for Report Generation

CCS: Clinical Consensus Selection for Radiology Report Generation

CogRad: A Cognitively-Inspired Multi-Agent Framework for Radiology Report Generation