SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learning

Halil Ibrahim Gulluk; Olivier Gevaert

cs.LGApr 10, 2026

SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learning

Authors: Halil Ibrahim Gulluk, Olivier Gevaert

Organizations: Department of Biomedical Data Science, Stanford University

Abstract

Medical vision-language datasets are often limited in size and biased toward negative findings, as clinicians report abnormalities mostly but might omit some positive/neutral findings because they might be considered as irrelevant to the patient's condition. We propose a self-supervised data enrichment method that leverages semantic clustering of report sentences. Then we enrich the findings in the medical reports in the training set by adding positive/neutral observations from different clusters in a self-supervised manner. Our approach yields consistent gains in supervised fine-tuning (5.63%, 3.04%, 7.40%, 5.30%, 7.47% average gains on COMET score, Bert score, Sentence Bleu, CheXbert-F1 and RadGraph-F1 scores respectively). Ablation studies confirm that improvements stem from semantic clustering rather than random augmentation. Furthermore, we introduce a way to incorporate semantic cluster information into the reward design for GRPO training, which leads to further performance gains (2.78%, 3.14%, 12.80% average gains on COMET score, Bert score and Sentence Bleu scores respectively). We share our code at https://github.com/igulluk/SemEnrich

Figures & tables

Panel of Figure 1 — Figure 1: SemEnrich pipeline. Phase 1 (top): every finding sentence in the training set is embedded by a frozen sentence transformer (MiniLM) and clustered with HDBSCAN. Each cluster is labeled normal or abnormal by an LLM, and cluster co-occurrence across patients defines a connectivity graph. Phase 2 (bottom): for every patient $(v_{j},F_{j})$ we sample a valid enrichment —a set of normal clusters $F_{cand}$ connected to every cluster already in $F_{j}$ —and append one sentence per added cluster to the findings. The enriched findings and the chest X-ray then train an LVLM via SFT or GRPO; in the GRPO setting the same clustering is reused as a cluster-F1 reward $\mathcal{R}_{cl}$ .

Figure 2: An example chest X-ray from the ReXGradient-160K dataset. Findings: The lungs are adequately inflated. There is no focal infiltrate. There is no pleural effusion. No pulmonary parenchymal nodules or masses are observed. The heart and pulmonary vascularity are normal. There is calcification in the wall of the thoracic aorta. The bony thorax exhibits no acute abnormality. Impression: No pulmonary nodules are observed. There is no acute cardiopulmonary abnormality. Thoracic aortic atherosclerosis.

Panel of Figure 3 — Figure 3: Visualization of the cluster-based findings representation. Each node represents a cluster of semantically similar sentences. Patient 1’s findings (consisting of multiple sentences) forms a complete subgraph where each node corresponds to the cluster containing one of the finding sentences. The blue subgraph shows an example findings with 4 sentences: the bold text shows the original sentence, and italicized text shows other sentences from the same cluster. The green and orange subgraphs represent findings from other patients.

Panel of Figure 4 — Figure 4: Illustration of the valid enrichment process. Left: the base findings $F_{j}=\{c_{1},c_{2},c_{3},c_{4}\}$ (inside the blue box) and candidate clusters (dashed nodes). Green nodes are normal clusters; red nodes are abnormal clusters. Candidate $c_{5}$ is valid (normal and connected to all base clusters). Candidate $c_{6}$ is invalid (abnormal). Candidate $c_{7}$ is invalid (not connected to $c_{1}$ and $c_{3}$ ). Candidates $c_{8}$ and $c_{9}$ form a valid pair (both normal, connected to all base clusters and to each other). Note that $c_{5}$ is not connected to $\{c_{8},c_{9}\}$ , so they cannot belong to the same enrichment. Right: two largest valid enrichments, $F_{j}\cup\{c_{5}\}$ and $F_{j}\cup\{c_{8},c_{9}\}$ . $F_{j}\cup\{c_{8}\}$ is a valid enrichment but not a largest one.

Figure 5: Model architecture. The chest X-ray is encoded by a Vision Transformer and mapped by a linear projector into $m$ image tokens $I_{1:m}$ . These are concatenated with the tokenized findings $y_{1:n}$ and passed to a LoRA-tuned large language model, trained with the next-token prediction loss on the finding tokens.

Metric	Method	DS-R1-Qwen-1.5B	Gemma3-4B	DS-R1-Llama-8B	Qwen3-8B	Mistral-7B	Mean
Sentence-BLEU-1	Baseline	27.58 $\pm$ 0.87	21.63 $\pm$ 0.70	27.45 $\pm$ 0.86	28.11 $\pm$ 1.17	29.87 $\pm$ 1.67	26.93
	K-means-1000	26.40 $\pm$ 0.66	22.80 $\pm$ 0.74	26.85 $\pm$ 1.00	26.88 $\pm$ 0.71	23.12 $\pm$ 0.52	25.21
	K-means-2000	28.74 $\pm$ 0.97	22.25 $\pm$ 0.70	29.09 $\pm$ 0.77	29.36 $\pm$ 0.94	29.58 $\pm$ 0.69	27.80
	K-means-5000	28.16 $\pm$ 0.62	23.32 $\pm$ 0.32	29.38 $\pm$ 0.61	29.40 $\pm$ 0.79	32.53 $\pm$ 1.15	28.56
	DBSCAN	27.03 $\pm$ 1.31	21.36 $\pm$ 0.83	28.44 $\pm$ 0.69	28.02 $\pm$ 0.56	28.89 $\pm$ 1.00	26.75
	HDBSCAN	29.51 $\pm$ 0.81	23.83 $\pm$ 0.91	29.11 $\pm$ 0.54	30.35 $\pm$ 1.35	32.64 $\pm$ 1.00	29.09

Table 1: Main results across five LLM backbones (mean

\pm

std over five seeds). Each block is one metric (in percent); the best method per backbone is bolded and the method with the best mean is shaded. Additional results are in Appendix A.12 .

Metric	Method	DS-R1-Qwen-1.5B	Gemma3-4B	DS-R1-Llama-8B	Qwen3-8B	Mistral-7B	Mean
COMET	Baseline	63.68 $\pm$ 0.49	62.05 $\pm$ 0.53	63.95 $\pm$ 0.48	63.80 $\pm$ 0.80	63.59 $\pm$ 0.77	63.41
	Random expansion	63.97 $\pm$ 0.48	61.97 $\pm$ 0.41	64.53 $\pm$ 0.57	64.35 $\pm$ 0.28	61.53 $\pm$ 0.47	63.27
	HDBSCAN	67.34 $\pm$ 0.41	64.59 $\pm$ 0.65	67.44 $\pm$ 0.31	67.71 $\pm$ 0.73	69.16 $\pm$ 0.57	67.25
BERTScore-F1	Baseline	67.60 $\pm$ 0.48	65.35 $\pm$ 0.31	67.83 $\pm$ 0.44	67.79 $\pm$ 0.67	67.86 $\pm$ 0.75	67.29
	Random expansion	67.06 $\pm$ 0.55	65.43 $\pm$ 0.31	67.91 $\pm$ 0.55	67.62 $\pm$ 0.11	66.50 $\pm$ 0.57	66.91
	HDBSCAN	69.66 $\pm$ 0.42	66.80 $\pm$ 0.60	69.80 $\pm$ 0.34	70.07 $\pm$ 0.69	71.35 $\pm$ 0.56	69.54

Table 2: Ablation: semantic (HDBSCAN) enrichment vs. random expansion (mean

\pm

std over five seeds, percent). The best method per backbone is bolded and the method with the best mean is shaded. Sentence-BLEU-1 and CheXbert-5 results are in Appendix A.12 .

Metric	Model	$\mathcal{R}_{cl}$	$\mathcal{R}_{exact}$
COMET	Qwen2.5-VL-3B	0.6179 $\pm$ 0.003	0.5973 $\pm$ 0.002
COMET	Qwen3-VL-4B	0.6125 $\pm$ 0.003	0.5989 $\pm$ 0.003
BERTScore-F1	Qwen2.5-VL-3B	0.7211 $\pm$ 0.002	0.6952 $\pm$ 0.002
BERTScore-F1	Qwen3-VL-4B	0.7193 $\pm$ 0.003	0.7000 $\pm$ 0.002
ROUGE-L	Qwen2.5-VL-3B	0.3204 $\pm$ 0.004	0.2792 $\pm$ 0.004
ROUGE-L	Qwen3-VL-4B	0.3203 $\pm$ 0.005	0.2818 $\pm$ 0.005

Table 3: GRPO results comparing cluster-based reward

\mathcal{R}_{cl}

and exact match reward

\mathcal{R}_{exact}

on the impression section. Additional results are in Appendix A.14 .

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Value
Total Sentences	66,477
Total Clusters	3,216
Average Cluster Size	20.67
Median Cluster Size	11.0
Min Cluster Size	5
Max Cluster Size	1,522

Appendix

Table 4: HDBSCAN clustering statistics for finding sentences.

Panel of Figure 6 — Figure 6: HDBSCAN clustering statistics: (a) Log histogram of cluster sizes, (b) Top clusters by size, (c) Percentage distribution of clusters.

Statistic	Value
Total Sentences	184,535
Total Clusters	5,000
Average Cluster Size	36.91
Median Cluster Size	31.0
Min Cluster Size	1
Max Cluster Size	296

Appendix

Table 5: K-means-5000 clustering statistics for finding sentences.

Panel of Figure 7 — Figure 7: K-means-5000 clustering statistics: (a) Log histogram of cluster sizes, (b) Top clusters by size, (c) Percentage distribution of clusters.

Model	COMET	BERTScore	S-BLEU-1	RadGraph
Med-Flamingo	51.18	62.63	6.79	10.47
LLaVA-Med	61.31	58.17	8.80	4.44
CXR-LLaVA	67.40	63.56	17.60	13.63
CheXagent	60.81	63.46	17.66	14.83
LLaVA-Rad	65.75	65.72	18.57	15.75
Mistral-7B (Baseline)	63.59 $\pm$ 0.77	67.86 $\pm$ 0.75	29.87 $\pm$ 1.67	30.21 $\pm$ 1.72

Appendix

Table 6: Comparison with open-source state-of-the-art medical VLMs on the ReXGradient-160K validation split (

2{,}500

reports). SOTA rows are single inference runs produced by each model’s released checkpoint; Mistral-7B rows are mean

\pm

std over five training seeds (taken from Table 1 ). RadGraph-F1 is the complete -match variant for all rows. Best value per metric is bolded. All scores are in percent.

Metric	Baseline	HDBSCAN (ours)	$\Delta$
Sentence-BLEU-1	23.88 $\pm$ 0.91	30.35 $\pm$ 0.18	$+6.47$
METEOR	27.04 $\pm$ 0.99	30.17 $\pm$ 0.32	$+3.13$
CheXbert-14	52.70 $\pm$ 0.44	55.80 $\pm$ 2.24	$+3.10$
RadGraph-F1	22.69 $\pm$ 1.17	25.28 $\pm$ 0.72	$+2.59$

Appendix

Table 7: Generalization of SemEnrich to MIMIC-CXR (

20

K training reports). Mistral-7B backbone, mean

\pm

std over four seeds; all scores in percent.

\Delta

is HDBSCAN minus Baseline.

Model	S-BLEU-1	METEOR	CheXbert-14	RadGraph
Med-Flamingo Moor et al. (2023)	13.05	13.62	54.00	18.59
LLaVA-Med Li et al. (2023a)	17.99	14.54	54.00	7.68
CXR-LLaVA Lee et al. (2025)	30.30	28.10	32.75	17.50
CheXagent Chen et al. (2024)	35.69	32.31	58.00	22.35
LLaVA-Rad Chaves et al. (2024)	37.84	33.64	55.25	19.35
Ours (Baseline)	23.88	27.04	52.70	22.69

Appendix

Table 8: Comparison with open-source medical VLMs on MIMIC-CXR. SOTA rows are single inference runs from released checkpoints; our rows are the Mistral-7B means from Table 7 . CheXagent and LLaVA-Rad were trained on MIMIC-CXR. Best value per metric is bolded; all scores in percent.

Encoder (dim)	S-BLEU-1	METEOR	CheXbert-14	RadGraph	Clust./Assig.
Baseline	23.88	27.04	52.70	22.69	—
MiniLM (384)	30.35	30.17	55.80	25.28	836 / 16,961
Stella (1024)	35.70	33.41	54.40	23.28	534 / 12,772
GTE-Qwen2 (1536)	33.76	31.97	52.90	22.05	576 / 11,399
PubMedBERT (768)	36.59	35.88	52.80	24.44	536 / 6,855
BioBERT (768)	24.75	27.70	54.10	23.13	628 / 8,179

Appendix

Table 9: Sentence-encoder ablation with the pipeline held fixed (Mistral-7B, MIMIC-CXR). “Clust./Assig.” is the number of clusters and the number of sentences assigned to a cluster. Best value per metric is bolded; scores in percent.

Recovered normal finding	% of reports
normal heart size	22.0
no osseous abnormalities	21.8
no pleural effusion or pneumothorax	19.6
no pneumothorax	11.0
normal pulmonary vasculature	8.2

Appendix

Table 10: Most frequently recovered normal findings on MIMIC-CXR, as a percentage of training reports into which they are inserted.

Statistic	%
Mean purity	97.9
Median purity	100.0
Clusters $100\%$ pure	92.5
Clusters $\geq 80\%$ pure	97.0
Insertion-weighted purity	99.78
Abnormal-injection rate	0.22

Appendix

Table 11: Polarity purity of the

133

inserted normal clusters (MIMIC-CXR). Insertion-weighted purity weights each cluster by how often it is inserted during training; the abnormal-injection rate is its complement.

Model	Baseline	DBSCAN	HDBSCAN	K-means-1000	K-means-2000	K-means-5000
DS-R1-Qwen-1.5B	17.26 $\pm$ 4.85	18.37 $\pm$ 2.99	21.19 $\pm$ 5.09	23.54 $\pm$ 4.17	23.23 $\pm$ 4.84	21.65 $\pm$ 4.93
Gemma3-4B	17.73 $\pm$ 1.85	9.34 $\pm$ 2.77	20.06 $\pm$ 2.05	17.17 $\pm$ 4.32	11.07 $\pm$ 2.18	12.88 $\pm$ 1.16
DS-R1-Llama-8B	31.90 $\pm$ 3.44	23.15 $\pm$ 2.27	24.93 $\pm$ 4.47	29.70 $\pm$ 2.37	26.30 $\pm$ 3.89	27.51 $\pm$ 5.28
Qwen3-8B	21.97 $\pm$ 5.39	22.48 $\pm$ 4.14	25.02 $\pm$ 3.51	26.22 $\pm$ 3.12	21.78 $\pm$ 4.80	25.77 $\pm$ 4.69
Mistral-7B	15.89 $\pm$ 4.30	17.01 $\pm$ 4.22	19.13 $\pm$ 5.79	24.82 $\pm$ 3.76	16.03 $\pm$ 3.01	21.67 $\pm$ 2.87
Mean	20.95	18.07	22.06	24.29	19.68	21.90

Appendix

Table 12: CheXbert-5 F1 scores comparing baseline with clustering methods (deferred from the main results).

Model	Baseline	DBSCAN	HDBSCAN	K-means-1000	K-means-2000	K-means-5000
DS-R1-Qwen-1.5B	26.36 $\pm$ 0.98	26.97 $\pm$ 1.70	30.13 $\pm$ 1.18	29.90 $\pm$ 1.27	30.94 $\pm$ 1.53	28.63 $\pm$ 1.20
Gemma3-4B	20.48 $\pm$ 0.64	20.70 $\pm$ 0.98	23.52 $\pm$ 1.16	25.68 $\pm$ 0.95	23.18 $\pm$ 0.64	23.04 $\pm$ 0.08
DS-R1-Llama-8B	26.26 $\pm$ 1.12	28.10 $\pm$ 1.21	29.72 $\pm$ 0.79	30.87 $\pm$ 1.43	30.51 $\pm$ 1.05	30.46 $\pm$ 1.27
Qwen3-8B	27.21 $\pm$ 1.43	27.57 $\pm$ 0.89	31.21 $\pm$ 1.97	29.92 $\pm$ 1.37	31.57 $\pm$ 1.36	30.38 $\pm$ 0.86
Mistral-7B	29.42 $\pm$ 1.83	27.87 $\pm$ 1.52	33.51 $\pm$ 1.66	30.67 $\pm$ 1.09	33.13 $\pm$ 1.41	33.78 $\pm$ 1.63
Mean	25.95	26.24	29.62	29.41	29.87	29.26

Appendix

Table 13: METEOR scores comparing baseline with clustering methods

Model	Baseline	DBSCAN	HDBSCAN	K-means-1000	K-means-2000	K-means-5000
DS-R1-Qwen-1.5B	35.39 $\pm$ 0.81	36.90 $\pm$ 1.48	39.40 $\pm$ 1.04	40.43 $\pm$ 0.94	40.88 $\pm$ 1.41	38.72 $\pm$ 1.03
Gemma3-4B	31.61 $\pm$ 0.66	32.40 $\pm$ 0.71	34.50 $\pm$ 1.15	37.03 $\pm$ 0.75	34.91 $\pm$ 0.57	34.34 $\pm$ 0.09
DS-R1-Llama-8B	35.33 $\pm$ 0.88	38.13 $\pm$ 1.07	38.99 $\pm$ 0.75	41.42 $\pm$ 1.22	40.83 $\pm$ 1.01	40.03 $\pm$ 1.18
Qwen3-8B	35.97 $\pm$ 1.43	37.66 $\pm$ 0.67	39.99 $\pm$ 1.60	40.69 $\pm$ 0.91	41.26 $\pm$ 1.15	39.85 $\pm$ 0.81
Mistral-7B	37.48 $\pm$ 1.65	38.85 $\pm$ 1.41	42.67 $\pm$ 1.42	41.83 $\pm$ 0.77	43.23 $\pm$ 1.32	43.25 $\pm$ 1.50
Mean	35.16	36.79	39.11	40.28	40.22	39.24

Appendix

Table 14: ChrF++ scores comparing baseline with clustering methods

Model	Baseline	DBSCAN	HDBSCAN	K-means-1000	K-means-2000	K-means-5000
DS-R1-Qwen-1.5B	25.59 $\pm$ 0.95	29.53 $\pm$ 1.42	32.44 $\pm$ 0.96	27.42 $\pm$ 0.67	31.05 $\pm$ 0.65	31.98 $\pm$ 0.86
Gemma3-4B	23.67 $\pm$ 0.69	24.77 $\pm$ 1.05	27.64 $\pm$ 0.95	23.19 $\pm$ 0.71	23.72 $\pm$ 0.74	26.93 $\pm$ 0.49
DS-R1-Llama-8B	26.73 $\pm$ 1.17	31.19 $\pm$ 1.30	32.02 $\pm$ 0.88	28.06 $\pm$ 1.19	31.86 $\pm$ 0.89	33.17 $\pm$ 0.91
Qwen3-8B	26.14 $\pm$ 1.21	29.93 $\pm$ 0.87	33.06 $\pm$ 1.37	28.45 $\pm$ 0.69	32.09 $\pm$ 0.90	33.19 $\pm$ 0.60
Mistral-7B	25.44 $\pm$ 1.51	29.89 $\pm$ 1.02	34.72 $\pm$ 1.29	23.76 $\pm$ 0.57	32.30 $\pm$ 0.74	34.71 $\pm$ 1.57
Mean	25.51	29.06	31.97	26.18	30.21	31.99

Appendix

Table 15: Corpus BLEU-1 scores comparing baseline with clustering methods (Pretrain).

Model	Baseline	DBSCAN	HDBSCAN	K-means-1000	K-means-2000	K-means-5000
DS-R1-Qwen-1.5B	37.79 $\pm$ 0.99	36.13 $\pm$ 1.34	40.54 $\pm$ 0.99	37.53 $\pm$ 0.75	39.83 $\pm$ 1.20	38.96 $\pm$ 0.86
Gemma3-4B	31.59 $\pm$ 0.86	30.52 $\pm$ 0.96	34.09 $\pm$ 1.21	33.51 $\pm$ 0.71	32.43 $\pm$ 0.44	33.32 $\pm$ 0.17
DS-R1-Llama-8B	37.80 $\pm$ 0.81	37.80 $\pm$ 0.83	40.47 $\pm$ 0.79	38.27 $\pm$ 1.04	40.24 $\pm$ 0.93	40.39 $\pm$ 0.85
Qwen3-8B	38.35 $\pm$ 1.30	37.09 $\pm$ 0.59	41.40 $\pm$ 1.52	38.10 $\pm$ 0.69	40.36 $\pm$ 0.99	40.42 $\pm$ 0.93
Mistral-7B	39.44 $\pm$ 1.68	36.86 $\pm$ 1.19	44.12 $\pm$ 1.15	34.75 $\pm$ 0.62	40.53 $\pm$ 0.96	43.86 $\pm$ 1.37
Mean	37.00	35.68	40.12	36.43	38.68	39.39

Appendix

Table 16: ROUGE-1-F scores comparing baseline with clustering methods (Pretrain).

Model	Baseline	DBSCAN	HDBSCAN	K-means-1000	K-means-2000	K-means-5000
DS-R1-Qwen-1.5B	19.04 $\pm$ 0.86	17.45 $\pm$ 1.57	19.04 $\pm$ 1.09	15.83 $\pm$ 1.00	18.48 $\pm$ 1.44	17.21 $\pm$ 0.96
Gemma3-4B	12.06 $\pm$ 0.51	10.05 $\pm$ 0.75	12.82 $\pm$ 0.87	11.55 $\pm$ 0.68	10.55 $\pm$ 0.68	11.87 $\pm$ 0.41
DS-R1-Llama-8B	18.76 $\pm$ 0.84	18.39 $\pm$ 1.06	18.58 $\pm$ 0.96	16.63 $\pm$ 1.33	18.24 $\pm$ 1.02	18.96 $\pm$ 1.04
Qwen3-8B	20.27 $\pm$ 1.74	18.22 $\pm$ 0.65	20.10 $\pm$ 1.75	16.29 $\pm$ 1.03	18.99 $\pm$ 1.38	18.74 $\pm$ 0.99
Mistral-7B	23.53 $\pm$ 1.87	20.96 $\pm$ 1.57	22.56 $\pm$ 1.53	15.82 $\pm$ 0.82	21.12 $\pm$ 1.26	23.34 $\pm$ 1.43
Mean	18.73	17.01	18.62	15.22	17.48	18.02

Appendix

Table 17: ROUGE-2-F scores comparing baseline with clustering methods (Pretrain).

Model	Baseline	DBSCAN	HDBSCAN	K-means-1000	K-means-2000	K-means-5000
DS-R1-Qwen-1.5B	30.32 $\pm$ 0.89	28.32 $\pm$ 1.22	30.49 $\pm$ 1.08	26.60 $\pm$ 0.72	29.39 $\pm$ 1.07	28.79 $\pm$ 0.57
Gemma3-4B	23.64 $\pm$ 0.52	21.89 $\pm$ 0.66	24.67 $\pm$ 0.88	22.54 $\pm$ 0.69	22.27 $\pm$ 0.55	23.85 $\pm$ 0.42
DS-R1-Llama-8B	29.96 $\pm$ 0.76	29.40 $\pm$ 0.98	30.32 $\pm$ 0.84	27.17 $\pm$ 0.97	29.48 $\pm$ 0.76	30.45 $\pm$ 0.82
Qwen3-8B	31.06 $\pm$ 1.22	29.22 $\pm$ 0.64	31.27 $\pm$ 1.33	26.99 $\pm$ 0.84	29.89 $\pm$ 1.05	30.14 $\pm$ 0.65
Mistral-7B	33.43 $\pm$ 1.51	30.70 $\pm$ 1.21	33.65 $\pm$ 1.23	24.56 $\pm$ 0.56	31.30 $\pm$ 1.01	34.26 $\pm$ 1.15
Mean	29.68	27.91	30.08	25.57	28.47	29.50

Appendix

Table 18: ROUGE-L-F scores comparing baseline with clustering methods (Pretrain).

Model	Baseline	HDBSCAN	Random expansion
DS-R1-Qwen-1.5B	17.26 $\pm$ 4.85	21.19 $\pm$ 5.09	20.86 $\pm$ 5.00
Gemma3-4B	17.73 $\pm$ 1.85	20.06 $\pm$ 2.05	18.98 $\pm$ 5.45
DS-R1-Llama-8B	31.90 $\pm$ 3.44	24.93 $\pm$ 4.47	21.68 $\pm$ 4.60
Qwen3-8B	21.97 $\pm$ 5.39	25.02 $\pm$ 3.51	26.82 $\pm$ 5.30
Mistral-7B	15.89 $\pm$ 4.30	19.13 $\pm$ 5.79	14.84 $\pm$ 2.02
Mean	20.95	22.06	20.63

Appendix

Table 19: Ablation study: CheXbert-5 F1 comparing Baseline, HDBSCAN, and Random expansion.

Model	Baseline	HDBSCAN	Random expansion
DS-R1-Qwen-1.5B	63.68 $\pm$ 0.49	67.34 $\pm$ 0.41	63.97 $\pm$ 0.48
Gemma3-4B	62.05 $\pm$ 0.53	64.59 $\pm$ 0.65	61.97 $\pm$ 0.41
DS-R1-Llama-8B	63.95 $\pm$ 0.48	67.44 $\pm$ 0.31	64.53 $\pm$ 0.57
Qwen3-8B	63.80 $\pm$ 0.80	67.71 $\pm$ 0.73	64.35 $\pm$ 0.28
Mistral-7B	63.59 $\pm$ 0.77	69.16 $\pm$ 0.57	61.53 $\pm$ 0.47
Mean	63.41	67.25	63.27

Appendix

Table 20: Ablation study: COMET comparing Baseline, HDBSCAN, and Random Expansion.

Model	Baseline	HDBSCAN	Random expansion
DS-R1-Qwen-1.5B	27.58 $\pm$ 0.87	29.51 $\pm$ 0.81	26.88 $\pm$ 1.01
Gemma3-4B	21.63 $\pm$ 0.70	23.83 $\pm$ 0.91	22.18 $\pm$ 0.49
DS-R1-Llama-8B	27.45 $\pm$ 0.86	29.11 $\pm$ 0.54	27.59 $\pm$ 0.71
Qwen3-8B	28.11 $\pm$ 1.17	30.35 $\pm$ 1.35	27.17 $\pm$ 0.44
Mistral-7B	29.87 $\pm$ 1.67	32.64 $\pm$ 1.00	28.85 $\pm$ 1.02
Mean	26.93	29.09	26.53

Appendix

Table 21: Ablation study: Sentence BLEU-1 comparing Baseline, HDBSCAN, and Random Expansion.

Model	Baseline	HDBSCAN	Random expansion
DS-R1-Qwen-1.5B	26.36 $\pm$ 0.98	30.13 $\pm$ 1.18	28.33 $\pm$ 1.41
Gemma3-4B	20.48 $\pm$ 0.64	23.52 $\pm$ 1.16	22.24 $\pm$ 0.61
DS-R1-Llama-8B	26.26 $\pm$ 1.12	29.72 $\pm$ 0.79	28.74 $\pm$ 1.24
Qwen3-8B	27.21 $\pm$ 1.43	31.21 $\pm$ 1.97	28.34 $\pm$ 0.60
Mistral-7B	29.42 $\pm$ 1.83	33.51 $\pm$ 1.66	32.33 $\pm$ 1.72
Mean	25.95	29.62	28.00

Appendix

Table 22: Ablation study: METEOR comparing Baseline, HDBSCAN, and Random Expansion.

Model	Baseline	HDBSCAN	Random expansion
DS-R1-Qwen-1.5B	37.79 $\pm$ 0.99	40.54 $\pm$ 0.99	36.73 $\pm$ 1.11
Gemma3-4B	31.59 $\pm$ 0.86	34.09 $\pm$ 1.21	31.77 $\pm$ 0.59
DS-R1-Llama-8B	37.80 $\pm$ 0.81	40.47 $\pm$ 0.79	37.79 $\pm$ 0.86
Qwen3-8B	38.35 $\pm$ 1.30	41.40 $\pm$ 1.52	37.28 $\pm$ 0.44
Mistral-7B	39.44 $\pm$ 1.68	44.12 $\pm$ 1.15	38.24 $\pm$ 1.12
Mean	37.00	40.12	36.36

Appendix

Table 23: Ablation study: ROUGE-1-F comparing Baseline, HDBSCAN, and Random Expansion.

Model	Baseline	HDBSCAN	Random expansion
DS-R1-Qwen-1.5B	19.04 $\pm$ 0.86	19.04 $\pm$ 1.09	17.42 $\pm$ 1.20
Gemma3-4B	12.06 $\pm$ 0.51	12.82 $\pm$ 0.87	11.95 $\pm$ 0.81
DS-R1-Llama-8B	18.76 $\pm$ 0.84	18.58 $\pm$ 0.96	18.21 $\pm$ 1.32
Qwen3-8B	20.27 $\pm$ 1.74	20.10 $\pm$ 1.75	17.82 $\pm$ 0.45
Mistral-7B	23.53 $\pm$ 1.87	22.56 $\pm$ 1.53	20.36 $\pm$ 1.32
Mean	18.73	18.62	17.15

Appendix

Table 24: Ablation study: ROUGE-2-F comparing Baseline, HDBSCAN, and Random Expansion.

Model	Baseline	HDBSCAN	Random expansion
DS-R1-Qwen-1.5B	30.32 $\pm$ 0.89	30.49 $\pm$ 1.08	28.04 $\pm$ 0.89
Gemma3-4B	23.64 $\pm$ 0.52	24.67 $\pm$ 0.88	23.28 $\pm$ 0.70
DS-R1-Llama-8B	29.96 $\pm$ 0.76	30.32 $\pm$ 0.84	29.19 $\pm$ 0.95
Qwen3-8B	31.06 $\pm$ 1.22	31.27 $\pm$ 1.33	28.55 $\pm$ 0.42
Mistral-7B	33.43 $\pm$ 1.51	33.65 $\pm$ 1.23	29.57 $\pm$ 1.08
Mean	29.68	30.08	27.72

Appendix

Table 25: Ablation study: ROUGE-L-F comparing Baseline, HDBSCAN, and Random Expansion.

Model	Baseline	HDBSCAN	Random expansion
DS-R1-Qwen-1.5B	35.39 $\pm$ 0.81	39.40 $\pm$ 1.04	37.56 $\pm$ 1.25
Gemma3-4B	31.61 $\pm$ 0.66	34.50 $\pm$ 1.15	33.39 $\pm$ 0.73
DS-R1-Llama-8B	35.33 $\pm$ 0.88	38.99 $\pm$ 0.75	37.80 $\pm$ 1.38
Qwen3-8B	35.97 $\pm$ 1.43	39.99 $\pm$ 1.60	37.63 $\pm$ 0.49
Mistral-7B	37.48 $\pm$ 1.65	42.67 $\pm$ 1.42	40.02 $\pm$ 1.46
Mean	35.16	39.11	37.28

Appendix

Table 26: Ablation study: ChrF++ comparing Baseline, HDBSCAN, and Random Expansion.

Model	Baseline	HDBSCAN	Random expansion
DS-R1-Qwen-1.5B	67.60 $\pm$ 0.48	69.66 $\pm$ 0.42	67.06 $\pm$ 0.55
Gemma3-4B	65.35 $\pm$ 0.31	66.80 $\pm$ 0.60	65.43 $\pm$ 0.31
DS-R1-Llama-8B	67.83 $\pm$ 0.44	69.80 $\pm$ 0.34	67.91 $\pm$ 0.55
Qwen3-8B	67.79 $\pm$ 0.67	70.07 $\pm$ 0.69	67.62 $\pm$ 0.11
Mistral-7B	67.86 $\pm$ 0.75	71.35 $\pm$ 0.56	66.50 $\pm$ 0.57
Mean	67.29	69.54	66.91

Appendix

Table 27: Ablation study: BERTScore-F1 comparing Baseline, HDBSCAN, and Random Expansion.

Model	$\mathcal{R}_{cl}$	$\mathcal{R}_{exact}$
Qwen2.5-VL-3B	0.2704 $\pm$ 0.0035	0.2361 $\pm$ 0.0042
Qwen3-VL-4B	0.2706 $\pm$ 0.0044	0.2354 $\pm$ 0.0034

Appendix

Table 28: Sentence-BLEU-1 scores, comparing

\mathcal{R}_{cl}

and

\mathcal{R}_{exact}

rewards (GRPO) on the impression section.

Model	$\mathcal{R}_{cl}$	$\mathcal{R}_{exact}$
Qwen2.5-VL-3B	0.2621 ± 0.0034	0.2335 ± 0.0054
Qwen3-VL-4B	0.2620 ± 0.0048	0.2344 ± 0.0032

Appendix

Table 29: METEOR scores, comparing

\mathcal{R}_{cl}

and

\mathcal{R}_{exact}

rewards (GRPO).

Model	$\mathcal{R}_{cl}$	$\mathcal{R}_{exact}$
Qwen2.5-VL-3B	0.1825 ± 0.0029	0.1599 ± 0.0027
Qwen3-VL-4B	0.1847 ± 0.0041	0.1578 ± 0.0023

Appendix

Table 30: Sentence-BLEU-4 scores, comparing

\mathcal{R}_{cl}

and

\mathcal{R}_{exact}

rewards (GRPO).

Radiology VLMs

Radiology vision-language models (VLMs) align medical images and text to support grounded diagnosis, reporting, retrieval, and clinical reasoning.

See all 109 papers

Jul 24, 2026

TextSLIP: Text Self-Supervised CLIP for Medical Report Generation

Automating radiology report generation is important for improving reporting consistency and clinical workflows . While Contrastive Language--Image Pretraining (CLIP) has advanced medical vision language modeling, existing CLIP-style approaches may still provide insufficient fine-grained semantic supervision for complex report generation. Standard CLIP primarily optimizes cross-modal alignment, without explicitly structuring the textual embedding space that guides visual representation learning. To address this limitation, we propose TextSLIP, a general medical vision-language pretraining framework that augments CLIP with intra-modal text contrastive learning. By improving textual embedding discriminability through self-supervised augmented text pairs, TextSLIP is designed to provide finer-grained linguistic supervision to the visual encoder. As an initial validation, we pretrain TextSLIP on a curated dataset of 7 million brain MRI image-text pairs and fine-tune the pretrained visual encoder within a report generation architecture. In controlled comparisons with CLIP-style baselines, TextSLIP shows consistent improvements on report generation metrics. Ablation studies further suggest that text-side self-supervision contributes to the observed gains. These results indicate that text-level contrastive learning is a promising direction for improving medical visual-textual alignment, while broader validation across additional medical domains remains an important next step.

Haoyu Jiang, Ziping Cong

Centre for Artificial Intelligence and Robotics (CAIR) · Hong Kong Institute of Science & Innovation · Chinese Academy of Sciences

Radiology Report Generation Self-Supervised Learning Medical VLMs Vision-Language Alignment Multimodal Contrastive Learning

Ask PDF

Feb 28, 2026

U-VLM: Hierarchical Vision Language Modeling for Report Generation

Automated radiology report generation is key for reducing radiologist workload and improving diagnostic consistency, yet generating accurate reports for 3D medical imaging remains challenging. Existing vision-language models face two limitations: they do not leverage segmentation-pretrained encoders, and they inject visual features only at the input layer of language models, losing multi-scale information. We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture: (1) progressive training from segmentation to classification to report generation, and (2) multi-layer visual injection that routes U-Net encoder features to corresponding language model layers. Each training stage can leverage different datasets without unified annotations. U-VLM achieves state-of-the-art performance on CT-RATE (F1: 0.414 vs 0.258, BLEU-mean: 0.349 vs 0.305) and AbdomenAtlas 3.0 (F1: 0.624 vs 0.518 for segmentation-based detection) using only a 0.1B decoder trained from scratch, demonstrating that well-designed vision encoder pretraining outweighs the benefits of 7B+ pre-trained language models. Ablation studies show that progressive pretraining significantly improves F1, while multi-layer injection improves BLEU-mean. Code is available at https://github.com/yinghemedical/U-VLM.

Pengcheng Shi, Minghui Zhang, Kehan Song +3

Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China

Radiology Report Generation Vision-Language Models CT Report Generation 3D VLMs

Ask PDF

Apr 22, 2026

Weighting What Matters: Boosting Sample Efficiency in Medical Report Generation via Token Reweighting

Training vision-language models (VLMs) for medical report generation is often hindered by the scarcity of high-quality annotated data. This work evaluates the use of a weighted loss function to improve data efficiency. Compared to standard cross-entropy loss, which treats all token prediction errors equally, the reweighted loss shifts the focus to semantically salient tokens with outsized clinical importance. In experiments on ophthalmological report generation, we show that this simple method improves efficiency across multiple data scales, achieving similar report quality with up to ten times less training data.

Alexander Weers, Daniel Rueckert, Martin J. Menten

TUM School of Computation, Information and Technology, Technical University of Munich, DE · Munich Center for Machine Learning (MCML), DE · Department of Computing, Imperial College London, UK

Medical Report Generation Healthcare Ophthalmology Adaptive Loss Weighting Medical VLMs Importance Weighting

Ask PDF

Abstract

Radiology VLMs

Explore similar work

TextSLIP: Text Self-Supervised CLIP for Medical Report Generation

U-VLM: Hierarchical Vision Language Modeling for Report Generation

Weighting What Matters: Boosting Sample Efficiency in Medical Report Generation via Token Reweighting