Automated radiology report generation requires models to do more than produce fluent clinical narratives: they must explicitly state supported diagnoses while avoiding unsupported ones. However, existing medical vision-language models (MedVLMs) are typically trained with token-level maximum likelihood objectives, which prioritize text agreement over diagnostically critical terms. As a result, models can miss key diagnoses and introduce wrong ones. We identify this as an objective misalignment between conventional language modeling and the asymmetric diagnostic requirements of radiology reporting. To encode these requirements, we introduce Diagnostic Loss (DLoss), a differentiable, architecture-agnostic objective that operates directly on decoder token probabilities. DLoss separately aggregates probability mass over vocabularies associated with diagnoses supported by the reference report and with unsupported alternatives. Using asymmetric pooling across token positions in the report, it encourages supported diagnoses to receive sufficient probability in at least one position while suppressing spurious probability spikes for unsupported diagnoses across all positions during training. The loss can be applied to existing 3D MedVLMs without modifying their vision backbones, language decoders, or inference procedures, and introduces no additional inference-time computation. We train and evaluate DLoss on 50,000 abdominal CT scans across several 3D MedVLM architectures on both in-distribution and out-of-distribution report-generation benchmarks. Across these settings, DLoss improves diagnostic faithfulness while maintaining competitive performance on conventional language-generation metrics, showing the value of asymmetric sequence-level supervision for radiology report generation. The code for DLoss is available at https://github.com/gmum/DLoss.
Figures & tables
Figure 1: Overview of DLoss and its workflow. DLoss supervises how probability is distributed over diagnostic targets alongside standard token prediction. (1) Integration. DLoss operates directly on decoder probabilities without modifying the architecture or decoding procedure (only the LoRA parameters and the projector are trained). Reports are decomposed into organ-level findings and mapped offline, via LLM tagging, to supported ( Di+ ) and unsupported ( Di− ) diagnostic targets in a controlled target space G . (2) Asymmetric pooling. Target probability masses mi,d,t across valid decoder positions are aggregated using distinct operators: noisy-OR for supported targets (saturates when present at any position) and temperature-controlled log-sum-exp for unsupported targets (ensures probability stays low everywhere). (3) Objective. The supported and unsupported target losses are added to cross-entropy with independent coefficients λ+ and λ− , respectively, and scaled by sample weight si . The diagnostic target sets are discarded at inference, so DLoss adds no inference-time computation.
Method
BLEU ↑
ROUGE-L ↑
METEOR ↑
CIDEr ↑
GREEN ↑
F1-RG ↑
M
E
M
E
M
E
M
E
M
E
M
E
Merlin
20.65
16.63
44.21
34.30
18.45
15.21
13.98
10.89
53.82
45.01
35.01
26.36
+ DLoss
+0.01
+0.09
+0.05
+0.26
+0.01
+0.07
+0.18
+0.50
+0.83
+1.24
− 0.01
+0.35
fVLM
39.58
18.61
50.02
34.06
26.32
15.75
36.25
18.15
53.91
46.78
46.53
29.32
+ DLoss
+7.04
+2.59
+6.70
+2.39
+4.04
+1.26
+6.56
+1.34
+8.99
+3.06
+6.56
+1.65
BTB3D
15.96
16.38
32.21
33.75
17.07
16.14
17.39
19.50
53.38
55.76
25.82
27.95
Table 1: Quantitative evaluation of 3D radiology report generation on internal and external benchmarks. We report language generation and clinical accuracy metrics on the Merlin ( M ) test set and an external ( E ) in-house dataset. F1-RG denotes the complete RadGraph score. Rows labeled + DLoss reflect signed changes relative to the baseline model after training with DLoss. Gains are shown in green and drops in red . Δˉ represents the mean change over the corresponding baseline.
Method
Step time (ms)
Peak allocated (GiB)
Peak reserved (GiB)
CE
+ DLoss
Δ (%)
CE
+ DLoss
Δ (%)
CE
+ DLoss
Δ (%)
fVLM
384.54
391.39
+1.82
3.913
3.976
+1.62
4.422
4.441
+0.44
BTB3D
434.88
453.96
+4.41
56.770
56.775
+0.01
58.375
60.055
+2.88
HLIP
414.97
413.24
−0.36
3.420
3.420
0.00
3.793
3.852
+1.54
RadFM
687.49
690.17
+0.43
27.204
27.411
+0.76
27.842
28.018
+0.63
Table 2: Measured training-step cost of CE and CE plus DLoss. Profiling used matched batches of eight in BF16 on a single NVIDIA GH200 GPU. Step times average 12 fresh-process runs per condition across six counterbalanced blocks; time Δ is the mean paired relative change across blocks. Peak allocated and reserved memory average three separate runs per condition, with Δ reporting relative changes between condition means. Allocated memory measures live tensors, while reserved memory includes allocator-held capacity. The optimized DLoss implementations used for profiling were validated against the training objective to ensure identical loss values and gradients.
Figure 2: Qualitative comparison of organ-level CT report generation. Each column shows a representative CT slice and excerpts from the reference, baseline, and baseline + DLoss findings for one anatomical organ. Green text indicates content consistent with the reference; red text marks baseline statements not supported by the reference.
Config
BLEU-4 (%)
GREEN (%)
F1-RG (%)
#1 CE
49.55
65.96
55.55
#2 S-only
−1.88
−1.11
−0.74
#3 U-only
−2.75
−2.62
−1.87
#4 S-max
−16.11
−14.76
−11.10
#5 S-mean
−2.27
−1.28
−1.91
#6 U-mean
−22.85
−25.17
−16.37
Table 3: Component and pooling ablations of DLoss on fVLM. S/U denote supported/unsupported targets. “Only” retains CE and the indicated DLoss component; “max” and “mean” replace that target’s pooling operator. The CE row gives absolute scores; other rows give relative percentage changes from CE.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Scans
Exams
Patients
Train
Test
Merlin
25,494
25,494
18,317
20,369
5,125
CancerVerse
24,168
22,867
13,863
24,168
-
In-house
2,706
2,706
1,948
-
2,706
Total
52,368
50,767
34,128
44,537
7,831
Appendix
Table 4: Overview of the 3D datasets used for training or testing. We provide the number of scans, exams, patients, and training and test splits for each dataset.
Figure 3: Example of report-to-structured-text conversion. Qwen3 maps a free-text CT report to a cleaned report and findings for each of the 11 organ groups. The cleaned report is omitted for display.
Figure 4: Example of radiology report preprocessing for a single CT study. The red panel shows the raw report excerpt, the green panel shows the cleaned current-study report text, and the blue panel shows selected organ-specific outputs and binary organ labels. Text highlights indicate removed phrasing, retained findings, and retained measurements. Only selected organs are shown for readability; the full preprocessing output contains all 11 organ groups.
Figure 5: Schematic overview of diagnostic target construction for the diagnostic loss. An organ-specific reference text is assigned structured semantic tags, mapped into a controlled diagnostic target space, and converted into supported and unsupported lexical target sets for the decoder loss. Supported targets are findings stated in the reference text; unsupported targets are alternatives for the same organ that are discouraged during training. The gallbladder example displays representative targets and lexical forms for one organ.
Figure 6: Conceptual comparison between token-level cross-entropy and the proposed diagnostic loss . Cross-entropy supervises each next-token prediction in the reference radiology report. Diagnostic loss increases probability assigned to supported targets at one or more decoder positions and suppresses probability assigned to unsupported targets across positions. For readability, tokens are shown as whole words, although model tokenizers may split words into smaller units.
Backbone
λ+
λ−
Merlin–RadLLaMA
0.001
0.0005
fVLM–Qwen
0.001
0.00025
BTB3D
0.001
0.00025
HLIP–Qwen
0.001
0.00025
RadFM-v3
0.01
0.01
SPECTRE–Qwen
0.01
0.005
Appendix
Table 5: Diagnostic Loss coefficients in the main experiments. The columns correspond to the supported- and unsupported-target coefficients in Eq. 9 .
Metric
Mean effect
Seed bootstrap
Seed–study bootstrap
BLEU-1
+0.0616
[+0.0046,+0.1242]
[+0.0050,+0.1238]
BLEU-2
+0.0698
[+0.0099,+0.1387]
[+0.0096,+0.1385]
BLEU-3
+0.0738
[+0.0114,+0.1460]
[+0.0116,+0.1450]
BLEU-4
+0.0752
[+0.0124,+0.1483]
[+0.0127,+0.1475]
METEOR
+0.0795
[+0.0169,+0.1480]
[+0.0164,+0.1483]
ROUGE-L
+0.0689
[+0.0138,+0.1275]
[+0.0137,+0.1277]
Appendix
Table 6: Paired BTB3D results on the full Merlin test set. Each effect is the mean DLoss-minus-CE difference over nine matched seeds. Intervals are 95% confidence intervals; positive values favor DLoss. Point estimates use all 56,375 organ rows per run.
Seed
BLEU-4
GREEN
RadGraph Complete
7
+0.0617
+0.0726
+0.0371
13
+0.2790
+0.0913
+0.2497
42
+0.0267
+0.0433
+0.0689
101
+0.0861
+0.0205
+0.0918
103
+0.0045
+0.0141
+0.0165
107
+0.2391
+0.1216
+0.2406
Appendix
Table 7: Within-seed effects for representative metrics. Each entry is the DLoss-minus-CE difference on the full Merlin test set, in the native metric scale. These are paired training runs, not independent test-row samples.
Radiology Report Generation aims to produce accurate and coherent diagnostics from medical images. Although large vision-language models improve report fluency and accuracy, they still suffer from hallucinations by generating plausible pathological descriptions that are not supported by the input images. Existing methods primarily rely on external knowledge guidance to facilitate the alignment between generated text and visual information. However, these approaches often ignore the inherent decoding priors and vision-language alignment biases in pretrained models and lack robustness due to reliance on constructed guidance. In this paper, we propose Layer-wise Expert-aligned Decoding, a method that directly intervenes in the internal decoding process of large vision-language models. A pathology-specific expert module is designed to extract discriminative pathological features, which are then injected into each decoder layer through a gated mechanism. This architecture enables the large language model to progressively incorporate expert features during generation through a learned layer-wise gating function, thereby mitigating decoding biases and steering generation toward factual consistency. Experiments on multiple public datasets demonstrate that the proposed method improves clinical accuracy and factual consistency while maintaining competitive report generation quality.
Ruixiao Yang, Yuanhe Tian, Di Dong +3
Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence +1
Automated radiology report generation is key for reducing radiologist workload and improving diagnostic consistency, yet generating accurate reports for 3D medical imaging remains challenging. Existing vision-language models face two limitations: they do not leverage segmentation-pretrained encoders, and they inject visual features only at the input layer of language models, losing multi-scale information. We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture: (1) progressive training from segmentation to classification to report generation, and (2) multi-layer visual injection that routes U-Net encoder features to corresponding language model layers. Each training stage can leverage different datasets without unified annotations. U-VLM achieves state-of-the-art performance on CT-RATE (F1: 0.414 vs 0.258, BLEU-mean: 0.349 vs 0.305) and AbdomenAtlas 3.0 (F1: 0.624 vs 0.518 for segmentation-based detection) using only a 0.1B decoder trained from scratch, demonstrating that well-designed vision encoder pretraining outweighs the benefits of 7B+ pre-trained language models. Ablation studies show that progressive pretraining significantly improves F1, while multi-layer injection improves BLEU-mean. Code is available at https://github.com/yinghemedical/U-VLM.
Pengcheng Shi, Minghui Zhang, Kehan Song +3
Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Reliable deployment of Vision-Language Models (VLMs) in radiology requires validation metrics that go beyond surface-level text similarity to ensure clinical fidelity and demographic fairness. This paper investigates a critical blind spot in current model evaluation: the use of decoding strategies that lead to high aggregate token-overlap scores despite succumbing to template collapse, in which models generate only repetitive, safe generic text and omit clinical terminology. Unaddressed, this blind spot can lead to metric gaming, where models that perform well on benchmarks prove clinically uninformative. Instead, we advocate for lexical diversity measures to check model generations for clinical specificity. We introduce Clinical Association Displacement (CAD), a vocabulary-level framework that quantifies shifts in demographic-based word associations in generated reports. Weighted Association Erasure (WAE) aggregates these shifts to measure the clinical signal loss across demographic groups. We show that deterministic decoding produces high levels of semantic erasure, while stochastic sampling generates diverse outputs but risks introducing new bias, motivating a fundamental rethink of how "optimal" reporting is defined.