Diagnostic Loss: Asymmetric Supervision for Radiology Report Generation

Ignacy Kolton; Wenxuan Li; Zongwei Zhou; Jarosław B. Ćwikła; Arkadiusz Sitek; Pedro R. A. S. Bassi; Szymon Płotka

cs.CVOct 22, 2026

Diagnostic Loss: Asymmetric Supervision for Radiology Report Generation

Authors: Ignacy Kolton, Wenxuan Li, Zongwei Zhou, Jarosław B. Ćwikła, Arkadiusz Sitek, Pedro R. A. S. Bassi, Szymon Płotka

Organizations: Jagiellonian University · Johns Hopkins University · GAMMED · Massachusetts General Hospital ·

Abstract

Automated radiology report generation requires models to do more than produce fluent clinical narratives: they must explicitly state supported diagnoses while avoiding unsupported ones. However, existing medical vision-language models (MedVLMs) are typically trained with token-level maximum likelihood objectives, which prioritize text agreement over diagnostically critical terms. As a result, models can miss key diagnoses and introduce wrong ones. We identify this as an objective misalignment between conventional language modeling and the asymmetric diagnostic requirements of radiology reporting. To encode these requirements, we introduce Diagnostic Loss (DLoss), a differentiable, architecture-agnostic objective that operates directly on decoder token probabilities. DLoss separately aggregates probability mass over vocabularies associated with diagnoses supported by the reference report and with unsupported alternatives. Using asymmetric pooling across token positions in the report, it encourages supported diagnoses to receive sufficient probability in at least one position while suppressing spurious probability spikes for unsupported diagnoses across all positions during training. The loss can be applied to existing 3D MedVLMs without modifying their vision backbones, language decoders, or inference procedures, and introduces no additional inference-time computation. We train and evaluate DLoss on 50,000 abdominal CT scans across several 3D MedVLM architectures on both in-distribution and out-of-distribution report-generation benchmarks. Across these settings, DLoss improves diagnostic faithfulness while maintaining competitive performance on conventional language-generation metrics, showing the value of asymmetric sequence-level supervision for radiology report generation. The code for DLoss is available at https://github.com/gmum/DLoss.

Figures & tables

Figure 1: Overview of DLoss and its workflow. DLoss supervises how probability is distributed over diagnostic targets alongside standard token prediction. (1) Integration. DLoss operates directly on decoder probabilities without modifying the architecture or decoding procedure (only the LoRA parameters and the projector are trained). Reports are decomposed into organ-level findings and mapped offline, via LLM tagging, to supported ( $\mathcal{D}_{i}^{+}$ ) and unsupported ( $\mathcal{D}_{i}^{-}$ ) diagnostic targets in a controlled target space $\mathcal{G}$ . (2) Asymmetric pooling. Target probability masses $m_{i,d,t}$ across valid decoder positions are aggregated using distinct operators: noisy-OR for supported targets (saturates when present at any position) and temperature-controlled log-sum-exp for unsupported targets (ensures probability stays low everywhere). (3) Objective. The supported and unsupported target losses are added to cross-entropy with independent coefficients $\lambda_{+}$ and $\lambda_{-}$ , respectively, and scaled by sample weight $s_{i}$ . The diagnostic target sets are discarded at inference, so DLoss adds no inference-time computation.

Method	BLEU $\uparrow$		ROUGE-L $\uparrow$		METEOR $\uparrow$		CIDEr $\uparrow$		GREEN $\uparrow$		F1-RG $\uparrow$
	M	E	M	E	M	E	M	E	M	E	M	E
Merlin	20.65	16.63	44.21	34.30	18.45	15.21	13.98	10.89	53.82	45.01	35.01	26.36
+ DLoss	+0.01	+0.09	+0.05	+0.26	+0.01	+0.07	+0.18	+0.50	+0.83	+1.24	$-$ 0.01	+0.35
fVLM	39.58	18.61	50.02	34.06	26.32	15.75	36.25	18.15	53.91	46.78	46.53	29.32
+ DLoss	+7.04	+2.59	+6.70	+2.39	+4.04	+1.26	+6.56	+1.34	+8.99	+3.06	+6.56	+1.65
BTB3D	15.96	16.38	32.21	33.75	17.07	16.14	17.39	19.50	53.38	55.76	25.82	27.95

Table 1: Quantitative evaluation of 3D radiology report generation on internal and external benchmarks. We report language generation and clinical accuracy metrics on the Merlin ( M ) test set and an external ( E ) in-house dataset. F1-RG denotes the complete RadGraph score. Rows labeled + DLoss reflect signed changes relative to the baseline model after training with DLoss. Gains are shown in green and drops in red .

\bar{\Delta}

represents the mean change over the corresponding baseline.

Method	Step time (ms)			Peak allocated (GiB)			Peak reserved (GiB)
	CE	+ DLoss	$\Delta$ (%)	CE	+ DLoss	$\Delta$ (%)	CE	+ DLoss	$\Delta$ (%)
fVLM	384.54	391.39	$+1.82$	3.913	3.976	$+1.62$	4.422	4.441	$+0.44$
BTB3D	434.88	453.96	$+4.41$	56.770	56.775	$+0.01$	58.375	60.055	$+2.88$
HLIP	414.97	413.24	$-0.36$	3.420	3.420	$0.00$	3.793	3.852	$+1.54$
RadFM	687.49	690.17	$+0.43$	27.204	27.411	$+0.76$	27.842	28.018	$+0.63$

Table 2: Measured training-step cost of CE and CE plus DLoss. Profiling used matched batches of eight in BF16 on a single NVIDIA GH200 GPU. Step times average 12 fresh-process runs per condition across six counterbalanced blocks; time

\Delta

is the mean paired relative change across blocks. Peak allocated and reserved memory average three separate runs per condition, with

\Delta

reporting relative changes between condition means. Allocated memory measures live tensors, while reserved memory includes allocator-held capacity. The optimized DLoss implementations used for profiling were validated against the training objective to ensure identical loss values and gradients.

Figure 2: Qualitative comparison of organ-level CT report generation. Each column shows a representative CT slice and excerpts from the reference, baseline, and baseline + DLoss findings for one anatomical organ. Green text indicates content consistent with the reference; red text marks baseline statements not supported by the reference.

Config	BLEU-4 (%)	GREEN (%)	F1-RG (%)
#1 CE	49.55	65.96	55.55
#2 S-only	$-1.88$	$-1.11$	$-0.74$
#3 U-only	$-2.75$	$-2.62$	$-1.87$
#4 S-max	$-16.11$	$-14.76$	$-11.10$
#5 S-mean	$-2.27$	$-1.28$	$-1.91$
#6 U-mean	$-22.85$	$-25.17$	$-16.37$

Table 3: Component and pooling ablations of DLoss on fVLM. S/U denote supported/unsupported targets. “Only” retains CE and the indicated DLoss component; “max” and “mean” replace that target’s pooling operator. The CE row gives absolute scores; other rows give relative percentage changes from CE.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Scans	Exams	Patients	Train	Test
Merlin	25,494	25,494	18,317	20,369	5,125
CancerVerse	24,168	22,867	13,863	24,168	-
In-house	2,706	2,706	1,948	-	2,706
Total	52,368	50,767	34,128	44,537	7,831

Appendix

Table 4: Overview of the 3D datasets used for training or testing. We provide the number of scans, exams, patients, and training and test splits for each dataset.

Panel of Figure 3 — Figure 3: Example of report-to-structured-text conversion. Qwen3 maps a free-text CT report to a cleaned report and findings for each of the 11 organ groups. The cleaned report is omitted for display.

Figure 4: Example of radiology report preprocessing for a single CT study. The red panel shows the raw report excerpt, the green panel shows the cleaned current-study report text, and the blue panel shows selected organ-specific outputs and binary organ labels. Text highlights indicate removed phrasing, retained findings, and retained measurements. Only selected organs are shown for readability; the full preprocessing output contains all 11 organ groups.

Figure 5: Schematic overview of diagnostic target construction for the diagnostic loss. An organ-specific reference text is assigned structured semantic tags, mapped into a controlled diagnostic target space, and converted into supported and unsupported lexical target sets for the decoder loss. Supported targets are findings stated in the reference text; unsupported targets are alternatives for the same organ that are discouraged during training. The gallbladder example displays representative targets and lexical forms for one organ.

Figure 6: Conceptual comparison between token-level cross-entropy and the proposed diagnostic loss . Cross-entropy supervises each next-token prediction in the reference radiology report. Diagnostic loss increases probability assigned to supported targets at one or more decoder positions and suppresses probability assigned to unsupported targets across positions. For readability, tokens are shown as whole words, although model tokenizers may split words into smaller units.

Backbone	$\lambda_{+}$	$\lambda_{-}$
Merlin–RadLLaMA	0.001	0.0005
fVLM–Qwen	0.001	0.00025
BTB3D	0.001	0.00025
HLIP–Qwen	0.001	0.00025
RadFM-v3	0.01	0.01
SPECTRE–Qwen	0.01	0.005

Appendix

Table 5: Diagnostic Loss coefficients in the main experiments. The columns correspond to the supported- and unsupported-target coefficients in Eq. 9 .

Metric	Mean effect	Seed bootstrap	Seed–study bootstrap
BLEU-1	$+0.0616$	$[+0.0046,+0.1242]$	$[+0.0050,+0.1238]$
BLEU-2	$+0.0698$	$[+0.0099,+0.1387]$	$[+0.0096,+0.1385]$
BLEU-3	$+0.0738$	$[+0.0114,+0.1460]$	$[+0.0116,+0.1450]$
BLEU-4	$+0.0752$	$[+0.0124,+0.1483]$	$[+0.0127,+0.1475]$
METEOR	$+0.0795$	$[+0.0169,+0.1480]$	$[+0.0164,+0.1483]$
ROUGE-L	$+0.0689$	$[+0.0138,+0.1275]$	$[+0.0137,+0.1277]$

Appendix

Table 6: Paired BTB3D results on the full Merlin test set. Each effect is the mean DLoss-minus-CE difference over nine matched seeds. Intervals are 95% confidence intervals; positive values favor DLoss. Point estimates use all 56,375 organ rows per run.

Seed	BLEU-4	GREEN	RadGraph Complete
7	$+0.0617$	$+0.0726$	$+0.0371$
13	$+0.2790$	$+0.0913$	$+0.2497$
42	$+0.0267$	$+0.0433$	$+0.0689$
101	$+0.0861$	$+0.0205$	$+0.0918$
103	$+0.0045$	$+0.0141$	$+0.0165$
107	$+0.2391$	$+0.1216$	$+0.2406$

Appendix

Table 7: Within-seed effects for representative metrics. Each entry is the DLoss-minus-CE difference on the full Merlin test set, in the native metric scale. These are paired training runs, not independent test-row samples.

Radiology Report Generation

Radiology report generation uses medical images and clinical context to produce structured, clinically accurate findings and impressions.

See all 159 papers

LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation
Feb 4, 2026Ruixiao Yang, Yuanhe Tian, Di Dong +3Radiology Report Generation Language Model Decoding
Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence +1
U-VLM: Hierarchical Vision Language Modeling for Report Generation
Feb 28, 2026Pengcheng Shi, Minghui Zhang, Kehan Song +3Radiology Report Generation Vision-Language Models
Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation
Mar 2, 2026Aditya Parikh, Aasa Feragen, Sneha Das +1VLM Evaluation Radiology Report Evaluation
DTU Compute, Technical University of Denmark

Feb 4, 2026J/K move · Enter open · S save

LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation

Ruixiao Yang, Yuanhe Tian, Di Dong, Xu Yang+2

Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence+1

Radiology Report Generation Language Model Decoding VLM Hallucination Vision-Language Alignment

Open paper AskPDF arXiv

Radiology Report Generation aims to produce accurate and coherent diagnostics from medical images. Although large vision-language models improve report fluency and accuracy, they still suffer from hallucinations by generating plausible pathological descriptions that are not supported by the input images. Existing methods primarily rely on external knowledge guidance to facilitate the alignment between generated text and visual information. However, these approaches often ignore the inherent decoding priors and vision-language alignment biases in pretrained models and lack robustness due to reliance on constructed guidance. In this paper, we propose Layer-wise Expert-aligned Decoding, a method that directly intervenes in the internal decoding process of large vision-language models. A pathology-specific expert module is designed to extract discriminative pathological features, which are then injected into each decoder layer through a gated mechanism. This architecture enables the large language model to progressively incorporate expert features during generation through a learned layer-wise gating function, thereby mitigating decoding biases and steering generation toward factual consistency. Experiments on multiple public datasets demonstrate that the proposed method improves clinical accuracy and factual consistency while maintaining competitive report generation quality.

Abstract

Radiology Report Generation

Explore similar work

LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation

U-VLM: Hierarchical Vision Language Modeling for Report Generation

Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation