Homogeneous Semantic Alignment and Hierarchical Expert Routing for Radiology Report Generation

Erjian Zhang; Jiayuan Ma; Liejun Wang; Yikemaiti Sataer; Xiaoming Tao; Zhiqing Guo

cs.CVOct 3, 2026

Homogeneous Semantic Alignment and Hierarchical Expert Routing for Radiology Report Generation

Authors: Erjian Zhang, Jiayuan Ma, Liejun Wang, Yikemaiti Sataer, Xiaoming Tao, Zhiqing Guo

Organizations: School of Computer Science and Technology, Xinjiang University, Urumqi, 830017, Xinjiang, China · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center, Urumqi, 830017, Xinjiang, China · Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China

Abstract

Radiology report generation (RRG) aims to convert medical images into diagnostic texts to assist in clinical decision-making and alleviate the workload of physicians. Although existing methods have made extensive progress in cross-modal interaction and the incorporation of external priors, the distribution shift of underlying representations and the undifferentiated rigid coupling of heterogeneous information cause weak visual abnormality cues to be easily diluted by massive text priors and generation inertia during decoding. To overcome this bottleneck, inspired by cognitive science, we propose a novel two-stage Homogeneous Semantic Alignment and Hierarchical Expert Routing (HSA-HER) framework. First, the model introduces an explicit homogeneous distribution constraint in the underlying latent space to effectively eliminate the cross-modal distribution shift between visual and textual features, thereby extracting purified visual features as semantic anchors that accurately align with diseases. Second, for heterogeneous clinical evidence composed of visual features, local entities, and global retrievals, we design a hierarchical expert routing mechanism guided by these disease semantic anchors. This mechanism abandons the undifferentiated rigid coupling paradigm. Specifically, it dynamically activates expert networks to perform targeted mining and semantic reconstruction on multi-source evidence, and adaptively allocates fusion weights. Extensive experiments on three mainstream benchmark datasets demonstrate that HSA-HER achieves state-of-the-art performance, accurately depicting complex imaging details and key diagnostic information.

Figures & tables

Figure 1: Comparison between the conventional paradigm and our HSA-HER framework. (a) The distribution shift of underlying representations and the undifferentiated rigid coupling of heterogeneous information cause weak visual abnormality cues to be easily diluted by massive text priors and generation inertia during decoding. (b) HSA-HER accurately depicts imaging details and diagnostic information through homogeneous semantic alignment and a hierarchical expert routing mechanism.

Figure 2: Overall architecture of the proposed HSA-HER framework. (Stage 1) The model eliminates distribution shifts in a unified latent space via an explicit homogeneous distribution constraint to establish precise disease semantic anchors. (Stage 2) Guided by these anchors, micro intra-group routing activates expert sub-networks to perform targeted reconstruction on heterogeneous evidence. Macro cross-source routing adaptively allocates fusion weights and outputs the final diagnostic evidence to drive the large language model for decoding.

Figure 3: Detailed architecture of intra-group gating. Guided by the context

\boldsymbol{C}_{x}

, the gating network dynamically allocates weights to aggregate expert outputs

\boldsymbol{H}_{x,m}

, yielding the reconstructed representation

\tilde{\boldsymbol{E}}_{x}

Method	Publication	NLG Metrics						CE Metrics
Method	Publication	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	CIDEr	Precision	Recall	F1
R2Gen [ 8 ]	EMNLP’20	0.301	0.179	0.118	0.081	0.246	0.077	0.318	0.200	0.181
WCL [ 50 ]	EMNLP’21	–	–	–	0.084	0.253	0.103	0.335	0.259	0.256
R2GenCMN [ 7 ]	ACL’21	0.321	0.195	0.128	0.087	0.256	0.102	0.329	0.241	0.231
XProNet [ 37 ]	ECCV’22	0.364	0.225	0.148	0.100	0.265	0.121	0.314	0.247	0.259
MSAT [ 44 ]	MICCAI’22	–	–	–	0.036	0.156	0.018	0.044	0.142	0.057

Table 1: Performance comparison on the CheXpert Plus test set. The best results are highlighted in bold.

Method	Publication	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	CIDEr
IU X-Ray
R2Gen [ 8 ]	EMNLP’20	0.470	0.304	0.219	0.165	0.371	–
PPKED [ 20 ]	CVPR’21	0.483	0.315	0.224	0.168	0.376	0.351
CMCL [ 19 ]	ACL’21	0.473	0.305	0.217	0.162	0.378	–
Clinical-BERT [ 51 ]	AAAI’22	0.495	0.330	0.231	0.170	0.376	0.432
GSKET [ 53 ]	MedIA’22	0.496	0.327	0.238	0.178	0.381	0.382

Table 2: Performance comparison on the IU X-Ray and MIMIC-CXR test sets. The best results are highlighted in bold.

Dataset	Method	# Clinically Significant Errors $\downarrow$						$\sum_{j=a}^{f}\#\mathrm{Error}_{j}\downarrow$	# Matched Findings $\uparrow$	GREEN $\uparrow$
Dataset	Method	(a)	(b)	(c)	(d)	(e)	(f)	$\sum_{j=a}^{f}\#\mathrm{Error}_{j}\downarrow$	# Matched Findings $\uparrow$	GREEN $\uparrow$
MIMIC-CXR	R2Gen [ 8 ]	1.310	3.089	0.103	0.201	0.082	0.142	4.926	1.803	0.283
	R2GenCMN [ 7 ]	1.383	2.963	0.127	0.228	0.081	0.161	4.942	1.935	0.297
	CvT2DistilGPT2 [ 23 ]	1.150	2.881	0.146	0.234	0.103	0.153	4.666	1.967	0.313
	HSA-HER (Ours)	1.539	2.317	0.161	0.288	0.150	0.102	4.557	2.066	0.323
CheXpert Plus	R2Gen [ 8 ]	1.235	3.638	0.150	0.122	0.021	0.103	5.269	0.938	0.168

Table 3: GREEN performance comparison on the MIMIC-CXR and CheXpert Plus test sets. The six clinically significant error categories are defined as follows: (a) False report of a finding, (b) Missing a finding, (c) Misidentification of anatomic location, (d) Misassessment of severity, (e) Mentioning a comparison not in the reference, and (f) Omitting a comparison detailing a change from a prior study. For clinically significant errors (a)–(f) and their total count, lower values are better (

\downarrow

). For matched findings and GREEN score, higher values are better (

\uparrow

Variant	$\mathcal{L}_{ITC}$	$\mathcal{L}_{CLS}$	$\mathcal{L}_{Homo}$	$\boldsymbol{P}_{cls}$	HER	NLG Metrics						CE Metrics
Variant	$\mathcal{L}_{ITC}$	$\mathcal{L}_{CLS}$	$\mathcal{L}_{Homo}$	$\boldsymbol{P}_{cls}$	HER	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	CIDEr	Precision	Recall	F1
#1						0.378	0.234	0.155	0.106	0.276	0.214	0.354	0.296	0.312
#2				✓	✓	0.379	0.232	0.153	0.103	0.273	0.206	0.377	0.301	0.323
#3	✓			✓	✓	0.380	0.238	0.159	0.110	0.279	0.213	0.375	0.301	0.316
#4	✓	✓		✓	✓	0.379	0.236	0.157	0.108	0.281	0.207	0.373	0.332	0.345
#5	✓	✓	✓			0.383	0.237	0.157	0.107	0.279	0.224	0.357	0.337	0.341

Table 4: Ablation study of Stage 1 and Stage 2 on the CheXpert Plus dataset. Variant #1 serves as the vanilla baseline. Variants #2, #3, #4, and #7 progressively validate the Stage 1 constraints (

\mathcal{L}_{ITC}

\mathcal{L}_{CLS}

\mathcal{L}_{Homo}

). Conversely, Variants #5, #6, and #7 progressively evaluate the Stage 2 modules, namely explicit diagnostic prompts (

\boldsymbol{P}_{cls}

) and hierarchical expert routing (HER), built upon the full Stage 1 setting.

Variant	$\boldsymbol{E}_{loc}$	$\boldsymbol{E}_{glob}$	Rigid Coupling	Micro Routing	Macro Routing	NLG Metrics						CE Metrics
Variant	$\boldsymbol{E}_{loc}$	$\boldsymbol{E}_{glob}$	Rigid Coupling	Micro Routing	Macro Routing	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	CIDEr	Precision	Recall	F1
#1	✓	✓	✓			0.379	0.237	0.158	0.109	0.280	0.213	0.381	0.300	0.316
#2		✓		✓	✓	0.383	0.237	0.157	0.107	0.277	0.218	0.379	0.316	0.336
#3	✓			✓	✓	0.381	0.235	0.156	0.107	0.277	0.216	0.377	0.317	0.335
#4	✓	✓		✓		0.383	0.240	0.160	0.110	0.279	0.220	0.389	0.300	0.320
#5	✓	✓		✓	✓	0.383	0.240	0.161	0.111	0.283	0.212	0.378	0.325	0.338

Table 5: Ablation study of Stage 2 heterogeneous evidence scheduling on the CheXpert Plus dataset. All variants are built upon the full Stage 1 setting. Variant #1 adopts an undifferentiated rigid coupling paradigm. Variants #2 and #3 evaluate the individual external evidence branches (

\boldsymbol{E}_{glob}

and

\boldsymbol{E}_{loc}

, respectively). Variants #4 and #5 progressively validate the micro intra-group routing and macro cross-source routing mechanisms.

Figure 4: Qualitative comparison on the CheXpert Plus test set. Blue indicates the core evidence in the ground truth. Green represents accurate generations that are semantically consistent. Orange denotes partial matches with deviations in location, severity, certainty, or granularity. Red highlights contradictory statements, such as erroneous negations or unsupported expansions. Dark gray indicates non-core, templated, or unverifiable content.

Radiology Report Generation

Radiology report generation uses medical images and clinical context to produce structured, clinically accurate findings and impressions.

See all 159 papers

RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation
Apr 30, 2026Yucheng Chen, Yang Yu, Yufei Shi +3Radiology Report Generation Radiology VLMs
MedVisAI Lab, Lee Kong Chian School of Medicine, Nanyang Technological University (NTU), and Centre of AI in Medicine, Singapore · Department of Computer Science and Engineering, the Chinese University of Hong Kong, Hong Kong SAR, China · Department of Machine Intellection, Institute for Infocomm Research (I2R), Agency for Science, Technology and Research, (A*STAR), Singapore
CliQ-RRG: Clinical-Knowledge Guided Disease-aware Visual-Textual Alignment for QA-Style Radiology Report Generation
Oct 9, 2026Chandravardhan Singh Raghaw, Shahid Shafi Dar, Arnav Nishith Sharma +2Radiology Report Generation Medical VLMs
Indian Institute of Technology Indore, Indore, India · University of Texas at Dallas, USA · University of Wisconsin - Madison, USA
Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Models
Aug 25, 2026Yingshu Li, Yunyi Liu, Zhanyu Wang +4Radiology Report Generation Radiology VLMs
University of Sydney Sydney, Australia · University of Wollongong Wollongong, Australia · University of Adelaide Adelaide, Australia

Apr 30, 2026J/K move · Enter open · S save

RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation

Yucheng Chen, Yang Yu, Yufei Shi, Conghao Xiong+2

MedVisAI Lab, Lee Kong Chian School of Medicine, Nanyang Technological University (NTU), and Centre of AI in Medicine, Singapore · Department of Computer Science and Engineering, the Chinese University of Hong Kong, Hong Kong SAR, China · Department of Machine Intellection, Institute for Infocomm Research (I2R), Agency for Science, Technology and Research, (A*STAR), Singapore

Radiology Report Generation Radiology VLMs Cross-Modal Alignment Vision-Language Alignment

Open paper AskPDF arXiv

Radiology report generation (RRG) has emerged as a promising approach to alleviate radiologists' workload and reduce human errors by automatically generating diagnostic reports from medical images. A key challenge in RRG is achieving fine-grained alignment between complex visual features and the hierarchical structure of long-form radiology reports. Although recent methods have improved image-text representation learning, they often treat reports as flat sequences, overlooking their structured sections and semantic hierarchies. This simplification hinders precise cross-modal alignment and weakens RRG accuracy. To address this challenge, we propose RIHA (Report-Image Hierarchical Alignment Transformer), a novel end-to-end framework that performs multi-level alignment between radiological images and their corresponding reports across paragraph, sentence, and word levels. This hierarchical alignment enables more precise cross-modal mapping, essential for capturing the nuanced semantics embedded in clinical narratives. Specifically, RIHA introduces a Visual Feature Pyramid (VFP) to extract multi-scale visual features and a Text Feature Pyramid (TFP) to represent multi-granularity textual structures. These components are integrated through a Cross-modal Hierarchical Alignment (CHA) module, leveraging optimal transport to effectively align visual and textual features across various levels. Furthermore, we incorporate Relative Positional Encoding (RPE) into the decoder to model spatial and semantic relationships among tokens, enhancing the token-level alignment between visual features and generated text. Extensive experiments on two benchmark chest X-ray datasets, IU-Xray and MIMIC-CXR, demonstrate that RIHA outperforms existing state-of-the-art models in both natural language generation and clinical efficacy metrics.

Abstract

Radiology Report Generation

Explore similar work

RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation

CliQ-RRG: Clinical-Knowledge Guided Disease-aware Visual-Textual Alignment for QA-Style Radiology Report Generation

Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Models