Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering

Ao Li; Rui Liu; Mingjie Li; Sheng Liu; Lei Wang; Xiaodan Liang; Lina Yao; Xiaojun Chang

cs.CVFeb 27, 2026

Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering

Authors: Ao Li, Rui Liu, Mingjie Li, Sheng Liu, Lei Wang, Xiaodan Liang, Lina Yao, Xiaojun Chang,

Organizations: University of New South Wales · Australian Artificial Intelligence Institute, University of Technology Sydney · Stanford University · School of Computing and Information Technology of University of Wollongong Australia ·

Abstract

Automated radiology report generation using vision-language models (VLMs) is limited by the risk of prior-comparison hallucination, where the model generates historical findings unsupported by the current study. We address this challenge with a training-free, inference-time control framework termed Semantically Decoupled Latent Steering (SDLS). Unlike generic activation steering, which often suffers from semantic entanglement, our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition followed by $QR$ -based orthogonalization. This orthogonalization step is critical. It leverages geometric constraints to filter out the clinical semantics often entangled in standard principal component analysis (PCA) directions, ensuring that the steering vector targets only the ``historical comparison" axis. We validate our method on the BiomedGPT foundation model, demonstrating that it overcomes the trade-off between hallucination suppression and clinical accuracy. Extensive experiments on MIMIC-CXR, and zero-shot transfer evaluation on CheXpert Plus and IU-Xray, demonstrate the robustness of our approach. Quantitative evaluations on MIMIC-CXR show that our approach significantly reduces the probability of historical hallucinations (FilBERT score decreases from 0.2373 to 0.1889) and improves clinical label fidelity (CheXpert macro-F1 increases from 0.2242 to 0.3208). Supplementary evaluations confirm that the structural integrity of the clinical narrative is maintained.

Figures & tables

Fig. 1 : Geometric Logic of SDLS. The latent space consists of two conflicting components: the Visual Manifold , which encodes grounded image features representing the clinical ground truth (e.g., current evidence of ‘pleural effusion’), and the Bias Subspace , which captures statistical language priors (e.g., the tendency to generate ‘no interval change’ regardless of visual evidence). Standard steering produces an Entangled Naive Vector (dashed) that inadvertently shifts representations on the manifold, creating a Semantic Error . Our Semantically Decoupled Intervention Vector(SDIV) utilizes $QR$ decomposition to isolate the strictly Orthogonal Component , projecting representations vertically to remove bias without spatial distortion.

Fig. 2 : Schematic overview of the Semantically Decoupled Latent Steering (SDLS) framework. The framework comprises an offline preparation phase (a-b) and an online deployment phase (c). (a) Contrastive Context Mining: Paired reports (hallucinated vs. clean) are encoded to isolate the raw bias representation. (b) Orthogonal Latent Decomposition: A $QR$ -based factorization extracts the pure hallucination direction ( SDIV , purple arrow) orthogonal to image semantics. (c) Inference-Time Steering: During test time, the SDIV is injected into decoder layers to suppress prior-comparison tokens without model retraining.

Category	Cue Phrases
Stability	stable, unchanged, no change, no significant change, no interval change, persistent, remains, similar, chronic, long-standing
Comparison	prior, previous, compared with, compared to, since the previous, again seen
Progression	worsened, increased, larger, more, progression, development of, now demonstrates
Improvement	improved, decreased, smaller, less, resolved, resolution, clearing
Note: A separate list of negative phrases (e.g., ’no prior’) is used for exclusion during analysis.

TABLE I : Dictionary of Cue Words and Phrases Used for HSR Calculation, Grouped by Semantic Category.

Position	Configuration on BiomedGPT	Configuration on IAMJB (VED)	Configuration on LLaVA-Med
B	Baseline	Baseline	Baseline
— GentleInject Methods (Varying Strength $\lambda$ ) —
1	GentleInject ( $\lambda=-25.0$ )	GentleInject ( $\lambda=-25.0$ )	GentleInject ( $\lambda=-25.0$ )
2	GentleInject ( $\lambda=-20.0$ )	GentleInject ( $\lambda=-20.0$ )	GentleInject ( $\lambda=-20.0$ )
3	GentleInject ( $\lambda=-15.0$ )	GentleInject ( $\lambda=-15.0$ )	GentleInject ( $\lambda=-15.0$ )
4	GentleInject ( $\lambda=-10.0$ )	GentleInject ( $\lambda=-10.0$ )	GentleInject ( $\lambda=-10.0$ )

TABLE II : Metrics Overview Figures. This table details the specific intervention method and configuration. Note that configurations may vary slightly across different backbone models.

Strategy	Mechanism and Implementation Detail
Global Injection	Injects the vector into the outputs of all decoder layers, including the word embedding layer. Aims for a global, persistent steering effect throughout the entire decoding stack.
SteerFair	A more targeted approach with two variants: LayerOutput injects into the final output of each transformer layer (excluding embeddings), while AttentionOutput injects deeper, specifically at the self-attention module’s output.
GentleInject	Applies a strong, non-norm-preserving additive shift ( hs[:, 0, :] += lam * vec ) only to the [CLS] token’s representation at the first decoder layer, testing the impact of an early, high-impact perturbation.
ICV-Token	Treats the vector as a pseudo-token. It is projected to the model’s hidden dimension and prepended to the decoder’s input sequence, influencing generation from the initial context.
Encoder-Concat	The vector is projected and concatenated to the visual encoder’s output sequence, directly modifying the visual context available to the decoder. This method was found to be incompatible with BiomedGPT’s .generate function and was thus excluded for that backend.

TABLE III : Intervention Strategies and Their Mechanisms.

Predictor	Coefficient	Std. Error	t-statistic	$P>\|t\|$
BERTScore	-2.368	0.004	-598.73	$<$ 0.001
HSR (token-level)	-1.379	0.004	-334.36	$<$ 0.001
Report Length	-0.001	1.41e-06	-825.70	$<$ 0.001
Intercept	2.981	0.003	881.77	$<$ 0.001
R-squared: 0.254, N = 2,792,822

TABLE IV : Linear Regression validating the need for a decoupled evaluation. The dependent variable is the probability of a report being history-free. The statistical significance of both BERTScore and HSR confirms that while generic similarity is a factor, task-aligned metrics are indispensable for evaluating this specific failure mode.

Model	Vector Source	Injection Strategy	$\lambda$	FilBERT ( $\downarrow$ )	$\Delta$	HSR ( $\downarrow$ )	$\Delta$	Macro-F1 ( $\uparrow$ )	$\Delta$	RadGraph	RadCliQ
In-Domain Evaluation: MIMIC-CXR
BiomedGPT	–	Baseline	–	0.2373	–	0.0081	–	0.2242	–	0.0035	0.0051
	SDIV (Ours)	GentleInject	-15.0	0.2350	-0.0024	0.0087	+0.0006	0.2913	+0.0671	–	–
	SDIV (Ours)	ICV-Token	0.0	0.2373	+0.0000	0.0081	+0.0000	0.2950	+0.0707	–	–
	SDIV (Ours)	Global Injection (All)	-0.3	0.2311	-0.0063	0.0046	-0.0035	0.3153	+0.0911	–	–
	SDIV (Ours)	SteerFair (Attn)	-0.3	0.1889	-0.0485	0.0079	-0.0002	0.3208	+0.0966	0.0026	0.0048

TABLE V : Comprehensive Performance Benchmark. We report the optimal operating point for each strategy. “

\Delta

” denotes the absolute change relative to the baseline. Blue highlights the primary reduction in hallucination probability for the optimal operating point. Bold values indicate the best performance per column. Metrics include HSR (History-comparison Span Rate, measuring the proportion of text containing historical cues) and FilBERT (a detector for hallucinated prior comparisons,

\downarrow

Dataset	Model	Base	SDIV	$\Delta$	Reduction (%)
MIMIC-CXR (Source)	BiomedGPT	0.22	0.16	0.06	26.2
CheXpert Plus (Zero-Shot)	BiomedGPT	0.18	0.17	0.01	2.0
	VED	1.80	1.27	0.53	40.9
IU-Xray (Zero-Shot)	BiomedGPT	0.24	0.14	0.10	40.6

TABLE VI : Hallucination Sentence Count (HSC). Average hallucinated sentences per report across datasets, showing SDIV’s reductions on foundation models.

Metrics	Ground Truth	Baseline (No Intervention)	SDIV
Case 1 HSR : 0.00 $\to$ 0.00 FilBERT : 2.00 $\to$ 0.00 F1 : 0.86 $\to$ 0.95	Pulmonary edema with bilateral pleural effusions, left greater than right.	ap chest compared to [DATE]: moderate - to - large left pleural effusion has increased since [DATE] . small right pleural effusion and mild - to - moderate pulmonary edema are stable . severe cardiomegaly is chronic . transvenous right atrial and right ventricular pacer leads in standard placements. no pneumothorax.	mild pulmonary edema with bilateral pleural effusions .
Case 2 HSR : 0.00 $\to$ 0.00 FilBERT : 2.00 $\to$ 0.00 F1 : 0.84 $\to$ 0.87	Diffuse bilateral opacities, most likely pulmonary edema, studies. However, superimposed infection cannot be excluded in the appropriate clinical setting.	in comparison with the study of [DATE], there is little overall change. again there are low lung volumes with enlargement of the cardiac silhouette and moderate pulmonary edema with bilateral pleural effusions and compressive basilar atelectasis.	mild pulmonary edema with small bilateral pleural effusions .
Case 3 HSR : 0.00 $\to$ 0.00 FilBERT : 2.00 $\to$ 0.00 F1 : 0.84 $\to$ 0.88	1. Right hilar mass, worrisome for neoplasm, warrants evaluation with a contrast enhanced chest ct. 2. Small bilateral pleural effusions. Notification : findings posted to the critical results dashboard on _ _ _.	in comparison with the study of [DATE], the patient has taken a better inspiration. continued enlargement of the cardiac silhouette without definite vascular congestion or acute focal pneumonia. blunting of the costophrenic angles is again seen .	small bilateral pleural effusions .
Case 4 HSR : 0.00 $\to$ 0.00 FilBERT : 3.00 $\to$ 0.00 F1 : 0.84 $\to$ 0.85	1. Free intraperitoneal air. _ _ _ was paged to discuss this finding on _ _ _ at 3 : 10 p. M. At the time of discovery. 2. Pulmonary edema. 3. Bibasilar atelectasis and small pleural effusions.	left picc line tip is at the level of mid svc. heart size and mediastinum are stable . bilateral pleural effusions are large, unchanged . there is interval progression of pulmonary edema.	1. left - sided picc line terminates in the left brachiocephalic vein. 2. small bilateral pleural effusions .

TABLE VII : Qualitative Case Studies. Comparison of generated reports for four representative test cases. Metrics shown are HSR (History-comparison Span Rate,

\downarrow

), FilBERT (Hallucination Probability,

\downarrow

), and F1 (CheXpert Macro-F1,

\uparrow

). Red text denotes historical hallucinations; Green text denotes valid clinical findings. Note how SDIV surgically removes the red comparisons while maintaining or improving the green F1 scores.

Panel of Figure 10 — Fig. 3 : Qualitative Visualization of Visual Grounding. We visualize the cross-attention maps during the generation of specific target tokens. (a, b) Factual terms: For grounded terms like “endotracheal” or “cardiopulmonary” , the model’s attention is strongly focused on the relevant anatomical regions (trachea, heart), indicating robust visual grounding. (c, d) Hallucinations: For historical comparison terms like “unchanged” or “stable” , the attention map is diffuse and failed to focus on meaningful visual features, confirming that these words are driven by language priors rather than visual evidence.

Fig. 4 : Dose-response curve showing FilBERT Score as a function of intervention strength ( $\lambda$ ) for SDIV on BiomedGPT. The curve demonstrates a clear optimal intervention zone, highlighting the controllability of the method.

Fig. 5 : Robustness and specificity checks. Standard negative controls (random, shuffled) show no suppression effect. Note that the style-orthogonalized ICV (orange dashed) underperforms the original baseline.

Radiology Report Generation

Radiology report generation uses medical images and clinical context to produce structured, clinically accurate findings and impressions.

See all 159 papers

Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models
May 24, 2026Farhad Nooralahzadeh, Benjamin Gundersen, Nicolas Deperrois +7Radiology Report Generation Healthcare
University of Zurich and University Hospital of Zurich, Switzerland · Zurich University of Applied Sciences, Switzerland · Kobe University, Japan +3
LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation
Feb 4, 2026Ruixiao Yang, Yuanhe Tian, Di Dong +3Radiology Report Generation Language Model Decoding
Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence +1
HalluCXR: Benchmarking and Mitigating Hallucinations in Medical Vision-Language Models for Chest Radiograph Interpretation
May 19, 2026Haoyu Wang, Zitong LiVision-Language Models Audio-Language Model Hallucination Detection
Department of Biostatistics & Health Informatics, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK

May 24, 2026J/K move · Enter open · S save

Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models

Farhad Nooralahzadeh, Benjamin Gundersen, Nicolas Deperrois, Hidetoshi Matsuom+6

University of Zurich and University Hospital of Zurich, Switzerland · Zurich University of Applied Sciences, Switzerland · Kobe University, Japan+3

Radiology Report Generation Healthcare Object Hallucination in VLMs Sparse Autoencoders Medical VLMs Activation Steering+1

Open paper AskPDF arXiv

Medical vision-language models (VLMs) often hallucinate findings when generating chest X-ray reports: they fabricate findings that are not present in the image, miss important ones, or locate them incorrectly. We mitigate this without weight updates by decoding-time residual steering on a per-token sparse autoencoder (SAE) basis: Top-

K

SAEs on late layers, causal steering against clinical errors, then combined suppress/boost intervention at inference time. On the MIMIC-CXR test split, our inference-only method improves the quality of generated reports for three radiology VLMs (RadVLM, LLaVA-Rad, and CheXOne), with relative improvements of +5.4%, +7.2%, and +17.0% in the clinical composite metric, and statistically significant GREEN gains on all backbones. A cross-model feature alignment shows that the quality-promoting (boost) directions overlap strongly across architectures, whereas hallucination-linked (suppress) directions are model-specific. Therefore, transferable steering must treat suppression per-backbone, rather than sharing a universal suppress list. The same recipe transfers zero-shot to IU-Xray (Green

+7.7\%

rel.) without retraining, confirming that the identified features are properties of the model, not of the training corpus. We release causal feature sets and an interactive feature dashboard: https://cxr-sparse-feature-dashboard.netlify.app/.

Abstract

Radiology Report Generation

Explore similar work

Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models

LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation

HalluCXR: Benchmarking and Mitigating Hallucinations in Medical Vision-Language Models for Chest Radiograph Interpretation