Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering
Authors: Ao Li, Rui Liu, Mingjie Li, Sheng Liu, Lei Wang, Xiaodan Liang, Lina Yao, Xiaojun Chang, Lei Xing
Organizations: University of New South Wales · Australian Artificial Intelligence Institute, University of Technology Sydney · Stanford University · School of Computing and Information Technology of University of Wollongong Australia · Sun Yat-sen University · University of Science and Technology of China
Automated radiology report generation using vision-language models (VLMs) is limited by the risk of prior-comparison hallucination, where the model generates historical findings unsupported by the current study. We address this challenge with a training-free, inference-time control framework termed Semantically Decoupled Latent Steering (SDLS). Unlike generic activation steering, which often suffers from semantic entanglement, our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition followed by QR-based orthogonalization. This orthogonalization step is critical. It leverages geometric constraints to filter out the clinical semantics often entangled in standard principal component analysis (PCA) directions, ensuring that the steering vector targets only the ``historical comparison" axis. We validate our method on the BiomedGPT foundation model, demonstrating that it overcomes the trade-off between hallucination suppression and clinical accuracy. Extensive experiments on MIMIC-CXR, and zero-shot transfer evaluation on CheXpert Plus and IU-Xray, demonstrate the robustness of our approach. Quantitative evaluations on MIMIC-CXR show that our approach significantly reduces the probability of historical hallucinations (FilBERT score decreases from 0.2373 to 0.1889) and improves clinical label fidelity (CheXpert macro-F1 increases from 0.2242 to 0.3208). Supplementary evaluations confirm that the structural integrity of the clinical narrative is maintained.
Figures & tables
Fig. 1 : Geometric Logic of SDLS. The latent space consists of two conflicting components: the Visual Manifold , which encodes grounded image features representing the clinical ground truth (e.g., current evidence of ‘pleural effusion’), and the Bias Subspace , which captures statistical language priors (e.g., the tendency to generate ‘no interval change’ regardless of visual evidence). Standard steering produces an Entangled Naive Vector (dashed) that inadvertently shifts representations on the manifold, creating a Semantic Error . Our Semantically Decoupled Intervention Vector(SDIV) utilizes QR decomposition to isolate the strictly Orthogonal Component , projecting representations vertically to remove bias without spatial distortion.
Fig. 2 : Schematic overview of the Semantically Decoupled Latent Steering (SDLS) framework. The framework comprises an offline preparation phase (a-b) and an online deployment phase (c). (a) Contrastive Context Mining: Paired reports (hallucinated vs. clean) are encoded to isolate the raw bias representation. (b) Orthogonal Latent Decomposition: A QR -based factorization extracts the pure hallucination direction ( SDIV , purple arrow) orthogonal to image semantics. (c) Inference-Time Steering: During test time, the SDIV is injected into decoder layers to suppress prior-comparison tokens without model retraining.
Category
Cue Phrases
Stability
stable, unchanged, no change, no significant change, no interval change, persistent, remains, similar, chronic, long-standing
Comparison
prior, previous, compared with, compared to, since the previous, again seen
Progression
worsened, increased, larger, more, progression, development of, now demonstrates
Note: A separate list of negative phrases (e.g., ’no prior’) is used for exclusion during analysis.
TABLE I : Dictionary of Cue Words and Phrases Used for HSR Calculation, Grouped by Semantic Category.
Position
Configuration on BiomedGPT
Configuration on IAMJB (VED)
Configuration on LLaVA-Med
B
Baseline
Baseline
Baseline
— GentleInject Methods (Varying Strength λ ) —
1
GentleInject ( λ=−25.0 )
GentleInject ( λ=−25.0 )
GentleInject ( λ=−25.0 )
2
GentleInject ( λ=−20.0 )
GentleInject ( λ=−20.0 )
GentleInject ( λ=−20.0 )
3
GentleInject ( λ=−15.0 )
GentleInject ( λ=−15.0 )
GentleInject ( λ=−15.0 )
4
GentleInject ( λ=−10.0 )
GentleInject ( λ=−10.0 )
GentleInject ( λ=−10.0 )
TABLE II : Metrics Overview Figures. This table details the specific intervention method and configuration. Note that configurations may vary slightly across different backbone models.
Strategy
Mechanism and Implementation Detail
Global Injection
Injects the vector into the outputs of all decoder layers, including the word embedding layer. Aims for a global, persistent steering effect throughout the entire decoding stack.
SteerFair
A more targeted approach with two variants: LayerOutput injects into the final output of each transformer layer (excluding embeddings), while AttentionOutput injects deeper, specifically at the self-attention module’s output.
GentleInject
Applies a strong, non-norm-preserving additive shift ( hs[:, 0, :] += lam * vec ) only to the [CLS] token’s representation at the first decoder layer, testing the impact of an early, high-impact perturbation.
ICV-Token
Treats the vector as a pseudo-token. It is projected to the model’s hidden dimension and prepended to the decoder’s input sequence, influencing generation from the initial context.
Encoder-Concat
The vector is projected and concatenated to the visual encoder’s output sequence, directly modifying the visual context available to the decoder. This method was found to be incompatible with BiomedGPT’s .generate function and was thus excluded for that backend.
TABLE III : Intervention Strategies and Their Mechanisms.
Predictor
Coefficient
Std. Error
t-statistic
P>∣t∣
BERTScore
-2.368
0.004
-598.73
< 0.001
HSR (token-level)
-1.379
0.004
-334.36
< 0.001
Report Length
-0.001
1.41e-06
-825.70
< 0.001
Intercept
2.981
0.003
881.77
< 0.001
R-squared: 0.254, N = 2,792,822
TABLE IV : Linear Regression validating the need for a decoupled evaluation. The dependent variable is the probability of a report being history-free. The statistical significance of both BERTScore and HSR confirms that while generic similarity is a factor, task-aligned metrics are indispensable for evaluating this specific failure mode.
Model
Vector Source
Injection Strategy
λ
FilBERT ( ↓ )
Δ
HSR ( ↓ )
Δ
Macro-F1 ( ↑ )
Δ
RadGraph
RadCliQ
In-Domain Evaluation: MIMIC-CXR
BiomedGPT
–
Baseline
–
0.2373
–
0.0081
–
0.2242
–
0.0035
0.0051
SDIV (Ours)
GentleInject
-15.0
0.2350
-0.0024
0.0087
+0.0006
0.2913
+0.0671
–
–
SDIV (Ours)
ICV-Token
0.0
0.2373
+0.0000
0.0081
+0.0000
0.2950
+0.0707
–
–
SDIV (Ours)
Global Injection (All)
-0.3
0.2311
-0.0063
0.0046
-0.0035
0.3153
+0.0911
–
–
SDIV (Ours)
SteerFair (Attn)
-0.3
0.1889
-0.0485
0.0079
-0.0002
0.3208
+0.0966
0.0026
0.0048
TABLE V : Comprehensive Performance Benchmark. We report the optimal operating point for each strategy. “ Δ ” denotes the absolute change relative to the baseline. Blue highlights the primary reduction in hallucination probability for the optimal operating point. Bold values indicate the best performance per column. Metrics include HSR (History-comparison Span Rate, measuring the proportion of text containing historical cues) and FilBERT (a detector for hallucinated prior comparisons, ↓ ).
Dataset
Model
Base
SDIV
Δ
Reduction (%)
MIMIC-CXR (Source)
BiomedGPT
0.22
0.16
0.06
26.2
CheXpert Plus (Zero-Shot)
BiomedGPT
0.18
0.17
0.01
2.0
VED
1.80
1.27
0.53
40.9
IU-Xray (Zero-Shot)
BiomedGPT
0.24
0.14
0.10
40.6
TABLE VI : Hallucination Sentence Count (HSC). Average hallucinated sentences per report across datasets, showing SDIV’s reductions on foundation models.
Metrics
Ground Truth
Baseline (No Intervention)
SDIV
Case 1 HSR : 0.00 → 0.00 FilBERT : 2.00 → 0.00 F1 : 0.86 → 0.95
Pulmonary edema with bilateral pleural effusions, left greater than right.
ap chest compared to [DATE]: moderate - to - large left pleural effusion has increased since [DATE] . small right pleural effusion and mild - to - moderate pulmonary edema are stable . severe cardiomegaly is chronic . transvenous right atrial and right ventricular pacer leads in standard placements. no pneumothorax.
mild pulmonary edema with bilateral pleural effusions .
Case 2 HSR : 0.00 → 0.00 FilBERT : 2.00 → 0.00 F1 : 0.84 → 0.87
Diffuse bilateral opacities, most likely pulmonary edema, studies. However, superimposed infection cannot be excluded in the appropriate clinical setting.
in comparison with the study of [DATE], there is little overall change. again there are low lung volumes with enlargement of the cardiac silhouette and moderate pulmonary edema with bilateral pleural effusions and compressive basilar atelectasis.
mild pulmonary edema with small bilateral pleural effusions .
Case 3 HSR : 0.00 → 0.00 FilBERT : 2.00 → 0.00 F1 : 0.84 → 0.88
1. Right hilar mass, worrisome for neoplasm, warrants evaluation with a contrast enhanced chest ct. 2. Small bilateral pleural effusions. Notification : findings posted to the critical results dashboard on _ _ _.
in comparison with the study of [DATE], the patient has taken a better inspiration. continued enlargement of the cardiac silhouette without definite vascular congestion or acute focal pneumonia. blunting of the costophrenic angles is again seen .
small bilateral pleural effusions .
Case 4 HSR : 0.00 → 0.00 FilBERT : 3.00 → 0.00 F1 : 0.84 → 0.85
1. Free intraperitoneal air. _ _ _ was paged to discuss this finding on _ _ _ at 3 : 10 p. M. At the time of discovery. 2. Pulmonary edema. 3. Bibasilar atelectasis and small pleural effusions.
left picc line tip is at the level of mid svc. heart size and mediastinum are stable . bilateral pleural effusions are large, unchanged . there is interval progression of pulmonary edema.
1. left - sided picc line terminates in the left brachiocephalic vein. 2. small bilateral pleural effusions .
TABLE VII : Qualitative Case Studies. Comparison of generated reports for four representative test cases. Metrics shown are HSR (History-comparison Span Rate, ↓ ), FilBERT (Hallucination Probability, ↓ ), and F1 (CheXpert Macro-F1, ↑ ). Red text denotes historical hallucinations; Green text denotes valid clinical findings. Note how SDIV surgically removes the red comparisons while maintaining or improving the green F1 scores.
Fig. 3 : Qualitative Visualization of Visual Grounding. We visualize the cross-attention maps during the generation of specific target tokens. (a, b) Factual terms: For grounded terms like “endotracheal” or “cardiopulmonary” , the model’s attention is strongly focused on the relevant anatomical regions (trachea, heart), indicating robust visual grounding. (c, d) Hallucinations: For historical comparison terms like “unchanged” or “stable” , the attention map is diffuse and failed to focus on meaningful visual features, confirming that these words are driven by language priors rather than visual evidence.
Fig. 4 : Dose-response curve showing FilBERT Score as a function of intervention strength ( λ ) for SDIV on BiomedGPT. The curve demonstrates a clear optimal intervention zone, highlighting the controllability of the method.
Fig. 5 : Robustness and specificity checks. Standard negative controls (random, shuffled) show no suppression effect. Note that the style-orthogonalized ICV (orange dashed) underperforms the original baseline.
Medical vision-language models (VLMs) often hallucinate findings when generating chest X-ray reports: they fabricate findings that are not present in the image, miss important ones, or locate them incorrectly. We mitigate this without weight updates by decoding-time residual steering on a per-token sparse autoencoder (SAE) basis: Top-K SAEs on late layers, causal steering against clinical errors, then combined suppress/boost intervention at inference time. On the MIMIC-CXR test split, our inference-only method improves the quality of generated reports for three radiology VLMs (RadVLM, LLaVA-Rad, and CheXOne), with relative improvements of +5.4%, +7.2%, and +17.0% in the clinical composite metric, and statistically significant GREEN gains on all backbones. A cross-model feature alignment shows that the quality-promoting (boost) directions overlap strongly across architectures, whereas hallucination-linked (suppress) directions are model-specific. Therefore, transferable steering must treat suppression per-backbone, rather than sharing a universal suppress list. The same recipe transfers zero-shot to IU-Xray (Green +7.7% rel.) without retraining, confirming that the identified features are properties of the model, not of the training corpus. We release causal feature sets and an interactive feature dashboard: https://cxr-sparse-feature-dashboard.netlify.app/.
Farhad Nooralahzadeh, Benjamin Gundersen, Nicolas Deperrois +7
University of Zurich and University Hospital of Zurich, Switzerland · Zurich University of Applied Sciences, Switzerland · Kobe University, Japan +3
Radiology Report Generation aims to produce accurate and coherent diagnostics from medical images. Although large vision-language models improve report fluency and accuracy, they still suffer from hallucinations by generating plausible pathological descriptions that are not supported by the input images. Existing methods primarily rely on external knowledge guidance to facilitate the alignment between generated text and visual information. However, these approaches often ignore the inherent decoding priors and vision-language alignment biases in pretrained models and lack robustness due to reliance on constructed guidance. In this paper, we propose Layer-wise Expert-aligned Decoding, a method that directly intervenes in the internal decoding process of large vision-language models. A pathology-specific expert module is designed to extract discriminative pathological features, which are then injected into each decoder layer through a gated mechanism. This architecture enables the large language model to progressively incorporate expert features during generation through a learned layer-wise gating function, thereby mitigating decoding biases and steering generation toward factual consistency. Experiments on multiple public datasets demonstrate that the proposed method improves clinical accuracy and factual consistency while maintaining competitive report generation quality.
Ruixiao Yang, Yuanhe Tian, Di Dong +3
Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence +1
Vision-language models (VLMs) are increasingly used for medical image interpretation, yet they frequently hallucinate, generating clinically plausible but factually incorrect findings that pose direct patient safety risks. We introduce HalluCXR, a benchmark evaluating six architecturally diverse VLMs across 856 stratified MIMIC-CXR chest radiographs and three query types, yielding 15,408 model evaluations. An eight-category hallucination taxonomy with clinical severity ratings and a two-layer detection pipeline are validated against 250 human annotations (auto-detection F1=0.959; LLM judge F1=0.907). We find that 61.9--82.3% of outputs contain hallucinations, with clinically dangerous errors in up to 80.2%. Three key patterns emerge: normal radiographs paradoxically attract the most severe hallucinations, common findings are systematically over-fabricated while rare findings go under-detected, and response length alone predicts hallucination risk (AUC up to 0.908). A six-model ensemble reduces fabrication by up to 84.8% at the cost of increased omission; a three-model subset retains comparable performance at half the cost. These results establish that hallucination auditing, verbosity-based risk monitoring, and ensemble-based safety layers are prerequisites for clinical deployment.
Haoyu Wang, Zitong Li
Department of Biostatistics & Health Informatics, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK