DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation

Yuhe Tian; Kun Zhang; Haoran Ma; Rui Yan; Yingtai Li; Rongsheng Wang; Shaohua Kevin Zhou

cs.CVMar 18, 2026

DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation

Authors: Yuhe Tian, Kun Zhang, Haoran Ma, Rui Yan, Yingtai Li, Rongsheng Wang, Shaohua Kevin Zhou

Organizations: Department of Electronic Engineering and Information Science, School of Information Science and Technology, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China · School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · State Key Laboratory of Precision and Intelligent Chemistry, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China

Abstract

While large language models (LLMs) have advanced CT report generation, existing methods typically encode 3D volumes holistically, failing to distinguish informative cues from redundant anatomical background. Inspired by radiological cognitive subtraction, we propose Differential Visual Prompting (DiffVP), which conditions report generation on explicit, high-level semantic scan-to-reference differences rather than solely on absolute visual features. DiffVP employs a hierarchical difference extractor to capture complementary global and local semantic discrepancies into a shared latent space, along with a difference-to-prompt generator that transforms these signals into learnable visual prefix tokens for LLM conditioning. These difference prompts serve as structured conditioning signals that implicitly suppress invariant anatomy while amplifying diagnostically relevant visual evidence, thereby facilitating accurate report generation without explicit lesion localization. On two large-scale benchmarks, DiffVP consistently outperforms prior methods, improving the average BLEU-1-4 by +10.98 and +4.36, respectively, and further boosts clinical efficacy on RadGenome-ChestCT (F1 score 0.421). All codes will be released at https://github.com/ArielTYH/DiffVP/.

Figures & tables

Figure 1: Comparison of conventional methods and our proposed framework. (a) Previous methods feed whole-volume visual tokens uniformly into an LLM for report generation. (b) In clinical practice, radiologists often compare the current scan image with a normal reference image to identify differences. (c) Inspired by this, our method leverages a normal reference prior to derive deviation-aware visual prompts, guiding the LLM to generate more accurate and fine-grained reports.

Figure 2: Overall architecture. The framework takes a target CT volume and a normal reference CT as inputs. A shared visual encoder and resampler produce aligned latent tokens ( $I$ and $I^{r}$ ). A difference-aware module then derives (a) a global delta $\Delta_{\text{global}}$ by applying a Transformer with a learnable query over $I$ and $I^{r}$ , and (b) a local delta $\Delta_{\text{local}}$ by aggregating token-wise residuals with distance-based importance weights. These two signals are fused by (c) a difference-to-prompt generator to form a visual difference prompt, which is prepended as a soft prefix to an LLM (LoRA-tuned) to guide medical report generation.

Dataset	Method	Year	BL-1	BL-2	BL-3	BL-4	RG-L	MTR
RadGenome- ChestCT	$\dagger$ RadFM	2023	44.20	34.49	28.06	23.65	31.53	39.94
	$\dagger$ M3D	2023	43.57	34.48	28.54	24.49	32.61	39.95
	R2GenGPT	2023	43.28	34.11	28.16	24.16	32.26	39.85
	MedVInT	2024	44.28	34.91	28.75	24.60	32.58	40.39
	CT2Rep	2024	44.42	34.43	27.94	23.56	30.99	40.16
	Reg2RG	2025	47.25	36.49	29.57	24.87	36.65	44.07

Table 1: Quantitative results on RadGenome-ChestCT and CTRG-Chest-548K. Models marked with

\dagger

are foundation models. We report BLEU (BL), ROUGE-L (RG-L), and METEOR (MTR) scores. The best, second best, and third best results are highlighted in dark blue, medium blue, and light blue, respectively.

Table 4

ID	Module Settings			NLG Metrics								CE Metrics
ID	$\Delta_{\text{global}}$	$\Delta_{\text{local}}$	E	B-1	B-2	B-3	B-4	M	R-1	R-2	R-L	Pre.	Rec.	F1
Baseline				55.29	46.55	39.36	33.17	44.11	62.12	41.95	31.94	0.295	0.109	0.159
#1			✓	55.62	46.89	39.65	33.44	44.80	62.31	42.19	32.35	0.311	0.217	0.256
#2	✓		✓	57.46	47.92	39.97	33.20	46.63	62.97	41.69	32.08	0.373	0.459	0.411
#3		✓	✓	57.68	48.24	40.47	33.89	47.67	63.19	42.25	32.76	0.360	0.464	0.406
Ours	✓	✓	✓	58.16	48.74	40.93	34.28	47.40	63.52	42.63	32.32	0.366	0.496	0.421

Table 4: Ablation study on the effectiveness of different modules on RadGenome-ChestCT. Global , Local , and E denote the Global Delta, Local Delta, and Classification Prompt modules, respectively. CE Metrics denote clinical efficacy metrics including Precision, Recall, and F1.

Figure 5: Visual semantic discrepancy and discrepancy-level comparison. (a) Two examples showing the input CT, the Visual Semantic Difference Map produced by our method, and the paired normal reference. (b) Performance comparison among w/o Diff , Pixel-level Diff (pixel-wise subtraction), and Ours on BLEU-1 and CE-F1.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Normal Reference Report
Dataset	Reference Content
RadGenome	Thoracic CT examination within normal limits. Findings within normal limits. No obvious pathology was observed in thorax CT examination. Findings within normal limits
CTRG-548K	Thorax is symmetrical. lung window shows clear lung markings. natural walking. good lung field transparency. no obvious consolidation shadow. and bilateral pulmonary hili are not large. The shape of heart shadow and heart big vessels is normal. and no obvious mass or enlarged lymph node is found in mediastinum. No pleural effusion or pleural thickening. Thorax is symmetrical. lung window shows clear lung markings. natural walking. good lung field transparency. no obvious consolidation shadow. and bilateral pulmonary hili are not large. The shape of heart shadow and heart big vessels is normal. and no obvious mass or enlarged lymph node is found in mediastinum. No pleural effusion or pleural thickening.

Appendix

Figure 7: Examples of normal references from the constructed pool. These reports describe standard physiological states without pathological findings.

Method	B1	B4	MTR	RG-L	F1	Recall	Precision
Baseline (Seed 21)	47.25	24.87	44.07	36.65	0.253	0.180	0.423
Baseline (Seed 22)	47.43	25.02	44.06	36.67	0.250	0.178	0.417
Baseline (Seed 23)	47.63	25.27	44.38	36.98	0.248	0.178	0.412
Baseline (Seed 24)	47.41	25.09	44.18	37.62	0.260	0.186	0.430
Baseline (Seed 25)	47.30	24.84	43.99	36.53	0.259	0.186	0.427
Ours (Seed 21)	58.01	35.82	50.76	39.48	0.398	0.440	0.362

Appendix

Table 5: Quantitative results and stability analysis on the test set. We report results across five random seeds for both Baseline and Ours, together with mean

\pm

std and two-sided

t

-test

p

-values (Ours vs. Baseline). Metric scores are reported in percentage (%).

Setting	BL-1	BL-4	MTR	RG-L	F1	Recall	Precision
Baseline (w/o $\Delta$ , w/o $E$ )	55.29	33.17	44.11	31.94	0.159	0.109	0.295
$+\Delta_{\text{global}}$ (w/o $E$ )	56.83	37.24	50.09	42.50	0.188	0.124	0.383
$+\Delta_{\text{global}}+\Delta_{\text{local}}$ (w/o $E$ )	57.23	33.99	46.67	32.83	0.238	0.175	0.374
$+\Delta_{\text{global}}+\Delta_{\text{local}}+E$ (Ours)	58.16	34.28	47.40	32.32	0.421	0.496	0.366

Appendix

Table 6: Ablation study without the diagnostic anchor

E

on RadGenome-ChestCT.

Judge Prompt (GPT-5)

You are a Senior Radiologist Consultant. You must compare Report A vs Report B against the Ground Truth (GT) on FIVE dimensions. IMPORTANT: You are NOT allowed to output ”Tie” in any dimension. You must choose either ”Report A” or ”Report B” for every winner field. This forced-choice design is required to compute consistent win-rate statistics. Focus strictly on what is written in the reports. Do not assume extra findings beyond GT. When judging ”detail_richness”, richness should NOT reward fabricated or unsupported facts. Prefer clinically plausible elaboration only if it remains consistent with the GT. Penalize clearly invented or contradictory details. Dimension 1: finding_coverage Question: Which report captured more GT POSITIVE findings (missed fewer true abnormalities)? How to judge: • Identify GT positive abnormalities and check which report mentions them correctly. • Missing a major GT abnormality is heavily penalized. Winner: the report that misses fewer true positives. Dimension 2: clinical_actionability Question: Which report is more clinically actionable while staying faithful to GT? How to judge: • Clear key impression / prioritization of important findings. • Less irrelevant clutter; avoids burying key abnormalities. • No major contradictions with GT. Winner: the report that is more useful for clinical decision-making. Dimension 3: detail_richness Question: Which report provides richer clinically-relevant detail while remaining faithful to GT? How to judge: • More descriptive radiology language, qualifiers, and contextual details. • Avoid empty generic statements. • Prefer plausible and GT-consistent details. • Penalize unsupported measurements, invented facts, or contradictions. Winner: the report with richer and informative yet GT-consistent descriptions. Dimension 4: anatomical_specificity Question: Which report uses more specific anatomical localization? How to judge: • Laterality (left/right), lobes/segments, sub-regions, organ parts. • Concrete localization beats vague phrases. Winner: the report with finer anatomical grounding. Dimension 5: quantification_detail Question: Which report includes more quantification and measurable descriptors? How to judge: • Measurements (mm/cm), counts (multiple/few), severity (mild/moderate/severe), intervals. Winner: the report with more quantification. Output format (MUST be a strict JSON object, no markdown, no extra text): { "finding_coverage_winner": "Report A" or "Report B", "clinical_actionability_winner": "Report A" or "Report B", "detail_richness_winner": "Report A" or "Report B", "anatomical_specificity_winner": "Report A" or "Report B", "quantification_detail_winner": "Report A" or "Report B", "evidence": { "GT_key_findings": ["<=5 short phrases summarizing the GT positives, if any; if GT is fully normal, write [’normal’]"], "ReportA_supporting_spans": ["2-4 short verbatim spans copied from Report A that support your decisions"], "ReportB_supporting_spans": ["2-4 short verbatim spans copied from Report B that support your decisions"] }, "reason": "1-3 sentences summarizing why the winners were chosen across the five dimensions." }

Appendix

Figure 8: Complete prompt used for LLM-as-judge evaluation (GPT-5, temperature

=0

Pool Size	B1	B4	M	RL	F1	Recall	Prec
1	58.20	35.82	50.71	39.38	0.403	0.451	0.365
10	58.33	36.07	50.83	39.50	0.406	0.451	0.370
40	58.40	36.31	50.92	39.74	0.414	0.458	0.379
60	58.25	35.89	50.66	39.47	0.403	0.450	0.365
124	58.08	35.73	50.55	39.38	0.398	0.445	0.360

Appendix

Table 7: Effect of normal reference pool size on RadGenome-ChestCT.

Setting	B1	B4	M	RL	F1	Recall	Prec
Baseline (No $\Delta$ )	55.29	33.17	44.11	31.94	0.159	0.109	0.295
Ours (In-domain $\Delta$ )	58.16	34.28	47.40	32.32	0.421	0.496	0.366
Ours (100% CTRG $\Delta$ )	56.53	34.95	48.03	36.50	0.128	0.082	0.289

Appendix

Table 8: In-domain vs. domain-shift comparison on RadGenome-ChestCT.

CTRG Mix Ratio	B1	B4	M	RL	F1	Recall	Prec
10%	56.73	35.05	48.17	36.66	0.130	0.083	0.299
50%	56.91	35.45	48.47	36.82	0.128	0.082	0.292
100%	56.53	34.95	48.03	36.50	0.128	0.082	0.289

Appendix

Table 9: Domain shift with different CTRG mixing ratios.

Abnormal Ratio	B1	B4	M	RL	F1	Recall	Prec
10%	57.03	36.51	48.41	36.34	0.166	0.118	0.282
50%	56.91	35.45	48.47	36.82	0.128	0.082	0.292
0% (ours)	58.16	34.28	47.40	32.32	0.421	0.496	0.366

Appendix

Table 10: Imperfect normal pool with mixed abnormal samples.

Explore similar work

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression
Mar 24, 2026V. K. Cody Bumgardner, Mitchell A. Klusty, Mahmut S. Gokmen +1Radiology Report Generation Computed Tomography
Center for Applied Artificial Intelligence, University of Kentucky, Lexington, KY 40506 USA
Enhancing Fine-Grained Spatial Grounding in 3D CT Report Generation via Discriminative Guidance
Apr 12, 2026Chenyu Wang, Weicheng Dai, Han Liu +2Radiology Report Generation Medical Image Grounding
Boston University, Boston, MA, USA · Siemens Healthineers, Princeton, NJ, USA
MedScribe: Clinically Grounded CT Reporting through Agentic Workflows
May 3, 2026Giuseppe A. Orlando, Paolo Papotti, Maria A. Zuluaga +2Radiology Report Generation Healthcare
Inria Centre at Université Côte d’Azur Sophia Antipolis, France · EURECOM Biot, France · Dep. of Nuclear Medicine, Centre Antoine Lacassagne Nice, France Université Côte d’Azur, CNRS, Inserm, iBV

Mar 24, 2026J/K move · Enter open · S save

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

V. K. Cody Bumgardner, Mitchell A. Klusty, Mahmut S. Gokmen, Evan W. Damron

Center for Applied Artificial Intelligence, University of Kentucky, Lexington, KY 40506 USA

Radiology Report Generation Computed Tomography Curriculum Learning

Open paper AskPDF arXiv

Automated radiology report generation from 3D computed tomography (CT) volumes is challenging due to extreme sequence lengths, severe class imbalance, and the tendency of large language models (LLMs) to ignore visual tokens in favor of linguistic priors. We present Ker-VLJEPA-3B, a four-phase curriculum learning framework for free-text report generation from thoracic CT volumes. A phased training curriculum progressively adapts a Llama 3.2 3B decoder to ground its output in visual features from a frozen, self-supervised encoder. Our visual backbone (LeJEPA ViT-Large) is trained via self-supervised joint-embedding prediction on unlabeled CTs, without text supervision. Unlike contrastive models (CLIP, BiomedCLIP), this language-free backbone yields modality-pure representations. Vision-language alignment is deferred to the curriculum's bridge and generation phases. This modality-agnostic design can integrate any self-supervised encoder into an LLM without paired text during foundation training. Methodological innovations include: (1) zone-constrained cross-attention compressing slice embeddings into 32 spatially-grounded visual tokens; (2) PCA whitening of anisotropic LLM embeddings; (3) a positive-findings-only strategy eliminating posterior collapse; (4) warm bridge initialization transferring projection weights; and (5) selective cross-attention freezing with elastic weight consolidation to prevent catastrophic forgetting. Evaluated on the CT-RATE benchmark (2,984 validation volumes, 18 classes), Ker-VLJEPA-3B achieves a macro F1 of 0.429, surpassing the state-of-the-art (U-VLM, macro F1 = 0.414) by 3.6%, and reaching 0.448 (+8.2%) with threshold optimization. Ablation studies confirm 56.6% of generation quality derives from patient-specific visual content. Code and weights are available.

Abstract

Explore similar work

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

Enhancing Fine-Grained Spatial Grounding in 3D CT Report Generation via Discriminative Guidance

MedScribe: Clinically Grounded CT Reporting through Agentic Workflows