Enhancing Fine-Grained Spatial Grounding in 3D CT Report Generation via Discriminative Guidance

Chenyu Wang; Weicheng Dai; Han Liu; Wenchao Li; Kayhan Batmanghelich

cs.CVApr 12, 2026

Enhancing Fine-Grained Spatial Grounding in 3D CT Report Generation via Discriminative Guidance

Authors: Chenyu Wang, Weicheng Dai, Han Liu, Wenchao Li, Kayhan Batmanghelich

Organizations: Boston University, Boston, MA, USA · Siemens Healthineers, Princeton, NJ, USA

Abstract

Vision--language models (VLMs) for radiology report generation (RRG) can produce long-form chest CT reports from volumetric scans and show strong potential to improve radiology workflow efficiency and consistency. However, existing methods face two key limitations: (i) training supervision is often coarse, aligning a whole CT volume with a full free-text report without explicit alignment for fine-grained attributes or pathology locations; and (ii) evaluation is typically holistic (lexical overlap, entity matching, or LLM-as-a-judge scores) and not diagnostic for spatial grounding. We propose Discriminative Cue-Prompting with Prompt Dropout (DCP-PD), a plug-and-play framework that distills fine-grained cues from free-text reports and uses them to guide report generation while mitigating shortcut reliance via prompt dropout. DCP-PD achieves state-of-the-art performance on CT-RATE, improving macro F1 from $=0.501$ to $0.603$ (20% relative), and substantially boosts out-of-distribution performance on Rad-ChestCT from F1 $=0.266$ to $0.503$ (89% relative). Finally, we introduce a hierarchical, location-aware question-set protocol (presence $\rightarrow$ laterality $\rightarrow$ lobe) to directly assess pathology-location grounding, showing that fine-grained spatial localization remains challenging even for models that score highly on current benchmarks.

Figures & tables

Figure 1: Overview of the data preparation, training, and inference pipeline of the DCP-PD framework. (a) Report structuring: we convert free-text CT reports into sectioned, anatomy-aware text using an LLM . (b) Automatic label extraction: we define three set of location-aware clinical questions, and use the same LLM to extract binary answers from each structured report, forming question–answer sets (QAS1–3). Here, QAS1 targets finding-level presence/absence, while QAS2–3 progressively probe finer localization. (c) Discriminative model training: using Multi-scale CT embeddings from the frozen image backbone, we train lightweight multi-label classifiers with weighted BCE to predict the QA-derived labels, with one classifier trained per question set. (d) Two-stage VLM training: the image encoder is frozen throughout. Stage 1 performs CT Image–Report alignment by tuning the visual projector for report generation. Stage 2 performs discriminative-guided SFT by prepending ground-truth (GT) cue prompts with prompt dropout and jointly tuning the projector and a LoRA-tuned LLM with a causal LM loss. (e) Inference: the trained classifier predicts cue prompts, which are injected through the same prompt interface alongside image tokens (and query tokens) to guide report generation.

Figure 2: Detailed inference schematic. The pre-processed multi-channel CT volume is encoded by a 3D convolutional stem to produce a fine-scale token grid, followed by progressive downsampling to construct multi-scale token hierarchies. These token grids are processed by the Atlas visual backbone with stacked MSA blocks to yield hierarchical representations. The aggregated embedding $h_{\mathrm{cls}}$ is fed into a linear classifier to produce DCP tokens via templating, while selected intermediate Atlas tokens $h^{(2)}_{2}$ are projected into LLM image tokens through a vision projector. The LLM generates the final report conditioned on the concatenation of image tokens, DCP tokens, and query tokens.

Figure 3: Per-pathology detection performance. We report F1 for the Base VLM and a linear discriminative classifier on 18 CT findings. Numbers above bars indicate

\Delta

F1 (Classifier

-

Base VLM). For readability, we shorten several long finding names on the x-axis: Arterial calcification (arterial wall calcification), Coronary calcification (coronary artery wall calcification), Septal thickening (interlobular septal thickening), Mosaic attenuation (mosaic attenuation pattern), Peribronchial thick. (peribronchial thickening), and Fibrotic sequela (pulmonary fibrotic sequela).

Training setting	Test setting	Prec. $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	CRG $\uparrow$	BLEU $\uparrow$	METEOR $\uparrow$
w/o DCP	No DCP	0.520	0.423	0.440	0.457	0.293	0.214
w/ GT DCP-PD	DCP-1	0.466	0.681	0.530	0.504	0.305	0.222
w/ GT DCP-PD	DCP-2	0.540	0.700	0.603	0.555	0.306	0.226

Table 1: Report generation performance under different test-time DCP variants. Models are trained without DCP or with our GT DCP-PD and evaluated with different test-time cue sources. DCP-PD consistently improves report quality and is robust to the choice of test-time DCP source, with additional gains when using stronger cue predictors.

Training setting	Test setting	Prec. $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	CRG $\uparrow$	BLEU $\uparrow$	METEOR $\uparrow$
w/ GT DCP	No DCP	0.393	0.071	0.094	0.349	0.268	0.200
w/ GT DCP-PD	No DCP	0.433	0.373	0.366	0.436	0.288	0.211

Table 2: Effect of cue prompts dropout training on shortcuts. We compare VLMs trained with GT DCP and GT DCP-PD and evaluate them under the No DCP test setting, where cue prompts are removed at inference. The shaded row highlights the shortcut failure case of GT DCP, which collapses when cue prompts are absent.

Training setting	$\boldsymbol{S_{\mathrm{text}}}$ (mean $\pm$ std)	$\boldsymbol{S_{\mathrm{image}}}$ (mean $\pm$ std)
w/ GT DCP	$0.441\pm 0.054$	$0.559\pm 0.054$
w/ GT DCP-PD	$0.316\pm 0.036$	$0.684\pm 0.036$

Table 3: Text and image attention reliance score under the DCP-2 test-time setting. We report mean

\pm

std over 500 validation samples on CT-RATE.

Method	Prec. $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	CRG $\uparrow$	BLEU(mean) $\uparrow$	METEOR $\uparrow$
CT2Rep † [ 16 ]	0.435	0.128	0.160	0.359	0.280	0.197
Merlin † [ 6 ]	0.295	0.112	0.160	0.352	0.260	0.148
CT-CHAT † [ 19 ]	0.450	0.158	0.184	0.368	0.272	0.215
BTB3D-16 † [ 18 ]	0.260	0.260	0.258	0.370	0.305	0.223
LLM+S-LMR+CSE ‡ [ 39 ]	0.468	0.166	0.189	-	0.286	0.219
MedVInT-TD ‡ [ 45 ]	0.408	0.097	0.121	-	0.252	0.184

Table 4: Comparison of report generation performance under different settings on the CT-RATE benchmark. We report clinical accuracy metrics (Prec., Recall, F1, CRG) and natural language generation metrics (BLEU(mean), METEOR). Best results per column are shown in bold. ( † show results from [ 18 ] ; ‡ show results from [ 39 ] ).

Figure 4: Per-pathology performance on CT-RATE. We compare our Base VLM and a DCP-PD -trained generator under two inference-time guidance settings ( DCP-1 and DCP-2 ) against prior baselines. We report per-pathology F1 for each finding. The value in parentheses next to each label indicates the positive rate (prevalence) in the CT-RATE validation set, and larger radii correspond to higher F1.

Model	Prec. $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
CT2Rep † [ 16 ]	0.299	0.139	0.133
Merlin † [ 6 ]	0.271	0.149	0.182
CT-CHAT † [ 19 ]	0.382	0.171	0.182
BTB3D-16 † [ 18 ]	0.272	0.329	0.266
Base VLM (Ours)	0.443	0.418	0.395
DCP-PD - DCP1 (Ours)	0.409	0.704	0.477

Table 5: Evaluation on Rad-ChestCT demonstrates strong out-of-distribution generalization. As only binary labels are available, text-based metrics are omitted. † show results from [ 18 ] .

Figure 5: Fine-grained evaluation on Laterality. We report F1 for the Base VLM, DCP-PD VLM, and trained classifier. The x-axis includes four findings: lung nodule (Nodule), consolidation (Cons), ground-glass opacity (GGO), and pleural effusion (PE), each evaluated on left (L) and right (R) sides. Numbers above bars indicate

\Delta

F1 (DCP-PD

-

Base).

Figure 6: Fine-grained evaluation on Lobe. We report F1 for the Base VLM, DCP-PD VLM, and trained classifier on the lobe-level question set. Lobe labels correspond to standard lung lobes: left upper lobe (LUL), left lower lobe (LLL), right upper lobe (RUL), right middle lobe (RML), and right lower lobe (RLL). Numbers above bars indicate

\Delta

F1 (DCP-PD

-

Base).

Dataset	Training reports	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	CRG $\uparrow$	BLEU $\uparrow$	MET $\uparrow$
CT-RATE (ID)	Unstructured	0.521	0.374	0.427	0.440	0.323	0.240
CT-RATE (ID)	Structured	0.520	0.423	0.440	0.457	0.293	0.214
Rad-ChestCT (OOD)	Unstructured	0.428	0.393	0.381	–	–	–
Rad-ChestCT (OOD)	Structured	0.443	0.418	0.395	–	–	–

Table 6: Effect of report structuring (ID vs. OOD). We compare the Base VLM trained on unstructured vs. structured reports on CT-RATE (ID) and Rad-ChestCT (OOD). Text-based metrics are unavailable for Rad-ChestCT and are marked as “–”.

Training with GT-DCP
Training setting	Test-time setting	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	CRG $\uparrow$	BLEU $\uparrow$	METEOR $\uparrow$
w/ GT DCP	GT-DCP	0.947	0.943	0.944	0.890	0.332	0.238
	DCP-2	0.534	0.716	0.606	0.561	0.294	0.231
	DCP-1	0.457	0.718	0.538	0.512	0.296	0.233
	No DCP	0.393	0.071	0.094	0.349	0.268	0.200
Training with DCP-PD

Table 7: Comparison of report generation performance under different classifier-augmented settings. DCP denotes models trained with a classifier-aware prompt prefix that conditions generation on predicted abnormality labels. GT-guided represents an oracle setting using ground-truth labels. DCP-2 uses an external classifier’s predicted labels, while DCP uses predictions from our trained discriminative model. No DCP removes the classifier prompt at inference time. Best results per column are shown in bold.

Model	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	CRG $\uparrow$	BLEU $\uparrow$	METEOR $\uparrow$
DCP-LLM	0.531	0.708	0.600	0.560	0.304	0.225
DCP-PD VLM	0.540	0.700	0.603	0.555	0.306	0.226

Table 8: Effect of visual tokens in discriminative-cue conditioned report generation on CT-RATE. We compare a cue-only generator ( DCP-LLM , no image tokens) against the full DCP-PD VLM . We report macro-averaged clinical metrics and report-level generation metrics.

Figure 7: Effect of visual tokens on fine-grained lobar grounding. We compare the full DCP-PD VLM against the cue-only DCP-LLM without image tokens. The x-axis includes three findings: lung nodule (Nodule), consolidation (Cons), and ground-glass opacity (GGO), each evaluated at the lobar level (LLL, LUL, RLL, RML, RUL). Numbers above bars indicate

\Delta

F1 ( DCP-PD VLM

-

DCP-LLM ), showing that adding visual tokens consistently improves fine-grained spatial grounding.

Method	Training setting	Testing setting	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$
DCP-1 classifier	-	-	0.439	0.794	0.544
Base VLM	w/o DCP	w/o DCP	0.520	0.423	0.440
DCP-1-guided training	DCP-1	DCP-1	0.531	0.414	0.438
GT-guided training	GT	DCP-1	0.457	0.718	0.538

Table 9: Effect of training/testing cue settings on CT-RATE. We report macro-averaged precision, recall, and F1 for the standalone classifier and report generation models under different cue configurations.

Figure 8: We visualize two examples comparing the Base VLM (no cues) and our DCP-PD model guided at inference by discriminative cue prompts. For each case, we show the ground-truth report (only relevant excerpts due to space), representative CT slices with anatomy overlays from LungMask [ 22 ] and pathology masks from ReXGroundingCT [ 3 ] , and the generated reports. Top: the Base VLM misses the lung nodules, while DCP-PD with object-level cues correctly identifies nodules consistent with the ground-truth report and highlighted regions. Bottom: the Base VLM fails to capture lung opacity and instead reports consolidation; DCP-PD with object-level cues detects the opacity but provides incomplete localization, whereas DCP-PD with fine-grained lobe-level cues produces a more faithful pathology–location description.

Figure 9: Example of full report generation on a CT-RATE validation case. The leftmost panel shows representative CT slices with anatomy overlays and pathology masks for GGO and consolidation; abnormal regions are highlighted in red and circled for clarity. The two report panels on the middle show the original unstructured ground-truth report and our processed structured ground-truth report. On the right, we compare full-report outputs from a baseline CT-CHAT model trained on unstructured reports and our DCP-PD model conditioned on fine-grained spatial location cues trained on structured reports. Compared with the baseline, DCP-PD produces a structured report with improved pathology–location grounding.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Set	Question	Pos	Neg
S1	Bronchiectasis	340	2699
S1	Cardiomegaly	309	2730
S1	Pericardial effusion	214	2825
S1	Atelectasis	809	2230
S1	Consolidation	567	2472
S1	Hiatal hernia	421	2618

Appendix

Table 10: Class distribution (Pos/Neg) of the global finding question set (S1) on the CT-RATE validation set.

Set	Question	Pos	Neg
S2	Pleural effusion (R)	303	2736
S2	Pleural effusion (L)	296	2743
S2	Consolidation (R)	457	2582
S2	Consolidation (L)	419	2620
S2	Ground-glass opacity (R)	883	2156
S2	Ground-glass opacity (L)	838	2201

Appendix

Table 11: Class distribution (Pos/Neg) of the laterality-aware question set (S2) on the CT-RATE validation set.

Set	Question	Pos	Neg
S3	Lung nodule (RUL)	314	2725
S3	Lung nodule (RML)	180	2859
S3	Lung nodule (RLL)	282	2757
S3	Lung nodule (LUL)	229	2810
S3	Lung nodule (LLL)	315	2724
S3	Consolidation (RUL)	143	2896

Appendix

Table 12: Class distribution (Pos/Neg) of the lobar question set (S3) on the CT-RATE validation set.

Set	Question	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$
S1	Bronchiectasis	0.266	0.688	0.383	0.789
S1	Cardiomegaly	0.412	0.890	0.564	0.937
S1	Pericardial effusion	0.250	0.790	0.380	0.858
S1	Atelectasis	0.538	0.693	0.606	0.805
S1	Consolidation	0.539	0.882	0.669	0.926
S1	Hiatal hernia	0.267	0.698	0.386	0.759

Appendix

Table 13: Per-question performance of DCP-1 on the global finding question set (S1) on the CT-RATE validation set . We report Precision (Prec), Recall (Rec), F1, and AUROC.

Set	Question	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$
S1	Medical material / device	0.545	0.741	0.628	0.932
S1	Arterial wall calcification	0.685	0.943	0.793	0.946
S1	Cardiomegaly	0.608	0.686	0.645	0.947
S1	Pericardial effusion	0.571	0.659	0.612	0.920
S1	Coronary artery wall calcification	0.709	0.916	0.799	0.952
S1	Hiatal hernia	0.525	0.571	0.547	0.847

Appendix

Table 14: Per-question performance of DCP-2 on the CT-RATE validation set for the global finding question set (S1). We report Precision (Prec), Recall (Rec), F1, and AUROC.

Set	Question	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$
S2	Pleural effusion (R)	0.680	0.931	0.786	0.982
S2	Pleural effusion (L)	0.667	0.936	0.779	0.978
S2	Consolidation (R)	0.461	0.860	0.600	0.922
S2	Consolidation (L)	0.445	0.905	0.596	0.923
S2	Ground-glass opacity (R)	0.687	0.804	0.741	0.898
S2	Ground-glass opacity (L)	0.655	0.796	0.719	0.893

Appendix

Table 15: Per-question performance of DCP-1 on the laterality-aware question set (S2) on the CT-RATE validation set . We report Precision (Prec), Recall (Rec), F1, and AUROC.

Set	Question	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$
S3	Lung nodule (RUL)	0.214	0.666	0.324	0.768
S3	Lung nodule (RML)	0.154	0.694	0.252	0.795
S3	Lung nodule (RLL)	0.201	0.684	0.311	0.762
S3	Lung nodule (LUL)	0.166	0.664	0.265	0.747
S3	Lung nodule (LLL)	0.239	0.686	0.355	0.783
S3	Consolidation (RUL)	0.172	0.818	0.285	0.903

Appendix

Table 16: Per-question performance of DCP-1 on the lobar question set (S3) on the CT-RATE validation set . We report Precision (Prec), Recall (Rec), F1, and AUROC.

Set	Question	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$
S1 ∗	Medical material	0.489	0.896	0.633	0.834
S1 ∗	Calcifications	0.838	0.879	0.858	0.802
S1 ∗	Cardiomegaly	0.341	0.759	0.470	0.870
S1 ∗	Pericardial effusion	0.255	0.744	0.380	0.758
S1 ∗	Hiatal hernia	0.238	0.609	0.342	0.745
S1 ∗	Lymphadenopathy	0.275	0.855	0.416	0.807

Appendix

Table 17: Per-question performance of DCP-1 on Rad-ChestCT for the global finding question set (S1 ∗ ). We report Precision (Prec), Recall (Rec), F1, and AUROC.

Set	Question	Prec. $\uparrow$	Rec. $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$
S1 ∗	Medical material	0.615	0.706	0.658	0.814
S1 ∗	Calcifications	0.803	0.920	0.857	0.818
S1 ∗	Cardiomegaly	0.399	0.716	0.512	0.897
S1 ∗	Pericardial effusion	0.586	0.538	0.561	0.849
S1 ∗	Hiatal hernia	0.368	0.668	0.475	0.828
S1 ∗	Lymphadenopathy	0.205	0.858	0.331	0.729

Appendix

Table 18: Per-question performance of DCP-2 on Rad-ChestCT for the global finding question set (S1 ∗ ). We report Precision (Prec), Recall (Rec), F1, and AUROC. Mosaic attenuation pattern is omitted because no positive samples are present in this evaluation set.

Explore similar work

DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation
Mar 18, 2026Yuhe Tian, Kun Zhang, Haoran Ma +4Radiology Report Generation Medical VLMs
Department of Electronic Engineering and Information Science, School of Information Science and Technology, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China +2
MedScribe: Clinically Grounded CT Reporting through Agentic Workflows
May 3, 2026Giuseppe A. Orlando, Paolo Papotti, Maria A. Zuluaga +2Radiology Report Generation Healthcare
Inria Centre at Université Côte d’Azur Sophia Antipolis, France · EURECOM Biot, France · Dep. of Nuclear Medicine, Centre Antoine Lacassagne Nice, France Université Côte d’Azur, CNRS, Inserm, iBV
Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression
Mar 24, 2026V. K. Cody Bumgardner, Mitchell A. Klusty, Mahmut S. Gokmen +1Radiology Report Generation Computed Tomography
Center for Applied Artificial Intelligence, University of Kentucky, Lexington, KY 40506 USA

Mar 18, 2026J/K move · Enter open · S save

DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation

Yuhe Tian, Kun Zhang, Haoran Ma, Rui Yan+3

Department of Electronic Engineering and Information Science, School of Information Science and Technology, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China+2

Radiology Report Generation Medical VLMs Visual Prompt Learning

Open paper AskPDF arXiv

While large language models (LLMs) have advanced CT report generation, existing methods typically encode 3D volumes holistically, failing to distinguish informative cues from redundant anatomical background. Inspired by radiological cognitive subtraction, we propose Differential Visual Prompting (DiffVP), which conditions report generation on explicit, high-level semantic scan-to-reference differences rather than solely on absolute visual features. DiffVP employs a hierarchical difference extractor to capture complementary global and local semantic discrepancies into a shared latent space, along with a difference-to-prompt generator that transforms these signals into learnable visual prefix tokens for LLM conditioning. These difference prompts serve as structured conditioning signals that implicitly suppress invariant anatomy while amplifying diagnostically relevant visual evidence, thereby facilitating accurate report generation without explicit lesion localization. On two large-scale benchmarks, DiffVP consistently outperforms prior methods, improving the average BLEU-1-4 by +10.98 and +4.36, respectively, and further boosts clinical efficacy on RadGenome-ChestCT (F1 score 0.421). All codes will be released at https://github.com/ArielTYH/DiffVP/.

Abstract

Explore similar work

DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation

MedScribe: Clinically Grounded CT Reporting through Agentic Workflows

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression