Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation

Renjie Liang; Yiling Ma; Yang Xing; Zhengkang Fan; Jinqian Pan; Chengkun Sun; Li Li; Kuang Gong

cs.CVMar 16, 2026

Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation

Authors: Renjie Liang, Yiling Ma, Yang Xing, Zhengkang Fan, Jinqian Pan, Chengkun Sun, Li Li, Kuang Gong, +1 more

Organizations: University of Florida · Yale University · University of Southern California

Abstract

Automated radiology report generation from 3D CT volumes often suffers from incomplete pathology coverage. We provide empirical evidence that this limitation stems from a representational bottleneck: contrastive 3D CT embeddings encode discriminative pathology signals, yet exhibit severe dimensional concentration, with as few as 2 effective dimensions out of 512. Corroborating this, scaling the language model yields no measurable improvement, suggesting that the bottleneck lies in the visual representation rather than the generator. This bottleneck limits both generation and retrieval; naive static retrieval fails to improve clinical efficacy and can even degrade performance. We propose \textbf{AdaRAG-CT}, an adaptive augmentation framework that compensates for this visual bottleneck by introducing supplementary textual information through controlled retrieval and selectively integrating it during generation. On the CT-RATE benchmark, AdaRAG-CT achieves state-of-the-art clinical efficacy, improving Clinical F1 from 0.420 (CT-Agent) to 0.480 (+6 points); ablation studies confirm that both the retrieval and generation components contribute to the improvement. Code is available at https://github.com/renjie-liang/Adaptive-RAG-for-3DCT-Report-Generation.

Figures & tables

Model	Embedding	Split	Total dim	dim 90	dim 95	PR
CLIP ViT-B/32 (ImageNet)	image emb.	reference	512	243.0	322.0	64.40
CT-CLIP (Zero-shot)	avg pool	train	512	2.0 [2.0,2.0]	4.0 [4.0,4.0]	1.43 [1.41,1.45]
	avg pool	valid	512	2.0 [2.0,2.0]	4.0 [4.0,4.0]	1.43 [1.41,1.45]
	max pool	train	512	109.8 [108.0,112.0]	181.3 [180.0,183.0]	6.70 [6.47,6.93]
	max pool	valid	512	103.6 [102.0,105.0]	171.1 [169.0,173.0]	6.65 [6.38,6.86]
CT-CLIP (VocabFine)	avg pool	train	512	1.0 [1.0,1.0]	2.0 [2.0,2.0]	1.16 [1.15,1.17]

Table 1: Effective dimensionality and train/validation stability of 3D CT embeddings. Gray brackets denote 95% intervals. dim 90 /dim 95 count components explaining 90%/95% of variance; PR is the participation ratio.

Lung (11)
	CT-CLIP (Zero-shot)		CT-CLIP (VocabFine)
Abnormality	avg	max	avg	max	FVLM	ViSD-Boost
Lung nodule	0.619	0.582	0.588	0.516	0.664	0.713
Mosaic attenuation	0.794	0.710	0.711	0.578	0.887	0.892
Peribronchial thick.	0.717	0.631	0.730	0.645	0.777	0.814
Consolidation	0.732	0.659	0.630	0.543	0.859	0.920

Table 2: Linear probe AUC on frozen contrastive embeddings. *

Organ	Img2Img	Img2Txt	Txt2Txt	Upper Bound
Lung	0.351	0.313	0.563	0.992
Heart	0.825	0.638	0.925	1.000
Esophagus	0.796	0.617	0.935	1.000
Aorta	0.741	0.617	0.666	1.000

Table 3: Organ-level retrieval precision (Jaccard@10) with ViSD-Boost image embeddings and its text encoder on organ paragraphs. Upper Bound ranks by ground-truth label overlap.

Figure 1: Overview of AdaRAG-CT. (a) A 3D CT volume is encoded into a global embedding (CT-CLIP) and four organ-specific embeddings (ViSD-Boost), which are projected into the LLM input space as visual tokens. (b) An organ-indexed sentence database is built from training-set reports and indexed with FAISS for efficient retrieval. (c) During report generation, the LLM autonomously emits a [RAG] token when it needs external evidence. Retrieved sentences are then injected into the context before the model continues generating.

		Clinical Efficacy			NLP
Method	Params	F1	P	R	B-1	B-4	R-L	MET	LLaMA Score
CT2Rep ( Hamamci et al., 2024 )	—	0.160	0.435	0.128	0.372	0.213	—	0.197	—
Merlin ( Blankemeier et al., 2024 )	7B	0.160	0.295	0.112	0.231	0.099	—	0.148	—
CT-CHAT ( Hamamci et al., 2026 )	8B	—	—	—	0.494	—	0.584	0.311	7.440
CT-CHAT ( Hamamci et al., 2026 )	70B	0.184	0.450	0.158	0.498	—	0.581	0.311	7.429
BTB3D ( Hamamci et al., 2025 )	8B	0.258	0.260	0.260	0.439	0.213	—	0.223	—

Table 4: Report generation performance on the CT-RATE validation set. Baseline numbers are from original publications.

Figure 2: Six evaluation metrics across training steps for Two-Stage and Text2Text retrieval pipelines on the 8B model, with bootstrap 95% confidence intervals.

Utilization Strategy	Clin-F1	$\Delta$	BLEU-4	ROUGE-L
Retrieval disabled	0.462	—	0.228	0.349
Fixed-interval ( $N$ =3)	0.453	$-$ 0.009	0.192	0.307
Fixed-interval ( $N$ =5)	0.494	+0.032	0.209	0.329
Fixed-interval ( $N$ =7)	0.482	+0.020	0.228	0.346
Adaptive RAG (ours)	0.480	+0.018	0.242	0.354

Table 5: Context utilization ablation (8B model). All variants share the same retrieval pipeline.

\Delta

is relative to the retrieval-disabled setting.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Domain / Model	top2 AUC	tail 1/2 AUC	$\Delta$
CLIP ViT-B/32 (ImageNet, 30 classes)	0.821	1.000	+0.178
CT-CLIP zero-shot, avg pool (CT-RATE, 18 abnormalities)	0.552	0.746	+0.193
CT-CLIP zero-shot, max pool (CT-RATE, 18 abnormalities)	0.533	0.739	+0.206

Appendix

Table 6: Cross-domain projection test (mean over all classes or abnormalities). top2: AUC using only the 2 highest-variance PCs. tail 1/2 : AUC using the lower half of all PCs.

\Delta

: tail 1/2

-

top2.

Abnormality	top2 AUC	tail 1/2 AUC	$\Delta$
Medical material	0.532	0.770	+0.238
Arterial wall calcification	0.514	0.708	+0.194
Coronary artery wall calc.	0.501	0.713	+0.212
Lymphadenopathy	0.519	0.677	+0.158
Lung nodule	0.545	0.651	+0.106
Pericardial effusion	0.514	0.817	+0.303

Appendix

Table 7: Per-abnormality projection test on CT-CLIP zero-shot. All 18 abnormalities show tail 1/2

>

top2.

Organ	Sentences	Unique reports	Avg. sents/report	Avg. words/sent	Parser F1
Lung	398,538	46,380	8.59	13.3	0.974
Heart	82,025	42,777	1.92	8.4	0.957
Esophagus	50,096	39,977	1.25	11.8	0.990
Aorta	41,632	25,093	1.66	10.0	0.955
Total	572,291	—	—	—	—

Appendix

Table 8: Per-organ sentence database statistics (training set only).

Lung (11)
Abnormality	Base F1	AdaRAG F1	$\Delta$
Lung nodule	0.565	0.591	+0.025
Mosaic attenuation	0.349	0.379	+0.030
Peribronch. thick.	0.245	0.160	$-$ 0.085
Consolidation	0.524	0.534	+0.009
Bronchiectasis	0.096	0.151	+0.054

Appendix

Table 9: Per-abnormality Clinical F1 breakdown: base model (8B, no RAG) vs. AdaRAG-CT (8B, two-stage).

\Delta

: change due to adaptive retrieval augmentation.

Setting	Clin-P	Clin-R	Clin-F1	B-1	B-4	R-L
AdaRAG-CT 8B	0.502	0.520	0.480	0.496	0.242	0.354
Random context	0.488	0.501	0.462	0.513	0.249	0.357
Wrong-organ abnormal	0.485	0.505	0.462	0.513	0.249	0.358

Appendix

Table 10: Retrieval sanity checks on CT-RATE validation. Corrupt-context rows are averaged over seeds 42, 66, and 2026.

		Clinical Efficacy			NLP
Method	Params	F1	P	R	B-1	B-4	R-L	MET	LLaMA
CT-CHAT [ Hamamci et al., 2026 ]	8B	0.224	0.331	0.366	0.421	0.188	0.303	0.193	6.73
CT-CHAT [ Hamamci et al., 2026 ]	70B	0.161	0.334	0.131	0.356	0.182	0.321	0.190	6.02
Ours (base)	8B	0.455	0.474	0.469	0.463	0.205	0.315	0.206	7.30
AdaRAG-CT	8B	0.480	0.502	0.520	0.496	0.242	0.354	0.246	7.75
Ours (base)	70B	0.405	0.468	0.373	0.449	0.213	0.334	0.208	7.10

Appendix

Table 11: Reproduced baselines under our unified evaluation protocol (CT-RATE validation set).

Figure 3: Qualitative comparison for an aorta abnormality. Left: base model output. Centre: AdaRAG-CT output with the regenerated sentence highlighted. Right: ground-truth report. Red highlights indicate hallucinated content. Green indicates correct clinical abnormalities.

Explore similar work

Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation
May 29, 2026Tom Maye-Lasserre, Yitong Li, Bailiang Jian +3Radiology Report Generation Report
Technical University of Munich (TUM), Germany · TUM Hospital, Germany · Munich Center for Machine Learning (MCML), Germany
RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography
Apr 16, 2026Mélanie Roschewitz, Kenneth Styppa, Yitian Tao +10Radiology Report Generation Medical Vision-Language Models
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland · ETH AI Center, Zurich, Switzerland · Department of Computer Science, ETH Zurich, Zurich, Switzerland +5
MonteRET: AI Agent Enhancing Multimodal LLMs with Multi-granularity Knowledge Retrieval for Chest CT Report Generation
Jul 15, 2026Yi Lin, Yihao Ding, Elana Benishay +8Radiology Report Generation Cone-Beam Computed Tomography
Department of Population Health Sciences, Weill Cornell Medicine, New York, USA · School of Physics, Mathematics and Computing, University of Western Australia, Crawley, Australia · Department of Radiology, Weill Cornell Medicine, New York, USA +4

May 29, 2026, cs.CVJ/K move · Enter open · S save

Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation

Tom Maye-Lasserre, Yitong Li, Bailiang Jian, Morteza Ghahremani+2

Technical University of Munich (TUM), Germany · TUM Hospital, Germany · Munich Center for Machine Learning (MCML), Germany

Radiology Report Generation Report Cone-Beam Computed Tomography Medical Vision-Language Models 3D Generative Models Representation Collapse

Open paper AskPDF arXiv

Modern 3D medical vision-language models (VLMs) can generate fluent radiology-style text while exhibit critically low pathology detection and output diversity, collapsing to generic templates that under-report rare yet critical findings. We identify this failure mode as Template Collapse. This failure stems from the unique constraints of 3D medical imaging, e.g., limited data, severe label imbalance, and weak signals from volumetric encoders. Under these constraints, text-generation objectives encourage shortcut learning and fluent but weakly grounded reports. We systematically diagnose the Template Collapse through clinical fidelity, output diversity, normal-template bias, and rare-finding survival. To mitigate it, we propose CLarGen, a decoupled framework that separates what to say (clinical detection) from how to say it (language synthesis). CLarGen uses (i) a Latent Query Transformer for multi-label pathology detection, (ii) pathology-guided retrieval for clinically matched exemplars, and (iii) a medical language model to synthesize the final report from detected findings and retrieved context. Across state-of-the-art 3D CT report generation baselines, CLarGen mitigates Template Collapse and substantially improves clinical accuracy (macro-F1 0.487 vs. 0.189; CRG 0.472 vs. 0.368) while maintaining fluent reporting. Our results suggest that explicit, measurable clinical grounding is essential for template-collapse-resistant 3D CT report generation. Code is available at https://github.com/ai-med/CLarGen.

Abstract

Explore similar work

Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation

RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography

MonteRET: AI Agent Enhancing Multimodal LLMs with Multi-granularity Knowledge Retrieval for Chest CT Report Generation