SelfCue: Making a 3D CT Report Generator Say What It Already Knows

Renjie Liang; Yang Yang; Jinqian Pan; Zhengkang Fan; Chengkun Sun; Jie Xu

cs.CVSep 25, 2026

SelfCue: Making a 3D CT Report Generator Say What It Already Knows

Authors: Renjie Liang, Yang Yang, Jinqian Pan, Zhengkang Fan, Chengkun Sun, Jie Xu

Organizations: University of Florida, Gainesville, FL, USA

Abstract

Progress in 3D CT report generation is usually sought in increasingly sophisticated architectures and larger pools of training data. We find instead that a 3D CT report generator already holds what its report leaves out, and loses it when the hidden state becomes tokens. Over the 18 CT-RATE abnormalities, this hidden-to-report surfacing gap is reflected by a drop in macro AUROC from 0.848 in the hidden states to 0.739 in the generated report. We propose SelfCue based on contrastive decoding. It promotes what the hidden state already supports and suppresses what it does not. It raises clinical efficacy F1 to 0.481 and the LLM-judged GREEN score to 0.510. Distilling that behaviour into the weights gives SelfCue-KD, a student that keeps most of the gain, needs nothing extra at inference, and drops into any pipeline already serving the baseline. Code is available at https://github.com/renjie-liang/SelfCue-CT.

Figures & tables

Figure 1: A. Macro AUROC over the 18 CT-RATE abnormalities at each stage of three report generators, the raw volume $1.0$ by assumption. B. The abnormality condition is read by a probe, and each next token is chosen by contrasting the conditioned and plain branches. C. The teacher’s reports train the student, which reads no condition. D. The two constructions of the training set conditions. Red marks a probe error.

Eligible works
System	[email protected]	$\mathrm{F1_{max}}$	AUROC	GREEN	BLEU-1	BLEU-4	ROUGE-L	METEOR
CT-CHAT ( Hamamci et al., 2026 )	$0.184$	–	–	–	$0.395$	–	$0.334$	–
RadAgent ( Roschewitz et al., 2026 )	$0.220$	–	–	–	–	–	–	–
BTB3D ( Hamamci et al., 2025 )	$0.258$	–	–	–	$0.439$	$0.213$	–	$0.223$
OmniCT ( Lin et al., 2026 )	$0.363$	–	–	–	–	–	–	–
U-VLM ( Shi et al., 2026 )	$0.414$	–	–	–	$0.474$	$0.256$	–	–

Table 1: Comparison on CT-RATE. SelfCue decodes at

\alpha=1.0

and

\tau=0.75

on all three encoder–backbone pairs.

M_{\mathrm{cond}}

is decoded with the condition in its prompt and no contrast.

Panel of Figure 2 — Figure 2: SelfCue with the gate on empty conditions over $\alpha$ and $\tau$ on COLIPRI with Llama-3.1-8B. $M_{\mathrm{cond}}$ is decoded with the condition at the same $\tau$ and no contrast.

	$\tau=0.70$			$\tau=0.75$			$\tau=0.90$
$\alpha$	[email protected]	$\mathrm{F1_{max}}$	GREEN	[email protected]	$\mathrm{F1_{max}}$	GREEN	[email protected]	$\mathrm{F1_{max}}$	GREEN
$0.25$	$0.000$	$0.000$	$-0.001$	$0.000$	$0.000$	$0.000$	$+0.001$	$+0.002$	$-0.002$
$1.0$	$0.000$	$0.000$	$+0.002$	$0.000$	$0.000$	$0.000$	$+0.004$	$+0.003$	$+0.003$
$2.0$	$+0.003$	$+0.003$	$+0.022$	$+0.004$	$+0.004$	$+0.029$	$+0.008$	$+0.009$	$+0.038$
$3.0$	$+0.007$	$+0.006$	$+0.057$	$+0.007$	$+0.005$	$+0.070$	$+0.015$	$+0.012$	$+0.097$

Table 2: Gate on empty conditions. It sets

\alpha

0

on volumes whose condition is empty, so they are decoded without contrast. Cells are gated minus ungated SelfCue on COLIPRI with Llama-3.1-8B.

	Condition source			Training set condition
Where it is used	[email protected]	$\mathrm{F1_{max}}$	GREEN	[email protected]	$\mathrm{F1_{max}}$	GREEN
Training $M_{\mathrm{cond}}$	$+0.008$	$+0.004$	$-0.001$	$-0.026$	$-0.021$	$-0.017$
SelfCue decoding	$+0.008$	$0.000$	$-0.003$	–	–	–
Teacher corpus	$+0.005$	$-0.003$	$-0.004$	$-0.151$	$-0.063$	$-0.013$

Table 3: Condition source. It is either the LLM hidden states or the visual embedding, and the training set conditions are ground truth with injected noise or 5-fold cross-fitted predictions. Each cell is the change when one choice is replaced by the other, on COLIPRI with Llama-3.1-8B.

	$M_{\mathrm{base}}$	$M_{\mathrm{cond}}$ , plain prompt	$M_{\mathrm{cond}}$	SelfCue	SelfCue-KD	Token-level KD
[email protected]	$0.395$	$0.299$	$0.430$	$0.481$	$0.462$	$0.452$
$\mathrm{F1_{max}}$	$0.482$	$0.438$	$0.514$	$0.542$	$0.534$	$0.532$
AUROC	$0.739$	$0.709$	$0.762$	$0.775$	$0.768$	$0.765$
GREEN	$0.466$	$0.495$	$0.510$	$0.510$	$0.510$	$0.512$
In condition, present	$0.505$	$0.364$	$0.697$	$0.901$	$0.729$	$0.695$
In condition, absent	$0.386$	$0.259$	$0.610$	$0.855$	$0.578$	$0.554$

Table 4: Sequence-level distillation. The last four rows are mention rates, split by whether the condition lists the abnormality and whether it is present, averaged over the 18 abnormalities. Results are on COLIPRI with Llama-3.1-8B.

	[email protected]	$\mathrm{F1_{max}}$	AUROC	GREEN
$M_{\mathrm{base}}$	$0.395$ $0.424\pm 0.008$	$0.482$ $0.487\pm 0.006$	$0.739$ $0.750\pm 0.006$	$0.466$ $0.431\pm 0.004$
$M_{\mathrm{cond}}$	$0.430$ $0.430\pm 0.006$	$0.514$ $0.510\pm 0.007$	$0.762$ $0.757\pm 0.001$	$0.510$ $0.510\pm 0.006$
SelfCue	$0.481$ $0.489\pm 0.012$	$0.542$ $0.541\pm 0.005$	$0.775$ $0.774\pm 0.005$	$0.510$ $0.503\pm 0.007$
SelfCue-KD	$0.462$ $0.463\pm 0.001$	$0.534$ $0.529\pm 0.002$	$0.768$ $0.763\pm 0.002$	$0.510$ $0.510\pm 0.008$

Table 5: Held-out selection. COLIPRI with Llama-3.1-8B on the validation split. Each cell gives the mean and standard deviation over three seeds, after the reported value in grey.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	all prompt positions	visual prompt only	last prompt position
COLIPRI, Llama-3.1-8B	$0.848$	$0.848$	$0.842$
VISD+CT-CLIP, Llama-3.1-8B	$0.845$	$0.838$	$0.835$
COLIPRI, MedGemma-27B	$0.852$	$0.851$	$0.847$

Appendix

Table 6: Probe pooling. Macro AUROC over the 18 CT-RATE abnormalities at each probe’s best layer.

Figure 3: Thresholds on RadBERT’s probability for COLIPRI with Llama-3.1-8B. (a) Markers give each curve’s peak and dotted lines the reported

\mathrm{F1_{max}}

. (b) One colour per abnormality.

Figure 4: Case study on one CT volume. Orange marks consolidation and the symbols show whether each report mentions it.

Figure 5: Clinical efficacy at every checkpoint of the conditional runs. Open circles mark the checkpoint carried into every other result in the paper.

COLIPRI $+$ Llama-3.1-8B
System	F1	$\mathrm{F1_{max}}$	Precision	Recall	Mentioned / volume
Condition ( $\tau=0.75$ )	$0.500$	–	$0.574$	$0.491$	$2.54$
$M_{\mathrm{base}}$	$0.395$	$0.482$	$0.509$	$0.338$	$2.37$
$M_{\mathrm{cond}}$	$0.430$	$0.514$	$0.590$	$0.365$	$2.01$
SelfCue	$0.481$	$0.542$	$0.577$	$0.452$	$2.40$
SelfCue-KD	$0.462$	$0.534$	$0.580$	$0.434$	$2.25$

Appendix

Table 7: Precision and recall as macro averages over the 18 abnormalities. The condition row is the probe at

\tau=0.75

3.43

abnormalities are present per volume on COLIPRI and

3.47

on VISD

+

CT-CLIP.

COLIPRI $+$ Llama-3.1-8B
System	$n$	GREEN	Matched	False finding	Missing	Location	Words
$M_{\mathrm{base}}$	$1548$	$0.470$	$5.01$	$1.47$	$3.45$	$0.22$	$194$
$M_{\mathrm{cond}}$	$1555$	$0.513$	$5.39$	$1.17$	$3.15$	$0.15$	$177$
SelfCue	$1560$	$0.511$	$5.36$	$1.19$	$3.16$	$0.17$	$174$
SelfCue-KD	$1558$	$0.512$	$5.63$	$1.09$	$3.47$	$0.14$	$172$
VISD $+$ CT-CLIP $+$ Llama-3.1-8B

Appendix

Table 8: GREEN components as mean counts per report.

n

excludes the reports whose judge output could not be parsed.

		COLIPRI, Llama-3.1-8B		VISD+CT-CLIP, Llama-3.1-8B		COLIPRI, MedGemma-27B
Abnormality	Prevalence	Gap	$\Delta\mathrm{F1_{max}}$	Gap	$\Delta\mathrm{F1_{max}}$	Gap	$\Delta\mathrm{F1_{max}}$
Pericardial effusion	$7.0\%$	$0.123$	$+0.151$	$0.151$	$+0.176$	$0.121$	$+0.073$
Interlobular septal thickening	$7.9\%$	$0.094$	$+0.059$	$0.055$	$+0.071$	$0.070$	$+0.074$
Mosaic attenuation pattern	$8.1\%$	$0.139$	$+0.148$	$0.144$	$+0.176$	$0.176$	$+0.165$
Medical material	$10.1\%$	$0.169$	$+0.105$	$0.107$	$+0.095$	$0.195$	$+0.279$
Cardiomegaly	$10.5\%$	$0.085$	$+0.090$	$0.089$	$+0.104$	$0.108$	$+0.171$

Appendix

Table 9: Gap and gain per abnormality. Both columns are read against

M_{\mathrm{base}}

, the gap as probe minus report in AUROC and the gain as SelfCue minus report in

\mathrm{F1_{max}}

. Prevalence is on the COLIPRI split.

Panel of Figure 6 — Figure 6: SelfCue with the gate on empty conditions over $\alpha$ and $\tau$ on the other two pairs. The threshold is swept from $0.70$ on MedGemma-27B.

	[email protected]	$\mathrm{F1_{max}}$	AUROC	GREEN
$M_{\mathrm{base}}$	$0.395$	$0.482$	$0.739$	$0.466$
SelfCue, probe condition	$0.480$	$0.542$	$0.775$	$0.510$
SelfCue, labels as the condition	$0.869$	$0.873$	$0.937$	$0.511$

Appendix

Table 10: The oracle ceiling. The same decoding rule with the condition read from the probe and from the ground-truth labels, on COLIPRI with Llama-3.1-8B.

Where the student starts
	[email protected]	$\mathrm{F1_{max}}$	AUROC
From $M_{\mathrm{cond}}$	$0.483$	$0.517$	$0.770$
From $M_{\mathrm{base}}$	$0.476$	$0.515$	$0.772$
From the stage-1 projector	$0.479$	$0.517$	$0.772$
From scratch	$0.474$	$0.510$	$0.769$
Which teacher reports it sees

Appendix

Table 11: Student variants. All rows are on COLIPRI with Llama-3.1-8B, and the first row of each block is the choice the paper makes.

CT Report Generation

See all 22 papers

Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation
Mar 16, 2026Renjie Liang, Yiling Ma, Yang Xing +6Healthcare Multimodal RAG
University of Florida · Yale University · University of Southern California
Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression
Mar 24, 2026V. K. Cody Bumgardner, Mitchell A. Klusty, Mahmut S. Gokmen +1CT Report Generation Radiology Report Generation
Center for Applied Artificial Intelligence, University of Kentucky, Lexington, KY 40506 USA
DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation
Mar 18, 2026Yuhe Tian, Kun Zhang, Haoran Ma +4CT Report Generation Radiology Report Generation
Department of Electronic Engineering and Information Science, School of Information Science and Technology, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China +2

Mar 16, 2026J/K move · Enter open · S save

Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation

Renjie Liang, Yiling Ma, Yang Xing, Zhengkang Fan+5

University of Florida · Yale University · University of Southern California

Healthcare Multimodal RAG Retrieval-Augmented Generation Radiology Report Generation Adaptive RAG Computed Tomography+1

Open paper AskPDF arXiv

Automated radiology report generation from 3D CT volumes often suffers from incomplete pathology coverage. We provide empirical evidence that this limitation stems from a representational bottleneck: contrastive 3D CT embeddings encode discriminative pathology signals, yet exhibit severe dimensional concentration, with as few as 2 effective dimensions out of 512. Corroborating this, scaling the language model yields no measurable improvement, suggesting that the bottleneck lies in the visual representation rather than the generator. This bottleneck limits both generation and retrieval; naive static retrieval fails to improve clinical efficacy and can even degrade performance. We propose AdaRAG-CT, an adaptive augmentation framework that compensates for this visual bottleneck by introducing supplementary textual information through controlled retrieval and selectively integrating it during generation. On the CT-RATE benchmark, AdaRAG-CT achieves state-of-the-art clinical efficacy, improving Clinical F1 from 0.420 (CT-Agent) to 0.480 (+6 points); ablation studies confirm that both the retrieval and generation components contribute to the improvement. Code is available at https://github.com/renjie-liang/Adaptive-RAG-for-3DCT-Report-Generation.

Abstract

CT Report Generation

Explore similar work

Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation