Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

V. K. Cody Bumgardner; Mitchell A. Klusty; Mahmut S. Gokmen; Evan W. Damron

cs.CVMar 24, 2026

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

Authors: V. K. Cody Bumgardner, Mitchell A. Klusty, Mahmut S. Gokmen, Evan W. Damron

Organizations: Center for Applied Artificial Intelligence, University of Kentucky, Lexington, KY 40506 USA

Abstract

Automated radiology report generation from 3D computed tomography (CT) volumes is challenging due to extreme sequence lengths, severe class imbalance, and the tendency of large language models (LLMs) to ignore visual tokens in favor of linguistic priors. We present Ker-VLJEPA-3B, a four-phase curriculum learning framework for free-text report generation from thoracic CT volumes. A phased training curriculum progressively adapts a Llama 3.2 3B decoder to ground its output in visual features from a frozen, self-supervised encoder. Our visual backbone (LeJEPA ViT-Large) is trained via self-supervised joint-embedding prediction on unlabeled CTs, without text supervision. Unlike contrastive models (CLIP, BiomedCLIP), this language-free backbone yields modality-pure representations. Vision-language alignment is deferred to the curriculum's bridge and generation phases. This modality-agnostic design can integrate any self-supervised encoder into an LLM without paired text during foundation training. Methodological innovations include: (1) zone-constrained cross-attention compressing slice embeddings into 32 spatially-grounded visual tokens; (2) PCA whitening of anisotropic LLM embeddings; (3) a positive-findings-only strategy eliminating posterior collapse; (4) warm bridge initialization transferring projection weights; and (5) selective cross-attention freezing with elastic weight consolidation to prevent catastrophic forgetting. Evaluated on the CT-RATE benchmark (2,984 validation volumes, 18 classes), Ker-VLJEPA-3B achieves a macro F1 of 0.429, surpassing the state-of-the-art (U-VLM, macro F1 = 0.414) by 3.6%, and reaching 0.448 (+8.2%) with threshold optimization. Ablation studies confirm 56.6% of generation quality derives from patient-specific visual content. Code and weights are available.

Figures & tables

Fig. 1 : Overview of the Ker-VLJEPA-3B architecture. CT slice embeddings from a frozen LeJEPA ViT-Large are compressed into 32 spatially-grounded tokens via zone-constrained cross-attention, projected to LLM space via the JEPA predictor with norm calibration, and grafted into Llama 3.2 3B at both the embedding level and intermediate layers (7, 14, 21) through gated cross-attention adapters. Auxiliary branches provide supervision via the JEPA embedding head (256-d whitened space) and an 18-class classifier.

Metric	Before (3072d)	After (256d)
Mean pairwise cosine sim.	0.949	$-$ 0.001
Discriminability ( $d^{\prime}$ )	1.36	16.03
Variance retained	—	97.3%

TABLE I : Effect of PCA Whitening on Text Embedding Quality

Fig. 2 : Four-phase curriculum learning pipeline. Each phase progressively builds capability from visual discrimination (Phase 1) through contrastive alignment (Phase 2) to free-text generation (Phase 3) and narrative adaptation (Phase 4). Colored boxes indicate trainable components; dashed boxes indicate frozen components. The warm bridge initialization (red dashed arrow) transfers 27 converged bridge components and 392 LoRA tensors from a prior Phase 3 run, providing immediate convergence and eliminating the cold-start problem.

Run	Best F1	Epochs to	Outcome
		collapse
Baseline	0.198	2	Never sustained
+ Cross-attn fix	0.304	1	Immediate collapse
+ Vis. dropout	0.262	1	Faster collapse
+ LLM visual cls.	0.259	4	Slight delay
+ Pos.-findings	0.427	none	Sustained $>$ 0.40

TABLE II : Phase 3 Posterior Collapse: Historical Experimental Comparison

Ver.	Configuration	Best F1
v1	No freeze, no EWC	0.297 $\downarrow$
v2	Freeze LoRA (3 epochs)	0.289 $\downarrow$
v3	EWC + JEPA + MMD	0.274
v4	EWC only	0.304 $\downarrow$
v5	Freeze cross-attn + EWC	0.418

TABLE III : Phase 4 Experimental History: Discovering the Correct Configuration

Method	Macro F1	Macro Prec	Macro Rec
CT-CLIP [ 2 ]	0.194	—	—
CT-CHAT [ 3 ]	0.287	—	—
BTB3D [ 4 ]	0.354	—	—
U-VLM [ 5 ]	0.414	0.491	0.429
Ker-VLJEPA-3B (P3)	0.422	0.380	0.517
Ker-VLJEPA-3B (P4)	0.429	0.389	0.524

TABLE IV : Comparison with State-of-the-Art on CT-RATE Benchmark (2,984 Validation Volumes)

Phase	Type	F1	Prec	Rec	AUC
1	Cls.	0.460	0.463	0.551	0.811
2	Cls.	0.465	0.490	0.532	0.816
3	Gen.	0.422	0.380	0.517	—
4	Gen.	0.429	0.389	0.524	—

TABLE V : Performance Progression Across Curriculum Phases (Full Validation Set)

Metric	Cold	Cold +	Warm
	Baseline	Better P2	Bridge
Best F1	0.427 (ep16)	0.424 (ep7)	0.446 (ep9)
Best Precision	0.391	0.366	0.437
Epoch 1 F1	0.360	0.362	0.425
Epochs to $>$ 0.42	5	5	1
Val. loss floor	1.130	1.069	1.056

TABLE VI : Cold vs. Warm Bridge Phase 3 Training Comparison

Class	Prec	Rec	F1	Support
Pleural effusion	0.574	0.789	0.664	370
Art. wall calcification	0.642	0.660	0.651	849
Coronary artery calc.	0.580	0.585	0.582	752
Cardiomegaly	0.437	0.743	0.550	315
Lung nodule	0.491	0.560	0.523	1344
Lung opacity	0.545	0.453	0.494	1173

TABLE VII : Per-Class Results (Phase 4, 2,984 Validation Volumes)

Stage	Dim	F1 (macro)
Raw LeJEPA embeddings	1024	0.447
Phase 1 (visual context)	1024	0.459
Phase 1 (norm-matched)	3072	0.488
Phase 2 (norm-matched)	3072	0.495
Phase 3 (norm-matched)	3072	0.495
Phase 4 (norm-matched)	3072	0.495

TABLE VIII : Linear Probe F1 Across Pipeline Stages

Condition	P3 F1	P3 Prec	P4 F1	P4 Prec
Normal	0.222	0.408	0.190	0.298
Zeroed	0.096	0.114	0.104	0.146
Random	0.121	0.156	0.121	0.129
Shuffled	0.122	0.204	0.119	0.137

TABLE IX : Generation Ablation: F1 Under Different Visual Token Conditions

Condition	P3 NLL	$\Delta$	P4 NLL	$\Delta$
Correct	1.200	—	1.196	—
Zeroed	1.211	+0.9%	1.209	+1.1%
Shuffled	1.264	+5.3%	1.259	+5.3%

TABLE X : NLL Ablation: Impact of Visual Token Manipulation

∗ Phase 4 generates raw narrative; direct comparison with
Configuration	F1	$\Delta$ vs. Base
Cross-attn fix only	0.304	—
+ Positive-findings-only	0.427	+0.123
+ Warm bridge	0.446	+0.142
+ Phase 4 (freeze xattn + EWC)	0.429 ∗	+0.125
positive-findings Phase 3 reflects task difference, not regression.

TABLE XI : Component Ablation: Impact of Key Innovations

Explore similar work

Mar 16, 2026

Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation

Automated radiology report generation from 3D CT volumes often suffers from incomplete pathology coverage. We provide empirical evidence that this limitation stems from a representational bottleneck: contrastive 3D CT embeddings encode discriminative pathology signals, yet exhibit severe dimensional concentration, with as few as 2 effective dimensions out of 512. Corroborating this, scaling the language model yields no measurable improvement, suggesting that the bottleneck lies in the visual representation rather than the generator. This bottleneck limits both generation and retrieval; naive static retrieval fails to improve clinical efficacy and can even degrade performance. We propose AdaRAG-CT, an adaptive augmentation framework that compensates for this visual bottleneck by introducing supplementary textual information through controlled retrieval and selectively integrating it during generation. On the CT-RATE benchmark, AdaRAG-CT achieves state-of-the-art clinical efficacy, improving Clinical F1 from 0.420 (CT-Agent) to 0.480 (+6 points); ablation studies confirm that both the retrieval and generation components contribute to the improvement. Code is available at https://github.com/renjie-liang/Adaptive-RAG-for-3DCT-Report-Generation.

Renjie Liang, Yiling Ma, Yang Xing +6

University of Florida · Yale University · University of Southern California

Radiology Report Generation Multimodal RAG Retrieval-Augmented Generation Healthcare Adaptive RAG Computed Tomography

Ask PDF

Mar 18, 2026

DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation

While large language models (LLMs) have advanced CT report generation, existing methods typically encode 3D volumes holistically, failing to distinguish informative cues from redundant anatomical background. Inspired by radiological cognitive subtraction, we propose Differential Visual Prompting (DiffVP), which conditions report generation on explicit, high-level semantic scan-to-reference differences rather than solely on absolute visual features. DiffVP employs a hierarchical difference extractor to capture complementary global and local semantic discrepancies into a shared latent space, along with a difference-to-prompt generator that transforms these signals into learnable visual prefix tokens for LLM conditioning. These difference prompts serve as structured conditioning signals that implicitly suppress invariant anatomy while amplifying diagnostically relevant visual evidence, thereby facilitating accurate report generation without explicit lesion localization. On two large-scale benchmarks, DiffVP consistently outperforms prior methods, improving the average BLEU-1-4 by +10.98 and +4.36, respectively, and further boosts clinical efficacy on RadGenome-ChestCT (F1 score 0.421). All codes will be released at https://github.com/ArielTYH/DiffVP/.

Yuhe Tian, Kun Zhang, Haoran Ma +4

Department of Electronic Engineering and Information Science, School of Information Science and Technology, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China +2

Radiology Report Generation Medical VLMs Visual Prompt Learning

Ask PDF

Aug 9, 2026

Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation

Vision-language models offer a promising path toward automating radiology report generation, but applying them to full 3D CT volumes poses substantial computational challenges. Modern foundation vision encoders (VEs) can produce tens of thousands of vision tokens per scan, making the visual sequence passed to the large language model (LLM) a primary computational bottleneck. Vision-to-language projectors can compress this sequence to reduce computation, but may discard clinically relevant detail; conversely, effective compression can accommodate higher-resolution inputs while keeping the downstream token count fixed. How this vision-token budget should be allocated across input field of view, spatial resolution, and vision-to-language projection therefore remains an open design question. We systematically evaluate four heterogeneous VEs (CNN- and ViT-based), five token-reducing projectors at up to 64x compression alongside a non-reducing MLP projector baseline, and five instruction-tuned LLMs (1.7B--4B) on two large-scale CT report datasets (CT-RATE and Merlin). At matched LLM token budgets, anatomy-guided region of interest cropping is the most consistent strategy, improving clinical macro F1 in 19 of 20 settings by +3.7 points on average for the 3D ViT Primus encoder and +1.1 for the slice-based 2D ViT Curia encoder. Increasing input resolution further is strongly projector-dependent: the PerceiverResampler, paired with higher-resolution Curia features, yields the strongest configuration in the resolution study on both datasets. Our best configurations achieve state-of-the-art clinical macro F1 on the test sets, reaching 49.5 on CT-RATE and 49.0 on Merlin. Code and models will be published upon publication.

Jonathan Suprijadi, Raphael Stock, Moritz Langenberg +10

Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany

Radiology Report Generation Radiology VLMs Computed Tomography Visual Token Compression

Ask PDF