cs.CVNov 18, 2026

Rethinking Contrastive Targets in Radiology Language-Image Pretraining

Authors: Yunhe Gao, Ashwin Kumar, Jiaming Liu, Chong Wang, Maya Varma, Fuying Wang, Stefania L. Moroianu, Louisa Fay, 

Organizations: Center for Artificial Intelligence in Medicine and Imaging (AIMI), Stanford University

Abstract

CLIP-style contrastive image-text pretraining is widely used in radiology data, where each image is paired with one report and every other report in the batch is treated as a negative. Under this recipe, two failures appear, neither of which is systematically addressed by existing contrastive learning theory: zero-shot evaluation peaks then degrades while training loss keeps decreasing, and training fails to learn from large batches. We hypothesize the cause is structural rather than radiology-specific: many unpaired reports share clinical findings with a given image, so CLIP's paired-identity target is systematically biased, not merely noisy. To test this, we construct CLEVR-Radiology, a synthetic testbed that preserves only this overlapping-fact structure and reproduces both failures. With evidence that the structure is sufficient to reproduce the failures, we develop a target-misspecification theory that explains both failures from a single deterministic gradient bias amplified by batch size, and use it to motivate a minimal compatibility-based correction, CorLIP: text is treated as positive whenever its clinical assertions agree with the image, with no architecture changes. Beyond benchmark improvements, CorLIP largely removes the post-peak degradation, makes large-batch training succeed where it previously collapsed, substantially amplifies the gains from more data, and is the only configuration that remains stable up to B=2048. The correction transfers from chest X-ray to 3D CT modalities.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Findings-aware Language-Image Pretraining for CT Volume Interpretation

    Nov 2, 2026Takuro Shimaya, Junya Inoh, Masahiro SaikoMedical VLMsMedical Image Representation Learning

  2. Meta-CLIP: Metadata-Conditioned Contrastive Learning for Zero-Shot Disease Inference

    Oct 24, 2026Haozhe Luo, Nirosh Sivanesan, Shelley Zixin Shu +2Medical VLMsContrastive Learning for VLMs

  3. TextSLIP: Text Self-Supervised CLIP for Medical Report Generation

    Jul 24, 2026Haoyu Jiang, Ziping CongRadiology Report GenerationSelf-Supervised Learning