cs.CVNov 13, 2026

Cracks in Embeddings: Robustness and Stability of JEPA-Based Chest X-Ray Representations

Authors: Zahid Mansur, Kalpana Yadav, Deepak Mishra

Organizations: Department of Computer Science and Engineering, IIT Jodhpur, India

Abstract

Self-supervised Joint Embedding Predictive Architecture (JEPA) models are increasingly used for medical imaging, however there is a limited understanding about robustness of these models toward real-world image changes. We test four JEPA-based chest X-ray encoders, trained on 78,468 NIH ChestX-ray14 images, under six tests: two black-box adversarial attacks (SPSA, Boundary Attack), three image corruptions (Gaussian noise, brightness shift, contrast shift), and a label-scarcity test, measuring embedding change and classification AUROC. These perturbations cause severe, model-dependent damage: encoder-level Cosine Similarity drops as low as 0.33, and AUROC falls by as much as 0.12 in the worst case (I-JEPA under the Boundary Test). We test two ways of training the small predictor on top of each frozen encoder to address this. Adversarial fine-tuning, trained on clean and perturbed images together, raises embedding consistency above 0.97 in nearly every case and improves AUROC, relative to the same perturbed baseline, for most models, with the largest gain of +0.10+0.10 on I-JEPA. A Vector-Quantized codebook method, trained only on clean images and evaluated on perturbed test images for the first time, instead degrades under distortion for all four models, most noticeably for I-JEPA (−0.15-0.15 AUROC on average), showing that its discrete bottleneck alone does not provide the robustness that comes from training on perturbed data directly.

Figures & tables

Explore similar work

CardsList
  1. RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture

    Jan 22, 2026Anas Anwarul Haq Khan, Mariam Husain, Pratik Jalan +1Radiology Report GenerationMedical Imaging

  2. CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning

    Oct 4, 2026Mintu Dutta, Ritesh Vyas, Mohendra Roy *Causal Representation LearningJoint-Embedding Predictive Architecture

  3. Extending ZACH-ViT to Robust Medical Imaging: Corruption and Adversarial Stress Testing in Low-Data Regimes

    Apr 7, 2026Athanasios Angelakis, Marta Gomez-BarreroVision TransformerMedical Image Classification