CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models

Xiang Chen; Fangfang Yang; Chunlei Meng; Yuxian Dong; Ang Li; Yiwei Wei; Jiahuan Long; Jiujiang Guo

cs.CVMar 19, 2026

CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models

Authors: Xiang Chen, Fangfang Yang, Chunlei Meng, Yuxian Dong, Ang Li, Yiwei Wei, Jiahuan Long, Jiujiang Guo,

Abstract

Medical vision--language models (MVLMs) are increasingly used as perceptual backbones in radiology pipelines and as the visual front end of multimodal assistants, yet their reliability under real clinical workflows remains underexplored. Prior robustness evaluations often assume clean, curated inputs or study isolated corruptions, overlooking routine acquisition, reconstruction, display, and delivery operations that preserve clinical readability while shifting image statistics. To address this gap, we propose CoDA, a chain-of-distribution framework that constructs clinically plausible pipeline shifts by composing acquisition-like shading, reconstruction and display remapping, and delivery and export degradations. Under masked structural-similarity constraints, CoDA jointly optimizes stage compositions and parameters to induce failures while preserving visual plausibility. Across brain MRI, chest X-ray, and abdominal CT, CoDA substantially degrades the zero-shot performance of CLIP-style MVLMs, with chained compositions consistently more damaging than any single stage. We also evaluate multimodal large language models (MLLMs) as technical-authenticity auditors of imaging realism and quality rather than pathology. Proprietary multimodal models show degraded auditing reliability and persistent high-confidence errors on CoDA-shifted samples, while the medical-specific MLLMs we test exhibit clear deficiencies in medical image quality auditing. Finally, we introduce a post-hoc repair strategy based on teacher-guided token-space adaptation with patch-level alignment, which improves accuracy on archived CoDA outputs. Overall, our findings characterize a clinically grounded threat surface for MVLM deployment and show that lightweight alignment improves robustness in deployment.

Figures & tables

Figure 1. Overview of CoDA. (a)~(c) Three pipeline stages: acquisition-like shading ( $A$ ), reconstruction and display remapping ( $R$ ), and delivery and export degradations ( $D$ ), illustrated on brain MRI, chest X-ray, and abdominal CT. (d) CoDA optimizes over seven stage compositions and within-stage parameters under masked similarity constraints to produce visually plausible adversarial samples. (e-1)~(e-2) Technical-authenticity auditing and reliability diagnostics. An overview figure of CoDA. Panels (a) to (c) illustrate three clinically plausible pipeline stages on brain MRI, chest X-ray, and abdominal CT: acquisition-like shading, reconstruction and display remapping, and delivery and export degradations. Panel (d) shows that CoDA searches over seven stage-composition families and their within-stage parameters under masked similarity constraints to generate visually plausible adversarial samples. Panels (e-1) to (e-2) present the technical-authenticity auditing setup and related reliability diagnostics. — Figure 1. Overview of CoDA. (a)(c) Three pipeline stages: acquisition-like shading ( $A$ ), reconstruction and display remapping ( $R$ ), and delivery and export degradations ( $D$ ), illustrated on brain MRI, chest X-ray, and abdominal CT. (d) CoDA optimizes over seven stage compositions and within-stage parameters under masked similarity constraints to produce visually plausible adversarial samples. (e-1)(e-2) Technical-authenticity auditing and reliability diagnostics. An overview figure of CoDA. Panels (a) to (c) illustrate three clinically plausible pipeline stages on brain MRI, chest X-ray, and abdominal CT: acquisition-like shading, reconstruction and display remapping, and delivery and export degradations. Panel (d) shows that CoDA searches over seven stage-composition families and their within-stage parameters under masked similarity constraints to generate visually plausible adversarial samples. Panels (e-1) to (e-2) present the technical-authenticity auditing setup and related reliability diagnostics.

Model	Setting	MRI							X-ray							CT
Model	Setting	$A$	$R$	$D$	$A{+}R$	$R{+}D$	$A{+}D$	$A{+}R{+}D$	$A$	$R$	$D$	$A{+}R$	$R{+}D$	$A{+}D$	$A{+}R{+}D$	$A$	$R$	$D$	$A{+}R$	$R{+}D$	$A{+}D$	$A{+}R{+}D$
BioMed-CLIP	Random ( $\tau=0.90$ )	11	40	33	35	54	26	38	71	61	75	33	45	75	24	69	73	67	66	68	67	74
	Random ( $\tau=0.80$ )	22	61	17	52	62	25	44	75	77	100	53	63	67	61	69	88	50	70	81	75	76
	Bayes ( $\tau=0.90$ )	0	61	25	45	58	7	45	69	68	90	44	42	100	37	62	76	60	64	68	87	79
	Bayes ( $\tau=0.80$ )	0	51	100	61	47	45	68	73	73	90	70	85	100	78	71	76	54	70	92	94	79
UniMed-CLIP	Random ( $\tau=0.90$ )	83	21	71	19	48	80	50	64	50	90	55	100	70	55	76	86	67	84	76	79	90

Table 1. Attack success rate (%). Each entry gives the success rate after optimizing the within-family parameters for its CoDA composition family. Bold indicates the best within each modality block (MRI/X-ray/CT) for the same model and setting.

Setting	BioMed-CLIP			UniMed-CLIP			BMC-CLIP			Rad-CLIP
Setting	MRI	X-ray	CT	MRI	X-ray	CT	MRI	X-ray	CT	MRI	X-ray	CT
Clean	81.5	64.0	57.5	65.0	58.5	54.0	78.5	50.5	54.0	93.0	85.5	50.0
$\tau=0.9$ / Random	62.5	52.5	31.0	51.5	51.0	21.0	62.0	40.5	44.0	76.0	37.0	44.5
$\tau=0.8$ / Random	50.5	32.0	25.0	47.5	46.5	11.5	57.5	38.0	42.5	56.5	31.5	33.0
$\tau=0.9$ / Bayes	56.5	45.5	29.0	48.0	48.5	15.5	59.5	38.0	44.5	71.5	33.5	44.0
$\tau=0.8$ / Bayes	44.0	24.5	22.0	44.0	42.5	8.5	57.0	32.5	40.0	50.5	28.0	30.0

Table 2. Zero-shot accuracy (%). Zero-shot classification performance of CLIP-style MVLMs on clean inputs and under CoDA shifts across MRI, X-ray, and CT, evaluated with random and Bayesian optimization at two plausibility thresholds.

Figure 2. Qualitative visualization of CoDA composition families. Representative clean and adversarial samples for MRI, X-ray, and CT under the CoDA shift space in Eq. 7 . A grid of examples for MRI, X-ray, and CT. Each column corresponds to one CoDA family. For each example, a clean sample (top) and its adversarial counterpart (bottom) are shown, along with the ground-truth label and the flipped prediction.

Attack	BioMed-CLIP			UniMed-CLIP			BMC-CLIP			Rad-CLIP
Attack	MRI	X-ray	CT	MRI	X-ray	CT	MRI	X-ray	CT	MRI	X-ray	CT
Only $A$	75.5	61.0	47.5	62.0	56.5	44.0	77.0	48.0	63.0	73.0	68.0	46.0
Only $R$	57.0	33.0	31.0	56.0	49.0	31.5	61.5	41.5	48.0	56.5	40.0	35.0
Only $D$	76.5	60.5	40.5	56.0	52.5	37.5	69.5	42.0	58.0	65.0	52.0	39.0
CoDA	44.0	24.5	22.0	44.0	42.5	8.5	57.0	32.5	40.0	50.5	28.0	30.0

Table 3. Ablation under Bayesian optimization (

\tau=0.8

). Zero-shot accuracy (%) under single-stage shifts (Only

A

/Only

R

/Only

D

) versus the full chained CoDA shift (

A{+}R{+}D

Figure 3. Auditing performance under clean vs. CoDA shifts. We compare proprietary and medical-specific MLLMs across MRI, X-ray, and CT, reporting clean accuracy, overall accuracy, and accuracy on CoDA-shifted inputs. A grid of horizontal bar charts for multiple models and modalities. Each subplot reports clean, overall, and adversarial accuracy, comparing proprietary multimodal models and medical-specific multimodal models under CoDA shifts.

Figure 4. Post-hoc token-space repair. (a) A frozen teacher optionally guides a lightweight student token adapter trained on clean data with task, view-consistency, and distillation losses. (b) Accuracy on archived CoDA outputs for the baseline and repaired models, comparing unguided (no T) vs. teacher-guided (teacher) adaptation across MRI, X-ray, and CT. A two-panel figure. Panel a shows a teacher-student diagram where the student uses a trainable token-space adapter and optionally matches teacher patch-token geometry. Panel b shows grouped bar charts for MRI, CT, and X-ray comparing baseline clean/attack performance to repair variants without teacher guidance and with teacher guidance.

Figure 5. Ablation of optimization iterations. Attack success rate under CoDA as a function of the optimization iteration budget, reported for MRI, X-ray, and CT across BioMed-CLIP, UniMed-CLIP, BMC-CLIP, and Rad-CLIP. A three-panel line chart for MRI, X-ray, and CT showing attack success rate versus iteration number, with one curve per model.

Figure 6. Ablation of repair hyperparameters. Robust accuracy under CoDA shifts as a function of $\lambda_{\mathit{cons}}$ and $\lambda_{\mathit{dist}}$ , compared against the clean baseline and the attack baseline, reported for MRI/X-ray/CT. A three-panel bar-chart figure for MRI, X-ray, and CT. Bars show robust accuracy under different settings of the view-consistency weight and teacher-guidance weight, with a clean baseline region and an attack baseline indicated by a red dashed line.

Medical Image Quality Assessment

See all 21 papers

Aug 26, 2026

What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes. We study this failure mode as a representation-level blind spot relevant to epistemic intelligence, without claiming a formal estimator of epistemic uncertainty. Using NIH ChestXray14 and CheXpert, we first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics. Using PadChest and OpenI, we then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information retained in frozen embeddings. Self-supervised visual initialization improves NIH-to-CheXpert transfer over supervised ImageNet initialization in matched ResNet-18 comparisons, whereas adversarial adaptation is useful only in a narrow regime and becomes unstable as adversarial pressure increases. Multimodal exact-pair retrieval remains low under external OpenI stress testing, and source-proxy information remains recoverable from learned representations. Qualitative nearest-neighbor and Grad-CAM analyses show clinically plausible cross-dataset structure and thoracic attention patterns in many cases, while device-heavy and false-positive cases remain ambiguous. Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking. Overall, the study shows that apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation of medical VLMs under distribution shift.

Ayoub Louaye Bouaziz, Lokmane Chebouba, Yassine Himeur

LaTIM, Inserm, University of Western Brittany, Brest, France · University of Constantine, Algeria · University of Dubai, UAE

Medical VLMs Radiology VLMs VLM Robustness Domain Generalization Shortcut Learning Image-Text Retrieval

Ask PDF

Mar 22, 2026

Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models

Consistency under paraphrase, the property that semantically equivalent prompts yield identical predictions, is increasingly used as a proxy for reliability when deploying medical vision-language models (VLMs). We show this proxy is fundamentally flawed: a model can achieve perfect consistency by relying on text patterns rather than the input image. We introduce a four-quadrant per-sample safety taxonomy that jointly evaluates consistency (stable predictions across paraphrased prompts) and image reliance (predictions that change when the image is removed). Samples are classified as Ideal (consistent and image-reliant), Fragile (inconsistent but image-reliant), Dangerous (consistent but not image-reliant), or Worst (inconsistent and not image-reliant). Evaluating five medical VLM configurations across two chest X-ray datasets (MIMIC-CXR, PadChest), we find that LoRA fine-tuning dramatically reduces flip rates but shifts a majority of samples into the Dangerous quadrant: LLaVA-Rad Base achieves a 1.5% flip rate on PadChest while 98.5% of its samples are Dangerous. Critically, Dangerous samples exhibit high accuracy (up to 99.6%) and low entropy, making them invisible to standard confidence-based screening. We observe a negative correlation between flip rate and Dangerous fraction (r = -0.89, n=10) and recommend that deployment evaluations always pair consistency checks with a text-only baseline: a single additional forward pass that exposes the false reliability trap.

Binesh Sadanandan, Vahid Behzadan

SAIL Lab, University of New Haven, West Haven, CT, USA

VLM Evaluation Medical VLMs Faithfulness in VLMs Prompt Sensitivity

Ask PDF

Mar 2, 2026

Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine

Large vision-language models (VLMs) often benefit from chain-of-thought (CoT) prompting in general domains, yet its efficacy in medical vision-language tasks remains underexplored. We report a counter-intuitive trend: on medical visual question answering, CoT frequently underperforms direct answering (DirA) across general-purpose and medical-specific models. We attribute this to a medical perception bottleneck: subtle, domain-specific cues can weaken visual grounding, and CoT may compound early perceptual uncertainty rather than correct it. To probe this hypothesis, we introduce two training-free, inference-time grounding interventions: (i) perception anchoring via region-of-interest cues and (ii) description grounding via high-quality textual guidance. Across multiple benchmarks and model families, these interventions improve accuracy, mitigate CoT degradation, and in several settings reverse the CoT--DirA inversion. Our findings suggest that reliable clinical VLMs require robust visual grounding and cross-modal alignment, beyond extending text-driven reasoning chains. Code is available here.

Yuan Wu, Zongxian Yang, Jiayu Qian +5

Department of Computer Science, City University of Hong Kong (Dongguan) · College of Computing and Data Science (CCDS), Nanyang Technological University · School of Computer Science, Northwestern Polytechnical University

Medical VLMs VLM Reasoning Medical VQA Vision-Language Grounding

Ask PDF

Abstract

Medical Image Quality Assessment

Explore similar work

What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models

Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine