cs.CVNov 19, 2026

Rank2Replace: VLM-Ranked Slot Replacement for Out-of-Distribution Generalization

Authors: Bo-Han Feng, Keng-Te Liao, Shou-De Lin

Organizations: National Taiwan University · National Tsing Hua University · National Taiwan University AI Center of Research Excellence

Abstract

Spurious correlations can give image predictors high in-distribution accuracy but poor generalization under distribution shift. We introduce Rank2Replace\textbf{Rank2Replace}, which transfers knowledge of task-relevant evidence from pretrained vision--language models (VLMs) to small predictors through training-time augmentation for out-of-distribution (OOD) generalization. The VLM ranks object-centric slots by task relevance and guides rank-dependent replacement with same-rank slot embeddings from another image. This augmentation retains the original image's label, encouraging reliance on preserved task evidence as incidental cues change, and supports combination with existing OOD methods. Across four datasets and six splits under shared frozen slot embeddings, the default cross-entropy configuration reaches macro-averaged average-group and worst-group accuracies of 80.43%80.43\% and 65.28%65.28\% without environment labels or dataset-specific replacement tuning. Slot corruption improves both macro-averaged metrics for all fourteen OOD training methods. Combining it with slot ranking and IRM reaches 81.96%81.96\% and 72.68%72.68\%. Controlled ablations support the ranking and replacement design, with gains across five ranking VLMs and six extractor--predictor combinations. Permutation-invariant predictors further improve both macro-averaged metrics over their unaugmented counterparts while using VLM guidance only during training for efficient inference.

Figures & tables

Explore similar work

CardsList
  1. Inductive Visual Logic for Few-Shot Out-Of-Distribution Adaptation in VLMs

    Sep 29, 2026Hung-Jen Chen, Yu-Heng Ho, Ting-Yao Huang +4Vision-Language ModelsFew-Shot Domain Adaptation

  2. Level, Sharpness, and Corpus: Why Zero-Shot OOD Detector Rankings Do Not Transfer

    Jul 29, 2026Ignacio M. De la Jara, Cristian Rodriguez-Opazo, Stephen Gould +1VLM EvaluationDistribution Shift Detection

  3. STAMP: Predicting Out-of-Distribution Generalization without Target Data

    Sep 26, 2026Md Kawsher Mahbub, Milon BiswasOOD Generalization