cs.CVNov 19, 2026

Rank2Replace: VLM-Ranked Slot Replacement for Out-of-Distribution Generalization

Authors: Bo-Han Feng, Keng-Te Liao, Shou-De Lin

Organizations: National Taiwan University · National Tsing Hua University · National Taiwan University AI Center of Research Excellence

Abstract

Spurious correlations can give image predictors high in-distribution accuracy but poor generalization under distribution shift. We introduce Rank2Replace\textbf{Rank2Replace}, which transfers knowledge of task-relevant evidence from pretrained vision--language models (VLMs) to small predictors through training-time augmentation for out-of-distribution (OOD) generalization. The VLM ranks object-centric slots by task relevance and guides rank-dependent replacement with same-rank slot embeddings from another image. This augmentation retains the original image's label, encouraging reliance on preserved task evidence as incidental cues change, and supports combination with existing OOD methods. Across four datasets and six splits under shared frozen slot embeddings, the default cross-entropy configuration reaches macro-averaged average-group and worst-group accuracies of 80.43%80.43\% and 65.28%65.28\% without environment labels or dataset-specific replacement tuning. Slot corruption improves both macro-averaged metrics for all fourteen OOD training methods. Combining it with slot ranking and IRM reaches 81.96%81.96\% and 72.68%72.68\%. Controlled ablations support the ranking and replacement design, with gains across five ranking VLMs and six extractor--predictor combinations. Permutation-invariant predictors further improve both macro-averaged metrics over their unaugmented counterparts while using VLM guidance only during training for efficient inference.

Figures & tables

Explore similar work

Sep 29, 2026

Inductive Visual Logic for Few-Shot Out-Of-Distribution Adaptation in VLMs

Generative vision-language models (VLMs) such as Qwen-VL and LLaVA achieve strong zero-shot performance on tasks overlapping with their pretraining distribution, yet fail on specialized domains where the required discriminative features were never learned, a regime we term distant out-of-distribution (OOD). Standard adaptation methods cannot overcome this representational absence because they operate within the encoder's existing feature space. However, VLMs retain a robust descriptive capacity even when discrimination collapses: a model that cannot classify a medical scan can still articulate its visual patterns. Exploiting this asymmetry, we introduce Inductive Visual Logic (IVL), a training-free framework that constructs classification knowledge from the model's surviving descriptive ability. IVL extracts visual traits from few-shot support images through dual-mode prompting, combining semantic descriptions with primitive visual observations, and organizes them into per-class trait dictionaries. At inference, hierarchical filtering identifies spatially grounded trait evidence for classification. Across multiple distant-OOD benchmarks, IVL achieves the highest aggregate accuracy under two VLM backbones while producing interpretable, trait-traceable predictions.
Jul 29, 2026

Level, Sharpness, and Corpus: Why Zero-Shot OOD Detector Rankings Do Not Transfer

Selecting a zero-shot out-of-distribution (OOD) detector for a new deployment is typically based on benchmark rankings, implicitly assuming that the highest-ranked detector will transfer across domains. We show that this assumption does not hold. Through a controlled portability audit across seventeen in-distribution datasets, three vision-language models, and seven representative zero-shot OOD detectors, we find that detector rankings reverse across deployments, every detector exceeds 80%80\% FPR95 on at least one domain, and the preferred detector depends on both the in-distribution data and the underlying VLM. We trace these reversals to complementary evidence channels in vision-language logits. Corpus-free detectors rely on different combinations of absolute match level and relative or spatial sharpness, while WordNet-based methods additionally depend on external semantic coverage. A simple proposition shows that level and sharpness cannot generally be recovered from one another, explaining why no single detector transfers reliably across deployments. Motivated by this diagnosis, we introduce the Complementary Evidence Guard (CEG), a detector-agnostic wrapper that preserves complementary evidence through a non-compensatory fusion of the base detector, level, and sharpness using only empirical in-distribution percentiles. Controls replacing these channels with entropy, logit variance, or random noise do not reproduce the gains. Without OOD samples, auxiliary corpora, or learned fusion, CEG reduces detector sensitivity and improves GL-MCM from 38.138.1 to 28.828.8 and MCM from 42.642.6 to 30.530.5 family-balanced FPR95.
Sep 26, 2026

STAMP: Predicting Out-of-Distribution Generalization without Target Data

Predicting whether a trained model will generalize under distribution shift remains difficult, especially when target-domain data are unavailable. We introduce STAMP (Semantic Temporal Augmented Model Prediction), a source-only, target-label-free criterion that estimates out-of-distribution (OOD) performance from paired source-domain images. STAMP computes the output-space correlation ratio η2=SB/STη^2=S_B/S_T by contrasting semantically stable pairs with random pairs: higher η2η^2 indicates that model outputs vary with semantic identity rather than nuisance variation. On 44 chest X-ray models spanning CNNs, ViTs, MetaFormers, foundation models, and SSL/VLM probes, temporal STAMP attains Spearman correlations of 0.8440.844--0.8550.855 with macro AUROC on VinDr-CXR, CheXpert, and MIMIC-CXR; a class-matched variant improves single-class RSNA from 0.3110.311 to 0.6630.663. STAMP attains the best average source-only medical ranking and outperforms the target-domain ATC and AoTL estimators without any target data. On 27 ImageNet models, temperature-scaled STAMPTS attains ρ=0.984ρ{=}0.984 on ObjectNet and ρ≥0.905ρ\geq0.905 on four additional distribution shifts, with partial correlations of 0.6620.662--0.9490.949 after controlling for ImageNet accuracy. Requiring approximately 12s per model on one GPU, STAMP is a practical pre-deployment model-selection and auditing tool.