cs.CVNov 1, 2026

Object-Only Is Not Context-Free: A Counterfactual Diagnostic for Context Removal in Object Retrieval

Authors: Boaz Meivar, Shai Avidan

Organizations: Tel Aviv University

Abstract

Object-centric retrieval pipelines often mask the background before pooling backbone features, aiming for a descriptor that reflects the object rather than its scene. Whether this truly yields an object-only descriptor, however, is rarely tested. We introduce the Keyhole Aperture Probe (KAP), a counterfactual diagnostic that repurposes the ground-truth masks of existing panoptic segmentation datasets (COCO, ADE20K) for retrieval: it holds the pooled object region fixed and varies only the pixels outside it, summarizing a backbone's context dependence in a single number. We find that the pooled object region has already integrated its surround. Replacing the context, with a neutral fill or even a different real scene, changes the descriptor, and when retrieval fails under a blank fill, the descriptor matches plain-background images rather than its own object. Object-only is not context-free. Across ten modern backbones, object-only descriptors retain only about a quarter to two-thirds of their full-image accuracy, and the effect is not unique to Transformer self-attention: a convolutional backbone shows it too. The dependence is also reducible: a lightweight adapter that encourages the descriptor to stay stable as context is removed lowers it and improves full-image instance retrieval on INSTRE, transferring to ILIAS with accuracy improved or preserved. A matched contrastive-fine-tuning control does not reproduce those gains. We release KAP as a simple, backbone-agnostic evaluation protocol to measure and reduce this dependence before deployment.

Explore similar work

CardsList
  1. ContextShift: A Controlled Benchmark for Context Dependence in Object Detection

    Jun 8, 2026Dan Zlotnikov, Alex Lazarovich, Ohad Ben-ShaharObject Detection

  2. Beyond Resolution: Object-to-Image Ratio Mismatch in Instance Retrieval

    Oct 8, 2026Boaz Meivar, Ofir Kedem, Amit Edenzon +2Image RetrievalRetrieval Robustness

  3. Mask-supervised Object-centric Representation Learning with LeJEPA

    Jul 2, 2026Jakob Geusen, Ender KonukogluContrastive LearningRepresentation Learning