cs.CVNov 9, 2026

Where Does Proposal Selection Fail? Diagnosing Frozen Vision-Language Queries over Detector Proposals

Authors: Bo Ma, Weiqi Yan, Jinsong Wu

Organizations: Auckland University of Technology · Guilin University of Electronic Technology

Abstract

In modular grounding a frozen detector proposes regions and a vision-language model (VLM) selects one, at lower cost than letting the VLM decode a box, but it is unclear where such systems fail. We hold perception fixed---every trained selector chooses among the same frozen proposals, which contain the referent for at least 99.8%99.8\% of RefCOCO/+/g queries---and vary the selection interface one factor at a time for two VLMs of one family (Qwen2.5-VL-7B, Qwen3-VL-8B). On RefCOCOg, what the selector reads matters more than how it is built: the VLM's image-conditioned query encodes the referent's location and beats a text-only query by 0.150.15--0.180.18 with every head tested (by less for InternVL3-8B, whose vision encoder differs; one head tested). The remaining errors are rankings among a few top candidates; a cross-attention selector supervised with every valid box reaches the VLM's coordinate decoding on RefCOCOg, and with Qwen2.5-VL on RefCOCO/+ when also trained on their data. Beyond that distribution it does not: on Ref-L4's Objects365 images, mostly of categories outside training, every trained head falls to 0.190.19--0.280.28 while Qwen decoding keeps about 0.80.8 (InternVL3's drops too), neither a wider detector prompt nor a class-agnostic proposer closes the gap, and on public reasoning benchmarks Ref-Adv-s and RefBench-PRO full-budget heads trail decoding by 0.190.19--0.370.37 although proposals cover most referents. A trained verifier reduces to an existence probe on the VLM's query state and does not beat the VLM's yes/no answer on public no-target benchmarks.

Figures & tables

Explore similar work

CardsList
  1. Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection

    Apr 18, 2026Dawar Jyoti Deka, Amit Sethi, Syed Mohammad AliOpen-Vocabulary Object DetectionVLM Robustness