Where Does Proposal Selection Fail? Diagnosing Frozen Vision-Language Queries over Detector Proposals
Organizations: Auckland University of Technology · Guilin University of Electronic Technology
Abstract
In modular grounding a frozen detector proposes regions and a vision-language model (VLM) selects one, at lower cost than letting the VLM decode a box, but it is unclear where such systems fail. We hold perception fixed---every trained selector chooses among the same frozen proposals, which contain the referent for at least of RefCOCO/+/g queries---and vary the selection interface one factor at a time for two VLMs of one family (Qwen2.5-VL-7B, Qwen3-VL-8B). On RefCOCOg, what the selector reads matters more than how it is built: the VLM's image-conditioned query encodes the referent's location and beats a text-only query by -- with every head tested (by less for InternVL3-8B, whose vision encoder differs; one head tested). The remaining errors are rankings among a few top candidates; a cross-attention selector supervised with every valid box reaches the VLM's coordinate decoding on RefCOCOg, and with Qwen2.5-VL on RefCOCO/+ when also trained on their data. Beyond that distribution it does not: on Ref-L4's Objects365 images, mostly of categories outside training, every trained head falls to -- while Qwen decoding keeps about (InternVL3's drops too), neither a wider detector prompt nor a class-agnostic proposer closes the gap, and on public reasoning benchmarks Ref-Adv-s and RefBench-PRO full-budget heads trail decoding by -- although proposals cover most referents. A trained verifier reduces to an existence probe on the VLM's query state and does not beat the VLM's yes/no answer on public no-target benchmarks.
Figures & tables
| RefCOCO | RefCOCO+ | RefCOCOg | Ref-L4 val | |||||||
| val | val | val | test | all | COCO cat. | other | Stress | Ref-Adv-s | RB-PRO | |
| Proposal recall | 0.999 | 0.999 | 1.000 | 0.998 | 0.958 | 0.999 | 0.886 | 1.000 | 0.907 | 0.884 |
| Detector-native | 0.847 | 0.737 | 0.783 | 0.781 | 0.634 | 0.649 | 0.609 | 0.255 | 0.271 | 0.524 |
| Qwen2.5-VL-7B | ||||||||||
| Text-only query | 0.683 | 0.517 | 0.672 | 0.676 | 0.442 | 0.587 | 0.194 | 0.312 | – | – |
| Alignment hook | 0.834 | 0.760 | 0.843 | 0.837 | 0.549 | 0.718 | 0.259 | 0.413 | 0.308 | 0.297 |
| Test | Ref-L4 | Stress | Test | Ref-L4 | Stress | ||
| Query source (learned bridge) | Supervision and readout (hook) | ||||||
| CLIP text | 0.553 .005 | 0.359 | 0.237 | Single positive, final token | 0.793 .007 | 0.464 | 0.384 |
| Detector text tower | 0.605 .006 | 0.388 | 0.250 | Every valid box positive | 0.805 .014 | – | – |
| projected | 0.611 .010 | 0.375 | 0.224 | Mean over expression | 0.721 .008 | – | – |
| VLM, text only | 0.593 .007 | 0.372 | 0.294 | Three VLM depths | 0.817 .001 | – | – |
| VLM, image-conditioned | 0.791 .013 | 0.465 | 0.423 | Three depths, every valid box | 0.842 .003 | – | – |
| RefCOCOg | ||||
| val | test | Ref-L4 | Stress | |
| Hook, top-1 | 0.787 | 0.788 | 0.451 | 0.390 |
| Correct box in hook’s top 5 | 0.973 | 0.973 | 0.757 | 0.736 |
| Correct box in detector’s top 5 | 0.980 | – | 0.898 | – |
| VLM over hook’s top 5: marks | 0.587 | – | 0.503 | 0.411 |
| box list | 0.760 | 0.762 | 0.576 | 0.524 |
| Claim | Main paper | Supplement |
|---|---|---|
| The query source decides accuracy under late fusion, because the image-conditioned query encodes the referent’s location; head form and capacity matter less than the query on the RefCOCO family (on RefCOCOg test the image-conditioned query adds – with each of three heads, a set-level head – over the hook; on Ref-L4 the head matters as much) | Tables 1, 2; Fig. 1 | Tabs. S36 , S20 , S46 , S50 , S2 , S51 , S61 and S62 |
| Remaining errors are ranking errors (correct proposal usually in the top five) | Tables 2, 3; Fig. 1 | Figs. S3 , S28 and S5 |
| Cross-attention scoring, token states, multi-layer readout and multi-positive supervision reach coordinate decoding on RefCOCOg at the full budget, and on RefCOCO/+ when trained on all three RefCOCO-family sets (not explained by the smaller RefCOCOg part of that union); VGent’s remaining lead is attributed by elimination, also against its class-agnostic proposer | Tables 1, 2; Fig. 1 | Tabs. S2 , S10 , S44 , S45 , S53 , S55 , S56 and S67 |
| Prompting the VLM to choose does not help on RefCOCOg; fine-tuning it to choose adds but ends level with fine-tuning it to decode on the same rows | Table 3 | Tabs. S23 , S24 , S25 , S26 and S27 |
| Ref-L4: collapse outside the trained categories, not lifted by a wider detector prompt or a class-agnostic proposer; training on the new categories helps, not through caption style | Table 1; Fig. 1 | Tabs. S9 , S10 , S13 , S14 , S15 and S16 |
| Released Jag and LOCUS weights on every column of Table 1: on Objects365 images far below decoding, with a smaller category term than our heads’ with size controlled | – | Tabs. S11 and S12 |
| Method | RefCOCO val | RefCOCO+ val | RefCOCOg val | RefCOCOg test | Ref-L4 val | Stress |
|---|---|---|---|---|---|---|
| Proposal recall (ceiling) | 0.999 | 0.999 | 1.000 | 0.998 | 0.958 | 1.000 |
| Learned bridge (CLIP query) | 0.511 .017 | 0.429 .008 | 0.542 .006 | 0.552 .009 | 0.357 .006 | 0.254 .022 |
| Index scorer | 0.759 .002 | 0.683 .004 | 0.770 .002 | 0.770 .007 | 0.444 .010 | 0.384 .032 |
| Index scorer, audited ∗ | 0.801 .003 | 0.722 .009 | 0.811 .003 | 0.815 .002 | 0.451 .027 | 0.434 .017 |
| Alignment hook | 0.789 .007 | 0.707 .010 | 0.787 .004 | 0.793 .007 | 0.464 .010 | 0.384 .035 |
| Relation-aware head, IoU-rank loss ∗ | 0.849 .003 | 0.772 .003 | 0.846 .002 | 0.849 .001 | 0.522 .007 | 0.388 .014 |
| Method | Afford. | Negation | World kn. | All |
|---|---|---|---|---|
| ( ) | ( ) | ( ) | ( ) | |
| Learned bridge (CLIP) | 0.251 .033 | 0.282 .028 | 0.210 .033 | 0.254 .022 |
| Detector-native | 0.256 | 0.282 | 0.200 | 0.255 |
| template interval | [.129, .386] | [.184, .359] | [.029, .481] | [.172, .337] |
| Index scorer | 0.296 .037 | 0.615 .064 | 0.229 .049 | 0.384 .032 |
| Alignment hook | 0.309 .038 | 0.568 .080 | 0.276 .066 | 0.384 .035 |
| Accuracy | Adversarial | |||||
| System | Rand. | Pop. | Adv. | Rec. | Hall. | Area |
| No verifier (affinity) | 0.573 | 0.586 | 0.589 | 0.521 | 0.343 | 0.341 |
| Default veto | 0.599 | 0.600 | 0.594 | 0.216 | 0.028 | 0.265 |
| Affinity only | 0.596 | 0.597 | 0.592 | 0.217 | 0.033 | 0.261 |
| Verifier only | 0.806 | 0.802 | 0.763 | 0.666 | 0.140 | 0.135 |
| Correctness label | 0.886 | 0.862 | 0.820 | – | – | – |
| Split | Proposal recall | (hook) | |
|---|---|---|---|
| RefCOCO val | 0.999 | 0.001 | 0.210 |
| RefCOCO+ val | 0.999 | 0.001 | 0.292 |
| RefCOCOg val | 1.000 | 0.000 | 0.213 |
| RefCOCOg test | 0.998 | 0.002 | 0.205 |
| Ref-L4 val | 0.958 | 0.042 | 0.516 |
| Reasoning stress | 1.000 | 0.000 | 0.616 |
| Method | RefCOCO val | RefCOCO+ val | RefCOCOg val | RefCOCOg test | Ref-L4 val | Stress | mIoU test |
|---|---|---|---|---|---|---|---|
| Learned bridge (CLIP query) | |||||||
| Index scorer | |||||||
| Alignment hook | |||||||
| Relation-aware head | |||||||
| Pooled query, cross-attention | – | – | |||||
| Token states, cross-attention | – | – |
| RefCOCO | RefCOCO+ | RefCOCOg (umd) | ||||||
| Method | val | testA | testB | val | testA | testB | val | test |
| Proposal recall (ceiling) | 0.999 | 0.999 | 0.998 | 0.999 | 0.999 | 0.999 | 1.000 | 0.998 |
| Alignment hook, k | 0.789 .007 | 0.827 .006 | 0.755 .011 | 0.707 .010 | 0.770 .007 | 0.625 .009 | 0.787 .004 | 0.793 .007 |
| Relation-aware head, k | 0.849 .003 | 0.881 .002 | 0.803 .005 | 0.772 .003 | 0.832 .006 | 0.696 .003 | 0.846 .002 | 0.849 .001 |
| Alignment hook, full set | 0.834 .006 | 0.857 .006 | 0.798 .008 | 0.760 .008 | 0.814 .008 | 0.693 .007 | 0.843 .004 | 0.837 .005 |
| Relation-aware head, full set | 0.872 .003 | 0.898 .002 | 0.846 .003 | 0.802 .006 | 0.855 .001 | 0.748 .003 | 0.877 .002 | 0.878 .003 |
| Ref-L4 test (subset, ) | Ref-L4 val | |||||||
| Method | [email protected] | [email protected] | [email protected] | mAcc | mAcc COCO cat. | mAcc other | [email protected] | mAcc |
| Proposal recall (ceiling) | 0.954 | 0.918 | 0.747 | 0.861 | 0.926 | 0.754 | 0.958 | 0.864 |
| Alignment hook, k | 0.455 | 0.402 | 0.330 | 0.389 | 0.505 | 0.198 | 0.464 | 0.396 |
| Relation-aware head, k | 0.517 | 0.458 | 0.377 | 0.444 | 0.587 | 0.206 | 0.522 | 0.446 |
| Alignment hook, full set | 0.542 | 0.483 | 0.401 | 0.468 | 0.623 | 0.213 | 0.549 | 0.473 |
| Relation-aware head, full set | – | – | – | – | – | – | 0.595 | 0.516 |
| Bucket | Cov. | Hook | Rel. | Token | Det. | Coord. | |
|---|---|---|---|---|---|---|---|
| COCO-80 category | 8483 | 0.999 | 0.592 | 0.680 | 0.784 | 0.649 | 0.900 |
| Other category | 4937 | 0.886 | 0.243 | 0.249 | 0.266 | 0.609 | 0.774 |
| Caption words | 4726 | 0.918 | 0.367 | 0.404 | 0.439 | 0.678 | 0.813 |
| Caption – words | 4706 | 0.970 | 0.505 | 0.567 | 0.642 | 0.643 | 0.860 |
| Caption words | 3988 | 0.991 | 0.530 | 0.607 | 0.718 | 0.572 | 0.892 |
| Small box (lower tercile) | 4474 | 0.900 | 0.237 | 0.220 | 0.236 | 0.553 | 0.754 |
| Images | Referent | Hook | Token | Det. | Coord. | |
|---|---|---|---|---|---|---|
| COCO | COCO-80 | 8483 | 0.592 | 0.784 | 0.649 | 0.900 |
| Objects365 | COCO-mappable | 1664 | 0.384 | 0.410 | 0.671 | 0.821 |
| Objects365 | other | 3273 | 0.171 | 0.192 | 0.578 | 0.749 |
| RefCOCO | RefCOCO+ | RefCOCOg | Ref-L4 val | |||||||
| val | val | val | test | all | COCO cat. | other | Stress | Ref-Adv-s | RB-PRO | |
| References | ||||||||||
| Qwen2.5-VL-7B, coordinate decoding | 0.893 | 0.838 | 0.865 | 0.866 | 0.853 | 0.900 | 0.774 | 0.623 | 0.520 | 0.662 |
| Qwen3-VL-8B, coordinate decoding | 0.918 | 0.868 | 0.886 | 0.890 | 0.880 | 0.918 | 0.815 | 0.671 | – | – |
| Qwen2.5-VL-7B, combined selector | 0.895 | 0.824 | 0.876 | 0.885 | 0.641 | 0.852 | 0.280 | 0.381 | 0.314 | 0.313 |
| all three train sets | 0.912 | 0.846 | 0.877 | 0.889 | 0.640 | 0.853 | 0.274 | 0.374 | – | – |
| Category term, size terciles | Category term, continuous size | Large target | Reading | |
| Jag | [ , ] | [ , ] | (c) | |
| LOCUS-0.8B v1.0, verified | [ , ] | [ , ] | (c) | |
| LOCUS-0.8B v1.0, draft | [ , ] | [ , ] | (a) | |
| LOCUS-2B v1.0, verified | [ , ] | [ , ] | (c) | |
| LOCUS-2B v1.0, draft | [ , ] | [ , ] | (c) | |
| LOCUS-0.8B v1.1, verified | [ , ] | [ , ] | (a) |
| Referent | Prompt | Coverage | Acc | Acc cov | |
|---|---|---|---|---|---|
| COCO-80 | COCO-80 | 5863 | 0.999 | 0.621 | 0.622 |
| extended | 0.998 | 0.616 | 0.617 | ||
| Other | COCO-80 | 3525 | 0.883 | 0.221 | 0.251 |
| extended | 0.957 | 0.261 | 0.273 |
| Head | Proposals | Training rows | RefCOCOg val | Ref-L4 val | COCO images | O365, COCO-mappable | O365, other |
|---|---|---|---|---|---|---|---|
| Proposal recall, COCO-80 prompt | 1.000 | 0.958 | 0.999 | 0.989 | 0.834 | ||
| Proposal recall, extended prompts | 0.998 | 0.984 | 0.998 | 0.984 | 0.947 | ||
| Hook | COCO-80 | k | 0.787 | 0.464 | 0.592 | 0.384 | 0.171 |
| Hook | ext. (not retrained) | k | 0.776 | 0.470 | 0.576 | 0.331 | 0.267 |
| Hook | ext. | k, ext. features | 0.775 | 0.472 | 0.582 | 0.364 | 0.241 |
| Hook | COCO-80 | k + Ref-L4 half | 0.794 | 0.561 | 0.617 | 0.543 | 0.425 |
| RefCOCOg | Ref-L4 val | |||||||
| Proposals, region features | val | test | all | COCO cat. | other | O365 map. | O365 other | Stress |
| Proposal recall | ||||||||
| Grounding DINO, COCO-80 prompt | 1.000 | 0.998 | 0.958 | 0.999 | 0.886 | 0.989 | 0.834 | 1.000 |
| UPN, no prompt | 0.997 | 0.996 | 0.986 | 0.998 | 0.966 | 0.986 | 0.955 | 0.991 |
| Alignment hook | ||||||||
| Grounding DINO, fused features | 0.787 .004 | 0.793 .007 | 0.464 .010 | 0.592 .018 | 0.243 .004 | 0.384 .017 | 0.171 .008 | 0.384 .035 |
| RefCOCOg | Ref-L4 val | |||||
| Training rows | val | all | COCO cat. | other | O365 map. | O365 other |
| Alignment hook | ||||||
| k RefCOCOg rows | 0.787 .004 | 0.464 .010 | 0.592 .018 | 0.243 .004 | 0.384 .017 | 0.171 .008 |
| + Ref-L4, COCO images | 0.792 .008 | 0.560 .008 | 0.749 .016 | 0.234 .006 | 0.374 .009 | 0.163 .005 |
| + Ref-L4, Objects365 images | 0.794 .003 | 0.561 .016 | 0.617 .022 | 0.465 .008 | 0.543 .002 | 0.425 .013 |
| Style term | ||||||
| RefCOCOg | Ref-L4 val | ||||||||
| Training rows | val | test | all | COCO cat. | other | O365 map. | O365 other | named | not named |
| Alignment hook | |||||||||
| k RefCOCOg rows | 0.787 .004 | 0.793 .007 | 0.464 .010 | 0.592 .018 | 0.243 .004 | 0.384 .017 | 0.171 .008 | 0.148 | 0.241 |
| + k Flickr30k Entities phrases | 0.799 .001 | 0.805 .001 | 0.484 .019 | 0.591 .021 | 0.299 .024 | 0.425 .037 | 0.235 .019 | 0.210 | 0.309 |
| Difference | |||||||||
| [.006, .019] | [.007, .017] | [.014, .026] | [ .009, .006] | [.046, .066] | [.022, .060] | [.052, .075] | [.050, .074] | [.044, .090] | |
| Split | Acc | Ratio | |||
|---|---|---|---|---|---|
| RefCOCO val | 10815 | 0.780 | 30.0 | ||
| RefCOCO+ val | 10742 | 0.700 | 30.1 | ||
| RefCOCOg val | 4896 | 0.779 | 32.6 | ||
| RefCOCOg test | 9583 | 0.780 | 32.7 | ||
| Ref-L4, COCO-80 referent | 5858 | 0.622 | 31.0 | ||
| Reasoning stress | 231 | 0.375 | 32.2 |
| Variant | RefCOCO val | RefCOCO+ val | RefCOCOg val | RefCOCOg test | Ref-L4 val | Stress |
|---|---|---|---|---|---|---|
| Alignment hook (reference) | ||||||
| Index scorer (image-conditioned VLM query) | ||||||
| As proposed (index embedding, 30 epochs) | ||||||
| No index embedding | ||||||
| Bilinear scorer | ||||||
| Uncovered positives masked | ||||||
| Configuration | Val | Test | Inst.-exact | R@5 |
| Default ( , , depth 1) | 0.958 | |||
| 0.953 | ||||
| 0.957 | ||||
| 0.957 | ||||
| Depth 2 | 0.952 | |||
| Depth 3 | 0.951 |
| Bucket | Continuous | Discrete | mid- | |
|---|---|---|---|---|
| All | 4896 | 0.787 | 0.681 | |
| Length words | 693 | 0.797 | 0.729 | |
| Length 5–8 words | 2079 | 0.783 | 0.667 | |
| Length words | 2124 | 0.789 | 0.678 | |
| Relational word | 2161 | 0.768 | 0.634 | |
| No relational word | 2735 | 0.803 | 0.717 |
| Emit iff agree | AURC | |||||||
|---|---|---|---|---|---|---|---|---|
| Split | Hook | Coord.-snap | Oracle | Ans. | Sel. acc. | Agree, then affinity | Detector score | Verifier |
| RefCOCOg val | 0.787 | 0.867 | 0.922 | 0.785 | 0.936 | 0.051 | 0.060 | 0.091 |
| RefCOCOg test | 0.793 | 0.866 | 0.922 | 0.791 | 0.934 | 0.048 | 0.057 | 0.097 |
| RefCOCO val | 0.789 | 0.895 | 0.935 | 0.785 | 0.958 | 0.032 | 0.035 | 0.085 |
| Ref-L4 val | 0.464 | 0.833 | 0.859 | 0.474 | 0.925 | 0.082 | – | 0.101 |
| Stress | 0.384 | 0.610 | 0.668 | 0.440 | 0.748 | 0.263 | 0.387 | 0.288 |
| RefCOCOg val | Stress | ||||||
|---|---|---|---|---|---|---|---|
| Variant | Acc | Ceil. | No-tgt. | Joint | Acc | Ceil. | Joint |
| Marks, hook, | 0.587 | 0.973 | 0.743 | 0.632 | 0.411 | 0.736 | 0.376 |
| Marks, hook, | 0.413 | 0.988 | 0.719 | 0.504 | 0.329 | 0.835 | 0.339 |
| Marks, detector, | 0.479 | 0.980 | 0.740 | 0.555 | 0.429 | 0.719 | 0.415 |
| Marks, detector, | 0.338 | 0.991 | 0.713 | 0.449 | 0.333 | 0.835 | 0.348 |
| Box list, hook, | 0.760 | 0.973 | 0.791 | 0.767 | 0.524 | 0.736 | 0.500 |
| RefCOCOg test | RefCOCO | RefCOCO+ | Ref-L4 | |||
| Variant | Acc | No-tgt. | Joint | val | val | val |
| Hook, top-1 | 0.790 | – | – | 0.779 | 0.706 | 0.481 |
| Detector, top-1 | 0.774 | – | – | 0.843 | 0.736 | – |
| Correct box in hook’s top 5 | 0.970 | – | – | 0.960 | 0.943 | 0.767 |
| Marks, hook, | 0.578 | 0.767 | 0.632 | 0.614 | 0.578 | 0.518 |
| Marks, hook, | – | – | – | – | – | 0.398 |
| RefCOCOg val | Ref-L4 | Stress | |
| Accuracy | 0.878 | 0.694 | 0.554 |
| Top-five ceiling | 0.973 | 0.757 | 0.736 |
| Accuracy given ceiling | 0.903 | 0.917 | 0.753 |
| “None” on present requests | 0.021 | 0.061 | 0.260 |
| Joint accuracy, native / tuned | 0.899 / 0.901 | – | 0.621 / 0.624 |
| vs. zero-shot box list | [.107, .130] | [.104, .132] | [ .026, .091] |
| RefCOCOg | RefCOCO | RefCOCO+ | Ref-L4 | ||||
| System | val | test | val | val | subset | Stress | |
| Hook, top-1 | – | 0.787 | 0.788 | 0.787 | 0.699 | 0.451 | 0.390 |
| Correct box in hook’s top 5 | – | 0.973 | 0.973 | 0.961 | 0.943 | 0.757 | 0.736 |
| Correct box in detector’s top 5 | – | 0.980 | – | – | – | 0.898 | – |
| Coordinate decoding, no fine-tuning | – | 0.865 | 0.866 | 0.893 | 0.838 | 0.838 | 0.623 |
| same pipeline as (a) | – | 0.865 | 0.868 | – | – | – | – |
| Difference | RefCOCOg val | RefCOCOg test | Ref-L4 subset | Stress |
|---|---|---|---|---|
| (b) (a), raw | [ .002, .017] | [ .001, .015] | – | – |
| (b) (a), snapped | [ .002, .017] | [ .001, .015] | – | – |
| (b) decoding | [.004, .022] | [.007, .024] | [ .172, .130] | [ .157, .033] |
| Gain of (a) ∗ | [ .002, .012] | [.002, .011] | – | – |
| Gain of (b) † | [.107, .131] | [.111, .130] | [.090, .129] | [ .039, .084] |
| Gain of (b) gain of (a) | [.100, .128] | [.103, .124] | – | – |
| RefCOCOg val | RefCOCOg test | |||
| Hook | Coord. | Hook | Coord. | |
| Error rate | 0.213 | 0.135 | 0.207 | 0.134 |
| near miss, in top-5 | 0.084 | – | 0.082 | – |
| other object, in top-5 | 0.087 | – | 0.083 | – |
| beyond top-5 | 0.042 | – | 0.040 | – |
| no covering proposal | 0.000 | – | 0.002 | – |
| Absent rate | FAR | ||||
|---|---|---|---|---|---|
| Coordinate-snap, always answer (val / test) | 0.824 / 0.823 | 0.781 / 0.779 | 0.607 / 0.606 | 0.434 / 0.433 | – |
| + VLM yes/no | 0.793 / 0.792 | 0.803 / 0.802 | 0.840 / 0.841 | 0.877 / 0.880 | 0.784 |
| + VLM yes-prob., tuned | 0.856 / 0.858 | 0.860 / 0.862 | 0.873 / 0.877 | 0.887 / 0.892 | 0.834 |
| + verifier | 0.861 / 0.861 | 0.866 / 0.866 | 0.888 / 0.888 | 0.909 / 0.910 | 0.860 |
| + verifier, hook’s pick | 0.866 / 0.866 | 0.870 / 0.870 | 0.886 / 0.887 | 0.902 / 0.904 | 0.861 |
| Hook, default veto | 0.714 / 0.724 | 0.727 / 0.737 | 0.780 / 0.788 | 0.833 / 0.840 | – |
| RefCOCOg val + absent (tuning split) | RefCOCOg test + absent (untouched) | |||||||||
| System | Joint | Target | No-tgt. | Ans. | AURC | Joint | Target | No-tgt. | Ans. | AURC |
| Coordinate-snap, always answer | 0.607 | 0.867 | 0.000 | 1.000 | 0.393 | 0.606 | 0.866 | 0.000 | 1.000 | 0.394 |
| + VLM yes/no | 0.840 | 0.784 | 0.970 | 0.626 | 0.134 | 0.841 | 0.782 | 0.978 | 0.624 | 0.135 |
| + VLM yes-probability, tuned | 0.873 | 0.853 | 0.921 | 0.708 | 0.134 | 0.877 | 0.854 | 0.929 | 0.708 | 0.135 |
| + verifier | 0.888 .002 | 0.856 | 0.962 | 0.696 | 0.148 | 0.888 .002 | 0.856 | 0.963 | 0.695 | 0.152 |
| + verifier at the hook’s proposal | 0.886 .001 | 0.862 | 0.942 | 0.712 | 0.161 | 0.887 .001 | 0.861 | 0.947 | 0.711 | 0.168 |
| Stress + absent | POPE | |||||
| System | Joint | No-tgt. | AURC | Random | Popular | Adversarial |
| Coordinate-snap, always answer | 0.427 | 0.000 | 0.573 | 0.500 | 0.500 | 0.500 |
| + VLM yes/no | 0.545 | 0.606 | 0.412 | 0.883 | 0.874 | 0.864 |
| + VLM yes-prob., tuned | 0.564 | 0.475 | 0.412 | 0.932 | 0.897 | 0.859 |
| + verifier | 0.528 .042 | 0.923 | 0.377 | 0.697 .033 | 0.695 .034 | 0.686 .030 |
| + verifier at hook’s proposal | 0.580 .031 | 0.859 | 0.394 | 0.806 .026 | 0.802 .024 | 0.763 .015 |
| No-target | Joint | ||||
|---|---|---|---|---|---|
| System | All | Present | Absent | Present | Absent |
| Coord.-snap + VLM yes/no | 0.970 | 0.908 | 0.991 | 0.796 | 0.834 |
| Coord.-snap + yes-prob., tuned | 0.921 | 0.807 | 0.959 | 0.848 | 0.879 |
| Coord.-snap + verifier | 0.962 | 0.869 | 0.993 | 0.857 | 0.889 |
| Coord.-snap + yes/no + verifier | 0.979 | 0.929 | 0.996 | 0.794 | 0.832 |
| Det.-native, score threshold | 0.467 | 0.193 | 0.558 | 0.679 | 0.689 |
| RefCOCOg test | gRefCOCO testB | |||||
| Training negatives | Joint | Hard | Stress | Pr@1 | N-acc | T-acc |
| Swaps (main recipe) | 0.888 | 0.895 | 0.512 | 0.479 | 0.646 | 0.893 |
| Swaps + same-image | 0.853 | 0.754 | 0.559 | 0.434 | 0.571 | 0.810 |
| Same-image only | 0.759 | 0.580 | 0.496 | 0.390 | 0.542 | 0.657 |
| Swaps + gRefCOCO | 0.887 | 0.870 | 0.544 | 0.475 | 0.821 | 0.728 |
| gRefCOCO only | 0.853 | 0.601 | 0.582 | 0.375 | 0.931 | 0.354 |
| Label | Gate | Val joint | Val AURC | Random | Popular | Advers. |
|---|---|---|---|---|---|---|
| Existence | veto | |||||
| Existence | verifier | |||||
| Correctness | veto | |||||
| Correctness | verifier | |||||
| Existence, post hoc | veto | |||||
| Existence, post hoc | verifier |
| RefCOCOg val | RefCOCOg test | Stress | |||
|---|---|---|---|---|---|
| Input | AUROC | Joint | AUROC | Hard AUROC | AUROC |
| Full | 0.993 | 0.833 | 0.994 | 0.982 | 0.860 |
| Query only | 0.993 | 0.831 | 0.995 | 0.983 | 0.894 |
| Region only | 0.935 | 0.757 | 0.931 | 0.863 | 0.692 |
| Affinity only | 0.978 | 0.812 | 0.978 | 0.935 | 0.715 |
| Probe on (no hook) | 0.993 | 0.833 | 0.994 | 0.982 | 0.916 |
| Split | Query | center | size | Snap | Hook |
|---|---|---|---|---|---|
| RefCOCOg val | image-cond. | 0.685 | 0.577 | 0.564 | 0.787 |
| text-only | 0.056 | 0.211 | 0.188 | ||
| RefCOCOg test | image-cond. | 0.672 | 0.570 | 0.563 | 0.793 |
| text-only | 0.055 | 0.211 | 0.189 | ||
| Ref-L4 val | image-cond. | 0.413 | 0.268 | 0.247 | 0.464 |
| Stress | image-cond. | 0.434 | 0.147 | 0.384 |
| Template | Hook | Coord. | Det. | |
|---|---|---|---|---|
| Affordance | ||||
| something that tells the time | 16 | 0.31 [.12, .52] | 0.69 [.44, .88] | 0.44 [.19, .69] |
| something to cut with | 15 | 0.24 [.07, .44] | 0.53 [.27, .80] | 0.00 [.00, .00] |
| something to eat with | 15 | 0.13 [.02, .27] | 0.33 [.13, .60] | 0.00 [.00, .00] |
| something to ride | 15 | 0.42 [.24, .62] | 0.60 [.33, .87] | 0.40 [.13, .67] |
| something to sit on | 15 | 0.47 [.24, .69] | 0.60 [.33, .87] | 0.27 [.07, .47] |
| GroundingME | RefCOCOg_rej (test images) | |||||||||
| System | Target | No-tgt. | Joint | Discr. | Spatial | Limited | Target | No-tgt. | Joint | Pair |
| Qwen2.5-VL-7B | ||||||||||
| Coordinate decoding | 0.243 | 0.000 | 0.194 | 0.338 | 0.153 | 0.267 | 0.853 | 0.000 | 0.427 | 0.000 |
| GroundingME prompt (null allowed) | 0.193 | 0.010 | 0.156 | 0.275 | 0.140 | 0.190 | 0.844 | 0.086 | 0.465 | 0.064 |
| at the checkpoint’s default resolution | 0.223 | 0.000 | 0.178 | 0.358 | 0.160 | 0.193 | – | – | – | – |
| Coordinate decoding + VLM yes/no | 0.111 | 0.224 | 0.133 | 0.206 | 0.090 | 0.067 | 0.771 | 0.870 | 0.820 | 0.657 |
| Hook | Set-level heads | |||||||||
| Referents | Recall | Decoding | k | full | Relation | Token | Comb. | Det. top | ||
| All present | 804 | 0.792 | 0.243 | 0.055 | 0.051 | 0.055 | 0.046 | 0.052 | 0.935 | 0.040 |
| Size at the detector input | ||||||||||
| px | 108 | 0.519 | 0.139 | 0.000 | 0.000 | 0.003 | 0.000 | 0.003 | 0.994 | 0.056 |
| – px | 64 | 0.672 | 0.188 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 1.000 | 0.031 |
| – px | 112 | 0.875 | 0.179 | 0.003 | 0.003 | 0.012 | 0.012 | 0.024 | 0.973 | 0.036 |
| Budget | Native | px | ||
|---|---|---|---|---|
| Proposal recall | 0.803 | 0.803 | 0.813 | |
| Coordinate decoding | 0.273 | 0.237 | 0.170 | |
| Detector’s top proposal | 0.040 | 0.040 | 0.037 | |
| Hook, k | 0.064 | 0.050 | 0.044 | |
| Hook, full | 0.061 | 0.058 | 0.064 | |
| Relation-aware head | 0.051 | 0.060 | 0.053 |
| Negation | Human-authored | Image source | Distractors | |||||||
| System | All | yes | no | yes | no | COCO | OpenImages | – | – | |
| Rows | ||||||||||
| Proposal recall | 0.907 | 0.932 | 0.891 | 0.976 | 0.867 | 0.981 | 0.886 | 0.895 | 0.905 | 0.977 |
| No task training | ||||||||||
| Coordinate decoding | 0.520 [.485, .555] | 0.554 | 0.498 | 0.475 | 0.546 | 0.409 | 0.553 | 0.557 | 0.479 | 0.419 |
| Coordinate-snap | 0.499 [.461, .536] | 0.532 | 0.477 | 0.466 | 0.519 | 0.402 | 0.528 | 0.539 | 0.444 | 0.419 |
| Attribute | Position | Interaction | Relation | Commonsense | Five tasks | |||||||
| System | @ | mAcc | @ | mAcc | @ | mAcc | @ | mAcc | @ | mAcc | @ | |
| Proposal recall | 0.927 | – | 0.887 | – | 0.908 | – | 0.902 | – | 0.794 | – | – | – |
| No task training | ||||||||||||
| Coordinate decoding | 0.725 | 0.595 | 0.693 | 0.566 | 0.622 | 0.524 | 0.607 | 0.510 | 0.662 | 0.494 | 0.662 [.645, .679] | 0.538 |
| Coordinate-snap | 0.702 | 0.589 | 0.658 | 0.546 | 0.597 | 0.513 | 0.577 | 0.499 | 0.562 | 0.444 | 0.619 [.601, .636] | 0.518 |
| Benchmark prompt | 0.724 | 0.591 | 0.676 | 0.547 | 0.628 | 0.524 | 0.574 | 0.482 | 0.647 | 0.486 | 0.650 [.633, .668] | 0.526 |
| Ref-Adv-s ( present) | RefBench-PRO ( present, reject) | |||||||
| Answer rate | ||||||||
| Gate | all | negated | other | Target | Reject | Target | Joint | Ans. |
| Gates on the VLM’s box | ||||||||
| Coordinate decoding, own refusal | 0.999 | 1.000 | 0.999 | 0.520 | 0.052 | 0.662 | 0.560 [.545, .575] | 0.996 |
| Coordinate-snap + VLM yes/no | 0.616 | 0.503 | 0.691 | 0.346 | 0.565 | 0.516 | 0.524 [.507, .540] | 0.755 |
| Coordinate-snap + yes-probability, tuned | 0.957 | 0.932 | 0.974 | 0.482 | 0.209 | 0.613 | 0.545 [.529, .561] | 0.976 |
| VLM, budget | Head | RefCOCO val | RefCOCO+ val | RefCOCOg val | RefCOCOg test | Ref-L4 val | Stress |
|---|---|---|---|---|---|---|---|
| Qwen2.5-VL, k | Text-only query | 0.557 .005 | 0.444 .012 | 0.579 .007 | 0.593 .007 | 0.372 .018 | 0.294 .033 |
| Alignment hook | 0.789 .007 | 0.707 .010 | 0.787 .004 | 0.793 .007 | 0.464 .010 | 0.384 .035 | |
| Matched hook | 0.811 .012 | 0.734 .014 | 0.820 .011 | 0.816 .011 | 0.497 .010 | 0.392 .007 | |
| Relation-aware head | 0.849 .003 | 0.772 .003 | 0.846 .002 | 0.849 .001 | 0.522 .007 | 0.388 .014 | |
| Pooled query | – | – | 0.850 .001 | 0.854 .001 | 0.502 .006 | 0.356 .007 | |
| Token states | – | – | 0.849 .004 | 0.852 .002 | 0.593 .004 | 0.426 .015 |
| Accuracy at | Recall@5 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| VLM | Head | RefCOCO | RefCOCO+ | RefCOCOg val | RefCOCOg test | Ref-L4 | Stress | RefCOCOg test | Ref-L4 |
| Qwen2.5-VL | Alignment hook | 0.770 | 0.702 | 0.760 | 0.761 | 0.490 | 0.355 | 0.971 | 0.761 |
| Relation-aware head | 0.810 | 0.745 | 0.802 | 0.808 | 0.536 | 0.278 | 0.960 | 0.725 | |
| Token states | 0.819 | 0.757 | 0.791 | 0.806 | 0.560 | 0.281 | 0.955 | 0.762 | |
| Combined | 0.829 | 0.765 | 0.797 | 0.814 | 0.573 | 0.300 | 0.953 | 0.752 | |
| Coordinate decoding | 0.791 | 0.742 | 0.749 | 0.757 | 0.724 | 0.468 | – | – | |
| k | Full training set | |||||
|---|---|---|---|---|---|---|
| Variant | Test | Ans. | Test | Ans. | Ref-L4 | Stress |
| Hook (final-token query) | 0.793 | 0.856 | 0.837 | 0.821 | 0.549 | 0.413 |
| Linear projections | 0.775 | 0.970 | 0.844 | 0.957 | – | – |
| Width | 0.790 | 0.749 | 0.836 | 0.952 | – | – |
| Three hidden layers | 0.787 | 0.000 | 0.813 | 0.553 | – | – |
| All-positive supervision | 0.805 | 0.636 | 0.849 | 0.767 | – | – |
| Split | Qwen2.5 | Qwen3 | Qwen2.5 | Qwen3 | Difference |
|---|---|---|---|---|---|
| RefCOCO val | 0.893 | 0.918 | 0.791 | 0.832 | [.020, .030] |
| RefCOCO+ val | 0.838 | 0.868 | 0.742 | 0.789 | [.023, .037] |
| RefCOCOg val | 0.865 | 0.886 | 0.749 | 0.801 | [.012, .029] |
| RefCOCOg test | 0.866 | 0.890 | 0.757 | 0.807 | [.017, .030] |
| Ref-L4 val | 0.853 | 0.880 | 0.724 | 0.774 | [.021, .032] |
| Qwen3-VL, full training set | Qwen2.5-VL, k | |||||
|---|---|---|---|---|---|---|
| System | RefCOCOg val | RefCOCOg test | Stress | RefCOCOg val | RefCOCOg test | Stress |
| Coord.-snap, always | 0.623 | 0.623 | 0.479 | 0.607 | 0.606 | 0.427 |
| + VLM yes/no | 0.862 | 0.860 | 0.612 | 0.840 | 0.841 | 0.545 |
| + VLM yes-prob. | 0.890 | 0.891 | 0.615 | 0.873 | 0.877 | 0.564 |
| + verifier | 0.906 | 0.907 | 0.558 | 0.888 | 0.888 | 0.528 |
| + verifier, hook’s pick | 0.906 | 0.906 | 0.638 | 0.886 | 0.887 | 0.580 |
| RefCOCOg val | Ref-L4 | Stress | |
|---|---|---|---|
| Hook, top-1 | 0.856 | 0.530 | 0.446 |
| Correct box in hook’s top 5 | 0.982 | 0.775 | 0.714 |
| VLM over hook’s top 5: box list | 0.848 | 0.690 | 0.550 |
| Coordinate decoding | 0.886 | 0.873 | 0.671 |
| RefCOCO | RefCOCO+ | RefCOCOg | Ref-L4 val | ||||||
| Head | Query | val | val | val | test | all | COCO cat. | other | Stress |
| Learned bridge | image | 0.839 | 0.760 | 0.834 | 0.838 | 0.546 | 0.714 | 0.256 | 0.417 |
| text | 0.683 | 0.517 | 0.672 | 0.676 | 0.442 | 0.587 | 0.194 | 0.312 | |
| difference | |||||||||
| Relation-aware head | image | 0.872 | 0.802 | 0.877 | 0.878 | 0.595 | 0.798 | 0.247 | 0.358 |
| text | 0.687 | 0.530 | 0.721 | 0.725 | 0.466 | 0.629 | 0.186 | 0.310 | |
| Qwen2.5-VL-7B | Qwen3-VL-8B | |||||||
|---|---|---|---|---|---|---|---|---|
| Split | Hook | Matched | Relation | Hook | Matched | Relation | ||
| RefCOCO val | 0.834 | 0.850 | 0.872 | [.014, .030] | 0.853 | 0.872 | 0.905 | [.027, .039] |
| RefCOCO+ val | 0.760 | 0.774 | 0.802 | [.018, .039] | 0.800 | 0.818 | 0.853 | [.030, .041] |
| RefCOCOg val | 0.843 | 0.851 | 0.877 | [.018, .034] | 0.853 | 0.873 | 0.900 | [.019, .035] |
| RefCOCOg test | 0.837 | 0.848 | 0.878 | [.025, .036] | 0.856 | 0.877 | 0.899 | [.017, .028] |
| Ref-L4 val | 0.549 | 0.577 | 0.595 | [.008, .028] | 0.531 | 0.572 | 0.605 | [.008, .070] |
| Qwen2.5-VL | Qwen3-VL | Coordinate decoding | ||||||
|---|---|---|---|---|---|---|---|---|
| Subset | Rows (images) | Hook, k | Hook, full | Relation, full | Hook, full | Relation, full | Qwen2.5 | Qwen3 |
| A | ( ) | 0.839 [.829, .850] | 0.877 [.867, .886] | 0.913 [.905, .922] | 0.900 [.892, .908] | 0.935 [.927, .942] | 0.908 [.898, .918] | 0.930 [.921, .938] |
| B | ( ) | 0.834 [.822, .845] | 0.875 [.865, .885] | 0.903 [.894, .913] | 0.891 [.882, .901] | 0.925 [.916, .934] | 0.905 [.895, .916] | 0.926 [.917, .936] |
| C | ( ) | 0.726 [.714, .736] | 0.778 [.768, .788] | 0.822 [.812, .832] | 0.796 [.786, .805] | 0.872 [.863, .881] | 0.875 [.866, .885] | 0.905 [.897, .914] |
| All | ( ) | 0.789 [.783, .796] | 0.834 [.828, .840] | 0.872 [.866, .878] | 0.853 [.848, .859] | 0.905 [.900, .911] | 0.893 [.888, .899] | 0.918 [.913, .924] |
| RefCOCO | RefCOCO+ | |||||
|---|---|---|---|---|---|---|
| val | testA | testB | val | testA | testB | |
| All rows | ( ) | ( ) | ( ) | ( ) | ( ) | ( ) |
| Pooled query, cross-attention | 0.876 .001 | 0.902 .001 | 0.845 .002 | 0.808 .001 | 0.867 .001 | 0.744 .004 |
| Token-state probe | 0.886 .002 | 0.915 .001 | 0.852 .005 | 0.815 .003 | 0.875 .003 | 0.764 .004 |
| Combined selector | 0.895 .003 | 0.921 .001 | 0.864 .004 | 0.824 .002 | 0.884 .001 | 0.772 .001 |
| Relation-aware head | 0.874 .001 | 0.898 .002 | 0.846 .003 | 0.802 .002 | 0.855 .001 | 0.748 .003 |
| Combined selector minus | Relation-aware head | ||||
|---|---|---|---|---|---|
| Split | Rows | relation-aware head | token-state probe | decoding | minus decoding |
| RefCOCO val | all | [.015, .026] | [.004, .014] | [ .005, .007] | [ .026, .014] |
| C | [.025, .044] | [.001, .016] | [ .024, .004] | [ .060, .038] | |
| RefCOCO testA | all | [.017, .030] | [.001, .012] | [ .013, .000] | [ .037, .022] |
| C | [.022, .044] | [.002, .020] | [ .027, .005] | [ .062, .036] | |
| RefCOCO testB | all | [.009, .026] | [.004, .020] | [.002, .023] | [ .016, .006] |
| RefCOCOg | RefCOCO val | RefCOCO+ val | Ref-L4 val | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| System | val | test | All | C | All | C | All | COCO cat. | other | Stress |
| Token-state probe, union | 0.875 | 0.880 | 0.909 | 0.891 | 0.842 | 0.817 | 0.643 | 0.858 | 0.274 | 0.364 |
| RefCOCOg only | 0.871 | 0.878 | 0.886 † | 0.852 † | 0.815 † | 0.779 † | 0.627 | 0.834 | 0.272 | 0.355 |
| RefCOCOg part | 0.870 | 0.876 | 0.884 | 0.856 | 0.810 | 0.777 | 0.620 | 0.825 | 0.269 | 0.372 |
| Combined selector, union | 0.877 | 0.889 | 0.912 | 0.893 | 0.846 | 0.820 | 0.640 | 0.853 | 0.274 | 0.374 |
| RefCOCOg only | 0.876 | 0.885 | 0.895 † | 0.861 † | 0.824 † | 0.784 † | 0.641 | 0.852 | 0.280 | 0.381 |
| Union RefCOCOg only | Combined selector decoding | |||
|---|---|---|---|---|
| Split | Token-state probe | Combined selector | Union | RefCOCOg only |
| RefCOCOg val | [ .003, .012] | [ .005, .007] | [.003, .021] | [.001, .020] |
| RefCOCOg test | [ .003, .007] | [.000, .009] | [.017, .030] | [.012, .025] |
| RefCOCO val † | [.019, .028] | [.012, .022] | [.013, .024] | [ .005, .007] |
| C † | [.031, .046] | [.025, .040] | [.009, .027] | [ .024, .004] |
| RefCOCO+ val † | [.022, .034] | [.017, .028] | [.001, .015] | [ .021, .007] |
| Ref-L4 val | ||||
| Selector | RefCOCOg val | all | COCO cat. | other |
| InternVL3-8B | ||||
| Hook, image | 0.735 .005 | 0.446 .007 | 0.581 .007 | 0.213 .007 |
| Hook, text-only | 0.601 .007 | 0.404 .008 | 0.525 .011 | 0.196 .008 |
| Decoding | 0.894 | 0.786 | 0.917 | 0.561 |
| Qwen2.5-VL-7B | ||||
| RefCOCO val | RefCOCO+ val | RefCOCOg val | RefCOCOg test | Ref-L4 val | COCO cat. | other | Stress | |
| Proposal recall, no RefC | 0.999 | 0.999 | 0.996 | 0.997 | 0.952 | 0.998 | 0.873 | 0.983 |
| Proposal recall, orig. | 0.999 | 0.999 | 1.000 | 0.998 | 0.958 | 0.999 | 0.886 | 1.000 |
| Hook, no RefC | 0.806 | 0.733 | 0.809 | 0.811 | 0.512 | 0.657 | 0.264 | 0.398 |
| Hook, orig. | 0.834 | 0.760 | 0.843 | 0.837 | 0.549 | 0.718 | 0.259 | 0.413 |
| Relation-aware head, no RefC | 0.870 | 0.792 | 0.870 | 0.873 | 0.590 | 0.790 | 0.247 | 0.397 |
| RefCOCO val | RefCOCO+ val | ||||
| Image subset | Rows (images) | Hook | Coord. decoding | Detector-native | Hook |
| A: among the hook’s training images | ( ) | 0.839 [.829, .850] | 0.908 [.898, .918] | 0.869 [.858, .881] | 0.750 [.738, .762] |
| B: in RefCOCOg train, never seen | ( ) | 0.834 [.822, .845] | 0.905 [.895, .916] | 0.872 [.860, .884] | 0.757 [.744, .770] |
| C: not in RefCOCOg | ( ) | 0.726 [.714, .736] | 0.875 [.866, .885] | 0.814 [.803, .825] | 0.645 [.633, .656] |
| A B (memorization) | +0.006 | +0.003 | 0.003 | 0.006 | |
| B C (image selection) | +0.108 | +0.030 | +0.058 | +0.112 | |
| Comparison | Split | Difference | Two-level interval | Worst seed pair |
|---|---|---|---|---|
| Relation-aware head (IoU-rank loss) matched hook, k | RefCOCO val | [ , ] | [ , ] | |
| RefCOCO+ val | [ , ] | [ , ] | ||
| RefCOCOg test | [ , ] | [ , ] | ||
| Ref-L4 val | [ , ] | [ , ] | ||
| Stress (templates) | [ , ] | – | ||
| Relation-aware head (IoU-rank loss) matched hook, full | RefCOCO val | [ , ] | [ , ] |
| Method | RefCOCO val | RefCOCO+ val | RefCOCOg val | RefCOCOg test | Ref-L4 val | Stress |
|---|---|---|---|---|---|---|
| Seed variation: -interval over three training seeds | ||||||
| Naive cosine (untrained hook) | 0.078 [ .074, .229] | 0.076 [ .074, .227] | 0.082 [ .079, .242] | 0.083 [ .081, .248] | 0.065 [ .072, .201] | 0.048 [ .072, .167] |
| Learned bridge (CLIP query) | 0.511 [.468, .553] | 0.429 [.409, .449] | 0.542 [.527, .558] | 0.552 [.531, .574] | 0.357 [.341, .373] | 0.254 [.200, .308] |
| Index scorer | 0.759 [.754, .765] | 0.683 [.672, .694] | 0.770 [.766, .775] | 0.770 [.753, .787] | 0.444 [.420, .468] | 0.384 [.303, .465] |
| Index scorer, audited | 0.801 [.792, .810] | 0.722 [.701, .744] | 0.811 [.804, .818] | 0.815 [.809, .821] | 0.451 [.384, .518] | 0.434 [.391, .478] |
| Alignment hook | 0.789 [.772, .807] | 0.707 [.682, .733] | 0.787 [.777, .797] | 0.793 [.776, .810] | 0.464 [.439, .489] | 0.384 [.297, .471] |
| Split | Holm | ||||
| Hook vs index scorer (perm.) | |||||
| RefCOCO val | 0.789 | 0.759 | +0.030 | 0.004 | |
| RefCOCO+ val | 0.707 | 0.683 | +0.025 | 0.004 | |
| RefCOCOg val | 0.787 | 0.770 | +0.016 | 0.005 | |
| RefCOCOg test | 0.793 | 0.770 | +0.023 | 0.004 | |
| Ref-L4 val | 0.464 | 0.444 | +0.020 | 0.004 | |
| Choice | Selected by | Untouched counterparts and measured optimism |
|---|---|---|
| Veto thresholds | Faithfulness F1 over a grid on RefCOCOg val (present referents) with an earlier hook of the same recipe ( Sec. S18 ); frozen since | Not re-tuned on the features reported here: with absent referents the default veto reaches joint accuracy on RefCOCOg val against a val oracle of , and against on test. Selection accuracy without abstention does not depend on the thresholds. |
| Detector score threshold | Joint accuracy on RefCOCOg val with absent referents | Equal to the val oracle ( ); on test against an oracle of ; on the stress set and POPE – below the oracle. |
| VLM yes-probability threshold (yes-minus-no score ) | Joint accuracy on RefCOCOg val with absent referents | On test against an oracle of ; on POPE – and on the stress set below the oracle. |
| Agreement signal: re-ranking threshold and emitted box | RefCOCOg val, per seed | Tuned minus default choice: on RefCOCOg test and RefCOCO val, on Ref-L4 val, on the stress set. |
| Index-scorer learning rate and epochs (three swept variants, six configurations each) | Selection accuracy on RefCOCOg val, per seed | Val-minus-test gap of the swept variants minus that of the four unswept variants: , i.e. no measurable selection optimism. |
| Reported relation-aware head (best of five variants) | Selection accuracy on RefCOCOg val | The same variant is best on five of the six splits and second on the stress set. |
| RefCOCOg val + absent ( rows) | Stress + absent ( rows) | |||
|---|---|---|---|---|
| System | Joint | AURC | Joint | AURC |
| Coordinate-snap, always answer | 0.607 [.596, .618] | 0.393 [.382, .404] | 0.427 [.376, .482] | 0.573 [.518, .624] |
| Coordinate-snap + VLM yes/no | 0.840 [.831, .848] | 0.134 [.126, .143] | 0.545 [.491, .597] | 0.412 [.342, .486] |
| Coordinate-snap + VLM yes-probability, tuned | 0.873 [.865, .881] | 0.134 [.126, .143] | 0.564 [.512, .618] | 0.412 [.342, .486] |
| Coordinate-snap + verifier | 0.888 [.880, .895] | 0.146 [.137, .156] | 0.528 [.479, .578] | 0.377 [.309, .447] |
| Coordinate-snap + verifier at the hook’s proposal | 0.886 [.879, .893] | 0.161 [.151, .172] | 0.580 [.531, .629] | 0.394 [.327, .463] |
| testA, Pr@(F1 1) | testB | ||||
|---|---|---|---|---|---|
| System | All | No-tgt. | Single | Multi | All |
| Coordinate decoding | 0.336 | 0.030 | 0.907 | 0.107 | 0.326 |
| + VLM yes/no | 0.509 | 0.875 | 0.849 | 0.097 | 0.519 |
| + VLM yes-prob. | 0.478 | 0.665 | 0.893 | 0.106 | 0.477 |
| + verifier | 0.484 | 0.744 | 0.860 | 0.102 | 0.478 |
| Hook, default veto | 0.400 | 0.788 | 0.704 | 0.000 | 0.412 |
| Pr@(F1 1) | ||||||
|---|---|---|---|---|---|---|
| System | All | No-tgt. | Single | Multi | N-acc | T-acc |
| Coordinate decoding | 0.326 | 0.071 | 0.773 | 0.095 | 0.071 | 0.988 |
| + VLM yes/no | 0.519 | 0.840 | 0.692 | 0.089 | 0.840 | 0.790 |
| + VLM yes-prob. | 0.477 | 0.622 | 0.746 | 0.095 | 0.622 | 0.929 |
| + verifier † | 0.478 | 0.641 | 0.737 | 0.090 | 0.641 | 0.894 |
| Hook, default veto † | 0.412 | 0.681 | 0.608 | 0.000 | 0.681 | 0.739 |
| RefCOCO | RefCOCO+ | RefCOCOg | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | val | testA | testB | val | testA | testB | val | test | RefC. data | Proposals |
| Qwen2.5-VL-7B [ 3 ] | 90.0 | 92.5 | 85.4 | 84.2 | 89.1 | 76.9 | 87.2 | 87.2 | undiscl. | none |
| Qwen3-VL-8B ‡ [ 2 ] | average ; per split not published | yes | none | |||||||
| Shikra-7B [ 6 ] | 87.0 | 90.6 | 80.2 | 81.6 | 87.4 | 72.1 | 82.3 | 82.2 | yes | none |
| Ferret-7B [ 80 ] | 87.5 | 91.4 | 82.5 | 80.8 | 87.4 | 73.1 | 83.9 | 84.8 | yes | none |
| Griffon-13B [ 85 ] | 89.4 | 92.5 | 84.6 | 83.3 | 88.4 | 76.0 | 85.1 | 86.1 | yes | none |
| Component | Needs detector | (ms) | (ms) | Req/s, batch 1 | Req/s, batched (size) | Peak memory (GB) |
|---|---|---|---|---|---|---|
| Detector pass (Swin-B, ) | yes | 143.1 | 156.3 | 7.1 | 5.4 (8) | 2.46 |
| VLM prefill, query readout | – | 92.4 | 112.0 | 10.3 | 6.1 (16) | 15.77 |
| with FlashAttention-2 | – | 74.9 | 81.9 | 13.2 | 18.8 (16) | 15.77 |
| Coordinate decoding | no | 585.9 | 617.0 | 1.7 | 4.9 (16) † | 15.68 |
| Yes/no answer | no | 96.6 | 111.8 | 10.1 | 6.1 (16) | 15.67 |
| Alignment hook (selection and verifier) | yes | 0.37 | 0.38 | 2,688 | 355,919 (256) | 0.08 |
| Result | Build |
|---|---|
| Margin and separability analysis ( Tab. S18 ) | Earlier build, hooks retrained on it |
| Extended-vocabulary proposals ( Tab. S13 ) | Earlier build; Ref-L4 re-detected with the extended prompts; hooks of the margin analysis |
| In-VLM selection on the remaining splits ( Tab. S24 ) | Earlier build, seed-42 hook retrained on it |
| Verifier with harder negatives ( Tab. S33 ) | Earlier build, hooks retrained on it; gRefCOCO features extracted for this run |
| gRefCOCO testB ( Tabs. S65 and S66 ) | testB features extracted with the earlier build; hooks retrained on it (testA: main build) |
| Discrete phrase interface ( Tabs. S3 and S21 ) | Its caches rebuilt with the detector re-run; covered-proposal flags identical to the main build |
| Item | Value / notes |
|---|---|
| Frozen detector | Grounding DINO, Swin-B backbone, the authors’ public base checkpoint; fixed request-independent prompt of the COCO category names; shorter image side , longer side at most pixels; the top of its decoder queries by score, without non-maximum suppression or a score threshold ( Sec. S18 ). |
| Region feature dimension | 256: the last decoder layer’s state of each selected query, before the hook’s projection. |
| Language model | Qwen2.5-VL-7B, frozen. Query: final-layer hidden state at the last position of the chat template whose user turn is the image followed by the bare request (no instruction), with the assistant prefix appended, i.e. the position that would begin the answer; no grounding prompt precedes the readout. Multi-layer readout: the same position after layers 28, 21 and 14 (of 28), concatenated and passed through the head’s query projection ( Sec. S18 ). Images resized by the model’s processor to at most pixels. |
| Query feature dimension | |
| Proposal budget | |
| Veto thresholds | , ; selected once by faithfulness F1 on RefCOCOg val and fixed for every reported split (definition and provenance in Sec. S18 ). |