Agreement versus Softmax under Spatial Disambiguation

Michał Zakrzewski; Mariia Shpir; Taras Rumezhak; Volodymyr Karpiv

cs.LGOct 9, 2026

Agreement versus Softmax under Spatial Disambiguation

Authors: Michał Zakrzewski, Mariia Shpir, Taras Rumezhak, Volodymyr Karpiv

Organizations: AGH University of Krakow, Krakow, Poland · SoftServe · ETH Zurich

Abstract

ScanRefer Multiple queries require spatial disambiguation: several instances of the target class are present, and the referencer must pick one. When per-query logits were never logged, a common substitute is cross-epoch agreement. On the full ScanRefer validation set (9,508 queries; GeoClassifier with ground-truth proposals), object-level softmax is similarly miscalibrated on Unique and Multiple (ECE 0.068 vs. 0.102; the scene-bootstrap gap interval includes zero). Fixed-final agreement on the same checkpoints reports ECE 0.041 vs. 0.272 (gap +0.231). Modal-vote agreement is 0.045 vs. 0.318. Most Multiple errors are same-class wrong-instance (45.6%); geometric miss (H2) is 59.6%, and wrong-category (H1) is 16.7%. A non-monotone H1 pattern does not appear under the default fixed-final protocol, and we retract it. Logit-free thresholds rank spatially hard queries poorly. If only agreement was logged, calibrate abstention on Multiple and report wrong-instance rates with H1.

Figures & tables

Split	$n$	Softmax ECE	Fixed-final ECE	Modal ECE
Unique	$1{,}759$	$0.068$	$0.041$	$0.045$
Multiple	$7{,}749$	$0.102$	$0.272$	$0.318$
Gap (M $-$ U)		$+0.034$	$+0.231$	$+0.273$

Table 1: Full ScanRefer val ECE. Softmax and fixed-final share the final-checkpoint prediction. Modal-vote uses a different identifier. Gaps are Multiple minus Unique.

Figure 1: Reliability on Unique and Multiple for native max-softmax and fixed-final agreement (same $9{,}508$ queries as Table 1 ). The dotted line is perfect calibration.

Split	Acc	H1	Wrong-instance	H2
Unique	$0.837$	$14.9\%$	$1.4\%$	$16.0\%$
Multiple	$0.377$	$16.7\%$	$45.6\%$	$59.6\%$

Table 2: Error types on the final-checkpoint identifier (fixed-final / softmax). Rates are fractions of queries, not of errors.

$\tau$	Coverage	Acc	Wrong-inst.	H1	H2
$0$	$1.000$	$0.377$	$0.456$	$0.167$	$0.596$
$5/9$	$0.671$	$0.472$	$0.387$	$0.141$	$0.502$
$8/9$	$0.325$	$0.595$	$0.316$	$0.089$	$0.371$
$1$	$0.184$	$0.647$	$0.266$	$0.087$	$0.319$

Table 3: Fixed-final abstention on Multiple (

n{=}7{,}749

\tau

is agreement with the final checkpoint among the other nine dumps.

Figure 2: ECE and accuracy by same-class candidate count on ScanRefer val. Count $1$ is the Unique split. Counts $2$ and above are Multiple.

Tier	$n$	Softmax ECE	Fixed-final ECE
Easy (count $2$ )	$4{,}118$	$0.177$	$0.279$
Hard (count $\geq 3$ )	$4{,}466$	$0.195$	$0.364$
Gap (H $-$ E)		$+0.018$	$+0.083$

Table 4: Nr3D val ECE by multiplicity tier (easy

=

count

2

, hard

=

count

\geq 3

). Gaps are hard minus easy.

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 420 papers

GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning
Aug 25, 2026Shuaishuai Cao, Min Huang, Meng Tang +3Spatial Reasoning Benchmarks Geospatial Reasoning
Central South University · Jiangxi Normal University · Hohai University +2
Confidence Calibration under Ambiguous Ground Truth
Mar 24, 2026Linwei Tao, Haoyang Luo, Minjing Dong +1Post-Hoc Calibration Annotator Disagreement
L. Tao and C. Xu are with the School of Computer Science, University of Sydney, Sydney, NSW 2006, Australia. · H. Luo and M. Dong are with the Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China.
Beyond Referring Expressions: Scenario Comprehension Visual Grounding
Apr 2, 2026Ruozhen He, Nisarg A. Shah, Qihua Dong +3Vision-Language Grounding Visually Grounded Reasoning
Rice University · Johns Hopkins University · Northeastern University

Aug 25, 2026J/K move · Enter open · S save

GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning

Shuaishuai Cao, Min Huang, Meng Tang, Xuan Liu+2

Central South University · Jiangxi Normal University · Hohai University+2

Spatial Reasoning Benchmarks Geospatial Reasoning Referring Image Segmentation Reasoning Segmentation Remote Sensing Benchmarks Geospatial Reasoning Benchmarks

Open paper AskPDF arXiv

Referring segmentation in overhead imagery is inherently relational: a query may ask for the buildings north of the road or the pond closest to a residential area, so the correct referent can contain one object, several objects, or none. Existing benchmarks mainly score mask overlap, which cannot verify whether a model actually resolved the stated spatial relation. We introduce GeoRefer-Bench, a benchmark for verifiable geospatial referring segmentation. Each query is represented by an executable logical form over a metric scene graph, and predictions are evaluated with Exact Query Success (EQS), which is satisfied only when the returned instance set exactly matches the set denoted by the query. GeoRefer-Bench contains 700 whole 2048x2048 UAV scenes (2.94 Gpx) at 12.5 and 25 cm ground sampling distance, 26,217 instances, 142,796 spatial relations, and 20,916 executable queries spanning five reasoning levels. It further includes three paraphrases per query, 24.0% unanswerable queries, 2,477 counterfactual pairs, and five leakage-controlled evaluation splits. An independent audit re-derives object geometry, mask ownership, relation values, query execution, and split provenance, finding zero issues across all 700 scenes. Relation-blind strategies can retain non-trivial mIoU while achieving at most 22.7 EQS overall, showing that overlap alone does not certify relational grounding. Across fifteen current models, the strongest reaches 74.1 EQS but drops from 98.9 at level 1 to 60.5 at level 5, while ten models score below 5 EQS on two-hop queries. GeoRefer-Bench turns geospatial referring segmentation from mask matching into verifiable reference resolution.

Abstract

Vision-Language Grounding

Explore similar work

GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning

Confidence Calibration under Ambiguous Ground Truth

Beyond Referring Expressions: Scenario Comprehension Visual Grounding