InstanceBench: Diagnosing Referential Reasoning and Target Identity in Referring Expression Segmentation

Yuchen Li; Shaoyang Zhou; Yiran Wang; Ruiyi Deng; Haoyu Wang; Ziru Wei; Zhen Zhao; Luping Zhou

cs.CVOct 7, 2026

InstanceBench: Diagnosing Referential Reasoning and Target Identity in Referring Expression Segmentation

Authors: Yuchen Li, Shaoyang Zhou, Yiran Wang, Ruiyi Deng, Haoyu Wang, Ziru Wei, Zhen Zhao, Luping Zhou

Organizations: The University of Sydney · The ATLAS Institute · Shanghai Artificial Intelligence Laboratory

Abstract

Referring Expression Segmentation (RES) links natural-language descriptions to pixel-level object masks. Yet standard evaluation provides limited insight into instance-level referential reasoning: it does not systematically distinguish referential logics, test target preservation across valid grounding paths, or separate target-selection from mask-generation errors. We introduce InstanceBench, an instance-centered diagnostic benchmark comprising 6,194 images, 9,264 target instances, and 25,077 human-verified expressions. Each target-centric expression set (TCES) fixes the image and target mask while pairing a minimal expression with a same-target variant that uses another valid cue or grounding path. A compact referential-logic taxonomy spans direct target evidence, same-class selection, relational and compositional grounding, and exclusion, while logic-critical construction suppresses simpler shortcuts. Identity-aware metrics measure target retention and set-level success while separating selection from mask-generation errors. Across 22 native-mask RES checkpoints from 18 model families, the strongest checkpoint reaches 67.1% mIoU but only 59.6% [email protected]. Controlled interventions confirm language sensitivity, while failure decomposition identifies target selection rather than mask decoding as the main bottleneck. On a controlled training subset, matched supervision improves identity-aware performance, showing that the diagnosed capability responds to targeted supervision. Collectively, InstanceBench supports a measure-diagnose-improve cycle: measuring target consistency across grounding paths, localizing failure sources, and evaluating targeted interventions.

Figures & tables

Figure 1: Overview of InstanceBench. (a) Independent RES scoring conflates target selection with mask quality. (b) A target-centric expression set (TCES) fixes the image and target while varying valid grounding paths. (c) Benchmark scope: the referential-logic taxonomy, evaluated models, and complementary metrics. (d) Diagnostics reveal reliability and identity gaps, localize failures to target selection, and assess matched supervision.

	Data		Diagnostic construction			Diagnostic evaluation
Benchmark	Real images	Pixel masks	Taxonomy scale	Controlled construction	Controlled same-target grounding-path variants	Logic-wise diagnosis	Set-level reliability	Selection–mask decomposition
RefCOCO/+/g	$\surd$	$\surd$	–
CLEVR-Ref+		$\surd$	5 categories	$\surd$		$\surd$
Cops-Ref	$\surd$		6 logic forms	$\surd$		$\surd$
FineCops-Ref	$\surd$		6 structures	$\surd$	$\surd$	$\surd$
gRefCOCO	$\surd$	$\surd$	–

Table 1: Comparison with representative published REC/RES benchmarks.

Figure 2: Referential-logic taxonomy. InstanceBench organizes nine categories into five evidence families. $G$ , $D$ , and $A/B$ denote the target, same-class distractors, and anchors; and , $B\!\rightarrow\!A\!\rightarrow\!G$ , and not denote joint constraints, nested anchor resolution, and exclusion.

Figure 3: InstanceBench construction pipeline.

Model / checkpoint	Venue	Group	oIoU	mIoU	[email protected]	[email protected]	[email protected]	Worst-path IoU
SAMTok-CO	CVPR’26	U	57.56	67.14	71.18	59.60	32.42	56.28
X2SAM	ECCV’26	U	55.77	65.92	68.50	56.60	38.73	54.43
UniPixel-7B	NeurIPS’25	U	54.68	62.78	63.18	51.82	24.61	52.07
Sa2VA-2B	TPAMI’26	U	53.44	58.83	59.13	45.37	21.83	45.87
X-SAM	AAAI’26	U	39.91	56.81	58.16	46.60	30.27	45.28
InstructSeg	ICCV’25	U	41.55	52.73	53.51	39.07	21.38	38.52

Table 2: Overall performance on the InstanceBench core test set (2,918 TCESs; 5,836 expressions). All values are percentages; best results are bold.

Figure 4: Referential-logic capability and failure localization. (a) Cells report expression-level mIoU across the nine canonical referential logics, revealing model-specific capability profiles. (b) At IoU 0.7, target-identity errors dominate failures on single-anchor, multi-anchor, and nested expressions. The dark inset denotes confirmed same-class switches, a strict subset of identity errors; the remaining green segment is correct-ID, poor-mask failure.

Model	Para. Ret.	Variant Ret.	Para. NCI	Variant NCI	$\Delta$ NCI [95% CI]	All-target-ID	Cond. mask IoU
SAMTok	97.55	88.47	0.565	0.151	$-0.414$ [ $-0.621,-0.242$ ]	62.47	97.48
X2SAM	96.75	89.18	0.662	0.201	$-0.462$ [ $-0.634,-0.295$ ]	58.32	98.05
UniPixel	97.42	89.16	0.770	0.287	$-0.483$ [ $-0.642,-0.322$ ]	58.98	92.31
Sa2VA	96.17	85.43	0.593	0.147	$-0.446$ [ $-0.612,-0.288$ ]	51.48	90.16
ConverSeg	96.11	87.22	0.660	0.340	$-0.320$ [ $-0.476,-0.169$ ]	51.62	90.30
PaDT	96.07	83.84	0.644	0.185	$-0.459$ [ $-0.602,-0.335$ ]	50.12	94.56

Table 3: Same-target consistency. Retention/NCI use 988 COCO-derived matched TCESs; dense All-target-ID and core-pair conditional mask IoU use 2,867 identity-eligible TCESs.

\Delta

NCI is variant minus paraphrase; brackets are 95% TCES-bootstrap CIs.

Figure 5: Qualitative diagnoses under a fixed target. Green: ground truth; blue: minimal prediction; red: wrong-instance variant; orange: incomplete target mask.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Subtype	Example	Identification requirement
Same-class set selection	Ordinal	“the second wine bottle from the left”	Order the same-class instances and select a specified index.
	Extremal	“the leftmost visible person”	Select the instance attaining a positional or visual extremum.
	Relative selection	“the larger of the two pizza slices”	Compare same-class instances using a visible property or distance.
	Subgroup-based	“the right pale-pink cruller among the two at the bottom”	Form a constrained subgroup before selecting one member from it.
Single-anchor relation	–	“the young girl immediately to the left of the woman cutting the pizza”	Resolve one independently identifiable anchor and apply one spatial, distance, contact, containment, facing, looking, or body-orientation relation.
Multi-anchor compositional grounding	Multi-anchor relation	“the dark suitcase between the wooden crate and the tan suitcase”	Jointly resolve at least two anchors whose combined constraints identify the target.

Appendix

Table 4: The complete InstanceBench referential-logic taxonomy. Categories are defined by the evidence and inference required to uniquely identify the target. An en dash indicates that a family is not further divided into subtypes.

Model	Group	Language backbone	Extra	Audited downstream training scope
X2SAM	U	Qwen3-VL	Yes	RefCOCO/+/g plus generic, reasoning, grounded-conversation, interactive, and visual-grounded segmentation over images and videos.
SAMTok	U	MLLM	Yes	Large-scale mask-tokenizer pretraining and mask–text conversations spanning RES, instance/scene parsing, region QA, and interactive reasoning.
UniPixel	U	MLLM	Yes	Joint image/video referring, reasoning, video segmentation, captioning, and pixel-QA-related tasks.
Sa2VA	U	Qwen3-VL	Yes	Image/video instruction tuning for chat, RES, referring-video segmentation, and grounded caption generation.
X-SAM	U	MLLM	Yes	Conversation alignment followed by generic, referring, reasoning, grounded-conversation, interactive, and visual-grounded segmentation.
HyperSeg / InstructSeg	U	MLLM	Yes	Joint generic/referring/reasoning and image/video segmentation mixtures.

Appendix

Table 5: Training-regime audit for representative evaluated checkpoints. “Extra” indicates downstream supervision beyond classic RefCOCO-family RES.

	Localization and mask quality			TCES reliability
Model	Box [email protected]	oIoU	mIoU	[email protected]	All-target-ID	Variant ret.
Qwen2.5-VL-7B	66.91	43.45	53.20	42.21	52.27	77.59
Qwen2.5-VL-32B-AWQ	73.57	49.20	58.98	48.99	59.73	81.42
Qwen2.5-VL-72B-AWQ	75.64	52.07	61.01	51.46	61.44	81.73
Qwen3-VL-4B	73.83	53.90	63.20	53.99	59.94	78.85
Qwen3-VL-8B-Thinking	71.36	55.56	60.84	52.04	57.67	79.14

Appendix

Table 6: Direct localization by general-purpose MLLMs on the InstanceBench test set. Each predicted box is converted to a mask by the same frozen SAM3 backend. The resulting two-component pipeline is diagnostic and is not directly comparable with native-mask RES models. All values are percentages.

Model	Expr.-ID	Minimal-ID	Variant-ID	All-target-ID	Variant ret.
Qwen2.5-VL-7B	43.95	46.69	41.21	30.61	65.57
Qwen2.5-VL-32B-AWQ	53.38	55.09	51.67	38.91	70.63
Qwen2.5-VL-72B-AWQ	54.11	56.03	52.20	41.63	74.30
Qwen3-VL-8B-Thinking	45.08	50.24	39.92	29.50	58.71
Qwen3.8-27B	66.07	68.41	63.74	54.22	79.26
Gemini-3.8-Flash	76.29	77.30	75.28	65.17	84.30

Appendix

Table 7: Oracle same-class candidate-ID diagnosis on the InstanceBench test set. The protocol supplies the source-annotated candidate inventory

\mathcal{C}_{t}

and therefore isolates target selection within the annotated instance space. It is an upper-context diagnostic rather than the standard RES input protocol. All values are percentages.

Architecture	Supervision	mIoU	[email protected]	All-target-ID
LAVT	Source text	$33.02{\pm}0.69$	$13.77{\pm}0.54$	$25.95{\pm}1.00$
	50/50 mixed	$36.77{\pm}1.49$	$18.68{\pm}1.91$	$32.26{\pm}0.78$
	Logic-aware tracks	$38.02{\pm}0.57$	$20.84{\pm}0.33$	$33.51{\pm}0.42$
DETRIS	Source text	$43.86{\pm}0.05$	$29.52{\pm}0.33$	$37.67{\pm}0.07$
	50/50 mixed	$45.64{\pm}0.06$	$32.07{\pm}0.27$	$42.00{\pm}0.11$
	Logic-aware tracks	$44.79{\pm}0.61$	$31.27{\pm}1.50$	$41.19{\pm}1.44$

Appendix

Table 8: Matched-supervision study on the controlled COCO-derived subset (mean

\pm

standard deviation over three seeds). Images, masks, rows, optimization steps, initialization, and seeds are fixed across supervision settings; only expression supervision changes.

Model	Target evidence (1,273)	Single anchor (1,830)	Extremal (401)	Subgroup (321)	Multi- anchor (635)	Ordinal (403)	Exclusion (653)	Nested (207)	Relative (113)
SAMTok	78.84	67.65	76.81	65.50	60.32	57.82	54.56	52.14	69.23
X2SAM	79.19	66.00	73.75	60.59	58.66	57.92	53.95	47.84	74.07
UniPixel	75.41	64.04	73.30	57.23	57.09	45.59	51.77	46.73	64.66
X-SAM	72.69	57.06	71.45	49.43	47.47	48.07	40.95	33.25	61.12
Sa2VA	76.21	58.04	71.67	52.89	49.31	42.87	45.37	42.91	64.34
ConverSeg	70.89	57.35	67.51	51.23	47.34	43.54	46.17	40.83	60.16

Appendix

Table 9: Expression-level mIoU by canonical referential-logic category on the InstanceBench test set. Counts are shared across models.

Figure 6: Supplementary identity-retention and supervision views. (a) Descriptive target-ID retention conditioned on a correct minimal prediction. Each segment connects the same model’s paraphrase and alternative-path retention; the strictly matched comparison and normalized consistency analysis are reported in Table 3 . (b) Paired gains over the source-text control in the three-epoch subset study. Points are mean within-seed differences and error bars are their standard deviations over seeds. Both panels are supplementary: panel (a) uses all eligible dense-track instances, while panel (b) measures a short continuation rather than general out-of-domain reasoning improvement.

Treated $-$ control	$N$	X2SAM	SAMTok	UniPixel	X-SAM
Ordinal $-$ extremal	349	$-10.27\;[-15.82,-4.89]$	$-17.02\;[-22.65,-11.53]$	$-25.52\;[-30.51,-20.29]$	$-20.86\;[-26.23,-15.51]$
Multi-anchor $-$ spatial	618	$-3.13\;[-7.27,1.04]$	$-1.63\;[-5.66,2.47]$	$-1.83\;[-5.28,1.78]$	$-2.17\;[-5.69,1.54]$
Nested $-$ spatial	200	$-13.09\;[-20.43,-6.02]$	$-7.33\;[-14.45,-0.83]$	$-6.61\;[-12.90,0.30]$	$-9.84\;[-15.87,-3.78]$

Appendix

Table 10: Matched referential-logic effects in mIoU percentage points. Each cell reports treated minus control mIoU with a 95% TCES-clustered bootstrap confidence interval. Candidate controls are matched exactly by normalized target category and source stream, then selected by nearest-neighbor distance over log target-area fraction, log same-class count, and expression length; controls may be reused.

N

is the number of matched treated TCESs.

Model	mIoU	Drop	Donor IoU	Donor pref.	Logic-explicit	Deliberation	SC context
	Category only		Text swap		Inference-time aid ( $\Delta$ mIoU)
X2SAM	24.29	$-44.48$	80.61	93.50	$-0.50$	$+0.13$	$+0.58$
SAMTok	24.84	$-44.91$	79.46	94.23	$-0.40$	$-0.16$	$-1.52$
UniPixel	21.24	$-43.78$	77.12	93.07	$-0.29$	$-0.44$	$+0.99$
X-SAM	22.34	$-36.95$	76.60	90.29	$-1.54$	$-2.05$	$+2.34$

Appendix

Table 11: Complete controlled-intervention results. Category-only drop is relative to minimal-expression mIoU. Donor preference measures whether a same-image, same-category text swap redirects the prediction toward the newly described instance. The final three columns report paired mIoU changes, in percentage points, relative to the corresponding original expressions.

Model / checkpoint	Minimal	Paraphrase	Same-target	Exclusion	Paired exclusion drop
SAMTok-CO	69.73	82.94	66.36	57.76	13.03
X2SAM	68.74	82.22	65.08	56.90	13.81
UniPixel-7B	65.00	80.00	62.25	54.46	13.96
Sa2VA-2B	62.22	78.63	56.97	48.04	17.52
X-SAM	59.30	79.59	56.57	44.02	14.42
ConverSeg-3B	59.06	75.59	56.35	48.35	12.43

Appendix

Table 12: Track-level mIoU on the 7,370-expression dense test set. Same-target denotes alternative valid grounding evidence, and exclusion is grouped by logic category. Paired exclusion drop compares 589 exclusion–minimal pairs from the same TCES. Optional tracks cover different TCES subsets, so their raw mIoUs are descriptive.

Model	COCO	OI	$\Delta$	COCO	OI	$\Delta$
	Expression-level mIoU			TCES-level [email protected]
SAMTok-CO	76.86	57.35	$-19.52$	75.34	43.74	$-31.60$
X2SAM	77.80	53.96	$-23.84$	75.46	37.62	$-37.84$
UniPixel-7B	74.61	50.86	$-23.75$	70.15	33.36	$-36.79$
Sa2VA-2B	71.82	45.74	$-26.08$	64.69	25.93	$-38.76$
ConverSeg-3B	70.07	43.52	$-26.56$	61.95	24.00	$-37.95$

Appendix

Table 13: Source-specific expression-level accuracy and TCES-level reliability on the core test (1,464 COCO and 1,454 OpenImages TCESs; two expressions per TCES).

\Delta

is OpenImages minus COCO. X2SAM and X-SAM are three-run means. Source differences are descriptive because the two subsets also differ in visual domain, object-category distribution, and referential-logic composition.

Model	Para. Ret.	Variant Ret.	Excl. Ret.	Para. [email protected]	Variant [email protected]	Excl. [email protected]
SAMTok	97.45	83.51	75.57	97.12	83.64	75.17
X2SAM	96.67	81.81	71.03	96.43	81.96	72.34
UniPixel	97.44	83.06	73.05	96.50	81.66	73.01
Sa2VA	95.97	77.73	67.03	95.34	74.99	63.75
ConverSeg	96.02	79.01	69.41	95.10	76.58	64.03
PaDT	96.09	78.60	68.77	95.98	82.57	71.77

Appendix

Table 14: Full dense-track identity retention and target-and-mask retention. Unlike the strictly matched analysis in Table 3 , each column here uses all TCESs containing that optional track, so these values are descriptive rather than a controlled path-change comparison.

Figure 7: Source-specific referential-logic profiles. Cells report expression-level mIoU; parentheses give the number of expressions in each source–logic slice. Across all nine models and in both sources, ordinal expressions score below extremal descriptions and nested grounding scores below single-anchor relations. The exact within-source ordering is not identical, but the principal difficulty pattern is stable.

		Ordinal $-$ extremal		Nested $-$ single-anchor
Model	$\rho$	COCO	OI	COCO	OI
SAMTok	0.867	$-25.52$	$-8.12$	$-14.10$	$-11.36$
X2SAM	0.833	$-17.78$	$-8.75$	$-16.23$	$-12.37$
UniPixel	0.950	$-29.91$	$-20.69$	$-23.68$	$-7.99$
X-SAM	0.750	$-28.36$	$-10.69$	$-16.87$	$-17.57$
Sa2VA	0.833	$-37.89$	$-14.49$	$-19.43$	$-6.02$

Appendix

Table 15: Cross-source stability of referential-logic difficulty.

\rho

is the Spearman correlation between each model’s nine-category mIoU rankings on COCO and OpenImages. Gap columns are mIoU percentage-point differences; negative values mean that ordinal is harder than extremal or that nested is harder than single-anchor, respectively.

Source	Logic category	$N$	Identity error	Same-class switch	Correct ID, poor mask
COCO	Single-anchor	984	66.46	59.45	33.54
COCO	Multi-anchor	205	77.56	69.76	22.44
COCO	Nested	132	78.79	70.45	21.21
OpenImages	Single-anchor	2,289	76.45	51.99	23.55
OpenImages	Multi-anchor	1,211	81.34	56.48	18.66
OpenImages	Nested	468	73.50	47.65	26.50

Appendix

Table 16: Source-specific failure decomposition at IoU 0.7, pooled over the five U-group models SAMTok, X2SAM, UniPixel, Sa2VA, and X-SAM.

N

counts model–expression failures, not unique benchmark expressions. Same-class switches are a strict subset of identity errors.

Figure 8: Full-dataset composition. (a) The six most frequent non-person target categories, with the person share reported in-panel. (b) Coverage of all 18,528 core expressions over the nine canonical referential-logic categories. Both panels summarize the complete 9,264-TCES release.

Figure 9: Supplementary test-set composition and grounding-path statistics. (a) COCO/RefCOCO and OpenImages contribute complementary image distributions and similar numbers of fixed target instances. (b) The full transition matrix reports the family of the minimal expression (rows) and same-target variant (columns) for all 2,918 test-set TCESs.

Logic category	Identity error	Same-class switch	Correct ID, poor mask
Single-anchor	73.45	54.23	26.55
Multi-anchor	80.79	58.40	19.21
Nested	74.67	52.67	25.33

Appendix

Table 17: Failure decomposition at IoU 0.7, aggregated over the five universal models in Table 3 . Same-class switches are a strict subset of identity errors.

Min. score	Min. margin	Core-pair All- target-ID	Identity error	Same-class switch	Unresolved	Correct ID, poor mask
0.05	0.00	59.77	73.28	58.04	15.24	26.72
0.05	0.05	59.04	75.31	55.51	19.80	24.69
0.10	0.00	59.60	73.64	57.67	15.98	26.36
0.10	0.05	58.92	75.55	55.17	20.38	24.45
0.10	0.10	58.53	76.73	53.73	22.99	23.27
0.15	0.05	58.86	75.80	54.81	20.99	24.20

Appendix

Table 18: Sensitivity to the identity-assignment score and margin thresholds, pooled over the five universal models used in Table 17 . Core-pair All-target-ID is computed over 14,335 model–TCES pairs. Failure columns use the 5,289 predictions below IoU 0.7 in the single-anchor, multi-anchor, and nested categories; identity error is the sum of confirmed same-class switches and unresolved assignments. Bold indicates the default setting.

Explore similar work

Aug 25, 2026cs.CV

GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning

Referring segmentation in overhead imagery is inherently relational: a query may ask for the buildings north of the road or the pond closest to a residential area, so the correct referent can contain one object, several objects, or none. Existing benchmarks mainly score mask overlap, which cannot verify whether a model actually resolved the stated spatial relation. We introduce GeoRefer-Bench, a benchmark for verifiable geospatial referring segmentation. Each query is represented by an executable logical form over a metric scene graph, and predictions are evaluated with Exact Query Success (EQS), which is satisfied only when the returned instance set exactly matches the set denoted by the query. GeoRefer-Bench contains 700 whole 2048x2048 UAV scenes (2.94 Gpx) at 12.5 and 25 cm ground sampling distance, 26,217 instances, 142,796 spatial relations, and 20,916 executable queries spanning five reasoning levels. It further includes three paraphrases per query, 24.0% unanswerable queries, 2,477 counterfactual pairs, and five leakage-controlled evaluation splits. An independent audit re-derives object geometry, mask ownership, relation values, query execution, and split provenance, finding zero issues across all 700 scenes. Relation-blind strategies can retain non-trivial mIoU while achieving at most 22.7 EQS overall, showing that overlap alone does not certify relational grounding. Across fifteen current models, the strongest reaches 74.1 EQS but drops from 98.9 at level 1 to 60.5 at level 5, while ten models score below 5 EQS on two-hop queries. GeoRefer-Bench turns geospatial referring segmentation from mask matching into verifiable reference resolution.

Shuaishuai Cao, Min Huang, Meng Tang +3

Central South University · Jiangxi Normal University · Hohai University +2

Spatial Reasoning Benchmarks Geospatial Reasoning Referring Image Segmentation Reasoning Segmentation Remote Sensing Benchmarks

Ask PDF

May 19, 2026cs.CV

SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction

Referring segmentation grounds natural-language queries to pixel-level masks, but extending it to complex scenarios with multiple instances, cross-category groups, or open-ended target sets remains challenging. Previous Large Vision Language Model (LVLM)-based methods represent referred targets with one or more special tokens sequentially, treating multiple targets as separate outputs rather than a coherent set and offering little incentive to capture set-level properties such as completeness and mutual exclusivity. We reformulate open-ended referring segmentation as explicit set-level concept prediction and propose Set-Concept Segmentation (SetCon), which uses LVLM-generated natural-language concepts, instead of segmentation-specific tokens, as semantic conditions for joint mask-set decoding. A hierarchical semantic decomposition first predicts a shared set-level concept defining the target scope and then refines it into fine-grained concept groups aligned with target subsets. To support this, a two-stage annotation pipeline augments existing reasoning segmentation datasets with hierarchical semantic supervision (236k samples, 784k concept phrases). SetCon achieves state-of-the-art results on image benchmarks (+3.3 gIoU on gRefCOCO, +12.1 gIoU on MUSE), with margins that grow as the number of referred targets increases. The concept interface also transfers to video under a detect-and-track setting, yielding new state-of-the-art results on seven referring video benchmarks, including +10.9 J&F on MeViS and +12.4 J&F on Ref-SeCVOS.

Zhixiong Zhang, Yizhuo Li, Shuangrui Ding +6

1Shanghai Jiao Tong University · 2Shanghai Innovation Institute · 3The Chinese University of Hong Kong +3

Image Segmentation Referring Video Object Segmentation Referring Image Segmentation

Ask PDF

May 8, 2026cs.CV

ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring

Referring expression grounding is a core problem in visual grounding and is widely used as a diagnostic of spatial grounding and reasoning in vision and language models, yet most prior work focuses on natural images. In contrast, existing chart referring expression grounding-related benchmarks remain limited: (1) they largely adopt bounding boxes, constraining localization precision for fine chart elements (2) they mostly assume a single and two referred target instances, failing to handle multi-instance target references; (3) the language expressions over-rely on textual cues or data-rank clues (4) they cover only a narrow range of chart types. To address these issues, we introduce a chart referring expression grounding benchmark that systematically supports multiple localization forms, multiple referred targets, diverse grounding cues and diverse chart types. Results across representative multimodal large models reveal a significant performance gap. We further introduce a code-driven synthesis pipeline that exploits the inherent alignment between plotting programs and rendered chart primitives to derive pixel accurate instance masks across chart element types and granularities. We train an instance segmentation model with the synthesized masks and integrate it into a general-purpose multimodal grounding framework. The resulting system consistently outperforms baselines on our benchmark and generalizes well to a ChartQA-derived real-chart grounding benchmark.

Tianhao Niu, Ziyu Han, Xuan Dong +2

Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, China

Referring Image Segmentation Visual Grounding Chart Understanding Multimodal Model Evaluation

Ask PDF