SCVIB: Editable State-Conditioned Visual Instance Binding for Multi-Turn Personalized Localization

Xiongtai Yang; Ziyan He; Tao Wang

cs.CVAug 14, 2026

SCVIB: Editable State-Conditioned Visual Instance Binding for Multi-Turn Personalized Localization

Authors: Xiongtai Yang, Ziyan He, Tao Wang

Organizations: Sichuan University, No. 24, South Section 1, First Ring Road, Wangjiang Campus, Chengdu, 610065, Sichuan, China

Abstract

Stored visual evidence can remain unchanged while its relevance to a current request changes with interaction state. To study this state-dependent source relevance, we introduce SCVIB, a controlled diagnostic benchmark for editable state-conditioned visual instance binding. SCVIB contains 1,050 manually verified support-query pairs and 1,500 multi-turn episodes across five visual domains and four target-state dependencies. Executable state semantics separate target resolution from evidence selection and query-side localization. Direct Qwen3-VL-8B inference achieves 99.53% target-resolution accuracy but 62.00% [email protected], revealing a 37.53-point resolution-localization gap; 563 of its 570 joint failures follow correct target identification. Routing comparisons and fixed-checkpoint support interventions diagnose a post-resolution evidence-utilization bottleneck: successful resolution must be coupled with relevant visual evidence and effective query-side grounding. We introduce TT-VG (Transition-Trace Visual Grounding), a state-conditioned visual grounding framework with an explicit and auditable state-to-evidence-to-grounding path. Target-State Transition Trace (TSTT) executes editable state and selects an auditable support record; Visual Evidence Grounding Adaptation (VEGA) learns support-conditioned grounding from target appearance, boxed support context, and the query. The canonical TT-VG configuration reaches 69.80% [email protected], improving over direct inference by 7.80 points at the combined evidence-utilization and grounding interface. Task adaptation improves the DINOv2 and MASA retrieval baselines to 51.47% and 55.67%, respectively, but both remain substantially below VEGA. These findings highlight state-dependent source selection and evidence utilization as consequential interfaces for interactive multimodal information fusion.

Figures & tables

Figure 1: Fixed-target versus editable state-conditioned localization. Left, a support specifies one target; right, state edits and rollback select relevant support. Red and green boxes mark support annotations and query targets.

Figure 2: SCVIB construction and target-state dependencies. Upper: 1,050 pairs across five domains and three curated strata. Lower: four dependencies, with red support boxes and illustrative green query targets. The pairs yield 1,500 controlled episodes.

Figure 3: TT-VG makes the state-to-evidence-to-grounding path explicit. TSTT selects the evidence record, the Visual Evidence Package combines its crop and boxed context with the query, and VEGA predicts the query box. The selected evidence is inspectable before localization.

Method	[email protected]	[email protected]	mIoU
GDINO–DINOv2	$49.33$	$41.53$	$48.34$
GDINO–MASA-R50	$53.80$	$48.60$	$50.07$
Box-initialized PerSAM	$31.20$	$28.53$	$31.40$
VEGA	69.80	62.33	61.52

Table 1: Query-side localization on 1,500 episodes at 100.00% target resolution under stated training conditions. All methods receive the same selected support and query. Scores are percentages; bold denotes the best. Table 4 provides retrieval adaptation controls using the same training pairs.

Figure 4: Qualitative examples of localization errors after correct target identification. Top: support reference (red annotation); bottom: query ground truth (solid green), Seq-free (dashed navy), and TT-VG (dotted rose). Supports are explanatory for Seq-free; display crops retain all boxes, and IoUs use full-image coordinates.

	Joint accuracy
Configuration	@0.5	@0.7	mIoU	TR
Seq-free	$62.00$	$52.40$	$55.14$	$99.53$
Router + VEGA	$69.33$	$61.87$	$61.22$	$99.53$
TT-VG	$69.80$	$62.33$	$61.52$	$100.00$

Table 2: Three-stage results on 1,500 episodes. Percentages are reported; Router + VEGA and TT-VG share VEGA weights.

Metric	$\Delta$	Episode CI	Cluster CI
[email protected]	+7.33	[4.87, 9.73]	[4.50, 10.07]
mIoU	+6.08	[4.23, 7.90]	[3.88, 8.21]

Table 3: Paired improvements with 95% bootstrap intervals. Differences are percentage points; cluster intervals resample episodes grouped by their target base pair.

Figure 5: Fixed-checkpoint support intervention on 150 pairs. Panels (a,b) show paired IoUs and success counts, with 0.5 thresholds and an equal-IoU diagonal; panel (c) illustrates correct versus mismatched support records, showing the support-context view from each full package (red annotations), query ground truth (solid green), and predictions (dashed rose). Changing the selected support record changes localization on the same query; the displayed example is illustrative.

	[email protected]		mIoU
Backend	Frozen	Adapted	Frozen	Adapted
DINOv2	$49.07$	$51.47$	$48.08$	$49.68$
MASA	$53.80$	$55.67$	$50.07$	$51.70$

Table 4: Paired retrieval-backend adaptation on all 1,500 episodes. Frozen and adapted conditions share selected supports and proposal pools; scores are percentages.

Figure 6: Failure composition and threshold sensitivity on 1,500 episodes. Stacked bars separate joint success, post-resolution localization failure, and resolution failure; curves retain all outputs across IoU thresholds. Most direct-inference failures occur after correct target identification.

Figure 7: Rollback comparison on all 91 episodes. Bars show [email protected] and success counts; Router + VEGA and TT-VG share VEGA weights. Restoration adds five successes (5.49 points); Appendix I gives all groups.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Event	Effect
introduce_only	Register; active unchanged.
set_active	Register; set active.
keep_active	Register distractor; active unchanged.
switch_active	Register; save active; switch.
mark_active	Set active to a registered name.
undo_previous_switch	Pop switch; restore active.

Appendix

Table A.1: Canonical executor events and their effects. The registry is append-only; rollback retains all introduced evidence.

Figure A.1: Visible-event distribution and an illustrative rollback execution. Panels (a,b) count compiled events in 1,500 histories on separate axes; panel (c) tracks the active target and available evidence through four edits. Undo restores the active target while preserving subsequently registered evidence.

Interface	Model-visible input and required output
Seq-free	Ordered support/update dialogue and final query; return one name-and-box JSON object.
State-Table	Same dialogue plus introduction order and a linearized state summary; return name and box.
VEGA	Target crop, full support with a red target box, then query; return {"bbox_2d":[x1,y1,x2,y2]} . No pseudo-name is supplied.

Appendix

Table A.2: Prompt interfaces and returned fields. Name references in the direct conditions retain the canonical name, active-target, or support-order instruction.

Interface	Convention
Seq-free / State-Table	Swap reversed endpoints, then clip to $[0,1000]$ . Missing or non-positive-area boxes receive zero IoU.
VEGA parser	Clip to $[0,1000]$ without swapping endpoints; invalid boxes receive zero IoU.
Single-turn input	Clip support/query annotations to pixel-index bounds $w-1,h-1$ before drawing, cropping, and serializing supervision.
Multi-turn scoring	Use the accompanying continuous query annotations.

Appendix

Table A.3: Coordinate handling and invalid-output rules. Proposal and mask outputs are normalized using query-image dimensions.

Localization method	Support representation	Query-side procedure
GDINO–DINOv2	Target crop	Grounding DINO proposals; DINOv2 similarity
GDINO–MASA-R50	Selected support	Grounding DINO proposals; MASA-R50 matching
Box-initialized PerSAM	Support box	Personalized segmentation; mask-to-box
VEGA	Visual Evidence Package	VEGA support-conditioned grounding

Appendix

Table A.4: Query-side localization interfaces. Every method receives the same TSTT-resolved support record and query image, fixing target resolution at 100.00%.

Parameter	Setting
Proposal model	GroundingDINO Swin-T OGC
Box / text threshold	0.05 / 0.05
Maximum proposals	100
Class-agnostic NMS IoU	0.70
Support / proposal crop margin	0.10
Matcher	DINOv2 ViT-B/14 + 4 registers

Appendix

Table A.5: GDINO–DINOv2 backend configuration. The full proposal vocabulary is specified alongside the table.

Setting	Value
LoRA	Rank 16; $\alpha=32$ ; dropout 0.05.
Adapted projections	Attention q/k/v/o ; MLP gate/up/down .
Supervision	Answer-token cross-entropy on the serialized normalized query box; prompt/padding tokens masked.
Optimizer	AdamW; learning rate $10^{-4}$ ; zero weight decay; gradient clipping 1.
Training schedule	One epoch; batch size 1; gradient accumulation 8.
Seed / precision	$s=3$ in Python, PyTorch and CUDA; bfloat16.

Appendix

Table A.6: Canonical VEGA adaptation and inference recipe. All routing conditions share the same checkpoint and Visual Evidence Package.

Source	Pairs	Episodes	Seq-free	TT-VG
Anime-Style	210	300	49.67	55.67
LaSOT	271	381	71.13	79.53
YFCC100M	147	212	65.57	74.06
Argoverse	86	127	62.20	74.80
BDD100K	99	141	49.65	63.12
Charades	27	39	84.62	94.87

Appendix

Table C.1: Descriptive [email protected] (%) by resolved underlying source. Underlying image sources and acquisition collections are distinguished in the inventory.

Configuration	[email protected]	[email protected]	mIoU	TR
Visible State-Table	58.13	48.20	51.98	99.40
Latest-Support + VEGA	19.07	16.33	30.52	28.00
Random-Support + VEGA	18.87	17.13	29.76	28.27
Seq-free (4B)	48.93	42.87	46.31	97.60

Appendix

Table D.1: Additional configurations evaluated on all 1,500 episodes. All reported scores are percentages.

Evidence	[email protected]	[email protected]	mIoU
Full	72.00	64.00	63.83
Boxed context	69.33	62.00	61.09
Crop	54.00	48.00	48.13
Mismatched support	23.33	22.00	23.02

Appendix

Table D.2: Fixed-checkpoint evidence controls on the same 150 pairs. Accuracy and mIoU are percentages; all conditions retain the full denominator.

Measure	Two-image Qwen	VEGA
[email protected] (%)	72.00	72.00
[email protected] (%)	62.00	64.00
mIoU (%)	61.99	64.03

Appendix

Table D.3: Matched-training-recipe pilot comparing two adaptations of Qwen3-VL-4B on 150 fixed pairs with correctly selected supports. These are grounding scores; the adaptations share training examples and optimizer settings, with distinct visual inputs and associated instructions.

Setting	DINOv2	MASA
Backbone	ViT-B/14 with four register tokens	R50
Updated component	Final transformer block and output norm	Existing 1024-to-256 embedding projection
Trainable parameters	7,090,944	262,400
Learning rate / decay	$10^{-5}$ / 0.04	$10^{-4}$ / 0
Visual preprocessing	Support/candidate crops with 10% margin; $224\times 224$ bicubic resize; ImageNet normalization	Full images with maximum side 1024 pixels; original ROI feature extraction
Computation	bfloat16 autocast in training and paired inference	Frozen upstream features cached for training

Appendix

Table D.4: Backend-specific adaptation settings. Visual preprocessing and trainable components follow the respective retrieval architectures.

Input	Configuration	[email protected]
Context + query	Frozen base model	29.05
Context + query	Trained context adapter	69.33
Crop + query	Frozen base model	53.24
Crop + query	Trained crop adapter; shuffled crops	16.29
Full package	VEGA	70.29

Appendix

Table D.5: Clean-input single-turn results across distinct training and input configurations. Context denotes boxed support; the crop-adapter row uses shuffled crops. Accuracy is in percent; VEGA uses 1,050 pairs.

Study	IDs	Texts/ID	Dependencies
Initial wording	80	1	32 / 14 / 25 / 9
Prompt revision	80	1	32 / 8 / 35 / 5
Wording transfer	40	4	16 / 6 / 16 / 2
Clarification	20	4	8 / 3 / 9 / 0

Appendix

Table E.1: Text-front-end sampling designs. Dependencies are Explicit-Name / Active-Target / Counter-Recency / Rollback; IDs are disjoint across studies.

Figure E.1: Text-front-end diagnostics on three separate sets. Each panel reports four passing-output counts out of 80; connected markers compare prompts on identical inputs. The initial and fresh-identifier sets contain 80 identifiers each, while clarification uses 20 further episodes with four renderings each. Revision improves the fresh-identifier scores; clarification has mixed effects.

Figure F.1: State edits on unchanged visual inputs. Each row fixes two supports and a query while varying one instruction; red boxes mark support annotations, solid green is query ground truth, and dashed rose is TT-VG. These selected examples illustrate active assignment and switch restoration; Table F.1 reports all 40 histories.

Measure	Frozen 4B	TT-VG
History-level evaluation
Target correct	40/40	40/40
[email protected] (%)	42.50	72.50
mIoU (%)	35.96	63.15
Pair-level evaluation (both branches)
Target correct	20/20	20/20

Appendix

Table F.1: Paired state-response diagnostic on 40 histories from 20 scene pairs. Pair-level metrics require correct predictions for both histories belonging to the same scene.

Measure	Two-image Qwen	VEGA
30 pairs: one adapter per process
Median image tokens	728	803
Median time (s)	1.814	1.890
Median peak allocated (GiB)	8.730	8.769
Maximum peak allocated (GiB)	8.741	8.858
Maximum peak reserved (GiB)	8.797	8.957

Appendix

Table F.2: Deployment measurements under two protocols. Ratios are median paired VEGA/two-image Qwen times. Single-adapter peaks measure process allocation (base model, adapter and activations), rather than total-device occupancy.

Figure G.1: Matched support-box sensitivity on the same 150 pairs. The panels give absolute accuracy and clean-minus-perturbed drops with stratified 95% intervals. All conditions share the checkpoint and renderer.

Figure G.2: Within-domain localization gains. Points show TT-VG minus Seq-free [email protected] with paired base-pair-cluster 95% intervals; each domain contains 300 episodes. Variation across domains characterizes the curated evaluation sample.

Figure G.3: Post-hoc scope sensitivity on fixed canonical predictions. Scores are percentages; points show TT-VG minus Seq-free with paired 95% intervals, and the right column gives the Router contrast. The TT-VG–Seq-free difference remains positive across the overlapping scopes.

Figure G.4: Localization scores and gains across query-area quartiles. Points show TT-VG minus Seq-free [email protected] with paired 95% intervals. The quartiles describe geometry separately from the curated strata.

IoU threshold	Difference	Paired 95% CI
0.30	5.53	[2.81, 8.34]
0.50	7.80	[4.87, 10.68]
0.70	9.93	[6.91, 13.07]
0.90	8.00	[5.21, 10.96]

Appendix

Table H.1: Selected IoU-threshold sensitivity. Differences are TT-VG minus Seq-free joint accuracy, in percentage points.

Dependency	$n$	Seq-free	Router	TT-VG
Explicit-Name	600	380	406	406
Active-Target	200	142	139	139
Counter-Recency	609	357	436	438
Rollback	91	51	59	64

Appendix

Table I.1: [email protected] success counts for all four dependencies. The three configurations share the same

n

within each row.

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

CoEvolve: Construct-to-Edit Visual Grounding with Bidirectional State Refinement
Oct 1, 2026Dongwei Sun, Yujie Zhang, Bowen Yao +3Object Detection Visual Grounding
School of Computer Science and Technology and the Ministry of Education Key Lab for Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an 710049, China · School of Computer Science and Technology, Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China · State Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China
FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization
May 29, 2026Mohammed Asad Karim, Vinay Kumar VermaVision-Language Grounding In-Context Learning
Amazon, Seattle, USA
Déjà Cue: Localizing States in Object Histories via Vocabulary-Relative Coordinates
Aug 3, 2026Haofan Cao, Zhichao You, Yunkai Yang +3Temporal Video Grounding Temporal IR
Southwest Jiaotong University, Chengdu, China · University of Leeds, Leeds, UK

Oct 1, 2026J/K move · Enter open · S save

CoEvolve: Construct-to-Edit Visual Grounding with Bidirectional State Refinement

Dongwei Sun, Yujie Zhang, Bowen Yao, Pei Liu+2

School of Computer Science and Technology and the Ministry of Education Key Lab for Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an 710049, China · School of Computer Science and Technology, Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China · State Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

Object Detection Visual Grounding

Open paper AskPDF arXiv

Visual grounding localizes an object described by language with a bounding box. Most multimodal grounding models compress target identification, spatial reasoning, and boundary estimation into one terminal prediction. Free-form rationales make reasoning linguistically explicit but do not necessarily expose measurable, editable spatial states. Intermediate localization errors are therefore difficult to diagnose and correct, allowing incorrect region choices and imprecise boundaries to persist in the final box. We introduce CoEvolve, a construct-to-edit framework that separates grounding into explicit state construction and state editing. Region-Evolution Reinforcement (RER) organizes grounding analysis into a progressive semantic--spatial trajectory, with each reasoning step committing to an explicit candidate region. Bidirectional Denoising Refiner (BDR) treats the reasoning text as fixed semantic context and refines the trajectory's coordinate fields through bidirectional same-position reconstruction. Geometry- and behavior-level objectives provide target geometry and edit-preference signals for consolidating reliable candidates, preserving accurate inputs, or correcting toward annotations. Evaluations cover natural-image and remote-sensing grounding. With a 9B backbone, CoEvolve rivals models up to 241B parameters in grounding accuracy. Under controlled corruption, a single BDR pass improves mean box overlap by over 27 percentage points, demonstrating strong recovery from substantial localization errors. State-source comparisons further support the complementarity of explicit state construction and source-matched editing. The project is at https://sundongwei.github.io/CoEvolve_Project/.

Abstract

Vision-Language Grounding

Explore similar work

CoEvolve: Construct-to-Edit Visual Grounding with Bidirectional State Refinement

FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimization

Déjà Cue: Localizing States in Object Histories via Vocabulary-Relative Coordinates