Ask About the Right Moment: Training-Free Object Identity Reasoning for Video-Language Models

Di Yang; Xiaochang Li; Muyang Liu; Yufei Wang; Yanhai Xiong

cs.CVOct 22, 2026

Ask About the Right Moment: Training-Free Object Identity Reasoning for Video-Language Models

Authors: Di Yang, Xiaochang Li, Muyang Liu, Yufei Wang, Yanhai Xiong

Organizations: College of William & Mary · SparcAI Inc.

Abstract

Many questions about a video concern one physical object. Where does the ball end up after the cups are shuffled? Which card is the queen? How often was the mug covered? Video-language models (Video-LMs) recognise such objects in single frames but do not keep them apart over time. On real shell-game videos, open models from 8B to 235B parameters, their thinking variants, a reinforcement-learned tool agent and three training-free tool agents all stay within ten points of chance. We separate semantic identity binding from temporal identity maintenance. An object's identity should be fixed at a moment when the evidence for it is visible, and carried afterwards by a tracker. Track, Then Bind implements this principle without training. A promptable video tracker follows every candidate object and draws it with a track number. A frozen vision-language model answers a static question about the identifying evidence, which binds the target to a track. The answer is then derived from the tracked state in text, without asking the model to reinterpret the original video during read-out. A single configuration covers all benchmarks. With Qwen3-VL-32B it solves both VET-Bench tasks, where a fine-tuned trajectory model reaches 92% and 90% and the same VLM voting over 16 samples stays at 30--32%. It answers 55.4% of the hardest Perception Test cup games, where the best published model answers 41.5%, and, using track-derived timelines, 48.6% of TOC-Bench questions, against 37.1% for the same model answering directly. Drawing the same tracks on the video and asking the question directly falls 12--19 points short of the agent on real cup games.

Figures & tables

Panel of Figure 2 — Figure 2: Method. A frozen VLM plans the question (1) and boxes its entities, which SAM2.1 tracks as numbered instances (2). If a target must be told apart from look-alikes or containers, the VLM answers one static question about the evidence moment on the numbered clip (3–4), and the answer is read out from the tracked state in text (5a; VET card monte card_003 ). Otherwise the tracks become a timeline that is appended to the benchmark prompt (5b; TOC-Bench 8b0c33d38075 ). Both examples are logged test runs with Qwen3-VL-8B.

Method	Type	VET cup	VET card	PT	PT-65	TOC
Random	–	33.3	33.3	33.3	33.3	30.3
Closed Video-LMs (cited: VET and PT from [ 1 ] , TOC from [ 3 ] )
Gemini-3-Pro [ 62 ]	direct answer	40.0	34.0	36.5	30.8	43.9 ∗
GLM-4.6V-Flash [ 64 ]	direct answer	28.0	34.0	43.9	41.5	–
Doubao-Seed-2.0-mini [ 65 ]	direct answer	32.0	30.0	46.7	40.0	–
GPT-5.5 [ 66 ]	direct answer	–	–	–	–	47.1 ∗

Table 1: Accuracy (%) on the test sets. VET: VET-Bench, 50 videos per task. PT: Perception Test cup games filtered by the VET-Bench authors (107 videos); PT-65: its 65 videos with three cups and at least one swap. TOC: TOC-Bench test split (1,845 questions). Closed models are cited; all other rows were evaluated by us with each benchmark’s prompt, frames and scorer. Best result per column in bold.

		8B		32B
Dimension	$n$	D	T	D	T
event existence	170	34.7	67.1	42.9	66.5
reappear vs. disappear	106	41.5	71.7	41.5	75.5
relative spatial change	131	24.4	51.9	31.3	52.7
event count	241	13.3	29.0	20.3	32.0
event ordering	89	12.4	27.0	13.5	32.6

Table 2: TOC-Bench test accuracy by dimension (%). D: Qwen3-VL answering directly (vLLM); T: Track, Then Bind with the same model.

VLM	Baseline	cup	card	PT	TOC	Calls	Tokens
8B	direct answer	28.0	28.0	33.6	25.1	1	12k
	self-consistency, as many calls	28.0	30.0	46.7	25.9	16/12/13/11	157k
	direct answer, more input tokens	24.0	30.0	27.1	24.7	1	34k
	Track, Then Bind	100	100	51.4	43.0	19/12/13/11	25k
32B	direct answer	32.0	38.0	35.5	37.1	1	12k
	self-consistency, as many calls	32.0	30.0	37.4	37.3	16/12/15/11	181k

Table 3: Call-matched and increased-input baselines (test sets, %). Self-consistency [ 34 ] :

k

answers sampled at the model’s recommended temperature and a majority vote, with

k

the agent’s mean number of VLM calls on that benchmark (at most 16). More input tokens : one answer from 16 fps and larger frames. SpatialClaw : as in Table 1 . Calls: VLM calls per question on cup/card/PT/TOC. Tokens: VLM input tokens per PT question.

Area	$n$	Direct	Track, Then Bind	$\Delta$
physics	689	60.1	64.0	$+3.9$
semantics	634	86.4	86.9	$+0.5$
abstraction	593	63.4	64.2	$+0.8$
memory	84	58.3	66.7	$+8.3$
all	2,000	69.3	71.5	$+2.1$
Video-MME short	900	78.7	78.1	$-0.6$

Table 4: General video questions (accuracy, %, Qwen3-VL-8B). Top: 2,000 Perception Test validation questions, stratified by area and type of reasoning. Bottom: Video-MME, short and medium videos. The direct answer is run in the same engine and the same pass as the agent.

VLM	Variant	cup	card	PT
8B	tracks drawn, question asked directly	44.0	40.0	32.7
	bind question on the whole video	100	100	53.3
	facts read out with the video	38.0	72.0	46.7
	Track, Then Bind	100	100	51.4
32B	tracks drawn, question asked directly	78.0	100	40.2
	bind question on the whole video	72.0	100	54.2

Table 5: Ablation of the three steps (test sets, accuracy %). Each variant keeps the agent’s plan, boxes and tracks and changes one step. Tracks drawn, question asked directly : the numbered clip of the whole video with the benchmark question, as in visual prompting with tracker IDs. Bind question on the whole video : no located moment. Facts read out with the video : the same facts are appended to the benchmark prompt, which is answered with the video.

Panel of Figure 3 — Figure 3: Qualitative results (test videos, Qwen3-VL-8B unless noted). (a) A success that no method without binding reproduces. (b, c) The two most common failures on real cup games: a wrong viewpoint fact with correct tracks, and a tracker that loses identical cups crossed by hand. Numbered boxes are the agent’s tracks as the VLM sees them.

Test set	Bind clip	8B	32B	Frames
VET cup	located (hidden)	100% (100)	100% (100)	12
VET card	located (in view)	100% (100)	100% (100)	6
PT	located (hidden)	20.6% (40.9)	11.2% (33.3)	12
PT	whole video	74.8% (55.0)	81.3% (57.5)	60
PT	no binding	4.7% (40.0)	7.5% (25.0)	–
TOC	no binding	100% (43.0)	99.9% (48.7)	–

Table 6: Bind clips selected by step 3 (test sets). Located : the VLM sees a 3 s clip around the evidence moment. Whole video : no moment is found, and the bind question is asked on the numbered clip of the whole video. No binding : the plan finds no target to tell apart, or fewer than two carrier tracks, and the question is answered with a timeline. Each cell gives the share of questions and, in parentheses, the accuracy on them. Frames: mean frames per bind call.

		Accuracy				Calls
VLM	Configuration	cup	card	PT	TOC	PT	TOC
8B	per benchmark	100	100	44.9	43.4	20	20.2
8B	single	100	100	51.4	43.0	13.4	10.8
32B	per benchmark	100	100	50.5	48.0	20	22.7
32B	single	100	100	52.3	48.6	14.7	11.1

Table 7: One configuration versus per-benchmark configurations (test sets): accuracy (%) and VLM calls per question.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Step	Setting	Value
1 Plan	video rate for the plan call	2 fps
	entities per plan	at most 3
2 Perceive	grounding frames	8 uniform, plus 8 in between for entities found on none
	tracking resolution	960 px on the long side
	first tracking pass	6 fps, at most 900 frames
	dense pass triggers	instance IoU $>0.05$ ; centre shift $>0.25$ box widths per sampled frame; carriers not all tracked after 90% of the video

Appendix

Table 8: Settings of the single configuration (all benchmarks).

p_{90}

: the 90th percentile of a track’s mask area over the video.

		VLM calls					VLM tokens		Time (s)
VLM	Set	video	frame	clip	text	all	in (k)	out	VLM	tracker	total
8B	cup	1.0	15.0	1.0	2.0	19.0	10.8	1058	40	69	110
	card	1.0	8.0	1.0	2.0	12.0	7.3	987	37	64	103
	PT	1.1	9.3	1.0	1.9	13.4	24.7	1289	50	71	126
	TOC	2.1	7.8	0.0	1.0	10.8	15.3	443	18	14	35
32B	cup	1.0	15.0	1.0	2.0	19.0	10.8	984	91	62	154

Appendix

Table 9: Cost per question on the test sets (means). VLM calls by input: video , the plan call and a read-out on the benchmark’s own frames; frame , grounding and the viewpoint question; clip , the bind call; text , the final-location question and the text-only read-out. Times are wall-clock seconds on A800 GPUs, one GPU per process at 8B and two at 32B, with other jobs sharing the node. The tracker time includes the dense pass.

Benchmark	Split	Use	Videos	Questions	Duration	Source
VET-Bench cup	test	test	50	50	12.4	VET-Bench [ 1 ] (synthetic)
VET-Bench card	dev	dev	56	56	12.1	rendered with the VET-Bench generator
VET-Bench card	test	test	50	50	12.1	VET-Bench [ 1 ] (synthetic)
Perception Test cup games	validation	dev	195	195	23.7	Perception Test [ 2 ]
Perception Test cup games	PT	test	107	107	15.6	filtered by the VET-Bench authors
three cups, $\geq 1$ swap	PT-65	test	65	65	–	subset of PT

Appendix

Table 10: Evaluation data. Duration: mean video length in seconds. Dev: used to select the design and the thresholds. Test: run once with the frozen configuration.

Baseline	Engine	Input	Decoding
Qwen3-VL, direct	vLLM 0.15.1 [ 67 ]	PT and VET: video at 8 fps, default resizing of qwen-vl-utils; TOC: 32 uniform frames	greedy; 2,048 new tokens (TOC: the benchmark’s budget per format)
Qwen3-VL, direct	Transformers 4.57.6	same	greedy
Self-consistency	vLLM	same	$k$ samples at temperature 0.7, top- $p$ 0.8, top- $k$ 20 (Qwen3-VL’s recommended settings), majority vote; $k$ equals the agent’s mean VLM calls, at most 16
More input tokens	vLLM	PT and VET: 16 fps, at most 288,000 pixels per frame, no resizing; TOC: 96 frames	greedy
Qwen3-VL-Thinking	vLLM (VET, PT); TOC-Bench runner (TOC)	as the direct answer	greedy; 16,384 new tokens
Qwen3-VL-235B-A22B	vLLM, 8 GPUs	video at 8 fps	greedy; 2,048 new tokens

Appendix

Table 11: Baseline settings. All baselines receive the benchmark’s prompt and are scored with its parser. Self-consistency and the larger input appear in Table 3 .

Set	$n$	Acc. (%)	VLM calls	Input tokens	Mask propagation (%)	Rerun after tool failure
cup	50	28.0	14.5	419k	26	0
card	50	36.0	9.0	229k	26	0
PT	107	26.2	17.7	508k	77	12
TOC	1845	35.7	10.2	300k	62	28

Appendix

Table 12: SpatialClaw with Qwen3-VL-32B on the test sets. Calls and input tokens are means per question. Mask propagation: share of questions whose code propagated a SAM3 mask through the video at least once without an error.

Panel of Figure 4 — Figure 4: SpatialClaw on VET card monte card_001 (Qwen3-VL-32B, logged test run). The images are the session’s own show() outputs. The queen’s mask is empty once the card turns over, and the model answers from the last frame. Track, Then Bind answers the same video correctly (Fig. 3 a).

VLM	Baseline	Set	Baseline (%)	Ours $-$ baseline [95% CI]	$p$
8B	Direct answer (vLLM)	cup	28.0	+72.0 [60.0, 84.0]	$<$ 0.001
		card	28.0	+72.0 [60.0, 84.0]	$<$ 0.001
		PT	33.6	+17.8 [3.7, 30.8]	0.02
		TOC	25.1	+17.9 [15.4, 20.4]	$<$ 0.001
	Direct answer (Transformers)	cup	22.0	+78.0 [66.0, 88.0]	$<$ 0.001
		card	34.0	+66.0 [52.0, 78.0]	$<$ 0.001

Appendix

Table 13: Ours minus each baseline with the same VLM (test sets, percentage points). CI: paired bootstrap 95% interval;

p

: exact McNemar test.

Dimension	$n$	Direct	Thinking	SpatialClaw	VideoAgent	VideoINSTA	Ours
event existence	170	42.9	42.9	45.9	38.2	40.6	66.5
reappear vs. disappear	106	41.5	43.4	55.7	34.0	33.0	75.5
relative spatial change	131	31.3	33.6	29.8	26.7	29.0	52.7
event count	241	20.3	19.9	7.9	35.3	39.4	32.0
event ordering	89	13.5	23.6	16.9	13.5	14.6	32.6
duration category	245	31.0	24.9	29.0	26.1	31.4	36.7

Appendix

Table 14: TOC-Bench test accuracy by dimension, Qwen3-VL-32B (%). Direct: answering directly in vLLM. Ours: Track, Then Bind. Best per row in bold.

Subset	$n$	D 8B	Ours 8B	D 32B	SpatialClaw	Ours 32B
2 cups	13	53.8	53.8	46.2	23.1	53.8
3 cups	90	30.0

Appendix

Table 15: Perception Test cup games by subset (accuracy, %). D: direct answer in vLLM.

Video Reasoning

Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.

See all 210 papers

Can Vision-Language Models Solve the Shell Game?
Mar 9, 2026Tiedong Liu, Wee Sun LeeVisual Object Tracking Temporal Video Understanding
National University of Singapore
TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models
May 11, 2026Junzhe Chen, Siyuan Meng, Yuxi Chen +3Video-Language Model Evaluation Video-Language Models
Tianjin University
Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models
May 9, 2026Tri Cao, Khoi Le, Thong Nguyen +7VLM Evaluation Object Hallucination in VLMs
National University of Singapore · VinUniversity · Nanyang Technological University

Mar 9, 2026J/K move · Enter open · S save

Can Vision-Language Models Solve the Shell Game?

Tiedong Liu, Wee Sun Lee

National University of Singapore

Visual Object Tracking Temporal Video Understanding VLM Evaluation State Tracking

Open paper AskPDF arXiv

Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io .

Abstract

Video Reasoning

Explore similar work

Can Vision-Language Models Solve the Shell Game?

TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models

Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models