Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io .
Figures & tables
Figure 1: Overview of VET-Bench.
Figure 2: Performance on VET-Bench, consisting of 50 cups-game and 50 cards-game videos featuring 3 objects and 5 swaps ( ∼ 12 seconds). Existing VLMs all perform near random chance. Molmo2-SGCoT is a fine-tuned model based on Molmo2 that leverages Spatiotemporal Grounded Chain-of-Thought (SGCoT) to solve the shell game (Section 5 ).
Figure 3: Performance of VLMs under different swap and object counts.
Model
Filtered
VET-Bench
Random
0.33
0.33
Gemini-3-Pro
0.31
0.34
Gemini-3-Flash
0.40
0.30
Qwen3-VL-8B
0.34
0.30
GLM-4.6V-Flash
0.42
0.34
Doubao-Seed-2.0
0.40
0.30
Table 1: Performance comparison on the filtered subset and VET-Bench.
Figure 4: Training and validation loss for direct-answer training on 500 synthetic VET-Bench cups-game videos.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Example frames from videos involving distinct cups in the Perception Test.
Figure 6: Example frames from videos involving transparent cups in the Perception Test.
Figure 7: Visual shortcuts in videos #7632 and #9896 . These videos contain unedited cut frames at the end that reveal the answer by showing the cups being lifted. This unintended disclosure provides a shortcut that allows the model to simply look at the final state to identify the object’s location, making the tracking of the shuffle irrelevant.
Figure 8: Comparison of the cups-game tasks in VideoReasonBench (left) and VET-Bench (right). VideoReasonBench provides explicit frame-level cues highlighting swap operations (indicated by red arrows), which are absent in VET-Bench.
Figure 9: Filtering process for the identical 3-cup setting in the Perception Test test split.
Figure 10: Distribution of swap counts in the filtered Perception Test subset, along with the number of correct responses from Gemini-3-Pro and Gemini-3-Flash.
Figure 11: Examples of Gemini-3-Pro responses on the VET-Bench cups-game task. (Top) A successful case where the model correctly tracks all the intermediate states. (Bottom) A failure case where perception errors (highlighted in red ) lead to an incorrect final prediction.
Figure 12: Examples of Gemini-3-Pro responses on the VET-Bench cards-game task. (Top) A successful case where the model correctly tracks all the intermediate states. (Bottom) A failure case where perception errors (highlighted in red ) lead to an incorrect final prediction.
Figure 13: Examples of model responses on VET-Bench. Errors are highlighted in red .
Figure 14: Examples of hallucination by Gemini-3-Flash in zero-swap cases. Gemini-3-Pro and Gemini-3-Flash often correctly identify the initial state but then hallucinate a long sequence of non-existent swaps (as shown in red ), causing the final prediction to be incorrect.
Figure 15: Examples of Molmo2’s prompts and responses on VET-Bench. Molmo2 responds in a direct-answer mode without explicitly performing spatiotemporal tracking, resulting in near-random performance.
Figure 16: Examples of Molmo2-SGCoT’s prompts and responses on VET-Bench. Top: cups game; Bottom: cards game.
Figure 17: Examples of Molmo2-SGCoT failure cases. Errors typically arise in SGCoT when the model fails to differentiate between visually identical objects. This manifests as abrupt “jumps” in the spatial (top) or temporal (bottom) domains (highlighted in red ), leading to incorrect terminal locations and subsequent failures in the final answer.
Base Model
Molmo2-8B
Training Method
QLoRA
Gradient Accumulation Steps
4
Batch Size
64
Learning Rate
1e-4
Training Samples
300
Number of Epochs
1
Appendix
Table 2: Molmo2-SGCoT Training Details. We apply QLoRA to fine-tune the language model while keeping the vision encoder weights frozen.
Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.
Many questions about a video concern one physical object. Where does the ball end up after the cups are shuffled? Which card is the queen? How often was the mug covered? Video-language models (Video-LMs) recognise such objects in single frames but do not keep them apart over time. On real shell-game videos, open models from 8B to 235B parameters, their thinking variants, a reinforcement-learned tool agent and three training-free tool agents all stay within ten points of chance. We separate semantic identity binding from temporal identity maintenance. An object's identity should be fixed at a moment when the evidence for it is visible, and carried afterwards by a tracker. Track, Then Bind implements this principle without training. A promptable video tracker follows every candidate object and draws it with a track number. A frozen vision-language model answers a static question about the identifying evidence, which binds the target to a track. The answer is then derived from the tracked state in text, without asking the model to reinterpret the original video during read-out. A single configuration covers all benchmarks. With Qwen3-VL-32B it solves both VET-Bench tasks, where a fine-tuned trajectory model reaches 92% and 90% and the same VLM voting over 16 samples stays at 30--32%. It answers 55.4% of the hardest Perception Test cup games, where the best published model answers 41.5%, and, using track-derived timelines, 48.6% of TOC-Bench questions, against 37.1% for the same model answering directly. Drawing the same tracks on the video and asking the question directly falls 12--19 points short of the agent on real cup games.
It is critical for vision-language models (VLMs) to comprehensively understand visual, temporal, and textual cues. However, despite rapid progress in multimodal modeling, video understanding performance still lags behind text-based reasoning. In this work, we find that progress is even worse than previously assumed: commonly reported long video understanding benchmarks contain 40-60% of questions that can be answered using text cues alone. Furthermore, we find that these issues are also pervasive in widely used post-training datasets, potentially undercutting the ability of post-training to improve VLM video understanding performance. Guided by this observation, we introduce VidGround as a simple yet effective solution: using only the actual visually grounded questions without any linguistic biases for post-training. When used in tandem with RL-based post-training algorithms, this simple technique improves performance by up to 6.2 points relative to using the full dataset, while using only 69.1% of the original post-training data. Moreover, we show that data curation with a simple post-training algorithm outperforms several more complex post-training techniques, highlighting that data quality is a major bottleneck for improving video understanding in VLMs. These results underscore the importance of curating post-training data and evaluation benchmarks that truly require visual grounding to advance the development of more capable VLMs. Project page: http://vidground.etuagi.com.
Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang +8
University of British Columbia · Vector Institute · Etude AI +4
Understanding a video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacity for visual state tracking is fundamental to video understanding, yet remains underexplored in current evaluations of Multimodal Large Language Models (MLLMs). We introduce Visual STAte Tracking benchmark (VSTAT), a video-based benchmark designed to diagnose visual state tracking in MLLMs. VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment, requiring continuous perception and integration of events across the entire video stream. Despite their strong performance on existing video benchmarks, we find that state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines. To analyze this gap, we compare MLLMs' thinking traces with the underlying video stream to understand why and when MLLMs fail on VSTAT. We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track. Finally, our preliminary evaluation suggests that recent agentic approaches, including MLLM-based video agents and coding agents, do not readily resolve these failures, still falling short on VSTAT.