Tracking a hidden object through a video is effortless for humans, yet it remains a bottleneck for vision language models (VLMs). In the shell game, where a ball is hidden under one of several identical cups that are then swapped, VLMs that answer directly stay near chance, thinking models fail on long games, and fine-tuning learns only the lengths it has seen. We argue that the limit is serial computation: each swap must be composed with the last result, and a VLM of fixed depth has only a fixed number of state transitions for it. To address this, we propose RecurrentVLM, which makes every attention layer of a pretrained VLM recur over time by computing the keys and values of finished frame groups from the layer's own output. With a single answer token, RecurrentVLM reaches 100% on VET-Bench and 99.6% on MET-Bench, and 99% on 50-swap games after training on at most 30 swaps. We also observe several interesting phenomena: length generalization emerges suddenly at an aha moment; without recurrence, each swap costs an attention layer; the ball's state forms in the first four attention layers; and recurrence keeps the latest state at the cost of details of earlier moments. These results suggest that recurrent VLMs are a potential solution for tracking the states of unobserved objects in video. Code will be released.
Figures & tables
Figure 2: Recurrent KV cache. (a) A standard attention layer of the LLM caches keys and values of its input tokens. (b) We cache keys and values of the layer’s own output, once per frame group, so every attention layer becomes a recurrence over time. (c) Each query reads keys and values of earlier tokens’ inputs in (a), but of earlier frame groups’ outputs in (b).
Figure 3: Frame-wise supervision. Each frame group ends with its final token, <|vision_end|> . In training, its output must name the ball’s current position (boxed), and the output after the question is the answer; at test time, only the answer is generated.
Figure 4: Length generalization emerges with an aha moment. (a, b) Training loss and accuracy on shell games: RecurrentVLM shows an abrupt drop in loss within 300 steps (shaded), an aha moment also seen in multi-step tasks [ 37 ] , while the baseline improves slowly. (c) Accuracy on 50 swaps, longer than in training, jumps right after the aha moment. (d) Accuracy against the number of swaps (gray: training lengths). Fine-tuned and zero-shot: Qwen3.5-4B.
Swaps
Model
3 cups, balls
5 cups, balls
1
2
3
1
2
3
30
Qwen3.5-4B (fine-tuned)
90
91
94
95
93
90
RecurrentVLM
100
100
100
96
98
93
50
Qwen3.5-4B (fine-tuned)
38
44
32
29
25
20
RecurrentVLM
100
99
100
87
92
92
Table 2: More cups and more balls. Accuracy (%) on our MET-Bench-style shell games with 3 or 5 cups and 1–3 balls, 100 games each (random: 33.3 and 20.0); both models train on the same mix of all settings, with up to 30 swaps per training game.
Figure 5: Probing the ball across attention layers. (a, b) Linear probes reading the ball position from every layer output (three GDN layers between attention layers) after each swap of 100-swap games, fit on the first 30 (inset: swaps 1–10). RecurrentVLM reads the ball after attention layer 4 at all swaps, while the baseline needs one more attention layer per about two swaps, which limits it to 10 swaps. (c) Memory transfer: midway through game A, we copy part of game B’s recurrent state and count answers switching to B.
Figure 6: Our hypothesis of why depth runs out. Each swap is combined with the last result. RecurrentVLM keeps it in one layer; the baseline reads earlier swaps one layer lower, so the result climbs and swap 6 finds only raw swaps (dashed).
Figure 7: Tracking the aha moment. (a) Over training: share of swaps on which the frame-wise answer equals the model’s own last answer updated by the current swap (black), and per-swap accuracy on swaps 31–50 (red). (b) Accuracy on 50-swap games with one head of attention layer 4 switched off; any single head of attention layer 3 or 5 keeps at least 60%. (c) Per-swap accuracy with head 2 of attention layer 4 off.
Figure 8: Shallow recurrence suits continuous motion; deep recurrence carries jumps further. Accuracy after 2,500 steps with recurrence in only 4 of the 8 attention layers (shallow, middle, deep: 1–4, 3–6, 5–8).
Variant
VET
MET
Cup × 16
Shell × 50
RecurrentVLM
100
99.6
91.0
99.0
(a) no recurrent KV cache
90.0
32.0
35.0
37.0
(b) no frame-wise supervision
36.0
33.8
35.0
32.0
(c) recurrent KV cache off
34.0
34.8
40.0
30.0
(d) keep last 8 frame groups
98.0
99.6
97.0
91.0
(e) keep last 4 frame groups
38.0
91.8
37.0
55.0
Table 4: Ablating components. Accuracy (%) after 2,500 steps, one change to RecurrentVLM per row: (a, b) in training, (c–e) only at inference; (a) is the baseline of table 1 at step 2,500. Shaded: unseen lengths.
Figure 10
Question type
Qwen3.5-4B
RecurrentVLM
zero-shot
fine-tuned
fine-tuned
Perception Test (2000)
64.8
78.5
76.7 ( − 1.8)
Advantage: the state after an action
Adversarial action (104)
42.3
56.7
62.5 (+5.8)
Object attributes (98)
57.1
74.5
78.6 (+4.1)
State recognition (166)
62.7
76.5
79.5 (+3.0)
Table 6: General video understanding. Accuracy (%) on the Perception Test [ 66 ] validation set by skill after fine-tuning; gray: question counts; parentheses: change vs. fine-tuned Qwen3.5-4B.
Video
Latency (s)
Mem. (GB)
Frames
Sec.
Qwen3.5-4B
RecurrentVLM
98
12
1.0
1.2 (1.2 × )
9.1
258
32
2.4
3.2 (1.4 × )
10.0
514
64
4.9
7.3 (1.5 × )
11.4
1,026
128
10.6
18.4 (1.7 × )
14.4
Table 7: Inference cost. Seconds per answer for 8-fps cup videos on one RTX A5000 (ratio to Qwen3.5-4B in parentheses) and peak GPU memory, which is the same for both models.
Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io .
Many questions about a video concern one physical object. Where does the ball end up after the cups are shuffled? Which card is the queen? How often was the mug covered? Open video-language models (Video-LMs) recognise such objects in single frames but do not keep them apart over time. On shuffled shell games, open models from eight families stay near chance, and so do published tool agents. We separate semantic identity binding from temporal identity maintenance. Binding should happen at a moment when the evidence for the object is visible, and maintenance should be left to a tracker. Track, Then Bind implements this principle without training. A promptable video tracker follows every candidate object and draws it with a track number. A frozen vision-language model (VLM) answers one question about the identifying evidence, which binds the target to a track. The answer is then derived from a text description of the tracks. With Qwen3-VL-32B, the agent solves both VET-Bench tasks, where a fine-tuned trajectory model reaches 92% and 90%. On real cup games from the Perception Test, it raises accuracy from 35.5% to 52.3%. Without retuning, the same configuration also solves VET-Bench with Gemma 4 and GPT-5.5. Its track-derived visibility timelines add 8.8-17.9 points on TOC-Bench for open models. GPT-6.1 already answers 92.5% of the real cup games directly but 71.0% through our pipeline, while our timelines still raise its TOC-Bench accuracy from 52.5% to 57.7%. The benefit of external tracking thus depends on the underlying model and the task.
Video vision-language models (VLMs) keep paying for visual state the stream already told us was stable. The factory wall did not move, but most VLM pipelines still hand the model dense RGB frames or a fresh prefix again. We study that waste as training-free anti-recomputation: reuse state when validation says it survives, and buy fresh evidence when the scene, query, or cache topology requires it. The largest measured win is after ingest. On frozen Qwen2.5-VL-7B-Instruct-4bit, adaptive same-video follow-up reuse preserves paired choices and correctness on a 93-query VideoMME breadth setting while reducing follow-up latency by 14.90-35.92x. The first query is still cold; the win starts when later questions reuse the same video state. Stress tests bound the result: repeated-question schedules hold through 50 turns, while dense-answer-anchored prompt variation separates conservative fixed K=1 repair from faster aggressive policies that drift. Fresh-video pruning is smaller but real. C-VISION skips timed vision-tower work before the first answer is generated. On Gemma 4-E4B-4bit, the clean 32f short cell reaches 1.316x first-query speedup with no paired drift or parse failures on 20 items; Qwen shows the fidelity/speed boundary. Stage-share ceiling (C-CEILING) is the accounting guardrail: a component speedup becomes an end-to-end speedup only in proportion to the wall-clock share it accelerates, so C-VISION and after-ingest follow-up reuse do not multiply. Candidate C-STREAM remains a native-rate target, not a headline result here. The broader direction is VLM-native media that expose change, motion, uncertainty, object state, sensor time, and active tiles directly, so models do not have to rediscover the world from dense RGB every frame.
JF Bastien, Sam D'Amico
Look, they said attention was all you needed, well we got your attention with this title.