Tracking a hidden object through a video is effortless for humans, yet it remains a bottleneck for vision language models (VLMs). In the shell game, where a ball is hidden under one of several identical cups that are then swapped, VLMs that answer directly stay near chance, thinking models fail on long games, and fine-tuning learns only the lengths it has seen. We argue that the limit is serial computation: each swap must be composed with the last result, and a VLM of fixed depth has only a fixed number of state transitions for it. To address this, we propose RecurrentVLM, which makes every attention layer of a pretrained VLM recur over time by computing the keys and values of finished frame groups from the layer's own output. With a single answer token, RecurrentVLM reaches 100% on VET-Bench and 99.6% on MET-Bench, and 99% on 50-swap games after training on at most 30 swaps. We also observe several interesting phenomena: length generalization emerges suddenly at an aha moment; without recurrence, each swap costs an attention layer; the ball's state forms in the first four attention layers; and recurrence keeps the latest state at the cost of details of earlier moments. These results suggest that recurrent VLMs are a potential solution for tracking the states of unobserved objects in video. Code will be released.
Figures & tables
Figure 2: Recurrent KV cache. (a) A standard attention layer of the LLM caches keys and values of its input tokens. (b) We cache keys and values of the layer’s own output, once per frame group, so every attention layer becomes a recurrence over time. (c) Each query reads keys and values of earlier tokens’ inputs in (a), but of earlier frame groups’ outputs in (b).
Figure 3: Frame-wise supervision. Each frame group ends with its final token, <|vision_end|> . In training, its output must name the ball’s current position (boxed), and the output after the question is the answer; at test time, only the answer is generated.
Figure 4: Length generalization emerges with an aha moment. (a, b) Training loss and accuracy on shell games: RecurrentVLM shows an abrupt drop in loss within 300 steps (shaded), an aha moment also seen in multi-step tasks [ 37 ] , while the baseline improves slowly. (c) Accuracy on 50 swaps, longer than in training, jumps right after the aha moment. (d) Accuracy against the number of swaps (gray: training lengths). Fine-tuned and zero-shot: Qwen3.5-4B.
Swaps
Model
3 cups, balls
5 cups, balls
1
2
3
1
2
3
30
Qwen3.5-4B (fine-tuned)
90
91
94
95
93
90
RecurrentVLM
100
100
100
96
98
93
50
Qwen3.5-4B (fine-tuned)
38
44
32
29
25
20
RecurrentVLM
100
99
100
87
92
92
Table 2: More cups and more balls. Accuracy (%) on our MET-Bench-style shell games with 3 or 5 cups and 1–3 balls, 100 games each (random: 33.3 and 20.0); both models train on the same mix of all settings, with up to 30 swaps per training game.
Figure 5: Probing the ball across attention layers. (a, b) Linear probes reading the ball position from every layer output (three GDN layers between attention layers) after each swap of 100-swap games, fit on the first 30 (inset: swaps 1–10). RecurrentVLM reads the ball after attention layer 4 at all swaps, while the baseline needs one more attention layer per about two swaps, which limits it to 10 swaps. (c) Memory transfer: midway through game A, we copy part of game B’s recurrent state and count answers switching to B.
Figure 6: Our hypothesis of why depth runs out. Each swap is combined with the last result. RecurrentVLM keeps it in one layer; the baseline reads earlier swaps one layer lower, so the result climbs and swap 6 finds only raw swaps (dashed).
Figure 7: Tracking the aha moment. (a) Over training: share of swaps on which the frame-wise answer equals the model’s own last answer updated by the current swap (black), and per-swap accuracy on swaps 31–50 (red). (b) Accuracy on 50-swap games with one head of attention layer 4 switched off; any single head of attention layer 3 or 5 keeps at least 60%. (c) Per-swap accuracy with head 2 of attention layer 4 off.
Figure 8: Shallow recurrence suits continuous motion; deep recurrence carries jumps further. Accuracy after 2,500 steps with recurrence in only 4 of the 8 attention layers (shallow, middle, deep: 1–4, 3–6, 5–8).
Variant
VET
MET
Cup × 16
Shell × 50
RecurrentVLM
100
99.6
91.0
99.0
(a) no recurrent KV cache
90.0
32.0
35.0
37.0
(b) no frame-wise supervision
36.0
33.8
35.0
32.0
(c) recurrent KV cache off
34.0
34.8
40.0
30.0
(d) keep last 8 frame groups
98.0
99.6
97.0
91.0
(e) keep last 4 frame groups
38.0
91.8
37.0
55.0
Table 4: Ablating components. Accuracy (%) after 2,500 steps, one change to RecurrentVLM per row: (a, b) in training, (c–e) only at inference; (a) is the baseline of table 1 at step 2,500. Shaded: unseen lengths.
Figure 10
Question type
Qwen3.5-4B
RecurrentVLM
zero-shot
fine-tuned
fine-tuned
Perception Test (2000)
64.8
78.5
76.7 ( − 1.8)
Advantage: the state after an action
Adversarial action (104)
42.3
56.7
62.5 (+5.8)
Object attributes (98)
57.1
74.5
78.6 (+4.1)
State recognition (166)
62.7
76.5
79.5 (+3.0)
Table 6: General video understanding. Accuracy (%) on the Perception Test [ 66 ] validation set by skill after fine-tuning; gray: question counts; parentheses: change vs. fine-tuned Qwen3.5-4B.
Video
Latency (s)
Mem. (GB)
Frames
Sec.
Qwen3.5-4B
RecurrentVLM
98
12
1.0
1.2 (1.2 × )
9.1
258
32
2.4
3.2 (1.4 × )
10.0
514
64
4.9
7.3 (1.5 × )
11.4
1,026
128
10.6
18.4 (1.7 × )
14.4
Table 7: Inference cost. Seconds per answer for 8-fps cup videos on one RTX A5000 (ratio to Qwen3.5-4B in parentheses) and peak GPU memory, which is the same for both models.