Tempo-Zero: Recursive Self-Play in Executable Worlds for Video Reasoning

Jingmin Zhu; Yanyan Zou; Xiuyu Li; James Bailey; Dinh Phung; Tao Yu; Yunyun Hou; Jun Liu

cs.CVOct 23, 2026

Tempo-Zero: Recursive Self-Play in Executable Worlds for Video Reasoning

Authors: Jingmin Zhu, Yanyan Zou, Xiuyu Li, James Bailey, Dinh Phung, Tao Yu, Yunyun Hou, Jun Liu,

Organizations: Monash University · Institute of Automation, Chinese Academy of Sciences · Communication University of China · Lancaster University ·

Abstract

Video reasoning requires understanding how events unfold over time, yet videos with accurate temporal annotations are expensive to collect. Self-play offers an alternative by letting a model generate and solve its own tasks without manual annotation. However, existing video self-play methods train on fixed sets of recorded videos, so they cannot alter the underlying event dynamics and rely on noisy pseudo-labels derived from the model's own outputs, such as majority voting, that risk reinforcing shared model errors. To overcome these limitations, we introduce Tempo-Zero, a self-improvement framework that closes a loop of executable environments, execution-verified self-play, and environment evolution, where executing a program renders a video with the ground-truth option and interval of every question. A single policy plays both Questioner and Solver, proposing questions and answering them with timestamped evidence lines, and is trained on execution-grounded rewards instead of model consensus. The Solver's performance steers a curriculum over tasks and difficulty levels, and the policy selects rule changes that enrich the event dynamics of later rounds. Despite being trained solely on simple programs, without supervised fine-tuning or external video data which prior methods typically require, Tempo-Zero outperforms the base models and recent self-play baselines on average across twelve real-world video benchmarks at different model scales, with particularly strong gains on temporal grounding tasks. Project page: https://alchemist-jo.github.io/tempo-zero/

Figures & tables

Figure 1: Limitations of video self-play and program-generated video. (a) On a shell-game video, majority voting over eight sampled answers selects a wrong label over the two correct ones, while execution gives the correct label. (b) Under a fixed rule, all Solver rollouts are correct; changing the cushion rule and re-executing the program yields a new path with exact labels.

Figure 2: Overview of Tempo-Zero. Programs in library

\mathcal{P}

execute and compile into video

v

, criteria

\mathcal{C}

, and canonical question

\bar{q}

. One policy proposes questions as the Questioner and answers them as the Solver; bank

\mathcal{B}

stores measured questions for replay, and execution-grounded rewards

r^{\mathrm{Q}}

and

r^{\mathrm{S}}

drive GRPO updates.

Figure 3: Environment compilation. Ground truth is recomputed on the sampled frames, and criteria are screened for temporal order. Colors identify entities, blue brackets mark ground-truth intervals, and the dark option is the ground-truth option.

Figure 4: Execution-verified self-play. The Questioner’s proposal is bound to criterion

c_{q}

and passes gate

g(q)

. The Solver’s rollouts give pass rate

p

, which sets

r_{\mathrm{learn}}

for the Questioner, and only an evidence line that states the criterion’s event earns interval credit.

Figure 5: Environment evolution. Pass rates guide allocation

\omega_{f,a}

(Eq. 6 ) and set level

\ell

, and the policy selects a rule change, which the executor validates before deployment.

	Method	ANet-RTL				ActivityNet Captions				Charades-STA				NExT-GQA		Avg.
	Method	mIoU	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	Avg.
	Base model	16.86	22.70	15.72	8.73	22.86	30.94	19.04	11.29	37.52	58.90	36.12	16.23	23.09	15.40	25.08
	VisPlay	16.58	23.14	13.97	8.73	22.89	30.98	19.12	11.31	37.59	58.82	36.59	16.42	23.14	15.33	25.05
	V-Zero	15.85	20.96	13.97	8.30	22.81	30.91	18.92	11.31	37.10	58.33	35.24	15.62	22.82	15.22	24.64
	Video-Zero	29.75	44.98	28.82	13.54	32.12	44.76	27.67	16.24	41.85	65.11	40.81	20.03	38.29	23.00	35.50
4B	Tempo-Zero	31.34 $\uparrow$ 14.48	48.03 $\uparrow$ 25.33	31.00 $\uparrow$ 15.28	13.10 $\uparrow$ 4.37	34.98 $\uparrow$ 12.12	49.35 $\uparrow$ 18.41	31.04 $\uparrow$ 12.00	17.26 $\uparrow$ 5.97	42.53 $\uparrow$ 5.01	66.29 $\uparrow$ 7.39	41.26 $\uparrow$ 5.14	19.49 $\uparrow$ 3.26	39.17 $\uparrow$ 16.08	24.50 $\uparrow$ 9.10	37.00 $\uparrow$ 11.92

Table 1: Temporal grounding on real-world benchmarks. Metrics in %; Avg. averages the mIoU of the first three benchmarks and [email protected]. Bold marks the best per column within each backbone, and underline marks the second best at 4B and 8B, where all methods are evaluated; arrows give the change over the base model, the only comparison at 30B-A3B and Gemma.

	Method	Long-Video Understanding				Video Reasoning				Avg.
	Method	LongVideo Bench	MLVU	Video-MME Long	LSDBench	LongVideo Reason	MMVU	VideoMath QA	VideoMMMU	Avg.
	Base model	57.70	62.60	51.60	54.68	69.90	64.32	19.05	57.56	54.68
	VisPlay	59.20	63.50	53.20	54.83	69.60	63.36	18.33	57.44	54.93
	V-Zero	59.50	63.50	51.90	54.68	68.20	64.48	20.95	58.33	55.19
	Video-Zero	59.10	63.40	53.60	56.52	70.30	64.96	21.67	61.56	56.39
4B	Tempo-Zero	61.00 $\uparrow$ 3.30	67.36 $\uparrow$ 4.76	53.30 $\uparrow$ 1.70	56.06 $\uparrow$ 1.38	72.40 $\uparrow$ 2.50	66.08 $\uparrow$ 1.76	22.38 $\uparrow$ 3.33	63.11 $\uparrow$ 5.55	57.71 $\uparrow$ 3.03

Table 2: Long-video understanding and video reasoning. Official accuracy in %; marks as in Table 1 .

Questioner
Variant	Temporal Grounding	Video QA g & Reasoning	Overall
w/o novelty reward	36.53	56.88	50.10
w/o learnability reward	35.25	57.27	49.93
w/o Questioner	31.86	55.63	47.71
Solver
w/o evidence score	35.34	57.33	50.00

Table 3: Ablations on Qwen3-VL-4B. Temporal Grounding and Video QA & Reasoning are the Avg. columns of Tables 1 and 2 , and Overall averages all twelve benchmarks; marks as in Table 1 .

Figure 6: Training dynamics. (A) Held-out accuracy. (B) Evidence (filled) and grounded (hollow) scores; B is the base model. (C) Proposals passing the gate. (D) Families whose rules change for each round.

Figure 7: Execution labels. (A) Majority vote versus execution labels on 7,420 Solver groups (%). (B) Rollouts credited only on wrong-event lines, and credit kept by

r_{\mathrm{grd}}

(%).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Backbone	Qwen3-VL-4B/8B/30B-A3B-Instruct and Gemma-4-E4B-it; LoRA with rank 64 and $\alpha=128$ on the q/k/v/o projections of the language model; the vision tower and the A3B experts carry no adapter
Environments	12 environment families, 3 difficulty levels; 5 rounds of 20 steps
Batch	32 instances per step; playback scale $\lambda_{r}=1.0,1.25,1.5,1.75,2.0$ for rounds $r=1,\ldots,5$
Questioner	4 proposals per instance; one to three located evidence lines
Solver	$M=8$ rollouts per question; one to eight evidence lines; criteria are those of the events the target rests on, or the target itself when it rests on none; at most 72 groups per step (32 canonical, up to 32 preselected proposals, 8 replayed)
Question bank	Capacity 640, shared across rounds; up to 8 replayed questions per step; at most one preselected proposal per instance; one prior example in the Questioner prompt

Appendix

Table 5: Training configuration. Settings shared by the 4B, 8B, 30B-A3B, and Gemma configurations.

Class	Tasks
Interval	Event grounding † , state interval † , closest interval † , tile motion interval † , interval relation, interval composition
Event sequence	Event order, adjacent event, path sequence, pattern extrapolation
Quantity comparison	Event count, extremum, temporal extremum, rate trend, speed comparison, closest pair, travel rank
Motion	Motion direction, relative motion, motion profile, trajectory shape, distance trend
State inference	Forward state, backward state, history query, occluded identity, tile endpoint, tile history, tile waypoint
Container update	Set update, multiset update, sequence update

Appendix

Table 6: Tasks by class. Tasks marked † are interval questions, answered with a free-form interval; all other tasks are multiple-choice.

Family	Observable dynamics	Reasoning targets
Billiards	Kinematic motion, elastic contact, and velocity shifts	Trajectory direction, event order, state transitions
Block stacking	Sequential placement and physical support	Object counts, stacking order, spatial relations
Checkout queue	Arrival, service progression, and departure	Queue duration, entity counts, interval relations
Dominoes	Cascading propagation along contact dependencies	Chain continuation, trigger ordering, collapse speed
Ferris wheel	Coupled rotating structures and cyclic cabins	Relative angular rates, phase shifts, event timing
Filling glasses	Liquid level shifts and volumetric transfers	Continuous quantities, extrema, volume change intervals

Appendix

Table 7: Twelve environment families. Each family supports three difficulty levels; the columns summarize its observable dynamics and the reasoning targets it supports.

Figure 8: Executable environments. Rendered frame pairs from Dominoes, Filling glasses, and Traffic lights; the labels inside the frames are part of the rendered video.

Figure 9: Training environments. (A) Key events and duration by level. (B) Static-question screen classes in %; only invariant criteria are removed.

Level	Instances	Count	fps	between	Criteria	Disagree	Unverified	Ordered	Invariant	Undet.
		Sampled frames		Events	Recomputation			Static-question screen
0	2,599	58.1	1.69	2.82	25,338	23	25	67.93	10.76	21.31
1	1,179	63.8	1.00	10.02	11,623	44	44	74.33	7.29	18.38
2	1,296	64.0	0.62	28.56	12,564	141	126	76.21	7.02	16.77
Overall	5,074	60.9	1.26	16.60	49,525	208	195	71.51	9.01	19.49

Appendix

Table 8: Verification checks on the training environments. Instances per level, mean sampled frames and effective frame rate, share in % of recorded events that fall between two sampled frames, criteria compiled and discarded by the recomputation, and shares in % of the static-question screen classes among the criteria that pass the recomputation. Invariant criteria are removed, and ordered and undetermined criteria are kept.

Panel of Figure 10 — Figure 10: A rule change in Billiards. Frames at 0.0, 4.0, 7.9, 11.9, 15.9, and 19.8 s of the same instance before and after the rule change; the overlays are part of the rendered frames.

	Majority voting					Interval credit
Backbone	Groups	Wrong	Outvoted	Unanimous	Tie	Wrong line	Kept
4B	3,703	56.82	23.17	21.33	7.24	48.78	38.01
8B	3,717	48.21	29.32	8.34	10.36	52.21	36.44
Pooled	7,420	52.51	26.25	14.82	8.80	50.50	37.18

Appendix

Table 9: Majority voting and interval credit. Left: Solver groups, with shares in % of groups whose majority label is wrong, whose majority label is wrong although at least one rollout chooses the correct option, whose eight rollouts agree on a wrong option, and whose vote ties. Right: Solver rollouts, with the share in % of rollouts credited by the content-blind overlap whose credit falls only on lines with a non-matching statement, and the share in % of the content-blind credit that

r_{\mathrm{grd}}

keeps.

Backbone	Metric	B	1	2	3	4	5
4B	Accuracy	22.9	27.9	32.7	32.4	33.2	31.1
	Evidence	24.3	40.4	42.1	41.7	43.7	47.0
	Grounded	3.1	7.6	8.9	7.1	10.0	10.8
8B	Accuracy	25.9	31.3	36.1	40.5	37.5	42.8
	Evidence	27.2	38.3	40.3	40.9	40.7	42.6
	Grounded	3.4	7.0	10.0	11.3	10.8	11.6

Appendix

Table 10: Held-out instances. Accuracy, evidence score, and grounded score in % on the 128 held-out instances with eight answers each, for the base model (B) and after each of the five rounds; bold marks the best round.

Backbone	2	3	4	5	Total	Selected	Failed check	Usable alternatives
	Changed					Kept
4B	10	12	7	3	32	12	4	129 / 129
8B	9	7	3	5	24	23	1	141 / 141
Total	19	19	10	8	56	35	5	270 / 270

Appendix

Table 11: Rule decisions. Families whose rules change for each round, decisions that keep the rules by the policy’s selection or after a failed check, and menu alternatives with usable previews. Each backbone makes 12 decisions per round.

Variant	Change from the full method	Component tested
w/o novelty reward	Remove the novelty reward $r_{\mathrm{nov}}$ from $r^{\mathrm{Q}}$	Diversity of proposed questions
w/o learnability reward	Remove the learnability reward $r_{\mathrm{learn}}$ from $r^{\mathrm{Q}}$	Targeting of questions at the Solver’s capability
w/o Questioner	Train the Solver only on canonical questions, without Questioner proposals	Self-proposed questions
w/o evidence score	Remove the evidence score $r_{\mathrm{evid}}$ from $r^{\mathrm{S}}$	Statement-level $F_{1}$ of evidence lines
w/o grounded score	Remove the grounded score $r_{\mathrm{grd}}$ from $r^{\mathrm{S}}$	Interval credit on matched lines
Answer reward only	Keep only the answer reward $r_{\mathrm{ans}}$ in $r^{\mathrm{S}}$	Both evidence terms

Appendix

Table 12: Ablation variants. Each variant changes one component of the full method.

Video Reasoning

Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.

See all 210 papers

Video-Zero: Self-Evolution Video Understanding
May 14, 2026Ruixu Zhang, Deyi Ji, Lanyun Zhu +4Temporal Video Grounding Long-Video Understanding
1Tsinghua University · 2Tencent · 3Tongji University +1
EvoVid: Temporal-Centric Self-Evolution for Video Large Language Models
May 21, 2026Shiqi Huang, Ziyue Wang, Zhongrong Zuo +3Vision-Language Models Video Reasoning
School of Electrical and Electronic Engineering, Nanyang Technological University · ByteDance
TimeThink: Reasoning with Time for Video LLMs
Jul 6, 2026Handong Li, Longteng Guo, Zikang Liu +8Temporal Video Grounding Process Reward Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.

May 14, 2026J/K move · Enter open · S save

Video-Zero: Self-Evolution Video Understanding

Ruixu Zhang, Deyi Ji, Lanyun Zhu, Xuanyi Liu+3

1Tsinghua University · 2Tencent · 3Tongji University+1

Temporal Video Grounding Long-Video Understanding Video Reasoning Temporal Video Understanding

Open paper AskPDF arXiv

Self-evolution offers a promising path for improving reasoning models without relying on intensive human annotation. However, extending this paradigm to video understanding remains underexplored and challenging: videos are long, dynamic, and redundant, while the evidence needed for reasoning is often sparse and temporally localized. Naively generating difficult question-answer pairs from full videos can therefore produce supervision that appears challenging but is weakly grounded, relying on static cues or language priors rather than temporal evidence. In this work, we argue that the key bottleneck of video self-evolution is not difficulty alone, but grounding. We propose Video-Zero, an annotation-free Questioner--Solver co-evolution framework that centers self-evolution on temporally localized evidence. The Questioner discovers informative evidence segments and generates evidence-grounded questions, while the Solver learns to answer and align its predictions with the supporting evidence. This closes an iterative loop of evidence discovery, grounded supervision, and evidence-aligned learning. Across 13 benchmarks spanning temporal grounding, long-video understanding, and video reasoning, Video-Zero consistently improves multiple video VLM backbones, demonstrating the effectiveness and transferability of evidence-centered self-evolution.

Abstract

Video Reasoning

Explore similar work

Video-Zero: Self-Evolution Video Understanding

EvoVid: Temporal-Centric Self-Evolution for Video Large Language Models

TimeThink: Reasoning with Time for Video LLMs