Video-CoE: Reinforcing Video Event Prediction via Chain of Events

Qile Su; Jing Tang; Rui Chen; Lei Sun; Xiangxiang Chu

cs.CVMar 16, 2026

Video-CoE: Reinforcing Video Event Prediction via Chain of Events

Authors: Qile Su, Jing Tang, Rui Chen, Lei Sun, Xiangxiang Chu

Organizations: AMAP, Alibaba Group

Abstract

Despite advances in the application of MLLMs for various video tasks, video event prediction (VEP) remains relatively underexplored. VEP requires the model to perform fine-grained temporal modeling of videos and establish logical relationships between videos and future events, which current MLLMs still struggle with. In this work, we first present a comprehensive evaluation of current leading MLLMs on the VEP task, revealing the reasons behind their inaccurate predictions, including lack of logical reasoning ability for future events prediction and insufficient utilization of visual information. To address these challenges, we propose Chain of Events (CoE) paradigm, which constructs temporal event chains to implicitly enforce MLLM focusing on the visual content and the logical connections between videos and future events, incentivizing model's reasoning capability with multiple training protocols. Experimental results on public benchmarks demonstrate that our method outperforms both leading open-source and commercial MLLMs, establishing a new state-of-the-art on the VEP task. Codes and models will be released soon.

Figures & tables

Panel of Figure 1 — Figure 1: Analysis of open-source MLLMs on video event prediction tasks. Fig. 1(a) illustrates the reasoning process, indicating the lack of logical reasoning capabilities in VEP task. Fig. 1(b) illustrates the attention distribution of the option tokens over input tokens demonstrating the insufficient utilization of visual information.

Figure 2: An illustration of our proposed CoE-SFT method within Qwen2.5-VL-72B. We provide the larger model with the video and the future event, and prompt it to generate the intermediate logical reasoning process that connects them. Training on such data encourages the model to develop logical reasoning abilities rather than relying on option-based analysis.

Figure 3: An illustration of our proposed CoE-GRPO method. The overall supervision signal consists of three components: $r_{e}$ encourages the model to follow the CoE reasoning paradigm and constrains the CoE length; $r_{s}$ supervises the alignment between event timestamps and textual descriptions while preventing reward hacking; and $r_{a}$ provides verifiable reward signals. The scissor icon indicates the temporal segmentation of video clips based on timestamps.

Model	Method	Frames	Futurebench $\uparrow$
	Method	Frames	1-Hop	2-Hop	3-Hop	Interp.	AVG
GLM-4.1V-9B [ 42 ]	Vanilla	32	29.9	41.9	52.2	47.3	44.38
LLavA-NeXT-Video [ 61 ]		32	48.8	49.3	40.0	44.4	45.24
Kimi-VL-A3B [ 40 ]		32	44.3	42.8	51.3	51.9	48.87
MiMo-VL-7B [ 54 ]		32	59.0	59.6	50.5	43.8	50.45
InternVL3-8B [ 46 ]		32	54.3	58.0	63.2	54.4	56.72

Table 1: Evaluation results of open-source/commercial MLLMs and our proposed CoE on FutureBench [ 43 ] .

Method	Verb (Test / Val) $\uparrow$	Noun (Test / Val) $\uparrow$			Action (Test / Val) $\uparrow$
		Precision	Recall	F1-Score	Precision	Recall	F1-Score
LLaVA-Video-7B [ 62 ]	5.67 / 6.27	44.50 / 44.96	41.95 / 41.64	43.19 / 43.24	3.13 / 3.57	2.95 / 3.30	3.04 / 3.43
Kimi-VL-A3B	5.12 / 5.30	31.79 / 31.46	36.43 / 37.04	33.95 / 34.02	2.40 / 2.83	3.27 / 3.16	2.77 / 2.99
InternVL3-8B	5.80 / 7.00	48.77 / 43.64	45.77 / 37.94	47.22 / 40.59	3.37 / 3.60	3.23 / 3.07	3.30 / 3.31
GLM-4.1V-9B	9.10 / 10.62	33.80 / 34.94	33.66 / 35.42	33.73 / 35.18	3.01 / 3.27	3.02 / 3.34	3.01 / 3.30
MiMo-VL-7B	10.23 / 9.57	43.46 / 39.54	32.07 / 35.10	36.91 / 37.19	6.96 / 5.54	5.08 / 3.86	5.87 / 4.55

Table 2: Evaluation results of open-source MLLMs and our proposed CoE on AVEP [ 39 ] .

Figure 4: Attention difference of visual tokens between different methods and the base model. Portions greater than 0 indicate an improvement in attention.

Method	SFT	CoT	GRPO	CoE-GRPO	CoE-SFT
Baseline	Qwen2.5-VL-7B-Instruct
WR $\uparrow$	0.32	0.44	0.59	0.77	0.93
IR(%) $\uparrow$	-3.33	+1.08	+1.47	+9.20	+15.11

Table 3: Attention variation of visual tokens. WR (Winning Rate) denotes the overall percentage of samples showing attention improvement compared to the base model, while IR (Improvement Rate) indicates the average numerical increase in attention weight compared to the base model.

Method	CoE-SFT	CoE-GRPO	Instruct	SFT	GRPO
WR(%) $\uparrow$	38.13	32.42	16.21	7.88	5.37

Table 4: The results of judge evaluation of different method. WR represents the ratio of samples where the model’s answers are more reasonable and accurate compared to other models.

Item	1-Hop	2-Hop	3-Hop	Interp.	AVG
Visual Attention Enhancement Methods
Prompt-guided	44.5	46.6	43.8	46.6	45.74
Constant-Bias	54.6	52.3	57.6	50.6	52.57
CoE (Ours)	80.9	83.9	71.6	71.4	75.00
Group Size $G$
$G=2$	57.8	64.8	60.7	59.9	60.61

Table 5: The results of the ablation studies.

Figure 5: The training curves of CoE-GRPO.

Model	FutureBench $\uparrow$	AVEP $\uparrow$
Model	AVG	Verb	Noun-F1	Action-F1
VideoChat-R1	46.59	8.31	42.22	3.95
Video-R1	67.47	9.47	47.04	4.32
Ours	75.00	18.75	64.03	9.88
EventFormer*	-	22.71	46.24	7.69

Table 6: Additional results, *denotes traditional method.

Figure 6: The data scaling curve of CoE

Figure 7: An example of judge model evaluation.

Figure 16: Some examples of attention differences comparing to the vanilla model.

Figure 17: Prompt template for training and inference.

Figure 18: Prompt template for CoE-SFT data generation and judge model evaluation.

Video Prediction

Video prediction forecasts future frames or scene states from observed video for applications in perception, planning, and simulation.

See all 69 papers

Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction
Jun 4, 2026Tianxiang Jiang, Linquan Wu, Sheng Xia +5Large Vision-Language Models Latent Visual Reasoning
University of Science and Technology of China · Shanghai AI Laboratory · City University of Hong Kong +4
VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction
Oct 5, 2026Qiutong Chen, Yuchan Guo, Zhenlong Yuan +10Video Prediction Causal Reasoning in Videos
Nankai University · Carnegie Mellon University · Xiaohongshu Inc. +4
Pop-Up Distractions Reveal Bag-of-Events Behavior in Video Large Language Models
May 26, 2026Oscar Chew, Serhii Honcharenko, Qian-Hui Chen +4Temporal Video Grounding Video-Language Model Evaluation
Texas A&M University · National Taiwan University · Stanford University +1

Jun 4, 2026J/K move · Enter open · S save

Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction

Tianxiang Jiang, Linquan Wu, Sheng Xia, Songze Li+4

University of Science and Technology of China · Shanghai AI Laboratory · City University of Hong Kong+4

Large Vision-Language Models Latent Visual Reasoning Video Prediction Visually Grounded Reasoning RL for VLMs

Open paper AskPDF arXiv

Video event prediction (VEP) requires models to infer unobserved future states from partial video evidence. Existing video MLLMs usually verbalize intermediate future reasoning in text space: once visual evidence is verbalized, fine-grained motion, geometry, and interaction cues can be lost, leading to plausible but visually ungrounded hallucinations. We introduce Future-L1, an interleaved latent visual reasoning framework that lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding. To train this capability, we construct Future-L1-50K by selecting examples where future visual hints help prediction and align latent states to future-frame embeddings, then further optimize sampled latent trajectories with LA-DAPO, a latent-aware RL objective with outcome-contrastive and temporal-diversity rewards. Future-L1 achieves new state-of-the-art results on both benchmarks: on FutureBench, it improves Qwen3-VL-8B from 61.0 to 85.4 and exceeds the previous best Video-CoE by 10.4 points; on TwiFF-Bench, it improves the average score from 2.44 to 3.04. These results suggest that future-oriented video reasoning benefits from preserving intermediate visual semantics in latent space rather than translating every reasoning step into text.

Abstract

Video Prediction

Explore similar work

Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction

VepAgent: Bridging Causal-Transition via Tool-Augmented Reinforcement Learning for Video Event Prediction

Pop-Up Distractions Reveal Bag-of-Events Behavior in Video Large Language Models