Organizations: East China Normal University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Shanghai Innovation Institute, Shanghai, China
Multimodal Large Language Models (MLLMs), particularly smaller, deployable variants, exhibit a critical deficiency in understanding temporal and procedural visual data, a bottleneck hindering their application in real-world embodied AI. This gap is largely caused by a systemic failure in training paradigms, which lack large-scale, procedurally coherent data. To address this problem, we introduce TPRU, a large-scale dataset sourced from diverse embodied scenarios such as robotic manipulation and GUI navigation. TPRU is systematically designed to cultivate temporal reasoning through three complementary tasks: Temporal Reordering, Next-Frame Prediction, and Previous-Frame Review. A key feature is the inclusion of challenging negative samples, compelling models to transition from passive observation to active, cross-modal validation. We leverage TPRU with a reinforcement learning (RL) fine-tuning methodology, specifically targeting the enhancement of resource-efficient models. Experiments show our approach yields dramatic gains: on our manually curated TPRU-Test, the accuracy of TPRU-7B soars from 50.33% to 75.70%, a state-of-the-art result that significantly outperforms vastly larger baselines, including GPT-4o. Crucially, these capabilities generalize effectively, demonstrating substantial improvements on established benchmarks. The codebase is available at https://github.com/Stephen-gzk/TPRU/ .
Figures & tables
Figure 1: An overview of our TPRU dataset. Unlike prior synthetic datasets (a), TPRU is built from real-world scenarios and structured into temporal tasks (b). As shown in the ability display (c), TPRU-7B achieves significant performance gains in temporal understanding.
Figure 2: The TPRU dataset construction and training pipeline. Chronological image sequences from embodied sources are curated with both positive and negative text descriptions. These image sequences are then formulated into three tasks (Ordering, Next Frame Prediction, and Previous Frame Review) to fine-tune MLLMs for enhanced temporal and procedural understanding.
Model
Action Underst.
Attribute Similarity
Cartoon Underst.
Counting
Diagram Underst.
Difference Spotting
Geographic Underst.
Image-Text Matching
Ordering
Scene Underst.
Visual Grounding
Visual Retrieval
Overall
Open-source
InternVL3-78B
48.17
61.73
44.87
50.85
83.17
55.59
60.00
79.31
31.25
69.35
44.05
66.10
64.65
InternVL3-38B
44.51
66.33
46.15
45.30
78.14
61.18
63.00
77.80
32.81
61.29
36.90
72.95
64.12
Qwen2.5-VL-72B
50.00
59.18
42.31
49.57
89.45
60.59
50.00
87.93
40.63
76.34
46.43
78.42
69.35
Qwen2.5-VL-32B
36.59
51.02
47.44
45.30
82.41
58.53
47.00
85.78
26.56
72.04
41.67
67.12
63.73
Qwen2.5-VL-7B
40.85
58.67
46.15
34.19
77.89
54.41
49.00
72.63
14.06
61.83
33.33
63.70
58.35
Table 1: Performance on MuirBench. The light gray rows show the absolute improvement (in percentage points) of our models over their corresponding Qwen2.5-VL base models. Gains are shown in red, and losses in blue.
Models
Height
Adjacency
Rotation
Multiview
Next-Step
Dependency
Rotation Stat.
Position
Backwards
Ordering
Outlier
Overall
Open-source
InternVL3-78B
52.0
63.0
36.0
54.0
64.0
80.0
58.0
29.0
25.0
22.0
37.0
47.3
InternVL3-38B
40.0
60.0
39.0
55.0
57.0
81.0
59.0
33.0
47.0
12.0
36.0
47.2
Qwen2.5-VL-72B
43.0
58.0
38.0
39.0
57.0
76.0
57.0
52.0
74.0
43.0
43.0
52.7
Qwen2.5-VL-32B
35.0
60.0
38.0
52.0
45.0
79.0
51.0
45.0
66.0
43.0
43.0
50.6
Qwen2.5-VL-7B
20.0
57.0
32.0
47.0
38.0
67.0
56.0
29.0
22.0
8.0
25.0
36.5
Table 2: Performance on LEGO-Puzzles. The light gray rows show the absolute improvement (in percentage points) of our models over their corresponding Qwen2.5-VL base models. Gains are shown in red, and losses in blue.
Model
MME- RealWorld-Lite
BLINK
RealWorld QA
MMCR
MMTBench
MMStar
MMMU-Dev
Overall
Open-source
InternVL3-78B
65.40
66.30
78.00
20.29
73.20
72.50
64.20
62.84
InternVL3-38B
67.30
64.49
75.60
21.38
71.80
71.50
62.00
62.01
Qwen2.5-VL-32B
45.96
59.34
68.89
37.32
58.89
54.93
34.67
51.43
Qwen2.5-VL-7B
44.55
54.76
68.10
22.83
61.46
61.67
44.40
51.11
Qwen2.5-VL-3B
41.94
48.97
65.36
19.20
60.47
54.40
44.67
47.86
Table 3: Evaluation on general multi-image benchmarks. Results show that our TPRU models maintain comparable performance to their base models, indicating that our specialized training does not degrade general capabilities.
Figure 3: Performance of different models on TPRU-test.
TPRU Data Components
Benchmark Accuracy (%)
Ordering
Previous Frame Review
Next Frame Predict
MuirBench
LEGO-Puzzles
✓
60.8
39.0
✓
61.7
38.0
✓
61.6
41.1
✓
✓
62.5
40.3
✓
✓
62.2
40.3
Table 4: Ablation study on the effect of different TPRU data components for the Qwen2.5-VL-7B model on MuirBench and LEGO-Puzzles benchmarks. The inclusion of a component is marked with a ✓. All scores are Overall Accuracy (%). The volume of data remains consistent.
Figure 4: Ablation analysis. (a) Ablation on negative samples. (b) and (c) show the performance with different training samples.
Training Paradigm
TPRU-Test
MuirBench
LEGO-Puzzles
Supervised Fine-Tuning (SFT)
72.88
63.03
40.60
TPRU-7B (GRPO)
75.70
65.04
42.82
Table 5: Ablation study on training paradigms. We compare the performance of standard Supervised Fine-Tuning (SFT) against our GRPO approach using the Qwen2.5-VL-7B backbone.
Model
MuirBench
LEGO-Puzzles
TPRU-Test
Overall
Ordering
Overall
Ordering
LLaVA-Video-Qwen2-7B
37.88
15.63
31.91
1.0
38.61
LLaVA-Video-Qwen2-72B
41.81
10.94
41.64
5.0
44.03
SmolVLM2-256M-Video
27.92
21.88
28.18
0.0
17.79
SmolVLM2-500M-Video
25.92
7.8
27.27
0.0
16.92
Long-VITA-16K
53.07
17.19
34.45
2.0
39.26
Table 6: Performance comparison with state-of-the-art Video LLMs on MuirBench, LEGO-Puzzles, and TPRU-Test datasets. The “Ordering” columns denote specific subtasks requiring precise sequential reasoning. Best results are highlighted in bold .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Training Strategy
Benchmark Accuracy (%)
Stage 1
Stage 2
Stage 3
MuirBench
LEGO-Puzzles
Ordering
Next Frame Prediction
Previous Frame Review
61.31
39.91
Ordering
Previous Frame Review
Next Frame Prediction
61.58
40.45
Previous Frame Review
Ordering
Next Frame Prediction
60.38
42.64
Next Frame Prediction
Ordering
Previous Frame Review
64.23
42.55
Next Frame Prediction
Previous Frame Review
Ordering
63.62
42.09
Appendix
Table 7: Ablation study on the training stage order for the Qwen2.5-VL-7B model. All scores are Overall Accuracy (%).
Figure 5: RL Training Reward Curves. The plots (a), (b) and (c) display the reward score across training steps. The curves demonstrate rapid initial convergence followed by a stable high-score plateau, indicating a stable optimization process without collapse or drastic oscillations.
Model
MuirBench
LEGO-Puzzles
TPRU-Test
Overall
Ordering
Overall
Ordering
TPRU-7B (No-Format Reward)
63.96
29.69
41.64
28.0
74.40
TPRU-7B (Ours)
65.04
34.4
42.82
32.0
75.70
Appendix
Table 8: Ablation study on the impact of format rewards. “TPRU-7B (No-Format Reward)” denotes the model trained solely with task-accuracy rewards, excluding format-specific constraints.