Modern video-language models are increasingly treated as general-purpose video reasoners, yet it remains unclear whether they robustly capture temporal dynamics in safety-critical settings such as driving. We study this question through controlled ego-motion tasks that require identifying actions from driving videos. Across video-language models, we find that standard prompting, visual prompting, few-shot prompting, and chain-of-thought prompting remain unreliable and highly task- and model-dependent, with persistent class biases, although newer models show substantial task-specific few-shot gains. In contrast, lightweight LoRA finetuning and simple probes trained on intermediate visual features recover strong motion-discriminative performance, indicating that motion-relevant cues are often present inside the models but are not reliably used by the final video-to-language pathway. These results suggest that, in driving, current video-language models remain unreliable temporal reasoners and often behave more like frame-level, bias-driven recognizers than dependable temporal world models.
Figures & tables
Fig. 1: Pipeline for probing ego-motion understanding in video-language models. We evaluate increasingly structured approaches, from training-free prompting to lightweight learning-based probes. In standard prompting , the model directly classifies the vehicle’s motion. Visual prompting adds a green bounding box around a static reference object to test whether explicit spatial grounding improves motion inference. In fusion of visual and text , labeled few-shot video examples are provided before the query. Chain-of-thought prompting explicitly relates temporal changes in the reference object’s position or size to ego-motion. We then apply LoRA finetuning to test whether motion distinctions can be learned under lightweight supervision, and feature extraction to train simple classifiers on intermediate visual representations and examine where motion information remains decodable. Overall, prompt-level methods remain strongly model- and task-dependent, whereas learning-based approaches recover substantially stronger motion discrimination.
Model
Action Pair
Prompt-level probing (accuracy %)
Learning-based probing (accuracy %)
Standard
Visual Prompt (VP)
Few-shot
Few-shot + CoT
Few-shot + CoT + VP
LoRA Finetune
Features (7a)
Features (7b)
Sim
Real
Qwen2-VL
Forward / Backward
50.00
50.00
65.00
58.80
62.06
62.06
100.00
88.75
93.75
Left / Right
50.00
50.00
50.00
48.57
50.00
49.27
96.25
93.75
97.50
Accelerate / Decelerate
51.25
35.00
51.00
50.00
50.00
48.50
100.00
87.50
88.75
Qwen2.5-VL
Forward / Backward
50.00
50.00
54.27
50.00
50.75
51.26
100.00
50.00
97.50
TABLE I: Accuracy on ego-motion tasks across video-language models under different prompting and learning-based strategies. Standard prompting is evaluated on both simulated (Sim) and real-world (Real) driving videos; all other results use Sim data. Features (7a) and (7b) report the 1-layer CNN probe without pooling.
Action Pair
Prompt-level probing (accuracy %)
Learning-based probing (accuracy %)
Standard
Visual Prompt
LoRA Finetune
Features (7a)
Features (7b)
Sim
Real
Forward / Backward
50.00
50.00
50.00
100.00
98.75
77.50
Left / Right
50.50
50.00
49.50
97.50
100.00
100.00
Accelerate / Decelerate
49.75
70.00
50.00
82.50
86.25
66.25
TABLE II: VideoLLaMA2 ego-motion accuracy. Standard prompting uses simulated (Sim) and real-world (Real) videos; all other results use Sim data. Feature columns follow Table I .
Action
VideoLLaMA2
Qwen2-VL
Qwen2.5-VL
Qwen3-VL
Intern-VL-3.5
MiniCPM-v4.5
Forward
100.0
100.0
100.0
100.0
100.0
100.0
Backward
0.00
0.00
0.00
0.00
0.00
0.00
Accelerate
99.50
27.50
99.50
99.00
98.50
0.00
Decelerate
0.00
75.00
3.50
13.00
10.50
100.0
Left
18.50
100.0
100.0
23.00
39.00
8.50
Right
82.50
0.00
0.00
97.00
95.00
90.50
TABLE III: Standard prompting accuracies for simulated data.
Fig. 2: Sim-to-real comparison of standard-prompting accuracy. The same class-level biases appear across CARLA and real-world videos. Overall class-level correlation is r=0.957 .
Fig. 3: Average ego-motion accuracy versus model release date. Standard and few-shot results are averaged across motion pairs. VideoLLaMA2 is excluded from the few-shot trend because its protocol is not comparable.
Model
Action
Few-shot
Few-shot with CoT
Few-shot with VP
Few-shot with CoT and VP
Qwen2-VL
Forward
95.48
91.96
94.96
91.96
Backward
22.11
32.16
21.16
32.16
Accelerate
100.0
100.0
100.0
96.98
Decelerate
0.00
0.00
0.00
0.02
Left
5.53
0.00
5.03
8.04
Right
91.60
100.0
98.50
90.50
TABLE IV: Comparison of few-shot prompting strategies across video-language models. VP denotes visual prompting and CoT denotes chain-of-thought prompting.
Model
Action Pair
1-layer CNN
1-layer CNN + pooling
Pre
Post
Pre
Post
VideoLLaMA2
Forw/Back
98.75
77.50
100.00
66.25
Acc/Dec
86.25
66.25
82.50
62.50
Left/Right
100.00
100.00
100.00
96.25
Qwen2-VL
Forw/Back
88.75
93.75
88.75
92.50
Acc/Dec
87.50
88.75
87.50
86.25
TABLE V: Training simple CNN probes on intermediate representations from different video-language models. Results are reported for features extracted before and after the visual feature compression stage. † Chance-level pooled-probe run.
Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.