RiskCueBench: Benchmarking Anticipatory Reasoning from Early Risk Cues in Video-Language Models

Sha Luo; Yogesh Prabhu; Timothy Ossowski; Kaiping Chen; Junjie Hu

cs.CVJan 6, 2026

RiskCueBench: Benchmarking Anticipatory Reasoning from Early Risk Cues in Video-Language Models

Authors: Sha Luo, Yogesh Prabhu, Timothy Ossowski, Kaiping Chen, Junjie Hu

Organizations: University of Wisconsin–Madison, Madison, WI, USA · University of California San Diego, San Diego, CA, USA

Abstract

With the rapid growth of video centered social media, the ability to anticipate risky events from visual data is a promising direction for ensuring public safety and preventing real world accidents. Prior work has extensively studied supervised video risk assessment across domains such as driving, protests, and natural disasters. However, many existing datasets provide models with access to the full video sequence, including the accident itself, which substantially reduces the difficulty of the task. To better reflect real world conditions, we introduce a new video understanding benchmark RiskCueBench in which videos are carefully annotated to identify a risk signal clip, defined as the earliest moment that indicates a potential safety concern. Experimental results reveal a significant gap in current systems ability to interpret evolving situations and anticipate future risky events from early visual signals, highlighting important challenges for deploying video risk prediction models in practice.

Figures & tables

Dataset	Temporal Reasoning	Object Grounding	Reasoning Metrics	Event Forecasting	Risk Events
	§ 4.3.3	§ 4.3.1	§ 4.3.2	§ 4.3.4	§ 3.3
TimeLogic	✓	✓	✗	✗	✗
ReXTime	✓	✓	✗	✗	✗
MotionBench	✓	✗	✗	✗	✗
MVBench	✓	✓	✗	✗	✗
VLM4D	✓	✗	✗	✗	✗

Table 1: Overview of benchmarks for spatio-temporal reasoning in vision–language models. Unlike prior work, our analysis centers on risk events, evaluating models’ ability to forecast under uncertainty, while also examining reasoning traces and object grounding.

Figure 1: Many existing benchmarks ask about events that occur during the video. Our RiskCueBench focuses on predicting future risky events (details in § 3 ).

Figure 2: Overview of our pipeline to curate risk signal clips from real-world incidents and evaluate VLM reasoning. Collection: We collect a large candidate set of YouTube videos using domain-relevant keywords. Scoring: The collected videos are filtered to only retain those with potential risk and high difficulty. Annotation: Human annotators label the filtered videos to identify the risk signal clip. Evaluation: Popular VLMs are presented with only the risk signal clip, and their output reasoning traces are evaluated.

Video Type	Average Signal Length (s)	# Videos
Car Crash (Normal)	3.49 $\pm$ 0.58	250
Car Crash (High Risk)	1.38 $\pm$ 0.77	252
Protest (Normal)	17.77 $\pm$ 14.81	267
Protest (High Risk)	5.99 $\pm$ 7.07	217
Arson (Normal)	15.83 $\pm$ 26.24	178
Arson (High Risk)	17.33 $\pm$ 26.29	231

Table 2: Risk signal length for RiskCueBench videos.

Non-Reasoning Models
Model	Car Crash				Protest				Arson				Cruise Collision
Model	F1	RGA	TRD	SCD	F1	RGA	TRD	SCD	F1	RGA	TRD	SCD	F1	RGA	TRD	SCD
Video-LLaVA	0.53	48.2	1.84	-	0.44	33.5	1.87	-	0.55	52.8	1.91	-	0.55	49.3	1.68	-
VideoLLaMA 3	0.59	54.1	1.65	-	0.46	39.2	2.42	-	0.57	53.2	1.65	-	0.56	52.0	1.61	-
Qwen3-VL-8B	0.64	57.3	1.48	-	0.48	42.8	2.23	-	0.62	55.3	1.59	-	0.63	57.7	1.72	-
MiMo-SFT	0.58	51.4	2.12	-	0.47	38.1	1.76	-	0.56	50.4	2.05	-	0.59	52.5	2.08	-

Table 3: Performance statistics of current SOTA models on binary risk prediction across the four domains. F1 denotes positive-class (risk) F1 computed separately within each domain; we do not report an overall or cross-domain average F1. The corresponding always-risk no-skill F1 baselines and additional TN-sensitive metrics including Accuracy, Specificity, and MCC are reported in Appendix 10. Random Guess reports a fixed-seed empirical random baseline; deviations from 0.50 reflect finite-sample variation, class imbalance, and seed noise. Certain metrics remain blank for Gemini-Flash-3 because its internal reasoning traces are not accessible for evaluation.

Figure 3: Percentages of correct and incorrect risk predictions across 8 relatedness score bins. Relatedness score here refers to the average over all predicted items by all models for each video in our dataset; see Eq. ( 1 ). Correct predictions (green bars) exhibit significantly higher mean relatedness scores than incorrect ones (red bars), indicating that accurate risk anticipation is strongly tied to better visual grounding.

Figure 4: Reasoning length distribution across incorrect and correct predictions. Incorrect predictions (red) are associated with significantly longer and more complex reasoning traces compared to correct ones (green), suggesting that models’ overthinking or circular deliberation often leads to performance degradation.

Figure 5: Taxonomy of VLM failure modes in situational risk assessment, categorized by perceptual, reasoning, and conclusion errors. Examples include perceptual errors (Type 1) that miss critical cues like aggressive shoving, reasoning errors (Type 2) like failing to interpret normal vs. abnormal behaviors, and incorrect answers (Type 3).

Video Type	Car Crash	Protest	Arson	Cruise Collision
Basic	$0.67\pm 0.2$	$0.59\pm 0.2$	$0.57\pm 0.1$	$0.62\pm 0.2$
Shuffled	$0.68\pm 0.1$	$0.57\pm 0.3$	$0.59\pm 0.1$	$0.61\pm 0.1$
Swap Halves	$0.65\pm 0.3$	$0.58\pm 0.3$	$0.56\pm 0.2$	$0.59\pm 0.3$
Reversed	$0.68\pm 0.3$	$0.58\pm 0.3$	$0.58\pm 0.3$	$0.62\pm 0.3$

Table 4: Model performance across temporal perturbations. Model performance is nearly identical across original and modified temporal sequences, suggesting that VLMs rely more on static frame content than genuine temporal reasoning for risk assessment.

Figure 6: Model accuracy vs. Risk signal distance for Protest data. F1 scores decline as the temporal gap between the risk cue and the incident increases from 1 to 20 seconds, suggesting that VLMs struggle to maintain predictive accuracy when cues are temporally far.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Year	Location	Specific Cities	Domain Synonyms	Example Query
Domain: Protests
2000	Palestine	Ramallah, Nablus, Hebron, …	civil unrest, protest, rally, sit-in, strike, march, demonstration	2000 Palestine civil unrest; 2000 Ramallah civil unrest…
2003	Global	New York City, London, Tokyo, …		2003 Global civil unrest; 2003 New York City march…
2011	Egypt	Cairo, Alexandria, Suez, …		2011 Cairo demonstration…
2014	Hong Kong	Hong Kong		2014 Hong Kong strike…
2020	United States	Minneapolis, New York City, …		2020 Minneapolis civil unrest…

Appendix

Table 5: Example List for Macro-Year-Location Query Construction

Domain	Vocabulary Group	Included Terms & Constraints
Arson	Evidence Terms	caught on camera, caught on tape, CCTV , surveillance video, security camera, doorbell camera, dashcam, footage
	Action Terms	set on fire, setting fire, pours gasoline, pouring petrol, lights himself on fire, sets ablaze, torching
	Actor Terms	suspect, arsonist, man, masked, criminal, blundering, wanted
	Target Terms	car, truck, vehicle, house, restaurant, store, church, mosque, gas station, strip mall, cop car, apartments
	Example Query	(surveillance video) AND (pours gasoline) AND (masked suspect) AND (“gas station”)
Cruise	Base Terms	cruise ship collision, passenger ship collision, ferry collision, maritime collision, vessel collision, cruise ship hits dock, ship docking accident, bridge strike ship

Appendix

Table 6: Example List for Micro-Anomaly Combination Queries (Arson and Cruise Collisions)

No.	Dimension	Criterion	Score 0	Score 1	Score 2
1	Logo Assessment	Video should have no news organization branding or features	Obvious news logos, watermarks, or media branding visible	Ambiguous logos or possible reposts with unclear branding	No branding visible; appears bystander- or CCTV-style
2	Location Information	Video should contain no location mentioned in text or audio	Explicit location references (city, state, country)	Ambiguous or hypothetical location references	No location information present
3	Time/Date Information	Video should contain no time or date mentioned in text or audio	Explicit time/date references (e.g., dates, timestamps)	Ambiguous or unrealistic temporal references	No time/date information present
4	Reporter Presence	Video should include no reporter, anchor, or journalist	Clearly identifiable reporter or journalist present	Possibly media-affiliated speaker or unclear role	No formal speaker; only participants or bystanders
5	SNS Engagement Overlays	Video should contain no social media overlays or engagement icons	Likes, shares, emojis, or UI elements visible	Minimal or transient overlays	No overlays or engagement elements
6	Natural Image Quality	Video should show no signs of professional editing	Heavy editing, stylized transitions, montage effects	Some editing artifacts or unclear transitions	Continuous, shaky, or natural camera movement

Appendix

Table 7: Visual Quality and Authenticity Scoring Criteria

Label	Protest (Risk)	Protest (No-Risk)	Arson (Risk)	Arson (No-Risk)
Risk Signal Start	00:03:00	00:04:00	00:01:15	00:02:00
Risk Signal End	00:12:58	00:08:55	00:08:12	00:05:00
Risk Visual Indicator	Throwing gestures, aggressive body postures, raised batons, riot shields.	Singing, dancing, sign-holding, and stationary police lines.	Plastic bottle, dousing liquid from the bottle, lighter.	Birthday cake, candles on cake, parent monitoring.
Risk Signal Description	First police vehicles appear and protestors intend to throw objects; then officers raise baton toward protesters.	Protesters hold signs still, some sing and dance while police observe from a distance without engagement.	First a man in black hoodie enter the room carrying a big plastic bottle, then he douse liquid all over the room from the bottle, afterward he takes out a lighter from pocket.	First a girl shows up sitting in front of a birthday cake, next her parents hold her hand to light up the candle on the cake with a burning match.
Accident Start Frame	00:12:59	/	00:08:13	/
Accident End Frame	00:15:00	/	00:12:00	/

Appendix

Table 8: Annotation Examples: Protests & Arson

Label	Car Crash (Risk)	Car Crash (No-Risk)	Cruise (Risk)	Cruise (No-Risk)
Risk Signal Start	00:05:12	00:39:38	00:02:45	00:10:00
Risk Signal End	00:10:11	00:44:00	00:07:30	00:15:00
Risk Visual Indicator	Crossing vehicle, red light, arrow on the ground indicating traffic flow.	Lane mark, steady speed, stop sign	Large cruise ship moving fast toward the dock, loud warning horn sounding continuously, people on the dock starting to run away in panic, ship appears too large to stop in time.	Slow controlled approach to port, steady movement, maintaining course toward the docking area, no visual signs of impact or distress.
Risk Signal Description	First a car enters the intersection against traffic flow; then ego continues when traffic light is red.	Vehicles slow appropriately at stop sign and proceed steadily according to traffic signals within lane mark correctly.	First, the cruise ship approaches the dock at a high speed with its horn blaring. Then, it becomes clear it cannot stop as people on the pier begin to scatter. Afterwards, the ship maintains its trajectory directly toward the stationary river boat and dock.	First, the cruise ship approached the pier slowly. Then, it maintained a steady path for docking. Afterwards, it successfully reached the designated area without any issues.
Accident Start Frame	00:10:12	/	00:07:31	/
Accident End Frame	00:13:00	/	00:11:00	/

Appendix

Table 9: Annotation Examples: Car Crash & Cruise Collisions

Figure 7: The histograms illustrate the distribution of risk signal lengths across four visual risk domains: car crashes, protests, arson, and cruise collisions. A “risk signal” is defined as the observable temporal window before an incident or outcome during which visible cues of potential danger are present.

Model Category	Model	F1	RGA	TRD	SCD
Non-Reasoning	Video-LLaVA	$0.54\pm 0.05$	$49.6\pm 5.48$	$1.86\pm 0.63$	-
	VideoLLaMA 3	$0.60\pm 0.04$	$55.4\pm 6.18$	$1.63\pm 0.58$	-
	Qwen3-VL-8B	$0.65\pm 0.04$	$58.7\pm 5.09$	$1.51\pm 0.52$	-
	MiMo-SFT	$0.59\pm 0.05$	$52.8\pm 7.31$	$2.09\pm 0.74$	-
	Apollo-7B	$0.58\pm 0.06$	$54.9\pm 6.77$	$1.85\pm 0.61$	-
Reasoning	InternVL-3.5	$0.62\pm 0.05$	$65.7\pm 4.91$	$2.89\pm 0.68$	$-0.27\pm 0.05$

Appendix

Table 11: Follow-up experiment on the Car Crash scenario using all 50 frames. Increasing the frame rate yields only marginal improvements in F1 and RGA, while TRD and SCD remain largely unchanged.

Figure 8: Crowdsourcing annotation interface for Arson.

Domain	Model	F1	Accuracy	Specificity	Balanced Accuracy	MCC
Car Crash	Video-LLaVA	0.53	0.59	0.62	0.58	0.17
	VideoLLaMA 3	0.59	0.59	0.60	0.59	0.18
	Qwen3-VL-8B	0.64	0.61	0.61	0.61	0.22
	MiMo-SFT	0.58	0.62	0.65	0.62	0.24
	Apollo-7B	0.57	0.59	0.61	0.59	0.18
	InternVL-3.5	0.61	0.64	0.66	0.64	0.28

Appendix

Table 12: Additional evaluation metrics beyond positive-class F1 across RiskCueBench domains.

Domain	Always-Risk F1	Random Guess F1
Car Crash	0.668	0.50
Protest	0.619	0.44
Arson	0.722	0.53
Cruise Collision	0.640	0.47

Appendix

Table 13: No-skill baseline comparison across RiskCueBench domains. The always-risk classifier predicts all samples as risky, providing a prevalence-dependent positive-class F1 baseline.

Model Category	Model	Car Crash				Protest
		F1	RGA	TRD	SCD	F1	RGA	TRD	SCD
Non-Reasoning	Video-LLaVA	0.52	47.6	1.90	–	0.43	32.9	1.92	–
	VideoLLaMA 3	0.58	53.4	1.69	–	0.45	38.7	2.49	–
	Qwen3-VL-8B	0.63	56.8	1.46	–	0.47	42.2	2.28	–
	MiMo-SFT	0.57	50.6	2.17	–	0.48	37.4	1.81	–
	Apollo-7B	0.58	52.9	1.89	–	0.44	36.8	2.66	–

Appendix

Table 14: Length-matched evaluation results for Car Crash and Protest domains. After matching risk and non-risk videos by duration, model performance remains comparable to the original evaluation, suggesting that observed performance differences are not primarily driven by duration imbalance.

	Perceptual	Reasoning	Conclusion
Perceptual	192 (57.1%)	31 (9.2%)	192 (57.1%)
Reasoning	31 (9.2%)	65 (19.3%)	65 (19.3%)
Conclusion	192 (57.1%)	65 (19.3%)	336 (100%)

Appendix

Table 15: Overlap matrix for qualitative error categories among 336 misclassified cases. Diagonal entries report the prevalence of each error category, while off-diagonal entries report pairwise co-occurrence. Because all analyzed cases are prediction failures, conclusion errors occur in every sample.

Model	CC V	CC 1st	CC Last	PR V	PR 1st	PR Last	AR V	AR 1st	AR Last	CR V	CR 1st	CR Last
Video-LLaVA	.53	.49	.55	.44	.42	.43	.55	.50	.58	.55	.50	.58
VideoLLaMA 3	.59	.55	.61	.46	.44	.45	.57	.52	.60	.56	.51	.59
Qwen3-VL-8B	.64	.61	.63	.48	.47	.46	.62	.59	.61	.63	.60	.62
MiMo-SFT	.58	.52	.62	.47	.43	.49	.56	.50	.60	.59	.53	.63
Apollo-7B	.57	.53	.59	.45	.43	.44	.58	.53	.61	.57	.52	.60
InternVL-3.5	.61	.63	.60	.47	.49	.46	.60	.61	.59	.59	.60	.58

Appendix

Table 16: Key-frame analysis across RiskCueBench domains. CC, PR, AR, and CR denote Car Crash, Protest, Arson, and Cruise Collision, respectively. V denotes the original full-video condition, while 1st and Last denote extreme-sparse baselines using only the first or last frame of the risk signal clip.

RGA Group	Verified (%)	$N$
High RGA	75	35
Low RGA	21	35

Appendix

Table 17: Human validation of RGA. Verified indicates the percentage of reasoning items judged by human annotators to correspond to visual evidence in the video and annotations.

Temporal Video Understanding

Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.

See all 371 papers

PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning
Jun 1, 2026Yusong Zhao, Yuejin Xie, Youliang Yuan +4Traffic Accident Anticipation Video-Language Models
The Chinese University of Hong Kong, Shenzhen · Tsinghua University
EgoSafe: A First-Person Mobile-Captured Benchmark for Visual Safety Understanding
Jul 29, 2026Yuyun Chen, Tianao Li, TianQuan Feng +4Egocentric Video Understanding Video Reasoning
South China University of Technology · Beihang University
RiskProp: Collision-Anchored Self-Supervised Risk Propagation for Early Accident Anticipation
Mar 28, 2026Yiyang Zou, Tianhao Zhao, Peilun Xiao +9Monotonicity Constraints Traffic Accident Anticipation
School of Computer Science, Wuhan University · Zhongguancun Academy, Beijing, China · Didi Chuxing +1

Jun 1, 2026J/K move · Enter open · S save

PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning

Yusong Zhao, Yuejin Xie, Youliang Yuan, Junjie Hu+3

The Chinese University of Hong Kong, Shenzhen · Tsinghua University

Traffic Accident Anticipation Video-Language Models Multimodal Model Evaluation Temporal Video Understanding

Open paper AskPDF arXiv

Between the first visible sign of danger and the moment an accident occurs, there is often a window where intervention remains possible. Video-capable multimodal large language models (MLLMs) could serve as always-on safety monitors that issue warnings during this window. Yet current benchmarks do not test this ability: they rely on static inputs, ignore timing precision, and omit false-positive measurement on safe scenes. We present PaSBench-Video, a 740-video benchmark with 481 risk and 259 no-risk videos across four domains: driving, healthcare, daily life, and industrial production. Risk videos are annotated with frame-level risk onset and accident boundaries. A model must observe the video causally and produce a warning that is both temporally calibrated and content-correct. Testing 13 MLLMs, we find that no model exceeds 20.0% on our strictest metric, and recall is tightly coupled with false-positive rate, with Pearson correlation 0.64: higher detection comes only at the cost of triggering warnings on the majority of safe clips. Performance splits sharply by domain: models achieve moderate recall at low false-positive rates in daily life, where risks are inherently anomalous, yet fire indiscriminately in driving, where routine and hazardous scenes look alike. These results indicate that current models rely on scene-level activity cues rather than reasoning about emerging harm.

Abstract

Temporal Video Understanding

Explore similar work

PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning

EgoSafe: A First-Person Mobile-Captured Benchmark for Visual Safety Understanding

RiskProp: Collision-Anchored Self-Supervised Risk Propagation for Early Accident Anticipation