From Cause to Effect: Generating Intermediate Event Chains in Videos

Ridouane Ghermi; Vicky Kalogeiton; Ivan Laptev

cs.CVOct 19, 2026

From Cause to Effect: Generating Intermediate Event Chains in Videos

Authors: Ridouane Ghermi, Vicky Kalogeiton, Ivan Laptev

Organizations: LIX, Ecole Polytechnique, IP Paris · MBZUAI

Abstract

Video-Language Models (VLMs) excel at visual recognition but struggle with causal reasoning. Current evaluations are flawed: multiple-choice QA allows shortcuts, and open-ended QA lacks robust automatic evaluation. We introduce Causal Chain Generation (CCG), a novel task where models have to generate the sequence of intermediate events connecting a cause and an effect. This structured setup demands genuine reasoning and enables reliable automatic scoring. We accompany this task with two complementary metrics: SODA-Embedding, which measures the semantic matching of a generated chain against a reference, and Causal-QA, which probes the chain's causal coherence by asking whether it contains the specific causal facts needed to reconstruct the narrative. We provide two test sets: SF20K-CC, a manually annotated diagnostic on 20 short films, and CMD-CC, a movie clip benchmark derived from audio descriptions. Our analysis reveals that state-of-the-art VLMs struggle with narrative causality understanding, frequently hallucinating or omitting key events. To address this, we fine-tune VLMs with GRPO using a composite reward that combines a format reward, SODA-Embedding, and Causal-QA accuracy, demonstrating substantial improvements over strong baselines.

Figures & tables

Dataset	Split	#Vid	#Mov	#Chains	Mean len.	#MCQs	Chain dur. (s)
SF20K-CC	Test	20	20	205	7.67	537	96.9
CMD-CC	Test	471	98	1,031	6.17	2,001	69.1
CMD-CC	Train	6,497	1,320	14,175	6.12	26,726	68.6

Table 1: Dataset statistics. #Vid counts distinct source videos (full short films for SF20K-CC, 2-minute movie clips for CMD-CC), #Mov counts distinct movies, Mean len. is the average number of events per chain, and chain duration is in seconds.

Panel of Figure 2 — Figure 2: Narrative causality types. One representative causal chain for each of the four types in our benchmark, Physical, Psychological, Motivational, and Enabling, read left to right.

Figure 3: Causal chain statistics. Left: Distribution of chain durations (in seconds). Right: Frequency distribution of chain lengths (in number of events).

Figure 5: SODA-Embedding validation. Top: robustness to perturbation (mean success rate over the four operators, where a success is

\textit{metric}(g,\tilde{p})<\textit{metric}(g,p)

). Bottom: pairwise agreement with human judgment. Embedding-based metrics dominate ngram metrics on both axes; in particular SODA-E beats its ngram ablation SODA-C.

					CMD-CC		SF20K-CC
Model	Size	Video	Frames	Subs.	SODA-E	CQA	SODA-E	CQA
Reference conditions
Random	–	–	–	–	35.6	13.0	36.5	2.4
Guessing (query-only)	–	–	–	$\times$	45.6	7.5	45.8	11.6
Blind (query + subtitles)	–	–	–	✓	52.9	20.8	45.0	11.6
Open-source VLMs

Table 2: Baseline performance. Zero-shot performance on CMD-CC and SF20K-CC, grouped by baseline family. The reference conditions isolate each input modality: Random shuffles reference chains across test instances, Guessing sees the query alone, Blind adds subtitles but no frames. Video : full video at 1 FPS, or a truncated segment centered on the causal chain (32 frames); the two give comparable scores for the same model (Appendix K ). Subs. : subtitle transcript provided. All models receive the ground-truth number of events. Best per column in bold .

	CMD-CC		SF20K-CC
Method	SODA-E	CQA	SODA-E	CQA
Base	54.6	21.5	52.7	16.5
SFT	62.3	24.6	56.6	19.3
GRPO	58.9	32.2	57.2	24.9
SFT + GRPO	63.3	29.4	57.9	22.1

Table 3: Training-algorithm ablation of Qwen3.5-9B: no fine-tuning (Base), SFT, GRPO from the base model, and SFT followed by GRPO. All GRPO runs use the full composite reward. Best per column in bold , second-best underlined .

Base (no fine-tuning)
Reward			CMD-CC		SF20K-CC
F	S	C	SODA-E	CQA	SODA-E	CQA
–	–	–	54.6	21.5	52.7	16.5
GRPO fine-tuning
$\checkmark$	$\times$	$\times$	55.4	17.7	53.9	12.9
$\checkmark$	$\checkmark$	$\times$	60.1	26.8	58.8	17.7

Table 4: Reward ablation. Performance of GRPO-fine-tuned Qwen3.5-9B (from the base model, no SFT initialization) as we vary the composition of the composite reward (Equation 1 ). We report the base (non-fine-tuned) Qwen3.5-9B row for reference.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

YouTube ID	Movie Title	Synopsis
qEaoaRpRRzo	After The Rain	A dog turn sheep’s wool into clouds.
nfK79Xblpus	Nullarbor	A man with road rage confronts an elderly driver.
2p1YK2w8j38	Boy In The Woods	A young boy is lost in the forest.
tzJl6XdByZo	Wild Love	The heart has its reasons of which nature knows nothing.
koRvRsflL4Y	The Sea Is Blue	The ocean gets its color.
nfK79Xblpus	Nate And John	A barber and his client form an unlikely bond that develops over their lifetimes.

Appendix

Table 5: SF20K-CC Benchmark metadata. Detailed list of the 20 movies curated from the SF20K dataset, including YouTube IDs, movie title, and synopsis.

Figure 6: Illustration of the SODA-Embedding metric. (a) A semantic similarity matrix is computed (predicted

M=3

, ground-truth

N=4

). (b) An optimal, order-preserving alignment path is found via dynamic programming, yielding a “soft” True Positive (

TP

) sum of

2.65

. (c) The final F1-Score is calculated from this

TP

sum, robustly handling different chain lengths.

Panel of Figure 7 — Figure 7: Causal-QA examples. Two items spanning different causality types. For each, the source frames are shown above the ground-truth chain (event boxes read left to right), and the Causal-QA question is answerable only from the intermediate events. The correct option is highlighted.

	CMD-CC			SF20K-CC
Method	SODA-E	CQA	Judge	SODA-E	CQA	Judge
Base	54.6	21.5	2.59	52.7	16.5	2.41
SFT	62.3	24.6	2.83	56.6	19.3	2.55
SFT + GRPO	63.3	29.4	2.92	57.9	22.1	2.58

Appendix

Table 6: Held-out evaluation. Our two metrics and a held-out judge (GPT-5.6-Luna, 0–5 scale) across the training pipeline. The held-out judge appears in no reward term.

		CMD-CC		SF20K-CC
Init	Reward	SODA-E	CQA	SODA-E	CQA
Scratch	F	55.4	17.7	53.9	12.9
	F+S	60.1	26.8	58.8	17.7
	F+C	57.0	37.8	54.9	26.0
	F+S+C	58.9	32.2	57.2	24.9
SFT	F	57.8	18.3	53.9	14.1

Appendix

Table 7: Reward ablation across initializations. GRPO-fine-tuned Qwen3.5-9B under all four reward compositions (

R_{\text{format}}

always on; S

=R_{\text{SODA-E}}

, C

=R_{\text{Causal-QA}}

), trained from the base model ( scratch ) or on top of the SFT-merged model ( SFT ), at a converged 8,192-example budget. The scratch block is the from-scratch ablation condensed in Table 4 . Best per column in bold ; the from-scratch F+C SF20K cell has no trained eval.

	CMD-CC		SF20K-CC
$k$	SODA-E	CQA	SODA-E	CQA
1	54.9	27.5	53.4	18.1
2	56.9	27.6	53.4	18.5
4	58.4	28.9	54.3	20.1
8	59.4	29.3	53.9	18.9
16	60.0	29.9	54.7	18.5

Appendix

Table 8: Test-time scaling of GRPO-fine-tuned Qwen3.5-VL-9B: sampling

k

candidate chains and selecting one by judge self-consistency. Evaluated on the 150-clip CMD-CC subset and the SF20K-CC clips supported by the selection pipeline; not directly comparable to the main tables. Best per column in bold .

Metric	Sim	LLM	Swapping	Shuffling	Intruder	Random
Metric	Sim	LLM	Swapping	Shuffling	Sentence	Sequence
Random	-	-	50.00	50.00	50.00	50.00
CIDEr	N	-	45.6	56.1	67.7	75.8
BLEU-1	N	-	40.3	40.3	67.7	87.0
ROUGE-L	N	-	35.0	49.1	70.9	90.3
METEOR	N	-	61.4	70.1	77.4	91.9

Appendix

Table 9: Perturbation analysis of the SODA-Embedding metric.

Metric	Family	Agree.	$\tau$
SODA-E (Qwen-4B)	embed	79.2	.584
SODA-E (Qwen-0.6B)	embed	78.5	.570
BERTScore	embed	77.3	.547
SODA-E (MiniLM)	embed	76.2	.523
ROUGE-L	lexical	72.0	.441
SODA-C	lexical	66.1	.323

Appendix

Table 10: Agreement with human judgment. Pairwise meta-evaluation over 256 A/B pairs: a metric agrees when the prediction it scores higher is the one annotators prefer. We report agreement, a 95% bootstrap confidence interval, and Kendall

\tau=2\,\text{acc}-1

on decided pairs. SODA-C is the lexical ablation of SODA-Embedding (same alignment, METEOR scorer). Embedding metrics are in bold .

		CMD-CC			SF20K
Metric	Model	Agr.	$\rho$	$\tau$	Agr.	$\rho$	$\tau$
SODA-E	0.6B	97	.97	.94	97	.98	.94
	8B	100	.99	1.00	100	1.00	1.00
	MiniLM †	94	.95	.89	91	.91	.83
Causal-QA	0.5B	89	.92	.78	52	$-$ .02	.03
	7B	97	.98	.94	89	.90	.77

Appendix

Table 11: Metric robustness. Per-model stability of system rankings when only the metric’s underlying model is varied. For each variant we report agreement with the default-model ranking (% of system pairs ordered identically), Spearman

\rho

, and Kendall

\tau

, on both test sets. References: Qwen3-Embedding-4B (SODA-E), Qwen2.5-3B-Instruct (Causal-QA).

\dagger

: cross-family model.

Figure 9: Baseline scaling for the Qwen3.5-VL family on the full video. Left: vs. model size (0.8B–27B); both metrics rise steeply up to 4B and then saturate. Right: vs. number of input frames for the 9B model; Causal-QA keeps rising while SODA-Embedding saturates around 128 frames.

	CMD-CC		SF20K-CC
Condition	SODA-E	CQA	SODA-E	CQA
Full video (1 FPS)	53.8	20.7	54.3	15.3
Truncated clip (32 frames)	54.6	21.5	52.7	16.5
$\Delta$	$+0.8$	$+0.8$	$-1.6$	$+1.2$

Appendix

Table 12: Full vs. truncated video for Qwen3.5-VL-9B (with subtitles). Sampling the full video at 1 FPS and sampling 32 frames from a clip centered on the chain give near-identical scores.

Figure 10: Where the difficulty lives. Causal-QA accuracy on CMD-CC sliced by (a) causality type, (b) gold chain length, and (c) dialogue availability, for GPT-5.4-mini, Qwen3.5-9B (full video with subtitles), and a text-only control that sees the subtitles but no frames. Dialogue is the sharpest axis: silent movies are significantly harder to understand.

Causality type	CMD-CC	SF20K-CC
Motivational	35.8%	33.7%
Psychological	30.6%	14.7%
Physical	24.2%	45.3%
Enabling	0 8.3%	0 5.3%
Other	0 1.2%	0 1.1%

Appendix

Table 13: Causality-type distribution across the CCG test sets (% of chains; CMD-CC over 1,031 chains). Per-type labels are annotated by an LLM classifier.

Panel of Figure 11 — Figure 11: Qualitative analysis. Comparison of ground-truth chains (top) vs. GPT-4.1-mini predictions (bottom). The examples illustrate successful physical (Row 1) and motivational reasoning (Row 2) contrasted with failures due to missed fine-grained visual details (Row 3) and hallucinations (Row 4).

Figure 12: Evaluation prompt. The unified prompt used to evaluate all baseline VLMs (GPT-4.1, Gemini-2.5, Qwen3-VL) on the Causal Chain Prediction task.

Figure 13: Synthetic data generation prompt. The prompt used to instruct the LLM to filter raw Audio Descriptions and extract structured causal chains for the training set generation pipeline.

Causal Reasoning in Videos

Causal reasoning in videos studies how events and interventions produce visual changes, helping models explain, predict, and edit dynamic scenes.

See all 51 papers

ChainReaction: Causal Chain-Guided Reasoning for Modular and Explainable Causal-Why Video Question Answering
Aug 28, 2025Paritosh Parmar, Eric Peh, Basura FernandoVideo QA Multimodal CoT Reasoning
Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore
CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering
May 22, 2026Mingfang Zhang, Jingjing Pan, Ashutosh Kumar +7VLM Evaluation Temporal Video Grounding
Woven by Toyota · The University of Tokyo
Event-Causal RAG: A Retrieval-Augmented Generation Framework for Long Video Reasoning in Complex Scenarios
May 7, 2026Peizheng Yan, Yu Zhao, Liang Xie +3Retrieval-Augmented Generation Streaming Video Understanding
Tianjin Key Lab of Intelligent Unmanned Swarm Tech & System Tianjin University Tianjin, China · Institute of Computing and Intelligence Harbin Institute of Technology (Shenzhen) Shenzhen, China · Tianjin Artificial Intelligence Innovation Center Tianjin, China +2

Aug 28, 2025J/K move · Enter open · S save

ChainReaction: Causal Chain-Guided Reasoning for Modular and Explainable Causal-Why Video Question Answering

Paritosh Parmar, Eric Peh, Basura Fernando

Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore

Video QA Multimodal CoT Reasoning Causal Reasoning Causal Explanation Causal Reasoning in Videos

Open paper AskPDF arXiv

Existing Causal-Why Video Question Answering (VideoQA) models often struggle with higher-order reasoning, relying on opaque, monolithic pipelines that entangle video understanding, causal inference, and answer generation. These black-box approaches offer limited interpretability and tend to depend on shallow heuristics. We propose a novel, modular paradigm that explicitly decouples causal reasoning from answer generation, introducing natural language causal chains as interpretable intermediate representations. Inspired by human cognitive models, these structured cause-effect sequences bridge low-level video content with high-level causal reasoning, enabling transparent and logically coherent inference. Our two-stage architecture comprises a Causal Chain Extractor (CCE) that generates causal chains from video-question pairs, and a Causal Chain-Driven Answerer (CCDA) that derives answers grounded in these chains. To address the lack of annotated reasoning traces, we introduce a scalable method for generating accurate causal chains from existing datasets. We construct human verified causal chains for 46K samples. We also propose CauCo, a new evaluation metric for causality-oriented captioning. Experiments on three large-scale benchmarks demonstrate that our approach not only outperforms state-of-the-art models, but also yields substantial gains in explainability, user trust, and generalization -- positioning the CCE as a reusable causal reasoning engine across diverse domains. Project page: https://paritoshparmar.github.io/chainreaction/

Abstract

Causal Reasoning in Videos

Explore similar work

ChainReaction: Causal Chain-Guided Reasoning for Modular and Explainable Causal-Why Video Question Answering

CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering

Event-Causal RAG: A Retrieval-Augmented Generation Framework for Long Video Reasoning in Complex Scenarios