CounterSketch: Physics-Grounded Counterfactual Reasoning for Sketch-to-Video Generation

Hien Trinh; Tan-Hiep To; Dinh-Khoi Vo; Tam V. Nguyen; Minh-Triet Tran; Trung-Nghia Le

cs.CVNov 3, 2026

CounterSketch: Physics-Grounded Counterfactual Reasoning for Sketch-to-Video Generation

Authors: Hien Trinh, Tan-Hiep To, Dinh-Khoi Vo, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le

Organizations: University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · University of Dayton, Ohio, USA

Abstract

Counterfactual sketch animation explores how a static sketch evolves under a hypothetical intervention. We present CounterSketch, a physics-grounded framework that transforms natural-language ``What if...?'' queries into physically plausible sketch animations. The framework integrates VLM-guided scene grounding, causal event-chain reasoning, constraint-based trajectory generation, and verification. It models physical transitions to update the scene graph, computes continuous object trajectories, and validates them against physical, geometric, and semantic constraints. The verified trajectories are then projected into object-level motion bounding boxes to guide downstream sketch-video synthesis. Experiments on CounterSketchBench (80 questions) demonstrate that physics-grounded trajectories provide stronger motion guidance than direct VLM predictions.

Figures & tables

Figure 1: CounterSketch interprets a counterfactual intervention, reasons about its physical consequences, and generates a sketch video whose motion follows the resulting trajectory.

Figure 2: Representative six types of counterfactual scenes in CounterSketchBench.

Method	Counterfactual Reasoning	Planning Basis	Motion Optimization
Sketch animation
FlipSketch [ 1 ]	$\times$	T2V prior	$\times$
Live-Sketch [ 6 ]	$\times$	T2V prior	$\checkmark$
GroupSketch [ 12 ]	$\times$	User keyframes	$\checkmark$
MoSketch [ 14 ]	$\times$	LLM	$\checkmark$
Physics-aware generation

Table 1: Comparison of representative methods for sketch animation, physics-aware generation, and constraint-based planning.

\checkmark

indicates support and

\times

indicates no support. Optimization denotes per-sketch iterative optimization.

Figure 3: CounterSketch overview. The physical reasoning layer is separated from the downstream sketch-video generator.

Figure 4: Physics-driven counterfactual event-chain reasoning. The intervention is grounded in the sketch scene graph and propagated through physically meaningful event states.

Figure 5: Constraint compilation, unified dynamics, hybrid rollout, and trajectory verification.

Category	Intervention	Example question	#
Object removal	Remove one or multiple objects or supporting structures	What if the table beneath the ball is removed?	19
Position	Translate, swap, reorder, or rearrange objects	What if the right ball is placed to the left of the left ball?	21
Orientation	Rotate an object, surface, or the entire scene	What if the ground is tilted by $15^{\circ}$ ?	10
Size / geometry	Scale an object or modify its geometric extent	What if the pendulum string is 50% longer?	15
Relation / constraint	Modify support, contact, attachment, or relative relations	What if the ball detaches from the string?	13
Dynamics	Modify an initial dynamic condition	What if the ball starts with a very large initial velocity?	2

Table 2: Counterfactual intervention categories in CounterSketchBench.

Scene Modification	Trajectory Generation	T2V Alignment $\uparrow$	S2V Alignment $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$
Direct VLM	Direct VLM	0.1682	0.9415	0.9830	0.5625
CounterSketch	Direct VLM	0.1729	0.9438	0.9825	0.7250
CounterSketch	CounterSketch	0.2011	0.9580	0.9834	0.7750

Table 3: Ablation of motion guidance for downstream synthesis using MoSketch. All variants share the identical renderer.

Figure 6: Qualitative results across five representative counterfactual intervention types.

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 377 papers

VideoPhysEdit: Physical Counterfactual Video Editing via Rigid-Body Physical Scene Reconstruction
Sep 28, 2026Conghan Yue, Yuanjie Chen, Yue Han +4Physical Consistency in Video Generation Video Editing
Institute of Trustworthy Embodied AI, Fudan University
PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
Aug 5, 2026Chen Yang, Shenxiang Zeng, Haoyang Zhao +6Counterfactual Reasoning Physical Reasoning in Video
1Tsinghua University · 2The University of Hong Kong
Sketch2Colab: Sketch-Conditioned Multi-Human Animation via Controllable Flow Distillation
Mar 2, 2026Divyanshu Daiya, Aniket BeraMarkov Chains Style-Conditioned Motion Generation
IDEAS Lab, Department of Computer Science, Purdue University

Sep 28, 2026J/K move · Enter open · S save

VideoPhysEdit: Physical Counterfactual Video Editing via Rigid-Body Physical Scene Reconstruction

Conghan Yue, Yuanjie Chen, Yue Han, Ya Gao+3

Institute of Trustworthy Embodied AI, Fudan University

Physical Consistency in Video Generation Video Editing Physical Reasoning in Video

Open paper AskPDF arXiv

Video editing has advanced substantially in recent years, with methods increasingly accounting for the visual consequences of edits, such as changes to shadows and occlusions. However, the physical consequences of edits, including changes to subsequent motion and interactions, remain less explored. We formulate this problem as physical counterfactual video editing (PCVE), which aims to generate a counterfactual video depicting the resulting motion and interactions given a source video, a physical edit, and its execution frame. PCVE is challenging because it requires understanding scene physics and inferring the downstream motion and interactions induced by a physical intervention, while paired factual and counterfactual data and dedicated evaluation metrics are lacking. We introduce VideoPhysEdit, a new training-free pipeline for PCVE in rigid-body scenes. It makes physical reasoning explicit through a novel physical scene reconstruction method that recovers a scene reproducing the observed motion and interactions under simulation, enabling the pipeline to apply physical edits as interventions and use the resulting trajectories to guide counterfactual video generation. We further construct PCVE-RigidBench, a synthetic benchmark with paired source and counterfactual target videos and physical ground truth, and introduce the Physical Edit Score. VideoPhysEdit achieves substantially higher physical edit accuracy than open-source methods and commercial models while maintaining competitive visual fidelity. Its Physical Edit Score is 0.376, the only positive score among the compared methods. Qualitative comparisons on real videos further show that VideoPhysEdit applies to real-world scenes and better depicts the downstream motion and interactions induced by the edits than the compared methods. Code: https://github.com/Hammour-steak/VideoPhysEdit

Abstract

Video Generation

Explore similar work

VideoPhysEdit: Physical Counterfactual Video Editing via Rigid-Body Physical Scene Reconstruction

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning

Sketch2Colab: Sketch-Conditioned Multi-Human Animation via Controllable Flow Distillation