VideoPhysEdit: Physical Counterfactual Video Editing via Rigid-Body Physical Scene Reconstruction

Conghan Yue; Yuanjie Chen; Yue Han; Ya Gao; Yunyan Xiao; WeiYao Zhang; Zhineng Chen

cs.CVSep 28, 2026

VideoPhysEdit: Physical Counterfactual Video Editing via Rigid-Body Physical Scene Reconstruction

Authors: Conghan Yue, Yuanjie Chen, Yue Han, Ya Gao, Yunyan Xiao, WeiYao Zhang, Zhineng Chen

Organizations: Institute of Trustworthy Embodied AI, Fudan University

Abstract

Video editing has advanced substantially in recent years, with methods increasingly accounting for the visual consequences of edits, such as changes to shadows and occlusions. However, the physical consequences of edits, including changes to subsequent motion and interactions, remain less explored. We formulate this problem as physical counterfactual video editing (PCVE), which aims to generate a counterfactual video depicting the resulting motion and interactions given a source video, a physical edit, and its execution frame. PCVE is challenging because it requires understanding scene physics and inferring the downstream motion and interactions induced by a physical intervention, while paired factual and counterfactual data and dedicated evaluation metrics are lacking. We introduce VideoPhysEdit, a new training-free pipeline for PCVE in rigid-body scenes. It makes physical reasoning explicit through a novel physical scene reconstruction method that recovers a scene reproducing the observed motion and interactions under simulation, enabling the pipeline to apply physical edits as interventions and use the resulting trajectories to guide counterfactual video generation. We further construct PCVE-RigidBench, a synthetic benchmark with paired source and counterfactual target videos and physical ground truth, and introduce the Physical Edit Score. VideoPhysEdit achieves substantially higher physical edit accuracy than open-source methods and commercial models while maintaining competitive visual fidelity. Its Physical Edit Score is 0.376, the only positive score among the compared methods. Qualitative comparisons on real videos further show that VideoPhysEdit applies to real-world scenes and better depicts the downstream motion and interactions induced by the edits than the compared methods. Code: https://github.com/Hammour-steak/VideoPhysEdit

Figures & tables

Figure 1 : Overview of VideoPhysEdit. We reconstruct an executable physical scene from the source video, apply the physical intervention, and use the simulated counterfactual trajectories and an edited reference image to guide counterfactual video generation.

Method	Physical Edit Accuracy			Visual Fidelity
Method	PES $\uparrow$	TE $\downarrow$	Mask IoU $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	FVD $\downarrow$
VACE	$-0.042$	146.26	0.273	14.06	0.728	0.447	0.795	1184.37
Ditto	$-0.120$	149.94	0.264	22.07	0.812	0.206	0.850	551.14
MiniMax H3	$-0.096$	152.00	0.250	24.87	0.870	0.127	0.906	246.45
Seedance 2.5	$-0.087$	144.99	0.231	28.85	0.928	0.080	0.932	246.67
No edit	0.000	143.13	0.289	31.23	0.974	0.036	0.957	249.68

Table 1 : Comparison on PCVE-RigidBench. Bold marks the best result among methods.

Method	Physical Edit Accuracy			Visual Fidelity
Method	PES $\uparrow$	TE $\downarrow$	Mask IoU $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	FVD $\downarrow$
VACE	$-0.001$	140.72	0.307	12.34	0.651	0.503	0.762	1782.50
Ditto	$-0.041$	153.98	0.255	21.13	0.805	0.246	0.790	892.24
MiniMax H3	0.383	96.24	0.361	25.23	0.889	0.104	0.914	256.52
Seedance 2.5	0.290	106.76	0.245	27.86	0.892	0.085	0.927	272.93
VOID	0.394	84.07	0.314	29.22	0.914	0.168	0.862	262.43

Table 2 : Results on the object removal tasks. Bold marks the best result among methods.

Output	Metric	Result
Stage 3 $\rightarrow$ 4	Mask IoU $\uparrow$	$0.883\rightarrow 0.887$
Stage 5 init. $\rightarrow$ opt.	Stage 5 Mask IoU $\uparrow$	$0.375\rightarrow 0.678$
Stage 5 init. $\rightarrow$ opt.	Stage 6 PES $\uparrow$	$0.269\rightarrow 0.403$
Stage 6 $\rightarrow$ 7	PES $\uparrow$	$0.398\rightarrow 0.412$
	TE $\downarrow$	$63.39\rightarrow 64.76$
	Mask IoU $\uparrow$	$0.391\rightarrow 0.412$

Table 3 : Accuracy across pipeline stages. Arrows indicate before and after values.

Figure 2 : Qualitative comparison on two synthetic (top) and two real (bottom) videos.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Edit	Tasks
Mass	32
Friction	21
Restitution	13
Initial velocity	26
Add	7
Delete	30

Appendix

Table 4 : Task distribution in PCVE-RigidBench.

Figure 3 : Examples from PCVE-RigidBench. The rows show the source video and three physical interventions at four frames.

Method	Resolution	Output frames	Evaluated frames	Tasks	Settings
VACE	$768\times 432$	96	96	129	30 steps, CFG 5.0
Ditto	$832\times 480$	73	73	129	VACE-14B with Ditto LoRA
MiniMax H3	768p	107	96	129	video editing API
Seedance 2.5	720p	89	89	129	video editing API
VOID	$672\times 384$	96	96	30	removal only
VideoPhysEdit	$720\times 480$	96	96	129	16 steps, CFG 1.0

Appendix

Table 5 : Output and evaluation settings.

Evaluation set	Tasks	Physical Edit Accuracy			Visual Fidelity
Evaluation set	Tasks	PES $\uparrow$	TE $\downarrow$	Mask IoU $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CLIP $\uparrow$	FVD $\downarrow$
Generated videos	116	0.418	63.68	0.421	27.47	0.921	0.107	0.928	198.79
No edit	129	0.000	143.13	0.289	31.23	0.974	0.036	0.957	249.68
Complete benchmark	129	0.376	66.70	0.421	27.51	0.925	0.104	0.929	182.46

Appendix

Table 6 : VideoPhysEdit results on generated videos and the complete benchmark.

Execution timing	Tasks	PES $\uparrow$	TE $\downarrow$	Mask IoU $\uparrow$
First frame	117	0.356	68.15	0.403
Partway	12	0.571	52.57	0.602

Appendix

Table 7 : VideoPhysEdit results by execution timing.

Method	Add $\uparrow$	Delete $\uparrow$	Set $\uparrow$
VACE	$-0.007$	$-0.001$	$-0.058$
Ditto	$-0.157$	$-0.041$	$-0.144$
MiniMax H3	$-0.521$	0.383	$-0.219$
Seedance 2.5	$-0.145$	0.290	$-0.205$
No edit	0.000	0.000	0.000
VideoPhysEdit	0.032	0.633	0.318

Appendix

Table 8 : PES by operation.

Method	Directly edited		Other affected		All measurable
Method	PES $\uparrow$	TE $\downarrow$	PES $\uparrow$	TE $\downarrow$	PES $\uparrow$	TE $\downarrow$
VACE	$-0.036$	179.32	$-0.045$	122.79	$-0.042$	146.26
Ditto	$-0.052$	165.86	$-0.198$	139.13	$-0.120$	149.94
MiniMax H3	0.009	161.44	$-0.156$	140.13	$-0.096$	152.00
Seedance 2.5	$-0.052$	166.37	$-0.132$	126.88	$-0.087$	144.99
VideoPhysEdit	0.451	70.82	0.276	72.15	0.376	66.70

Appendix

Table 9 : Physical edit accuracy by object group.

Method	Friction $\uparrow$	Initial velocity $\uparrow$	Mass $\uparrow$	Presence $\uparrow$	Restitution $\uparrow$
VACE	$-0.083$	$-0.012$	$-0.071$	$-0.002$	$-0.080$
Ditto	$-0.148$	$-0.017$	$-0.245$	$-0.063$	$-0.140$
MiniMax H3	$-0.258$	$-0.127$	$-0.249$	0.212	$-0.267$
Seedance 2.5	$-0.166$	$-0.086$	$-0.324$	0.208	$-0.215$
No edit	0.000	0.000	0.000	0.000	0.000
VideoPhysEdit	0.315	0.435	0.307	0.520	0.116

Appendix

Table 10 : PES by edited property.

Stage	Mean	Median	25th percentile	75th percentile
Stage 3 canonical and anchor scenes	0.883	0.899	0.832	0.945
Stage 4 motion prior	0.887	0.908	0.861	0.944
Stage 5 physical rollout	0.678	0.733	0.602	0.799

Appendix

Table 11 : Mask IoU across reconstructed source scenes.

Figure 4 : Mask IoU by scene for Stages 3 to 5. Lines connect results from the same reconstructed scene.

Stage	PES $\uparrow$	TE $\downarrow$	Mask IoU $\uparrow$
Stage 6 projected trajectories	0.398	63.39	0.391
Stage 7 generated video	0.412	64.76	0.412

Appendix

Table 12 : Counterfactual motion before and after video generation.

Figure 5 : Adaptive temporal scaling and interaction ROI in Stage 7.

Figure 6 : Object insertion and projected point trajectories in Stage 6.

Stage	Observation	Method
2	Unreliable or missing masks	Weight motion estimates by observation confidence and identify stable intervals, transition episodes, and unresolved observations
3	Fragmented support planes	Merge compatible planes, refit their combined 3D points, and refine finite boundaries using image outlines
3	Sparse 3D correspondences	Use 2D correspondences and dense object points to supplement sparse 3D correspondences when fitting pose and scale
3	Depth scale across frames	Align each motion anchor frame to the canonical scene using the static background and keep object scale fixed
3	Multiple support assignments	Check contact, nonpenetration, and support plane boundaries; reconcile support relations across motion anchor frames
4	Missing motion observations	Fit motion models to stable intervals and connect them with transition curves constrained by boundary states

Appendix

Table 13 : Handling incomplete and ambiguous observations across the pipeline.

Figure 7 : Canonical frame selection in drop_centered . In the 3D views, orange shows the reconstruction and blue wireframes show the aligned ground truth. The red dashed line marks the center error.

Figure 8 : Support plane reconstruction in tennis_flight . Compatible fragments are refitted as one plane while retaining the ball’s support relation.

Figure 9 : Motion prior reconstruction and physical inversion. Stage 4 reconstructs motion through missing observations, and Stage 5 improves the physical rollout.

Figure 10 : Scene reconstruction errors in bowling and domino_chain .

Figure 11 : Physical inversion errors after contact.

Figure 12 : Physical inversion with closely spaced contacts in dining_chain .

Figure 13 : A video generation error after the Stage 6 trajectory leaves the image.

Variant	Stage 5 Mask IoU $\uparrow$	Stage 6 PES $\uparrow$	Stage 6 TE $\downarrow$	Stage 6 Mask IoU $\uparrow$
Calibrated initialization	0.375	0.269	76.70	0.324
Search and refinement	0.678	0.403	62.37	0.392

Appendix

Table 14 : Ablation of Stage 5 physical inversion.

Stage	Average time (s)	Peak GPU memory
1 Object identification and tracking	32.7	4.1 GiB
2 Motion observation analysis	27.6	11.9 GiB
3 Canonical and anchor scene reconstruction	117.5	12.0 GiB
4 Motion prior reconstruction	70.1	CPU only
5 Physical inversion	246.7	CPU only
6 Physical intervention	22.6	depends on operation

Appendix

Table 15 : Runtime and peak GPU memory by stage.

Method	Heavy block	Grippy car	Heavy red ball	Strong push
Seedance 2.5	$-0.022\!\rightarrow\!-0.060$	$0.205\!\rightarrow\!-0.015$	$-1.000\!\rightarrow\!-1.000^{\dagger}$	$0.001\!\rightarrow\!-0.215$
MiniMax H3	$0.010\!\rightarrow\!-0.006$	$0.001\!\rightarrow\!-0.271^{\dagger}$	$-1.000^{\dagger}\!\rightarrow\!-1.000$	$0.000\!\rightarrow\!0.026$
VACE	$0.003\!\rightarrow\!0.001$	$-0.001\!\rightarrow\!0.005$	$0.016\!\rightarrow\!0.008$	$-0.001\!\rightarrow\!-0.001$
Ditto	$-0.034\!\rightarrow\!-0.032$	$-0.044\!\rightarrow\!-0.046$	$-0.390\!\rightarrow\!-0.261$	$0.005\!\rightarrow\!0.004$

Appendix

Table 16 : PES with the original physical edit and with explicit downstream consequences appended. Each cell reports original

\rightarrow

explicit consequences.

\dagger

denotes an invalid track of the evaluated object.

Figure 14 : Effect of explicit downstream consequences on Seedance 2.5 and MiniMax H3. The bold text after each arrow is added to the original edit instruction. Each baseline is evaluated with and without the added text; VideoPhysEdit uses the original instruction. All methods are shown at the same three time points for each task.

Figure 15 : Effect of explicit downstream consequences on VACE and Ditto. Tasks, time points, and layout match Figure 14 .

Figure 16 : Additional comparisons on PCVE-RigidBench.

Figure 17 : Comparison with VOID on four removal edits applied from the first frame. The top two examples are from PCVE-RigidBench; the bottom two are real videos without paired counterfactual targets. Each row shows the same time point across methods. Crops are fixed within each video and aligned by ruler markings in the last example.

Figure 18 : Additional comparisons on real videos.

Figure 19 : VideoPhysEdit results for restitution, removal, and insertion edits.

Figure 20 : VideoPhysEdit results for initial velocity, mass, gravity, and removal edits.

Explore similar work

May 22, 2026cs.CV

CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models

Video prediction is increasingly viewed as a path toward generalizable world models, yet it remains unclear whether these systems learn underlying causal structure or merely exploit superficial visual correlations for future prediction. We introduce CRONOS, an intervention-based benchmark designed to evaluate counterfactual physical consistency: whether a model's predictions of physical events respond appropriately to controlled changes in the visual input, such as variations of scene context, viewpoint, object appearance, and object category. Built in a photorealistic Unreal Engine environment, CRONOS enables controlled, high-fidelity generation of videos across diverse scenes and dynamics. In contrast to previous benchmarks, CRONOS systematically intervenes on four key factors - viewpoint, scene, object category, and object appearance - while keeping the underlying physical event type, such as a collision, occlusion, or fall, fixed. Our evaluation of recent open-source video generators reveals substantial failures in counterfactual physical consistency: prediction quality for the same physical event type is affected by appearance, environment, and, particularly by viewpoint changes. CRONOS provides a controlled and reproducible testbed for diagnosing how the quality of generated videos changes for different interventions, establishing a concrete target for developing models that perform consistently across changes of multiple conditions. The dataset and code are available at our project page.

León Begiristain, Olaf Dünkel, Adam Kortylewski

University of Freiburg · Freiburg im Breisgau, Germany · Max Planck Institute for Informatics +3

World Models Chronos

Ask PDF

Sep 29, 2026cs.CV

Reimagine Video Dynamics

Most video editing methods focus on changing the appearance of the source video, while offering limited control over its dynamics. We introduce Reimagine Video Dynamics (RVD), a framework that disentangles a compact, editable dynamics token from visual context. We learn this token through self-supervised reconstruction: given the first frame as visual context, a renderer must recover the original video from the dynamics token, encouraging it to capture how the scene evolves rather than how it looks. This disentanglement allows video dynamics to be edited directly while preserving visual context. We develop a language-guided dynamics-token editor that transforms source dynamics into target dynamics, and train it with a scalable counterfactual video-pair pipeline and a two-stage training strategy. Extensive experiments show that RVD enables effective video dynamics editing, training-free retiming, and appearance-controlled re-rendering.

Yu Yuan, Yawen Lu, Guoxian Song +5

Adobe · Purdue University

Video Editing

Ask PDF

Jun 25, 2026cs.CV

PhyEditBench: A Real-World Multi-Stage Benchmark for Physics-Aware Image Editing

While instruction-based image editing, enabled by multi-modal generative models, has advanced significantly, existing benchmarks lack a comprehensive evaluation of physics-based reasoning, a critical capability for handling real-world scenarios. To address this, we introduce PhyEditBench, a benchmark designed to assess the physical understanding of editing models. Guided by a hierarchical taxonomy, we establish 4 primary classes and 12 subclasses. It comprises 238 high-quality, high-resolution, real-world instances meticulously extracted from videos to capture authentic physical dynamics, alongside 35 synthetic Anti-Physics instances. Our empirical analysis of current SOTA editing methods exposes substantial limitations in their physics-based reasoning. We further propose a training-free baseline named PhyWorld that uses test-time scaling and a latent reduction strategy. PhyWorld outperforms comparable models and suggests that the video generation process can effectively serve as a reasoning mechanism for image editing. The project page is available at https://github.com/Previsior/PhyEditBench.

Shengbin Guo, Shaokang He, Chaoyue Meng +4

Nanjing University · SDU · HRBEU +2

Image Editing Generative Video Models Generative Models Real-World

Ask PDF