MOSIV: Multi-Object System Identification from Videos

Chunjiang Liu; Xiaoyuan Wang; Qingran Lin; Albert Xiao; Haoyu Chen; Shizheng Wen; Hao Zhang; Lu Qi

cs.CVMar 6, 2026

MOSIV: Multi-Object System Identification from Videos

Authors: Chunjiang Liu, Xiaoyuan Wang, Qingran Lin, Albert Xiao, Haoyu Chen, Shizheng Wen, Hao Zhang, Lu Qi,

Organizations: CMU · Georgia Tech · Harvard · ETH Zurich ·

Abstract

We introduce the challenging problem of multi-object system identification from videos, for which prior methods are ill-suited due to their focus on single-object scenes or discrete material classification with a fixed set of material prototypes. To address this, we propose MOSIV, a new framework that directly optimizes for continuous, per-object material parameters using a differentiable simulator guided by geometric objectives derived from video. We also present a new synthetic benchmark with contact-rich, multi-object interactions to facilitate evaluation. On this benchmark, MOSIV substantially improves grounding accuracy and long-horizon simulation fidelity over adapted baselines, establishing it as a strong baseline for this new task. Our analysis shows that object-level fine-grained supervision and geometry-aligned objectives are critical for stable optimization in these complex, multi-object settings. The source code and dataset will be released.

Figures & tables

Figure 1: From multi-view observations of multi-object scenes (left), prior approaches select from a fixed library of expert constitutive models via categorical prediction, leading to visually implausible and weakly calibrated physics dynamics. MOSIV instead performs geometric reconstruction, per-object system identification of continuous constitutive parameters, enabling both faithful reproduction of observed interactions and accurate prediction of future behaviors (right).

Figure 2: (1) Geometric reconstruction. From multi-view RGB videos, we reconstruct object geometry and disentangle material-specific motion via optimizing 4D Gaussian Splatting (4DGS) with object masks. (2) Continuum simulation. The reconstructed Gaussians are lifted into object-specific continuums, which serve as the initial states for a differentiable MPM. Geometry-aligned losses on surfaces and silhouettes drive physics parameter optimization under inter-material contact and friction. (3) Applications. The calibrated model generalizes to novel interaction scenarios, enabling physically faithful rollouts and long-horizon predictions of complex multi-object dynamics.

Figure 3: Novel Interaction. Left—MOSIV original GT video sequence. Right—rollout after swapping object physics parameters while keeping initial conditions unchanged. Rows show time.

Figure 4: Qualitative comparison of multi-object interactions. The first four columns shows a plasticine–fluid (P–F) example; the last four columns shows a sand–sand (S–S) example.

Figure 5: Qualitative comparison between MOSIV and baselines.

Figure 6: Qualitative comparison of trajectory. The trajectories illustrate how well each method captures long-term dynamics.

Supervision Granularity	$\mathcal{L}_{\text{CD}}$	$\mathcal{L}_{\alpha}$	PSNR $\uparrow$	SSIM $\uparrow$	CD $\downarrow$	EMD $\downarrow$
Scene-wise losses (naive)	✗	✓	26.59	0.964	53.21	0.132
	✓	✗	27.59	0.959	40.29	0.119
	✓	✓	27.89	0.968	22.13	0.091
Object-aware losses (ours)	✗	✓	30.18	0.975	0.985	0.045
	✓	✗	29.86	0.975	1.17	0.043
	✓	✓	30.24	0.977	0.696	0.041

Table 3: Supervision granularity ablation. Comparison of scene-wise vs. object-wise supervision while toggling the Chamfer term

\mathcal{L}_{\mathrm{CD}}

and silhouette term

\mathcal{L}_{\alpha}

	$\text{Young's Modulus }(E)^{\dagger}$	$\text{Poisson's Ratio }(\nu)^{\dagger}$	$\text{Density }(\rho)^{\dagger}$
Elastic	$[4.75\times 10^{4},5.25\times 10^{4}]$	$[20,30]$	$[800,1200]$
Elastoplastic	$[4.75\times 10^{4},5.25\times 10^{4}]$	$[20,30]$	$[800,1200]$
Snow	$[4.75\times 10^{4},5.25\times 10^{4}]$	$[20,30]$	$[800,1200]$
Liquid	$[4.75\times 10^{4},5.25\times 10^{4}]$	$[20,30]$	$[800,1200]$
	$\text{Shear Modulus }(\mu)^{\dagger}$	$\text{Yield Stress Range }(\tau_{Y})$	$\text{Friction Angle }(\theta)$
Elastic	–	–	–

Table 4: Physical parameter values for objects in the Genesis Multi-Object Dataset (10 Geometry Shapes)

Figure 7: MOSIV Dataset (2 Objects) Example.

Figure 8: MOSIV Dataset (3 Objects) Example.

Metric \ Method	CoupNeRF ∗	OPGS	MOSIV
Training Time (s) $\downarrow$	9591.09	5263.70	5021.46
Peak GPU Memory (GB) $\downarrow$	31.14	61.08	29.79

Table 6: Runtime and memory cost

	Observable Simulation				Future Simulation
Method	PSNR $\uparrow$	SSIM $\uparrow$	CD $\downarrow$	EMD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	CD $\downarrow$	EMD $\downarrow$
MOSIV	23.98	0.941	2.401	0.043	19.81	0.904	5.800	0.095

Table 7: Quantitative Evaluation of MOSIV Dataset Extension

Noise level( $\alpha$ )	PSNR $\uparrow$	SSIM $\uparrow$	CD $\downarrow$	EMD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	CD $\downarrow$	EMD $\downarrow$
	Observable Simulation				Future Simulation
0.005	27.86	0.96	0.22	0.03	27.47	0.96	0.29	0.03
0.020	26.06	0.95	0.41	0.03	25.43	0.95	0.52	0.05
0.050	23.73	0.94	1.27	0.05	23.39	0.93	1.72	0.07

Table 8: Sensivity analysis of i.i.d. noise

Noise level ( $\alpha$ )	PSNR $\uparrow$	SSIM $\uparrow$	CD $\downarrow$	EMD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	CD $\downarrow$	EMD $\downarrow$
	Observable Simulation				Future Simulation
0.005	27.82	0.96	0.24	0.03	27.43	0.96	0.35	0.03
0.020	26.17	0.95	0.38	0.03	25.57	0.95	0.49	0.04
0.050	23.92	0.93	1.14	0.04	23.52	0.93	1.50	0.06

Table 9: Sensitivity analysis of shared noise

Figure 9: Qualitative Comparison with material type elastic/fluid (E/F).

Figure 10: Qualitative Comparison with material type elastic/plasticine (E/P).

Figure 11: Qualitative Comparison with material type elastic/sand (E/S).

Figure 12: Qualitative Comparison with material type fluid/sand (F/S).

Figure 13: Qualitative Comparison with material type plasticine/fluid (P/F).

Figure 14: Qualitative Comparison with material type sand/plasticine (S/P).

Figure 15: Qualitative Comparison with material type fluid/fluid (F/F).

Figure 16: Qualitative Comparison with material type plasticine/plasticine (P/P).

Figure 17: Qualitative Comparison with material type elastic/elastic (E/E).

Figure 18: Qualitative Comparison with material type sand/sand (S/S).

System Identification

System identification infers mathematical models and parameters of dynamical systems from input-output, trajectory, sensor, or video data.

See all 158 papers

IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video
Mar 17, 2026Rasul Khanbayov, Mohamed Rayan Barhdadi, Erchin Serpedin +1Parameter Identifiability System Identification
Hamad Bin Khalifa University · Texas A&M University
OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Models
Sep 28, 2026Hao Wang, Tao Yu, Liuzhou Zhang +13World Model Evaluation Video World Models
USTC · Infrec · CASIA +10
MatPhys: Learning Material-Aware Physics Parameters for Deformable Object Simulation from Videos
May 19, 2026Yang Yang, Yiyan Wang, Zheming Liu +1Dynamic Scene Reconstruction Materials Property Prediction
1The University of Osaka · 3Huawei Technologies Japan K.K. · 2The University of Tokyo

Mar 17, 2026J/K move · Enter open · S save

IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video

Rasul Khanbayov, Mohamed Rayan Barhdadi, Erchin Serpedin, Hasan Kurban

Hamad Bin Khalifa University · Texas A&M University

Parameter Identifiability System Identification Inverse Dynamics Inverse Problems

Open paper AskPDF arXiv

Unsupervised physical parameter estimation from video lacks a common benchmark: existing methods evaluate on non-overlapping synthetic data, the sole real-world dataset is restricted to single-body systems, and no established protocol addresses governing-equation identification. This work introduces IRIS, a high-fidelity benchmark comprising 240 real-world videos captured at 4K resolution and 60fps, spanning both single- and multi-body dynamics with independently measured ground-truth parameters and uncertainty estimates. Each dynamical system is recorded under controlled laboratory conditions and paired with its governing equations, enabling principled evaluation. A standardized evaluation protocol is defined encompassing parameter accuracy, identifiability, extrapolation, robustness, and governing-equation selection. Multiple baselines are evaluated, including a multi-step physics loss formulation and four complementary equation-identification strategies (VLM temporal reasoning, describe-then-classify prompting, CNN-based classification, and path-based labelling), establishing reference performance across all IRIS scenarios and exposing systematic failure modes that motivate future research. The dataset, annotations, evaluation toolkit, and all baseline implementations are publicly released.

Abstract

System Identification

Explore similar work

IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video

OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Models

MatPhys: Learning Material-Aware Physics Parameters for Deformable Object Simulation from Videos