Vid2Sid: Videos Can Help Close the Sim2Real Gap

Kevin Qiu; Yu Zhang; Marek Cygan; Josie Hughes

cs.ROFeb 22, 2026

Vid2Sid: Videos Can Help Close the Sim2Real Gap

Authors: Kevin Qiu, Yu Zhang, Marek Cygan, Josie Hughes

Organizations: University of Warsaw · IDEAS NCBR · EPFL · Nomagic

Abstract

Calibrating a robot simulator's physics parameters (friction, damping, material stiffness) to match real hardware is often done by hand or with black-box optimizers that reduce error but cannot explain which physical discrepancies drive the error. When sensing is limited to external cameras, the problem is further compounded by perception noise and the absence of direct force or state measurements. We present Vid2Sid, a video-driven system identification pipeline that couples foundation-model perception with a VLM-in-the-loop optimizer that analyzes paired sim-real videos, diagnoses concrete mismatches, and proposes physics parameter updates with natural language rationales. We evaluate our approach on a tendon-actuated finger (rigid-body dynamics in MuJoCo) and a deformable continuum tentacle (soft-body dynamics in PyElastica). On sim2real holdout controls unseen during training, Vid2Sid achieves the best average rank across all settings, matching or exceeding black-box optimizers while uniquely providing interpretable reasoning at each iteration. Sim2sim validation confirms that Vid2Sid recovers ground-truth parameters most accurately (mean relative error under 13% vs. 28--98%), and ablation analysis reveals three calibration regimes. VLM-guided optimization excels when perception is clean and the simulator is expressive, while model-class limitations bound performance in more challenging settings.

Figures & tables

Fig. 2: Experimental platforms. (a) CAD model of tendon-driven finger. (b) Physical finger with tracked marker. (c) CAD model of soft tentacle. (d) Underwater tentacle setup. These platforms span rigid-body and continuum dynamics, enabling evaluation across qualitatively different calibration regimes.

Fig. 3: Centerline extraction pipeline. (a) Simulated centerline (10 equidistant points, base to tip). (b) Real video frame with SAM3 segmentation mask and extracted centerline overlay. The shared 10-point representation enables direct point-wise error computation between sim and real without manual annotation.

Method	Finger (px) $\downarrow$	Tentacle–Air (px) $\downarrow$	Tentacle–Water (px) $\downarrow$ †	Avg. Rank $\downarrow$
Random	27.8 $\pm$ 17.2 (12.1)	54.3 $\pm$ 1.9 (51.6)	76.1 $\pm$ 3.8 (71.2)	4.3
Nelder-Mead	17.2 $\pm$ 3.5 (14.2)	57.3 $\pm$ 4.2 (54.3)	80.7 $\pm$ 2.9 (77.3)	5.3
Golden-CD	12.1 $\pm$ 0.2 (11.9)	53.4 $\pm$ 4.8 (49.5)	77.8 $\pm$ 3.7 (73.2)	3.3
BO	16.1 $\pm$ 2.4 (14.4)	62.4 $\pm$ 10.4 (54.5)	71.7 $\pm$ 0.1 (71.5)	3.7
CMA-ES	15.3 $\pm$ 4.9 (11.8)	52.1 $\pm$ 2.9 (49.9)	77.6 $\pm$ 3.4 (72.9)	2.7
Vid2Sid (Ours)	10.9 $\pm$ 6.3 ( 4.9 )	53.0 $\pm$ 5.3 ( 48.8 )	73.3 $\pm$ 3.1 ( 71.1 )	1.7

TABLE I: sim2real holdout generalization across experimental settings. We report mean

\pm

std over

N=3

training seeds, with the best single-seed result in parentheses. Each seed value is the mean over 4 holdouts

\times

3 hardware repeats (12 datapoints). Best mean in bold , second-best underlined . The water column uses a shared-body protocol (environment parameters only) and is included as a stress test. All methods perform similarly due to model-class limitations.

Fig. 4: Finger qualitative alignment after Vid2Sid calibration. (a) Simulated finger at $t{=}6$ s. (b) Corresponding real video frame with tracked marker. (c) Tip vertical position over time: calibrated simulation (blue) closely tracks real finger motion (orange). The narrow residual (shaded) confirms that Vid2Sid captures both the amplitude and timing of real finger dynamics.

Fig. 5: Ablation study on (a) finger and (b) tentacle. Each bar removes one component. Dots show individual seeds. Removing video increases error by 66% on the finger but decreases it by 38% on the tentacle, indicating that prompt design is domain-dependent.

Method	Finger (mm) $\downarrow$	Tentacle (px) $\downarrow$
Random	15.67 $\pm$ 13.80	10.63 $\pm$ 9.59
Nelder-Mead	46.41 $\pm$ 8.77	11.65 $\pm$ 11.05
Golden-CD	16.36 $\pm$ 17.79	11.44 $\pm$ 12.44
BO	9.70 $\pm$ 7.49	10.02 $\pm$ 11.03
CMA-ES	23.77 $\pm$ 19.23	10.02 $\pm$ 7.15
Vid2Sid (Ours)	7.13 $\pm$ 2.56	8.91 $\pm$ 9.52

TABLE II: sim2sim holdout generalization (mean

\pm

std over

N=12

datapoints: 3 seeds

\times

4 holdouts). Finger error is in mm (2D tip position); tentacle error is in px (2D centerline). Without model misspecification, Vid2Sid recovers parameters most consistently across both platforms, suggesting that residual sim2real error is dominated by perception and model-class limits.

Fig. 6: Normalized distance to ground truth over iterations ( sim2sim , best seed per method). Vid2Sid consistently converges toward the true parameters on both platforms, while baselines oscillate or settle at compensating values far from ground truth. This confirms that VLM reasoning recovers physically meaningful parameters rather than exploiting compensating combinations.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Specification
CPU	Intel Core i7-10700K
RAM	32 GB DDR4
GPU (SAM3)	NVIDIA A100 40GB
VLM	Google Gemini 2.5 Pro
OS	Ubuntu 22.04 LTS

Appendix

TABLE III: Compute resources.

	Vid2Sid	Baselines
Shared components	35 s	35 s
Optimizer/VLM	11 s	$<$ 1 s
Total per iteration	46 s	36 s
10-iteration calibration	7.7 min	6.0 min

Appendix

TABLE IV: Per-iteration wall-clock time (tentacle platform). Shared components (hardware capture: 15 s, simulation: 8 s, SAM3: 12 s) total 35 s for all methods. Vid2Sid adds 11 s VLM inference (28% overhead), keeping total calibration under 10 minutes.

Parameter	Min	Max	Nominal	Description
Friction loss	0	150	50	Joint dry friction torque (Nm)
Damping	10	200	100	Velocity-dependent viscous damping
Armature	0.1	5.0	1.0	Rotor inertia added to each joint
Link density	1.0	20.0	5.0	Density of body geometries (kg/m 3 )
MCP amplitude	0	60	10	MCP joint oscillation amplitude (deg)
PIP amplitude	0	60	10	PIP/DIP joint oscillation amplitude (deg)

Appendix

TABLE V: Parameter bounds for the finger platform.

Parameter	Min	Max	Nominal	Description
Body parameters
Young’s modulus	$5\times 10^{3}$	$5\times 10^{6}$	$2\times 10^{5}$	Rod stiffness (Pa)
Rod density	500	5000	1050	Material density (kg/m 3 )
Poisson’s ratio	0.2	0.5	0.5	Lateral-to-axial strain ratio
Damping constant	0	100	3.0	Linear (Rayleigh) damping coefficient
Environment parameters

Appendix

TABLE VI: Parameter bounds for the tentacle platform (in-air calibration).

Parameter	Max	Nominal	Description
Fluid density	2000	1000	Surrounding fluid density (kg/m 3 )
Perpendicular drag	100	10	Perpendicular fluid drag coefficient
Tangential drag	100	10	Tangential fluid drag coefficient

Appendix

TABLE VII: Hydrodynamic environment parameter bounds for the underwater tentacle stress test. Body parameters are frozen from the in-air experiment.

	Finger	Tentacle
	( $A_{0}$ , $A_{1}$ )	( $A$ , $f$ )
H1	$50^{\circ}$ , $50^{\circ}$	0.9 rad, 0.5 Hz
H2	$50^{\circ}$ , $8^{\circ}$	0.9 rad, 0.15 Hz
H3	$8^{\circ}$ , $50^{\circ}$	0.3 rad, 0.5 Hz
H4	$8^{\circ}$ , $8^{\circ}$	0.3 rad, 0.15 Hz

Appendix

TABLE VIII: Holdout control profiles for each platform. Finger profiles vary joint amplitudes with fixed frequencies (

f_{0}{=}0.3

Hz,

f_{1}{=}0.35

Hz, phase offset

=45^{\circ}

). Tentacle profiles vary amplitude and frequency.

Component	Tokens	Cost (USD)
Input (per iteration)	6,200	$0.008
Output (per iteration)	600	$0.006
Per iteration	6,800	$0.014
Per training run (10 iter)	68,000	$0.14
Per experiment (3 seeds)	204,000	$0.42

Appendix

TABLE IX: VLM cost analysis for Gemini 2.5 Pro. A full 10-iteration calibration run costs $0.14, negligible relative to hardware and compute time.

Setting	Value
Model	Google Gemini 2.5 Pro
Temperature	1.0 (default)
Top- $P$	0.95 (default)
Top- $K$	64 (default)
Max output tokens	65,536
Max input tokens	1,048,576

Appendix

TABLE X: VLM inference settings. All values are Gemini 2.5 Pro defaults, held fixed across every seed, domain, and experimental condition.

Fig. 7: VLM recommendation success rate by confidence threshold. Higher confidence reliably predicts better outcomes. At confidence $\geq 0.9$ , recommendations succeed 89% of the time. The red dashed line indicates random chance (50%).

Prompt Type	Success (%)	Manual Annotation
Text	97.2	None
Point	98.5	Seed point
ROI	94.1	Bounding box

Appendix

TABLE XI: SAM3 segmentation success rate by prompt type.

Fig. 8: Training convergence across methods. Top row: finger sim2real . Bottom row: tentacle sim2real . Bold lines show mean error; shaded regions show $\pm$ 1 std; thin traces show individual seeds. Dots mark the best mean iteration. Vid2Sid reaches low error within 5 iterations on both platforms, matching or exceeding the convergence speed of black-box baselines.

Method	Seed 1	Seed 2	Seed 3	Mean	Std
Random	7.3	11.8	5.6	8.3	3.2
Nelder-Mead	24.9	46.2	5.0	25.4	20.6
Golden-CD	21.0	26.2	1.4	16.2	13.1
BO	12.0	12.9	5.6	10.2	4.0
CMA-ES	4.1	34.2	3.4	13.9	17.6
Vid2Sid (Ours)	10.3	10.6	5.0	8.6	3.2

Appendix

TABLE XII: Finger sim2real : best training error (px) within 10 iterations per seed. Best mean in bold , second-best underlined .

Method	Seed 1	Seed 2	Seed 3	Mean	Std
Random	55.9	56.5	55.2	55.9	0.7
Nelder-Mead	55.7	189.3 †	57.1	100.7	76.8
Golden-CD	54.0	73.0	56.6	61.2	10.3
BO	55.1	58.4	55.0	56.2	1.9
CMA-ES	54.5	67.0	54.8	58.8	7.1
Vid2Sid (Ours)	55.9	58.7	57.9	57.5	1.4

Appendix

TABLE XIII: Tentacle sim2real : best training error (px) within 10 iterations per seed. Best mean in bold , second-best underlined .

Method	Seed 1	Seed 2	Seed 3	Mean $\pm$ Std
Random	12.1	51.7	19.5	27.8 $\pm$ 17.2
Nelder-Mead	14.2	22.2	15.3	17.2 $\pm$ 3.5
Golden-CD	12.3	12.2	11.9	12.1 $\pm$ 0.2
BO	14.4	14.5	19.5	16.1 $\pm$ 2.4
CMA-ES	11.9	22.3	11.8	15.3 $\pm$ 4.9
Vid2Sid (Ours)	8.2	4.9	19.5	10.9 $\pm$ 6.3

Appendix

TABLE XIV: Finger sim2real : per-seed holdout error (px). Each value is the mean over 4 holdouts

\times

3 hardware repeats. Best mean in bold , second-best underlined .

Method	Seed 1	Seed 2	Seed 3	Mean $\pm$ Std
Random	55.3	56.0	51.6	54.3 $\pm$ 1.9
Nelder-Mead †	54.3	–	60.3	57.3 $\pm$ 4.2
Golden-CD	50.4	49.5	60.2	53.4 $\pm$ 4.8
BO	77.1	54.5	55.8	62.4 $\pm$ 10.4
CMA-ES	50.3	49.9	56.2	52.1 $\pm$ 2.9
Vid2Sid (Ours)	48.8	49.9	60.5	53.0 $\pm$ 5.3

Appendix

TABLE XV: Tentacle sim2real : per-seed holdout error (px). Best mean in bold , second-best underlined .

Method	Seed 1	Seed 2	Seed 3	Mean $\pm$ Std
Random	80.5	76.7	71.2	76.1 $\pm$ 3.8
Nelder-Mead	84.4	77.3	80.4	80.7 $\pm$ 2.9
Golden-CD	82.2	73.2	77.9	77.8 $\pm$ 3.7
BO	71.7	71.8	71.5	71.7 $\pm$ 0.1
CMA-ES	80.8	72.9	79.0	77.6 $\pm$ 3.4
Vid2Sid (Ours)	77.6	71.1	71.1	73.3 $\pm$ 3.1

Appendix

TABLE XVI: Water tentacle (env-only, shared body): per-seed holdout error (px). Best mean in bold , second-best underlined .

Fig. 9: Per-holdout error breakdown for all methods across experiments. Each cell shows mean error over 3 seeds $\times$ 3 hardware repeats (sim2real) or 3 seeds (sim2sim). Vid2Sid performs consistently well across holdout conditions, particularly excelling on H2 and H3 for the finger, while black-box methods show more variable performance across holdouts.

Ablation	Seed 1	Seed 2	Seed 3	Mean $\pm$ Std
Full Vid2Sid	3.68	12.38	13.63	9.89 $\pm$ 4.42
w/o Active Learning	3.45	12.46	3.48	6.46 $\pm$ 4.24
w/o History	3.43	3.75	19.51	8.90 $\pm$ 7.51
w/o CoT	3.60	10.49	19.51	11.20 $\pm$ 6.52
w/o Video	12.24	17.05	20.04	16.44 $\pm$ 3.21

Appendix

TABLE XVII: Finger sim2real ablation: per-seed holdout error (px). Each cell is the mean over 4 holdouts

\times

3 HW repeats for the best iteration of that seed. Best mean in bold , second-best underlined .

Ablation	Seed 1	Seed 2	Seed 3	Mean $\pm$ Std
Full Vid2Sid	60.26	50.30	66.21	58.92 $\pm$ 6.56
w/o Active Learning	32.24	37.08	37.59	35.64 $\pm$ 2.41
w/o History	37.80	37.04	32.59	35.81 $\pm$ 2.30
w/o CoT	49.29	55.96	33.03	46.09 $\pm$ 9.63
w/o Video	33.07	41.70	35.63	36.80 $\pm$ 3.62

Appendix

TABLE XVIII: Tentacle sim2real ablation: per-seed holdout error (px). Best mean in bold , second-best underlined .

	Iteration
Seed	1	2	3	4	5	6	7	8	9	10	Min	Mean
1	.31	.75	.35	.60	.65	.40	.45	.35	.35	.33	.31	.45
2	.96	.50	.60	.50	.70	.48	.60	.50	.48	.55	.48	.59
3	.39	.75	1.0	1.0	1.0	1.0	1.0	.90	.95	.42	.39	.84

Appendix

TABLE XIX: Tentacle sim2real : VLM-recommended motor amplitude (rad) per iteration. Bounds: [0.2, 1.0]. Amplitudes remain well above the lower bound across all seeds, ruling out degeneracy.

		Iteration
Seed	Joint	1	2	3	4	5	6	7	8	9	10	Min	Mean
1	MCP	25	18	28	20	24	20	23	19	18	19	18	21
1	PIP	34	25	35	27	30	26	30	26	25	27	25	28
2	MCP	38	25	21	28	35	24	35	39	25	22	21	29
2	PIP	35	30	24	33	40	28	40	46	29	26	24	33
3	MCP	8	40	28	45	12	18	10	22	10	25	8	22

Appendix

TABLE XX: Finger sim2real : VLM-recommended joint amplitudes (deg) per iteration. Bounds: [0, 60]. Both joints maintain substantial amplitude throughout optimization.

Parameter	GT	Random	Nelder-Mead	Golden-CD	BO	CMA-ES	Vid2Sid
frictionloss	90.4	45.2 (50%)	20.8 (77%)	84.3 (7%)	124.5 (38%)	23.1 (74%)	88.0 (3%)
damping	114.3	156.0 (37%)	176.1 (54%)	171.0 (50%)	137.4 (20%)	194.1 (70%)	120.0 (5%)
armature	3.60	2.1 (42%)	4.0 (11%)	3.8 (6%)	1.6 (56%)	3.4 (6%)	3.5 (3%)
density	11.4	8.3 (27%)	5.6 (51%)	5.8 (49%)	12.2 (7%)	3.8 (67%)	10.5 (8%)
Mean rel. error (%)	–	42.1	53.8	28.4	35.6	58.2	8.7

Appendix

TABLE XXI: Finger sim2sim parameter recovery (best seed). Relative error (%) in parentheses. Best per parameter in bold ; red indicates

>

50% error.

Parameter	GT	Random	Nelder-Mead	Golden-CD	BO	CMA-ES	Vid2Sid
youngs_mod	$2.1{\times}10^{6}$	$3.5{\times}10^{6}$ (67%)	$4.3{\times}10^{6}$ (105%)	$2.8{\times}10^{6}$ (33%)	$4.1{\times}10^{6}$ (95%)	$5.0{\times}10^{6}$ (138%)	$\bm{2.3{\times}10^{6}}$ (10%)
rod_density	1665	2100 (26%)	3993 (140%)	2200 (32%)	3110 (87%)	2517 (51%)	1850 (11%)
poisson_ratio	0.353	0.31 (12%)	0.28 (21%)	0.35 (1%)	0.30 (15%)	0.29 (18%)	0.30 (15%)
damping_const	40.5	55.0 (36%)	46.4 (15%)	52.0 (28%)	53.6 (32%)	68.3 (69%)	42.0 (4%)
Mean rel. error (%)	–	48.2	95.1	35.8	72.4	98.6	12.4

Appendix

TABLE XXII: Tentacle sim2sim parameter recovery (best seed). Relative error (%) in parentheses. Best per parameter in bold ; red indicates

>

50% error.

Fig. 10: Finger sim2sim parameter trajectories (best seed per method). Parameters are normalized to $[0,1]$ within bounds. Black dashed line indicates ground truth. Vid2Sid (blue) converges toward ground truth on all four parameters, while baselines often settle at compensating values far from the true parameters.

Fig. 11: Tentacle sim2sim parameter trajectories (best seed per method). Vid2Sid reaches ground truth on Young’s modulus and damping constant, while most baselines remain far from the true values despite achieving low error through compensating parameter combinations.

Robot System Identification

See all 18 papers

RigPI: Dynamic Parameter Identification of Rigid Body via VLM-Seeded Differentiable Simulation
Jun 23, 2026Xincheng He, Rongrong Zhang, Wei Jiang +1Robot Dynamics Identification Robotic Manipulation
Way Robotics
Swim2Real: VLM-Guided System Identification for Sim-to-Real Transfer
Mar 21, 2026Kevin Qiu, Kyle Walker, Mike Y. Michelis +2Robot System Identification Sim-to-Real Transfer
University of Warsaw · IDEAS NCBR · EPFL +2
Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents
Jul 21, 2026Guanxiong Chen, Qianjun Xia, Jiawei Peng +24World Models for Robotics Robotics Simulation
University of British Columbia. · National University of Singapore. · Johns Hopkins University. +4

Jun 23, 2026J/K move · Enter open · S save

RigPI: Dynamic Parameter Identification of Rigid Body via VLM-Seeded Differentiable Simulation

Xincheng He, Rongrong Zhang, Wei Jiang, Wenqiang Xu

Way Robotics

Robot Dynamics Identification Robotic Manipulation System Identification

Open paper AskPDF arXiv

Accurate physical parameter identification of manipulated objects is fundamental to advanced robotic manipulation and the construction of faithful digital twins. However, acquiring physically consistent inertial and frictional properties from real-world interactions remains challenging due to sensing noise, modeling errors, and limited prior knowledge. This paper presents RigPI, a systematic framework for identifying dynamic parameters of both unconstrained rigid bodies and multi-link rigid bodies during robot-object interaction. RigPI integrates vision-based semantic priors, force-torque measurements, and motion observations within a differentiable simulation pipeline. A vision-language model (VLM) provides informed initialization and a constrained search space, while gradient information from a differentiable physics simulator enables efficient and stable parameter refinement. The proposed two-stage optimization strategy alleviates sensitivity to noise and avoids physically implausible solutions. Extensive real-world experiments on objects with revolute and prismatic joints demonstrate that RigPI achieves accurate and stable parameter estimates, and successfully reproduces manipulation trajectories on a real robot with parameter-aware predictive validity. These results highlight the effectiveness and robustness of RigPI for real-world robotic system identification tasks.

Abstract

Robot System Identification

Explore similar work

RigPI: Dynamic Parameter Identification of Rigid Body via VLM-Seeded Differentiable Simulation

Swim2Real: VLM-Guided System Identification for Sim-to-Real Transfer

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents