We present Swim2Real, a pipeline that calibrates a 16-parameter robotic fish simulator from swimming videos using vision-language model (VLM) feedback, requiring no hand-designed search stages. Calibrating soft aquatic robots is particularly challenging because nonlinear fluid-structure coupling makes the parameter landscape chaotic, simplified fluid models introduce a persistent sim-to-real gap, and controlled aquatic experiments are difficult to reproduce. Prior work on this platform required three manually tailored stages to handle this complexity. The VLM compares simulated and real videos and proposes parameter updates. A backtracking line search then validates each step size, tripling the accept rate from 14% to 42% by recovering proposals where the direction is correct but the magnitude is too large. Swim2Real calibrates all 16 parameters simultaneously, most closely matching real fish velocities across all motor frequencies (MAE = 7.4 mm/s, 43% lower than the next-best method), with zero outlier seeds across five runs. Motor commands from the trained policy transfer to the physical fish at 50 Hz, completing the pipeline from swimming video to real-world deployment. Downstream RL policies swim 12% farther than those from BayesOpt-calibrated simulators and 90% farther than CMA-ES. These results demonstrate that VLM-guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification, a step toward automated, general-purpose simulator tuning for underwater robotics.
Figures & tables
Fig. 2 : One iteration of the backtracking line search. The VLM diagnoses a physical discrepancy and proposes updated parameters θ′ . The full step ( β0=1.0 ) overshoots, but halving the step size ( β1=0.5 ) yields an improvement over Lbest and is accepted. This triples the accept rate from 14% to 42% compared to evaluating only the full step.
Parameter group
Count
Bounds
Unit
Fluid coefficients
5
[0,10]
–
Motor arm length
1
[0.01,0.06]
m
Hinge stiffness (per joint)
5
[0.1,5.0]
N ⋅ m/rad
Hinge damping (per joint)
5
[0.0,2.0]
N ⋅ m ⋅ s/rad
TABLE I : The 16-dimensional parameter space covers fluid dynamics, motor geometry, and per-joint mechanical properties.
Fig. 3 : (a) CAD cross-section showing the antagonistic tendon arrangement that crosses at the tail midpoint to produce an S-bend. (b) Block diagram of the onboard electronics. (c) The tendon-driven fish robot platform with annotated components. A single motor drives the full range of swimming gaits used for calibration and RL deployment.
Method
Best (mm) ↓
Worst (mm) ↓
AUC (mm) ↓
Random
82.0±34.3
141.8
129.3±26.6
CMA-ES
112.7±83.9
254.2
156.4±91.7
BayesOpt
52.4±2.1
55.6
94.2±17.6
Swim2Real (Ours)
51.3±1.2
53.2
85.9±30.9
w/o line search
52.4±2.8
57.3
–
w/ Gemini 3.1
54.6±4.2
62.0
–
TABLE II : Calibration performance across 5 seeds. Best: mean ± std of per-seed final L2 marker error (averaged across all markers). Worst: highest per-seed final error. AUC: mean best-so-far error over 40 evaluations (omitted for ablations, whose different per-round costs make the metric non-comparable). Swim2Real achieves the lowest error and tightest variance, with all five seeds falling within a 3 mm range.
Fig. 4 : Chronophotography of swimming at 1.5 Hz (3 snapshots, increasing opacity). Top: real fish (overhead). Middle: Swim2Real-calibrated simulator (51 mm error). Bottom: CMA-ES calibration (123 mm error). Swim2Real reproduces the body shape and forward progression of the real fish, while CMA-ES exhibits incorrect posture and reduced thrust.
Fig. 5 : Best-so-far L2 error (mean across 5 seeds, with dots showing individual seed final values). (a) All five Swim2Real seeds fall within 50.2–53.2 mm, while CMA-ES collapses on 2 of 5. (b) The line search triples the accept rate (42% vs. 14%), so Swim2Real reaches ∼51 mm in ∼ 16 VLM calls while the no-line-search ablation requires 39.
Fig. 6 : Velocity error ∣vsim−vreal∣ across motor frequencies. Swim2Real tracks the real fish most closely (MAE = 7.4 mm/s, half that of BayesOpt at 12.9 mm/s), with the gap widening at higher frequencies where calibration quality matters most.
Fig. 7 : Sim-to-real transfer of the Swim2Real-calibrated RL policy. (a) Simulated target-reaching trajectories to diverse goals (green dots). (b) Real-world deployment with motor commands transferred open-loop at 50 Hz. Color encodes time (purple = start, yellow = end). The leftward arc in (b) results from open-loop execution and a hardware steering bias (see text). Despite this, the fish produces directed swimming from commands generated entirely in simulation.
We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. Given synchronized multi-view videos of an individual fish, the pipeline reconstructs a metrically scaled deformable mesh from a VLM-selected canonical frame, generates internal articulation adapted to that individual's morphology through a VLM actor-critic loop, and extracts a Biological Locomotion Manifold (BLM) from the fish's observed midline curvature. The BLM provides a low-dimensional action space bounded by real-fish motion, enabling an individual swimming policy to be learned for each reconstructed fish. We release two paired datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the dataset supports a benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual in trajectory shape, body curvature, and tail-beat frequency, across trajectory following, reward-free swimming behavior transfer from video, and a downstream case study in which a simulated BlueROV underwater robot captures one of the assets. We find that task success and locomotion fidelity do not necessarily improve together: the method achieving the highest task completion is not the method achieving the highest locomotion fidelity, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Project website: https://hangongchen.github.io/video2swimfish-web/.
Hangong Chen, Linfeng Cheng, Tahsin Zaman Jilan +4
Department of Computer Science, North Carolina A&T State University · Department of Computer Science, University of Maryland, College Park
Calibrating a robot simulator's physics parameters (friction, damping, material stiffness) to match real hardware is often done by hand or with black-box optimizers that reduce error but cannot explain which physical discrepancies drive the error. When sensing is limited to external cameras, the problem is further compounded by perception noise and the absence of direct force or state measurements. We present Vid2Sid, a video-driven system identification pipeline that couples foundation-model perception with a VLM-in-the-loop optimizer that analyzes paired sim-real videos, diagnoses concrete mismatches, and proposes physics parameter updates with natural language rationales. We evaluate our approach on a tendon-actuated finger (rigid-body dynamics in MuJoCo) and a deformable continuum tentacle (soft-body dynamics in PyElastica). On sim2real holdout controls unseen during training, Vid2Sid achieves the best average rank across all settings, matching or exceeding black-box optimizers while uniquely providing interpretable reasoning at each iteration. Sim2sim validation confirms that Vid2Sid recovers ground-truth parameters most accurately (mean relative error under 13% vs. 28--98%), and ablation analysis reveals three calibration regimes. VLM-guided optimization excels when perception is clean and the simulator is expressive, while model-class limitations bound performance in more challenging settings.
Complex tasks for underwater robots remain limited by the capabilities of their controllers. Learning a better one for a soft, underactuated robotic fish trades simulator cost against fidelity. We show that an intentionally low-fidelity simulator is enough: a stateless, quasi-steady fluid model with no wake and no added-mass history suffices to learn a general, closed-loop controller that transfers to hardware without tuning. Our platform is a soft, single-motor, tendon-driven fish whose policy observes only what the hardware can measure. A staged pipeline grounds the simulator in two independent identifications, fixing the tail dynamics and a stateless fluid model; the policy then acts through a band-limited rhythmic trajectory generator rather than commanding the tail directly. Deployed unchanged in an outdoor pool, a single policy performs closed-loop target reaching, disturbance rejection, and out-of-distribution target acquisition and tracking. The transfer rests on the constraint rather than the fidelity: the generator cannot leave the band over which the fluid was identified. This raises the question of how much of the physics can reside in the controller rather than in the simulator.
Liam Maloney, Simon Ramchandani, Mike Y. Michelis +2
Soft Robotics Lab, ETH Zurich, Switzerland · ETH AI Center, ETH Zurich, Switzerland