Learning Soft Robotic Dynamics with Active Exploration

Hehui Zheng; Bhavya Sukhija; Chenhao Li; Klemens Iten; Andreas Krause; Robert K. Katzschmann

cs.ROOct 31, 2025

Learning Soft Robotic Dynamics with Active Exploration

Authors: Hehui Zheng, Bhavya Sukhija, Chenhao Li, Klemens Iten, Andreas Krause, Robert K. Katzschmann

Organizations: Soft Robotics Lab, D-MAVT, ETH Zurich, Zurich, Switzerland · ETH AI Center, ETH Zurich, Zurich, Switzerland · Learning & Adaptive Systems Group, D-INFK, ETH Zurich, Zurich, Switzerland

Abstract

Soft robots offer adaptability and safety in unstructured environments, yet their compliant, high-dimensional, and nonlinear dynamics make modeling for control difficult. Existing data-driven approaches often fail to generalize, constrained by narrowly focused task demonstrations or inefficient random exploration. We introduce an uncertainty-aware active exploration framework that autonomously learns task-agnostic and generalizable dynamics models of soft robotic systems. We evaluate this framework on three simulated platforms---a continuum arm, an articulated fish in fluid, and a musculoskeletal leg---and on a real-world pneumatically actuated continuum arm. Compared with random exploration and task-specific baseline, our method achieves higher modeling accuracy by actively collecting informative data. This improved model quality yields higher returns in zero-shot control on unseen tasks and robust performance under sensing noise, actuation delays, and material nonlinearities. These results demonstrate that uncertainty-driven active exploration enables data-efficient dynamics learning across soft robotic morphologies, supporting autonomous and adaptable control for compliant robots.

Figures & tables

Figure 1: Learning soft robotic dynamics with active exploration. ( A – D ) Soft robotic platforms studied in this work: (A) soft continuum arm, (B) deformable fish in fluid, (C) musculoskeletal (MSK) leg with electrohydraulic actuation (yellow arrows), and (D) real-world pneumatically actuated arm (SoPrA). Their nonlinear, high-dimensional, and deformable dynamics pose significant challenges for generalizable model learning. ( E ) We address this with SoftAE , an active exploration strategy that autonomously collects informative and diverse data for training dynamics models. During dynamics learning (top), Random exploration is unguided, and task-oriented method overfits. SoftAE instead targets regions of high model uncertainty, improving data coverage and model accuracy. This enables robust generalization to diverse downstream tasks (bottom), as ( F ) demonstrated across all platforms.

Environment	Task	State Space Dimensions	Action Space Dimensions
Soft Continuum Arm	(i) reach close target	$58$	$12$
Soft Continuum Arm	(ii) reach far target	$58$	$12$
Articulated Fish in Fluid	(iii) swim along +x direction	$15$	$4$
	(iv) swim along -x direction	$15$	$4$
	(v) swim to target	$18$	$4$
	(vi) swim away from target	$18$	$4$

Table 1: Simulated soft robotic environments for dynamics learning. The table summarizes three environments investigated in this work: soft continuum arm, articulated fish with deformable skin in fluid, and musculoskeletal leg, together with their associated tasks. For each environment, the state space dimensions correspond to the size of the observation vector, while the action space dimensions denote the number of independent actuation signals. Detailed task rewards and decomposition of state–action space are provided in Supplementary Tables 1 – 5 .

Figure 2: Active exploration enables robust zero-shot task performance across diverse soft robotic systems. We evaluate SoftAE against baseline exploration strategies across three soft robotic platforms: ( A ) a soft continuum arm, ( B ) a deformable fish swimming in fluid, and ( C ) a musculoskeletal (MSK) leg actuated by electrohydraulic muscles and a DC motor. H-UCRL , a task-specific model-based baseline, is trained only on tasks (i) , (iii) , (v) , and (vii) , while the remaining tasks are held out as unseen. Across tasks, SoftAE consistently produces task-aligned and physically plausible behaviors, while Random and H-UCRL often fail to generalize to the hard or unseen scenarios. Returns are averaged over 10 random seeds for tasks (i) and (ii) , and 5 seeds for the remaining tasks; shaded regions denote $\pm$ one standard error of the mean.

Figure 3: Active exploration improves data coverage, model accuracy, scales to high-dimensional dynamics, and captures soft–rigid contact. (A) Spatial coverage of collected data for the soft continuum arm, with tip positions projected onto the $x$ – $z$ plane. Heatmaps show visitation frequency per spatial bin. SoftAE explores more broadly and uniformly than Random and H-UCRL . (B) Normalized mean squared error (MSE) of learned dynamics models on a held-out set of 17,451 transitions. SoftAE achieves the lowest error. (C) Return curves for reaching (i) close and (ii) far targets on a soft arm with a 136-dimensional state space. SoftAE maintains high performance in this high-dimensional setting, matching H-UCRL on its trained task and outperforming baselines on the unseen task. Returns are averaged over 10 seeds; shaded regions denote $\pm$ one standard error of the mean. ( D ) Autoregressive 10-step prediction (from $t=0.1$ s to $t=0.2$ s) of learned dynamics models in an environment with a rigid spherical obstacle (green). A model trained without obstacle exposure (left) predicts the arm passing through the obstacle, while a model trained in the obstacle environment (right) accurately reproduces the contact-induced deformation. ( E ) Downstream reaching tasks with rigid obstacles, where SoftAE and Random explore in the obstacle environment and H-UCRL is trained on task (i) . Top: reaching under a horizontal bar. Bottom: reaching the opposite side of a vertical wall through a gap near its base. Only SoftAE consistently succeeds at both contact-rich tasks.

Figure 4: Real-world dynamics learning and task performance on a pneumatically actuated soft arm. ( A ) SoPrA: a fiber-reinforced, pneumatically actuated continuum arm with motion capture via marker rings. ( B ) Reaching tasks for (ix) close and (x) far targets. Top row: front view; bottom row: bottom view. Models are trained using Random , H-UCRL (task-specific, trained only on (ix) ), or SoftAE (ours). Yellow dots indicate target positions. ( C ) Return curves show all methods perform similarly on task (ix) , while SoftAE outperforms on the harder, unseen task (x) . Returns are averaged over 3 random seeds, shaded regions denote standard error of the mean. ( D ) Quantitative evaluation over the same set of 20 random targets shows that SoftAE achieves the highest mean return and lowest tip position error compared to baselines. ( E ) Trajectory tracking performance on two reference trajectories: square (top row) and lemniscate (bottom row). SoftAE tracks both shapes more accurately than baselines.

		$\displaystyle{\color[rgb]{0,0,0}(\bm{\pi}_{n},\bm{\eta}_{n})\in\argmax_{\begin{subarray}{c}\bm{\pi}\in\Pi,\\ \bm{\eta}(\cdot)\in[-\beta_{n-1},\beta_{n-1}]^{d_{{\bm{s}}}}\end{subarray}}\mathbb{E}\bigg[\sum_{t=0}^{T-1}\big\\|\bm{\sigma}_{n-1}}$		$\displaystyle{\color[rgb]{0,0,0}\bigl(\widetilde{{\bm{s}}}_{t},\bm{\pi}(\widetilde{{\bm{s}}}_{t})\bigr)\big\\|\bigg]}$
		➤ Prepare optimistic policy via ( 9 )
		$\displaystyle\quad\text{{\color[rgb]{0,0,0}s.t.}}\quad\,\,\,\,\,\ {\color[rgb]{0,0,0}\widetilde{{\bm{s}}}_{t+1}=\bm{\mu}_{n-1}\bigl(\widetilde{{\bm{s}}}_{t},\bm{\pi}(\widetilde{{\bm{s}}}_{t})\bigr)+\bm{\sigma}_{n-1}}$		$\displaystyle{\color[rgb]{0,0,0}\big(\widetilde{{\bm{s}}}_{t},\bm{\pi}(\widetilde{{\bm{s}}}_{t})\big)\bm{\eta}(\widetilde{{\bm{s}}}_{t})}$
		➤ Propagate optimistic dynamics via ( 10 )
		$\displaystyle(\bm{S},\bm{A},\bm{S}^{\prime})\leftarrow\textsc{Rollout}(\bm{\pi}_{n})$		➤ Collect measurements
		$\displaystyle{\mathcal{D}}_{n}\leftarrow{\mathcal{D}}_{n-1}\cup(\bm{S},\bm{A},\bm{S}^{\prime})$		➤ Add rollout to dataset
		$\displaystyle(\bm{\mu}_{n},\bm{\sigma}_{n},\beta_{n})\leftarrow\textsc{UpdateModel}({\mathcal{D}}_{n})$		➤ Update model via ( 13 )

Algorithm 1 SoftAE : Soft-robot Active Exploration.

Supplementary Figure 1 : Random exploration with extended interaction budget on the simulated soft continuum arm. Return plots for the two downstream reaching tasks: task (i) reach close, and task (ii) reach far. Random is granted 500 exploration episodes, five times the budget given to SoftAE and H-UCRL (100 episodes each, identical to Fig. 2 A). On the easier task (i) , Random surpasses SoftAE ’s performance at 100 exploration episodes and approaches the performance of H-UCRL , though without fully matching it. On the harder task (ii) , Random plateaus well below SoftAE even with the extended budget. Returns are averaged over 10 random seeds.

Supplementary Figure 2 : Comparison of different uncertainty-driven active exploration strategies. Same as return plots in Fig. 2 but with two additional active exploration baselines: Mean-AE and PETS-AE . Mean-AE plans trajectories using the mean model

\bm{\mu}_{n}

, PETS-AE uses trajectory sampling from the model posterior to improve robustness to model inaccuracies. Returns are averaged over 10 random seeds for tasks (i) and (ii) , and 5 seeds for the remaining tasks.

Supplementary Figure 3 : Effect of actuation cost on uncertainty-driven active exploration in the simulated soft continuum arm. Same as the return plots in Supplementary Fig. 2 A, but with the intrinsic exploration reward augmented by an actuation penalty,

r_{n}^{\mathrm{exp,act}}({\bm{s}}_{t},{\bm{a}}_{t})=\left\|\bm{\sigma}_{n-1}({\bm{s}}_{t},{\bm{a}}_{t})\right\|-c_{\mathrm{act}}\left\|{\bm{a}}_{t}\right\|_{2}^{2}

. Rows show downstream returns for tasks (i) reach close (top) and (ii) reach far (bottom); columns sweep

c_{\mathrm{act}}\in\{0.0,0.25,0.5,0.75,1.0\}

(left to right). Without a penalty, all three active exploration methods perform comparably. Under moderate penalties (

c_{\mathrm{act}}\in\{0.25,0.5,0.75\}

), SoftAE maintains its penalty-free performance while Mean-AE and PETS-AE degrade heavily, as planning with the mean or sampled dynamics favors conservative trajectories. At

c_{\mathrm{act}}=1.0

, average SoftAE performance declines as a subset of seeds yields conservative trajectories similar to Mean-AE and PETS-AE , indicating that a sufficiently strong penalty can suppress informative exploration even under optimistic planning. Returns are averaged over 20 random seeds.

Supplementary Figure 4 : Return comparison of policy optimization in the soft continuum arm simulation. All methods use a learned dynamics model to train policies with SAC. Returns are averaged over 10 random seeds. SoftAE-MBPO employs uncertainty-based rewards for active exploration, while MBPO and MBPO-TS baselines are trained on task (i) . The random baseline collects trajectories from random actuation. SoftAE-MBPO achieves performance comparable to task-specific baselines on the training task, but substantially outperforms them on unseen downstream tasks, highlighting the benefit of active exploration for generalization.

Supplementary Figure 5 : Comparison with model-free reinforcement learning on the real-world SoPrA arm. Same as return plots in Fig. 4 C but with the additional model-free baseline: Soft Actor-Critic ( SAC ). SoftAE , Random , and H-UCRL use model-based planning with iCEM trajectory optimization on a learned dynamics model, while SAC is a model-free baseline trained directly through online interaction with the physical arm, without any learned dynamics model. SAC (fixed target) is trained on the single fixed target of task (ix) only (dashed), while SAC (goal-conditioned) is trained on reaching randomized targets and evaluated on the fixed targets of tasks (ix) and (x) . Goal-conditioned SAC fails to make meaningful progress within 200 episodes on both tasks, reflecting the high sample cost of model-free learning on real soft robotic hardware. Returns are averaged over 3 random seeds, with shaded regions indicating the standard error of the mean, except for SAC (fixed target), which is a single run.

State Entry	Symbol	Dimensions (58D)	Dimensions (136D)
element positions	$\mathbf{p}$	$0{:}15$	$0{:}33$
element linear velocities	$\dot{\mathbf{p}}$	$15{:}30$	$33{:}66$
element orientations	$\mathbf{R}$	$30{:}46$	$66{:}106$
element angular velocities	$\bm{\omega}$	$46{:}58$	$106{:}136$
Action Entry	Symbol	Dimensions
normal torques	$\bm{u}_{n}$	$0{:}6$

Supplementary Table 1 : State and action space decomposition of the soft continuum arm. This table details the individual state and action entries corresponding to the two observation discretization levels described in Table 1 . State entries include element positions, velocities, orientations, and angular velocities, while action entries correspond to applied torques at six equidistant points along the arm. Dimensions are provided for both the 5-observed-point (58D) and 11-observed-point (136D) configurations.

State Entry	Symbol	Dimensions (15D)	Dimensions (18D)
relative position of target	$\mathbf{p}_{\mathrm{target}}-\mathbf{p}$	-	$0{:}3$
base linear velocity	$\mathbf{v}$	$0{:}3$	$3{:}6$
head orientation	$\theta$	$3$	$6$
head angular velocity	$\omega$	$4$	$7$
joint positions	$\mathbf{q}$	$5{:}10$	$8{:}13$
joint velocities	$\dot{\mathbf{q}}$	$10{:}15$	$13{:}18$

Supplementary Table 2 : State and action space decomposition of the articulated fish with deformable skin. This table follows the same format as Supplementary Table 1 . Dimensions are listed for task (iii–iv) with a 15D state space and task (v–vi) with an 18D state space, respectively.

State Entry	Symbol	Dimensions
boom yaw	$\psi$	$0$
boom pitch	$\theta$	$1$
boom roll	$\phi$	$2$
leg joint positions	$\mathbf{q}$	$3{:}5$
boom angular velocities	$\bm{\omega}$	$5{:}8$
leg joint velocities	$\dot{\mathbf{q}}$	$8{:}10$

Supplementary Table 3 : State and action space decomposition for the musculoskeletal (MSK) leg with hybrid actuation. This table follows the same format as Supplementary Table 1 .

State Entry	Symbol	Dimensions
tip capture ring position	$\mathbf{p}$	$0{:}3$
tip capture ring orientation	$\mathbf{R}$	$3{:}7$
midpoint capture ring position	$\mathbf{p}_{\mathrm{mid}}$	$7{:}10$
midpoint capture ring orientation	$\mathbf{R}_{\mathrm{mid}}$	$10{:}14$
tip capture ring linear velocity	$\dot{\mathbf{p}}$	$14{:}17$
tip capture ring angular velocity	$\bm{\omega}$	$17{:}20$

Supplementary Table 4 : State and action space decomposition for the real-world pneumatically actuated soft arm (SoPrA). This table follows the same format as Supplementary Table 1 .

Environment	Task	Reward $r_{t}$
Arm	(i) - (ii) reach target	$g(\left\\|\mathbf{p}_{t}-\mathbf{p}_{\mathrm{target}}\right\\|_{2})$ 1 1 1 For arm reach tasks, we employ a shaped long-tail reward function that maps the distance to target into the range $(0,1]$ . Specifically, $g(\left\\|\mathbf{p}_{t}-\mathbf{p}_{\mathrm{target}}\right\\|_{2})=1$ when $\left\\|\mathbf{p}_{t}-\mathbf{p}_{\mathrm{target}}\right\\|_{2}\leq 5\times 10^{-3}$ .
Fish	(iii) swim forward along +x direction	$v_{+x,t}$
	(iv) turn and swim along -x direction	$v_{-x,t}$
	(v) swim to target	$-\left\\|\mathbf{p}_{t}-\mathbf{p}_{\mathrm{target}}\right\\|_{2}$
	(vi) swim away from target	$\left\\|\mathbf{p}_{t}-\mathbf{p}_{\mathrm{target}}\right\\|_{2}$
MSK Leg	(vii) hop counterclockwise	$\dot{\phi_{t}}$

Supplementary Table 5 : Downstream task rewards for simulated and real-world environments. This table summarizes the reward definitions for all simulated environments described in Table 1 as well as for the real-world SoPrA platform. Each task reward is formulated as a function of position, linear velocity, or angular displacement, and follows the notation introduced in Supplementary Tables 1 – 4 , depending on the environment and objective.

Hyperparameters	Arm	Fish	MSK Leg	SoPrA
Exploration horizon	$200$	$200$	$500$	$200$
Downstream task horizon	$200$	$200$	$500$	$20$
Hidden layers	$4$	$4$	$4$	$4$
Neurons per layer	$256$	$256$	$256$	$256$
Number of ensembles	$5$	$5$	$5$	$5$
Batch size	$64$	$64$	$64$	$64$

Supplementary Table 6 : Hyperparameters used in the experiments. This table lists the dynamics model training hyperparameters for all soft robotic environments. The reported values cover exploration and task horizons, dynamics model network architecture and ensemble size, and training schedule.

Hyperparameters	Arm	Fish - (iii) & (iv)	Fish - (v) & (vi)	MSK Leg	SoPrA
Number of samples $P$	$200$	$200$	$200$	$200$	$200$
Horizon $H$	$10$	$10$	$100$	$100$	$5$
Size of elite-set $K$	$20$	$20$	$20$	$20$	$20$
Colored-noise exponent	$0.25$	$0.25$	$0.25$	$0.25$	$1.0$
Number of particles	$10$	$10$	$10$	$10$	$10$
CEM-iterations	$5$	$5$	$5$	$5$	$5$

Supplementary Table 7 : Hyperparameters of the iCEM optimizer. This table reports the optimizer settings used in this work, including the number of samples, horizon length, elite-set size, colored-noise exponent, number of particles, CEM iterations, and the fraction of elites reused. Values are shown separately for each environment and task group.

Hyperparameters	Arm
Exploration horizon	$200$
Downstream task horizon	$200$
Hidden layers	$3$
Neurons per layer	$1024$
Number of ensembles	$5$
Batch size	$256$

Supplementary Table 8 : Hyperparameters of the SAC algorithm in the model-based RL setting. This table lists the training hyperparameters used for the SAC agent, including exploration and task horizons, network architecture, ensemble size, batch size, learning rate, and imagined rollout horizon.

Hyperparameters	SoPrA
Episode horizon	$200$
Training episodes	$200$
Total environment transitions	$40{,}000$
Random warm-up transitions	$0$
Update-to-data ratio	$1$
Hidden layers	$2$

Supplementary Table 9 : Hyperparameters of the SAC algorithm in the online model-free RL setting. This table lists the training hyperparameters used for the SAC baseline of Supplementary Fig. 5 . The code implementation is the same as in the model-based setting (Supplementary Table 8 ), but this model-free baseline is trained on real environment transitions only. The reward is the SoPrA reach target reward of Supplementary Table 5 .

Robot Dynamics Learning

Robot dynamics learning builds models that predict how robots and objects move under actions, enabling control, planning, and adaptation.

See all 99 papers

Trajectory Generation for Underactuated Soft Robot Manipulators using Discrete Elastic Rod Dynamics
Mar 23, 2026Beibei Liu, Akua K. Dickson, Ran Jing +1Robot Motion Generation Trajectory Generation
Division of Systems Engineering, Boston University, Boston MA, USA. · Department of Mechanical Engineering, Boston University, Boston MA, USA.
Learning-Based Modeling of Soft Robots via Cosserat Rod Theory
Jun 18, 2026Mohammad Ali, Nithin Senthur Kumar, Eric J. Barth +1Dynamical Systems Port-Hamiltonian Systems
Department of Computer Science, Vanderbilt University, Nashville, TN 37235, USA · Department of Mechanical Engineering, Vanderbilt University, Nashville, TN 37235, USA
Rapidly Learning Soft Robot Control via Implicit Time-Stepping
Nov 10, 2025Andrew Choi, Dezhong Tong, Xiaonan HuangRobot Policy Learning Robotics Simulation
Horizon Robotics, Cupertino, CA, USA · Robotics Department at the University of Michigan, Ann Arbor, MI, USA

Mar 23, 2026J/K move · Enter open · S save

Trajectory Generation for Underactuated Soft Robot Manipulators using Discrete Elastic Rod Dynamics

Beibei Liu, Akua K. Dickson, Ran Jing, Andrew P. Sabelhaus

Division of Systems Engineering, Boston University, Boston MA, USA. · Department of Mechanical Engineering, Boston University, Boston MA, USA.

Robot Motion Generation Trajectory Generation Soft Robotics Continuum Robot Control

Open paper AskPDF arXiv

Soft robots are well suited for contact-rich tasks due to their compliance, yet this property makes accurate and tractable modeling challenging. Planning motions with dynamically-feasible trajectories requires models that capture arbitrary deformations, remain computationally efficient, and are compatible with underactuation. However, existing approaches balance these properties unevenly: continuum rod models provide physical accuracy but are computationally demanding, while reduced-order approximations improve efficiency at the cost of modeling fidelity. To address this, our work introduces a control-oriented reformulation of Discrete Elastic Rod (DER) dynamics for soft robots, and a method to generate trajectories with these dynamics. The proposed formulation yields a control-affine representation while preserving certain first-principles force-deformation relationships. As a result, the generated trajectories are both dynamically feasible and consistent with the underlying actuation assumptions. We present our trajectory generation framework and validate it experimentally on a pneumatic soft robotic limb. Hardware results demonstrate consistently improved trajectory tracking performance over a constant-curvature-based baseline, particularly under complex actuation conditions.

Abstract

Robot Dynamics Learning

Explore similar work

Trajectory Generation for Underactuated Soft Robot Manipulators using Discrete Elastic Rod Dynamics

Learning-Based Modeling of Soft Robots via Cosserat Rod Theory

Rapidly Learning Soft Robot Control via Implicit Time-Stepping