Organizations: University of Tsukuba, Tsukuba, Japan. · CNRS-AIST JRL (Joint Robotics Laboratory), IRL3218, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan.
Manipulating visually identical objects with different physical properties requires adapting actions based on information acquired through interaction. We address the problem of learning from demonstrations how to adapt manipulation actions, such as regrasping and changing contact locations, based on physical-property information acquired during interaction. To this end, we propose a learning-from-demonstration method that learns latent representations of physical properties needed to predict object responses and conditions a policy on online estimates of these representations. A dynamics model and object-specific latent representations are jointly learned from demonstrations that include behaviors adapting to the objects' physical properties, and the learned representations are then used to condition an imitation policy. During execution, the dynamics model and policy weights remain fixed, while the latent representation is updated based on the discrepancy between predicted and actual observations, allowing interaction with the object to inform action generation. Real-world and simulation experiments with objects whose centers of mass differ suggest that online adaptation through latent representations of physical properties can improve task success rates. However, in simulation, success rates on center-of-mass locations absent from the demonstrations remained around 30%, indicating limited adaptation to unseen physical properties. Project page: https://isri-aist.github.io/latent-dynamics-project-page/
Figures & tables
Fig. 1: Overview of the proposed approach. ① Teleoperated demonstrations include adaptive actions based on observed object responses, such as regrasping or changing contact locations. ② Using the same demonstrations, we first jointly learn a dynamics model and object-specific physical representations, then train an imitation policy conditioned on the learned representations. This learns both dynamics and adaptive actions that depend on physical properties. ③ Starting from a reference physical representation, the robot updates it from prediction errors and conditions subsequent actions on the estimate. The figure illustrates uniform density as the initial assumption.
Fig. 2: Training and deployment pipeline of the proposed method. t denotes the discrete time-step index, and H is the prediction horizon in steps.
Fig. 3: Manipulation tasks with objects whose centers of mass differ: (a) Pouch Placement and (b) Box Pushing on a real robot, and (c) Horizontal Lifting in simulation.
Task
Demonstrations
Evaluation rollouts
Pouch Placement
12 (36)
3 (9)
Box Pushing
10 (50)
2 (22)
Horizontal Lifting
25 (75)
10 (70)
TABLE I: Demonstration and evaluation counts. Each entry gives the count per object configuration, with the total in parentheses. Evaluation counts are per training seed.
Fig. 4: Dynamics model evaluation. Left: prediction errors on validation demonstrations excluded from all training. Solid lines use the representation of the actual object; dashed lines use those of other objects. Errors are measured in image features, except in (c-1), which shows position error. Panel (b) also compares predicted and measured force Fy along the pushing direction. Panels (c-1) and (c-2) use the same episode. Center: prediction-error differences averaged over all validation data for each object, relative to using its corresponding di∗ . Rows indicate the actual objects and columns the input representations. Positive values indicate larger errors when using another object’s representation. Values in the heatmaps are reported in units of 10−3 . Right: online estimates d^ for all evaluation rollouts on seen objects only, with one curve per rollout. Horizontal dashed lines mark the learned, fixed representations di∗ ; curves and reference lines for the same object share a color. For unseen objects with weight positions between those of two seen objects, convergence to values between the corresponding learned representations is desirable. Box Pushing trajectories are shown in Fig. 6 . The objects with centered weights have indices 1, 2, and 0 in Pouch Placement, Box Pushing, and Horizontal Lifting, respectively.
Fig. 5: Task success rates for Pouch Placement and Box Pushing on a real robot. “Proposed + retraining” denotes retraining the VAE and dynamics model on the augmented dataset while keeping the policy fixed. Error bars denote approximate 95% Wilson confidence intervals computed from rollouts pooled across three training seeds, treating rollouts as independent. All seeds were evaluated on the same fixed test configurations. Successes per seed: Pouch Placement (of 9), DP 4, 4, 3; Proposed 6, 6, 6. Box Pushing (of 22), DP 9, 11, 12; Proposed 10, 10, 13; Proposed + retraining 13, 13, 17.
Fig. 6: Online estimates d^ for Box Pushing before (left) and after (right) reinitializing and retraining the VAE and dynamics model on the augmented dataset. Colors indicate object types. Dashed horizontal lines denote the learned di∗ for objects 0–4 (blue, orange, green, red, and purple, respectively).
Method
Seen Objects
Unseen Objects
Total
DP
85.6±5.1
21.7±5.2
49.0±0.8
DP (long history)
85.6±5.1
6.7±2.9
40.5±3.0
Proposed (state-based)
96.7±3.3
30.0±6.6
58.6±2.9
Proposed (image-based)
92.2±3.8
32.5±2.5
58.1±0.8
TABLE II: Success rates (%) for Horizontal Lifting, reported as mean ± sample standard deviation across three training seeds.
While pretrained robotic policies exhibit impressive capabilities in controlled environments, unobserved physical properties and dynamics require these policies to rapidly adapt during deployment. Existing test-time adaptation methods typically rely on sparse scalar rewards, failing to exploit the rich geometric and dynamic feedback from the environment during physical interaction. To address this challenge, we propose SCOUT, a dynamics-aware meta-learning framework that enables manipulation policies to rapidly adapt by continuously revising their internal beliefs about environment dynamics. Our approach couples an action-prediction policy with a forward dynamics model via a shared belief latent space. During meta-training, an inner loop updates this shared belief latent by minimizing the dynamics prediction error against the observed action outcome, while the outer loop optimizes the network for action selection. At deployment, this structure allows the agent to infer and adapt to unknown physical dynamics on the fly. By updating its latent belief based on action-outcome mismatches, the policy automatically adapts without risking catastrophic forgetting. We demonstrate that SCOUT significantly accelerates online adaptation across simulated manipulation benchmarks and achieves robust sim-to-real transfer in the real world. Project webiste can be found here: https://liy1shu.github.io/SCOUT/
Yishu Li, Liyuan Geng, Xinyi Mao +2
Robotics Institute, Carnegie Mellon University · Computer Science and Technology, Tsinghua University
Pretrained robotic policies can suffer substantial performance degradation under out-of-distribution (OOD) conditions encountered during deployment, motivating post-training through real-world interaction. However, reinforcement-learning (RL)-based post-training typically requires substantial environment interactions, a burden that is especially significant in manipulation, where each trial can be slow, costly, or destructive. Therefore, we present PEARS, a physics-prior-guided hybrid RL framework for sample-efficient online adaptation of pretrained policies with tactile feedback. After each episode, its physics-guided force reasoning (PFR) module uses physical priors encoded in a vision-language model (VLM) to diagnose failures from the visual outcome and tactile interaction history and update task-appropriate contact-force bounds. A high-frequency hybrid force-position controller then enforces these bounds during contact. Complementarily, tactile-conditioned diffusion steering reinforcement learning adjusts the latent noise of the frozen flow-matching policy to correct errors in free-space motion and contact timing without updating the base model. In simulation, PEARS improves success rates by 12.4-37.4 percentage points over the strongest per-task baselines. PEARS also reduces the number of interaction episodes required for a certain success threshold by up to 53.2% relative to the fastest baseline. In real-world experiments, PEARS achieves success rates of 95% on Whiteboard Erasing and 90% on Pipette Liquid Aspiration. These results show that combining the PFR module with policy steering can accelerate adaptation while reducing costly interactions. The project website is available at https://song-kun.github.io/pears.
Kun Song, Yiming Wang, Yilin Chen +5
The University of Hong Kong · Shanghai Jiao Tong University · Xense Robotics +1
Recent vision-based action models have demonstrated strong capabilities in complex manipulation, but they rarely leverage explicit object physical properties to adapt their policies. We introduce ViTacPhys, a visual-tactile framework and data acquisition system that estimates object mass and friction-coefficient classes, together with continuous stiffness, from human manipulation demonstrations. Trained on data from 60 rigid and deformable objects, ViTacPhys combines temporal visual-tactile modeling, cross-attention multimodal fusion, and a semantic prior derived from a vision-language model. On seen objects, it achieves 97.2% mass classification accuracy, 98.8% friction-coefficient classification accuracy, and a stiffness mean absolute percentage error (MAPE) of 5.51%. On held-out objects from known categories, it achieves 87.5% mass accuracy, 97.5% friction-coefficient accuracy, and a stiffness MAPE of 9.08%. We transfer ViTacPhys from the human domain to the robot domain using limited robot teleoperation data, robot-style video augmentation, and human demonstrations with matched actions, and deploy it as an online module for adaptive grasping. The resulting physical-property-conditioned policy achieves total grasping success rates of 95.0% on in-distribution objects and 83.4% on out-of-distribution objects. For out-of-distribution objects successfully grasped by both methods, its force profiles are more consistent with human teleoperation than those produced by ACT. These results demonstrate the feasibility of explicitly estimating and conditioning on object physical properties for real-world adaptive grasping.