Shake to Learn: Dynamic Interrogation of Hidden Object Physics for Robotic Manipulation with Physical Reservoir Computing
Authors: Wen Sin Lor, Jun Wang, Suyi Li
Organizations: Department of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24060, USA · Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI 48109, USA
Many physical properties relevant to robotic manipulation are hidden from vision. A sealed object, for example, may reveal little about its center of mass (COM) or internal contents until it is lifted, shaken, or otherwise dynamically perturbed. This study shows that such interactions can enable a new modality of robotic perception and learning, in which interaction-induced dynamic responses are used to infer object physics that is inaccessible to conventional sensing. We implement this idea using an origami-inspired soft robotic arm that functions as a physical reservoir computer. After grasping an object, the arm is excited by a fixed shaking input at its base, and the resulting ringdown response is recorded through either camera tracking or embedded sensors. Because the input is held constant across trials, hidden object properties, such as the COM position, are encoded through their effect on the dynamics of the coupled robot-object system. A lightweight linear readout can then decode these dynamics to recover interpretable information about the hidden object physics. Using this framework, the soft robotic arm reservoir completed three tasks of increasing difficulty: inferring the orientation of the object's hidden COM, inferring the COM distance from the grasp point, and using the inferred COM information to guide a subsequent regrasp. We further develop a dynamic summary representation of the ringdown response that improves prediction accuracy. Together, these results establish shake-to-learn mechanical interrogation as a promising strategy for robotic systems to convert brief physical interactions into actionable cues about hidden object properties for downstream manipulation.
Figures & tables
Figure 1: ”Shake-to-learn” dynamic interrogation —an overview of the proposed robotic learning modality based on interaction-induced dynamics. ( A ) The approach draws inspiration from how children physically manipulate an object to elicit dynamic cues and uses them to infer what’s inside. ( B ) In the robotic analogue, a soft robotic arm—doubling as a physical reservoir computer—grasps an object whose physical state is visually hidden, delivers a brief dynamic probe, and records the resulting body response. The reservoir computing power embodied in the soft arm projects the robot–object interaction into a separable, high-dimensional ringdown response (i.e., reservoir states), from which a linear readout extracts the hidden physical properties. Here, we focus specifically on the object’s center-of-mass (COM) position and how it shifts.
Figure 2: Experiment setup for dynamic interrogation . ( A ) Anatomy of the modular soft robotic arm/physical reservoir. ( B ) 3D-printed cross-shaped container, showing the hidden payload mass m and the corresponding coordinate setup. ( C ) 3D-printed bar-shaped container. ( D ) The ringdown test procedure on a three-module robotic arm using camera-based motion tracking. ( E ) Another test on a four-module arm using embedded bending strain sensors.
Figure 3: Predicting the angular orientation of the hidden COM. ( A–C ) Experimental snapshot, tracked marker displacement, and confusion matrix using external camera tracking. With a 100 g payload, the camera-tracking method yields 44/48 correct leave-one-out predictions. ( D–F ) Experimental snapshot, recorded sensor reading, and confusion matrix using embedded bending sensors. With a 100 g payload, the embedded bending sensors yield 78/80 correct leave-one-out predictions. ( G ) With 40 g and 20 g payloads, the camera-tracking strategy yields 44/48 and 41/48 correct predictions, respectively. ( H ) With 40 g and 20 g payloads, the embedded sensors yield 68/80 and 54/80 correct predictions, respectively. ( I, J ) Frequency-resolved spectral separability of the ringdown response across different COM orientations. A larger Jm(f) indicates that the four COM-orientation classes produce more distinguishable frequency-domain ringdown signatures, while the area under the curve, Am , summarizes the total separable spectral content across the frequency band. Trends in Jm(f) and Am support the interpretation that lighter payloads produce weaker orientation-dependent spectral signatures in the shake-induced ringdown.
Figure 4: Predicting the offset of hidden COM relative to the grasping point. ( A ) A robotic grasp centered on an object can perform poorly if the true COM is located far away—a common challenge in realistic robotic manipulation tasks. ( B ) Definition of the dynamic summary, illustrated on a raw ringdown response from an embedded sensor. The early and late windows are shaded with different colors. ( C ) The physical reservoir’s prediction of COM position using raw sensor data (left) and the dynamic summary (right). The prediction performance is also assessed using the ”Leave-One-Out” (LOO) protocol: we conducted 12 experimental trials, all with true center-of-mass (COM) positions at +2 or −2 . We held out one trial for testing and used the remaining trials for readout training; the outcome yˉ(i) for this held-out trial is shown as a marker in the plot. To classify whether this yˉ(i) prediction is correct, we also apply nearest-target decoding o^(i)=argminyˉ(i)−co2 , where o=±2,±3 in this Task II. We repeated this process until every trial had been used for testing once. ( D ) Analogous test results where the true COM is at +3 or −3 . ( E ) An additional set of test results illustrating interpolation. Here, the readout is trained using trials with COM at ±3 only, but the testing trials can have COM at ±2 or ±3 . ( F ) PLS analysis of raw sensor data (up) and dynamic-summary features (bottom).
Figure 5: Object grasping and re-grasing according inferred center-of-mass positions. ( A ) During the first grasp, the robot starts from standby, grasps the bar-shaped container at its geometric center, then shakes the bar to infer the center-of-mass position (at o=+4 in this case). ( B ) In the second grasp, the robot returns to standby, bends towards the inferred COM position, and then grasp and lift the container again. The bar-container’s angle decreased from 43.15∘ to 21.51∘ . ( C ) Readout accuracy for the COM-offset unseen in training. The +4 condition is repeated ten times to evaluate whether the readout trained on o=±3 data gives a consistent prediction.
Figure 6: Exploratory Studies for Future Investigations . ( A ) Quantitatively accurate COM-distance prediction. The readout weights are trained on o=−3,−1,+1,+3 and tested on held-out o=−2,+2 trails. The results reaches 55% exact-cell accuracy, above the seven-cell chance level of 14.3% . For target COM positions at −2 and +2 , the predictions were −2.5±0.22 and 1.6±0.16 , respectively. ( B ) The overall prediction accuracy changes according the the length of the ringdown dynamics recording.
In animals such as elephants and octopuses, acquiring non-visual information about an object and physically engaging with it are inseparable processes mediated by rich, large-area interactions between compliant appendages and the environment. Soft robots provide a natural platform for translating this principle into engineered systems. Yet current robotic intelligence makes limited use of physical interaction, treating it primarily as a disturbance to be rejected or, at best, as a means of compensating for object misalignment. Here, we introduce a physical intelligence framework in which distributed compliant interactions jointly reveal task-relevant information and organize manipulation behavior. This results in an intrinsically partially observable problem: key task-relevant information is never measured directly, but must instead be inferred from the history of physical interactions. We propose a reinforcement-learning architecture that addresses this challenge by learning a memory-based control policy end-to-end. The key innovations making this possible are (i) a pretrained exploration policy that provides a reference for broad workspace exploration, (ii) joint optimization that integrates exploration and grasping objectives within a single recurrent policy, and (iii) a two-stage sim-to-real adaptation including observation mapping and policy fine-tuning. We demonstrate this principle through blind whole-arm grasping with a hybrid rigid-soft robotic arm that we equip with IMUs embedded directly within its compliant structure, providing its only source of proprioceptive sensing. The learned policy successfully identifies and grasps various objects by autonomously coordinating workspace exploration, object encounter and localization, inference of grasp-relevant properties, and stable whole-arm wrapping.
Cognitive Robotics Department, Faculty of Mechanical Engineering, Delft University of Technology, Delft 2628 CD, The Netherlands. · School of Engineering, Newcastle University, UK.
We present PhysCaP, a Physics-Informed Code-as-Policy agent system for active perception in robotic manipulation. While vision-language-action policies excel at imitating demonstrations, they rely on passive observation and fail to infer latent physical properties critical for manipulation. PhysCaP augments code-as-policy frameworks with a physics-informed exploration layer that enables explicit information-seeking through interaction. Our method introduces training-free physical property extraction modules that estimate object mass and stiffness from robot proprioception without additional sensors. To balance exploration costs and the efficiency of information obtained, PhysCaP employs a multi-agent design: a Planner that decides when to explore and when to stop, and a Prioritizer that filters implausible interactions and ranks the remainder using a heuristic priority score, enabling efficient, targeted exploration. We evaluate PhysCaP on three real-world tabletop manipulation tasks and a simulated task in LIBERO. The results show that existing passive and naive interactive baselines either fail when physical properties are hidden or over-explore, whereas PhysCaP achieves comparable performance with fewer interactions and reduced execution time. Ablation studies further validate the effectiveness of the proposed physical property extraction modules. Project page: https://physcap.github.io
Chen-Yu Lin, Jing-Wen Chen, Hsueh-En Chang +8
National Taiwan University · NVIDIA Research · National Yang Ming Chiao Tung University
Recovering physical properties of unknown objects through non-prehensile interaction is challenging because no single manipulation primitive reveals all relevant parameters. Planar pushing couples mass and friction, while conventional tipping cannot recover friction and may fail entirely when low-friction or curved-base objects slide or rotate instead of tipping. We introduce a multi-modal estimation framework that combines a sliding interaction with a press-and-pull tipping primitive to recover object mass, center of mass height, and surface friction. The press-and-pull interaction increases the object-table sliding threshold and stabilizes the pivot, enabling controlled tipping without any prior geometric object model. Wrist force/torque sensing, RGB-D perception, and robot proprioception are fused to estimate the physical parameters from the two complementary interaction modes. Experiments on an ABB IRB120 across four objects with varied geometry, mass, center of mass, and friction achieve low relative error, while successfully operating on curved-base objects that fail under conventional forward tipping. The results demonstrate that complementary non-prehensile interactions can recover a compact set of physical parameters without grasping, a prior object model, or learned interaction dynamics.