Authors: Haonan Chen, Feiyang Wu, Yuxiang Ma, Mustafa Mete, Pengfei Ye, Junxuan Shen, Cheng Zhu, Aurora Ruggeri, +6 more
Organizations: Harvard University, Cambridge, MA, USA. · Massachusetts Institute of Technology, Cambridge, MA, USA. · University of Pennsylvania, Philadelphia, PA, USA. · Stanford University, Stanford, CA, USA.
Force-aware manipulation typically relies on specialized force or tactile sensors. We show that force-aware manipulation can instead be achieved through visual force prediction from the deformation of a compliant Fin Ray gripper. Our approach trains two models. First, we train a visual force estimator on calibration data and use it to annotate task demonstrations with force estimates. Second, we train an action--force proposal policy on these force-augmented demonstrations to jointly generate candidate robot actions and their associated forces. At test time, we sample candidate actions and the forces they are expected to produce, then execute the action whose predicted force is closest to a target from the demonstrations. We evaluate our approach on berry picking, empty-can grasping, in-hand reorientation, and plug insertion. Our results show that visual force prediction can guide inference-time action selection for contact-rich manipulation without requiring force or tactile sensors at deployment.
Figures & tables
Fig. 1: Force-aware manipulation from visual gripper deformation. Humans naturally regulate force when handling fragile objects (left). We use wrist-camera observations of a compliant gripper (right, inset) to infer interaction force from visible deformation and enable gentle robotic manipulation without dedicated force or tactile sensing at deployment.
Fig. 2: Overview of visual force estimation and inference-time force search. (a) Visual force estimation converts a wrist-camera image into a masked Sobel edge representation and uses gϕ to estimate the axial interaction force from the visible deformation of the compliant Fin Ray gripper. (b) During inference-time force search, the policy jointly samples candidate end-effector actions and force values. We execute the action from the sample whose generated force value most closely matches the target force.
Fig. 3: Hardware setups used in our experiments. (a) Visual force data collection setup, where an Intel RealSense D405 camera observes the deformation of the hand-held Fin Ray gripper and a six-axis F/T sensor records the ground-truth interaction force through 3D-printed contact fixtures. (b) The six-axis F/T sensor and 3D-printed contact fixtures used to record ground-truth interaction force during data collection. (c) Robotic manipulation platform used for policy evaluation, consisting of an ARX robotic arm equipped with wrist and side-view cameras.
Fig. 4: Force-estimation accuracy. (a) Reference load cell. (b) Force trajectories. (c) Estimated versus measured force (three episodes, 442 frames); dashed line: perfect agreement. Mean episode MAE: motor current, 4.20N ; ViewForce, 0.91N (equal episode weights).
Fig. 5: Manipulation success rates. Success rate (%) over 15 trials per task for single-sample policy inference and our force-search method, both using the same policy checkpoint.
Fig. 6: Representative failure modes under single-sample policy inference. From top left to bottom right: a dented empty soda can, a crushed berry, unsuccessful in-hand reorientation, and plug slippage within the gripper. These examples correspond to the four tasks reported in Fig. 5 .
Fig. 7: Representative successful rollouts of four force-sensitive manipulation tasks using force-based candidate selection. Each row shows a task sequence progressing from left to right. For empty soda can grasping and berry picking, the robot applies sufficient gripping force to lift and transfer the object without crushing it. During in-hand reorientation, the robot maintains contact while allowing the metal plate to pivot into an upright configuration. During plug insertion, the robot maintains sufficient contact friction to prevent slippage and fully inserts the plug into the socket.
Fig. 8: Force Search. Distribution of 32 force predictions sampled from the policy for a held-out soda-can observation. The teal star marks the prediction selected by force search ( 4.45N ). Points are offset vertically for readability.