Authors: Haonan Chen, Feiyang Wu, Yuxiang Ma, Mustafa Mete, Pengfei Ye, Junxuan Shen, Cheng Zhu, Aurora Ruggeri, +6 more
Organizations: Harvard University, Cambridge, MA, USA. · Massachusetts Institute of Technology, Cambridge, MA, USA. · University of Pennsylvania, Philadelphia, PA, USA. · Stanford University, Stanford, CA, USA.
Force-aware manipulation typically relies on specialized force or tactile sensors. We show that force-aware manipulation can instead be achieved through visual force prediction from the deformation of a compliant Fin Ray gripper. Our approach trains two models. First, we train a visual force estimator on calibration data and use it to annotate task demonstrations with force estimates. Second, we train an action--force proposal policy on these force-augmented demonstrations to jointly generate candidate robot actions and their associated forces. At test time, we sample candidate actions and the forces they are expected to produce, then execute the action whose predicted force is closest to a target from the demonstrations. We evaluate our approach on berry picking, empty-can grasping, in-hand reorientation, and plug insertion. Our results show that visual force prediction can guide inference-time action selection for contact-rich manipulation without requiring force or tactile sensors at deployment.
Figures & tables
Fig. 1: Force-aware manipulation from visual gripper deformation. Humans naturally regulate force when handling fragile objects (left). We use wrist-camera observations of a compliant gripper (right, inset) to infer interaction force from visible deformation and enable gentle robotic manipulation without dedicated force or tactile sensing at deployment.
Fig. 2: Overview of visual force estimation and inference-time force search. (a) Visual force estimation converts a wrist-camera image into a masked Sobel edge representation and uses gϕ to estimate the axial interaction force from the visible deformation of the compliant Fin Ray gripper. (b) During inference-time force search, the policy jointly samples candidate end-effector actions and force values. We execute the action from the sample whose generated force value most closely matches the target force.
Fig. 3: Hardware setups used in our experiments. (a) Visual force data collection setup, where an Intel RealSense D405 camera observes the deformation of the hand-held Fin Ray gripper and a six-axis F/T sensor records the ground-truth interaction force through 3D-printed contact fixtures. (b) The six-axis F/T sensor and 3D-printed contact fixtures used to record ground-truth interaction force during data collection. (c) Robotic manipulation platform used for policy evaluation, consisting of an ARX robotic arm equipped with wrist and side-view cameras.
Fig. 4: Force-estimation accuracy. (a) Reference load cell. (b) Force trajectories. (c) Estimated versus measured force (three episodes, 442 frames); dashed line: perfect agreement. Mean episode MAE: motor current, 4.20N ; ViewForce, 0.91N (equal episode weights).
Fig. 5: Manipulation success rates. Success rate (%) over 15 trials per task for single-sample policy inference and our force-search method, both using the same policy checkpoint.
Fig. 6: Representative failure modes under single-sample policy inference. From top left to bottom right: a dented empty soda can, a crushed berry, unsuccessful in-hand reorientation, and plug slippage within the gripper. These examples correspond to the four tasks reported in Fig. 5 .
Fig. 7: Representative successful rollouts of four force-sensitive manipulation tasks using force-based candidate selection. Each row shows a task sequence progressing from left to right. For empty soda can grasping and berry picking, the robot applies sufficient gripping force to lift and transfer the object without crushing it. During in-hand reorientation, the robot maintains contact while allowing the metal plate to pivot into an upright configuration. During plug insertion, the robot maintains sufficient contact friction to prevent slippage and fully inserts the plug into the socket.
Fig. 8: Force Search. Distribution of 32 force predictions sampled from the policy for a held-out soda-can observation. The teal star marks the prediction selected by force search ( 4.45N ). Points are offset vertically for readability.
Accurate pre-contact grasp force selection is critical for safe and reliable robotic manipulation. Adaptive controllers regulate force after contact but still require a reasonable initial estimate. Starting a grasp with too little force requires reactive adjustment, while starting a grasp with too high a force risks damaging fragile objects. This trade-off is particularly challenging for compliant grippers, whose contact mechanics are difficult to model analytically. We propose Exp-Force, an experience-conditioned framework that predicts the minimum feasible grasping force from a single RGB image. The method retrieves a small set of relevant prior grasping experiences and conditions a vision-language model on these examples for in-context inference, without analytic contact models or manually designed heuristics. On 129 object instances, ExpForce achieves a best-case MAE of 0.43 N, reducing error by 72% over zero-shot inference. In real-world tests on 30 unseen objects, it improves appropriate force selection rate from 63% to 87%. These results demonstrate that Exp-Force enables reliable and generalizable pre-grasp force selection by leveraging prior interaction experiences. http://expforcesubmission.github.io/Exp-Force-Website/
Contact-rich manipulation requires policies to generate precise actions by reasoning over contact forces, robot configurations, and interaction histories beyond visual observations. Existing methods passively condition on force feedback rather than actively predicting future contact dynamics, limiting their ability to generate high-precision actions. To address this problem, we introduce Predictive Sensorimotor Representation (PSR) learning, a framework that learns a hierarchy of predictive representations from multimodal sensorimotor signals and integrates them into the action stream of a visuomotor policy. Specifically, during a pretraining stage, a multimodal Transformer is trained to learn a hierarchy of predictive representations by jointly forecasting future interaction dynamics. The learned hierarchy subsequently augments the action stream, enabling the resulting policy to exploit contact-relevant cues at multiple depths. We further instantiate PSR within a Vision-Language-Action (VLA) model, resulting in PSR-VLA, and evaluate it on six real-world contact-rich manipulation tasks. Experimental results show that PSR-VLA achieves 91.7% overall success, improving over π0.5, ForceVLA-π0.5, and ForceVLA2-π0.5 by 30.0, 22.5, and 19.2 percentage points, respectively. These results demonstrate the effectiveness of the proposed PSR for force-aware, contact-rich manipulation. Videos of the tasks and stability tests are available at https://psr-vla.pages.dev/.
Vision-Language-Action (VLA) models have emerged as general-purpose robotic manipulation policies. However, in dexterous hand manipulation, contact forces are typically provided as separate states or force-specific representations, making it difficult to explicitly represent the spatial correspondence between force and their corresponding visual locations. In this work, we propose VisForce, which visually grounds the current and desired forces at their corresponding fingertip locations. VisForce renders current and desired visual force cues on the current wrist image and a task-specific goal image, and combines the two representations through goal-conditioned cross-attention to generate force-aware actions. We evaluate VisForce using a real UR10 robot equipped with an RH56F1 dexterous hand through force-conditioned grasping and three multi-stage manipulation tasks. In force-conditioned grasping experiments, VisForce exhibited a consistent grip-force response as the desired force increased, and achieved grasp-and-lift success rates of 70% and 80% for an egg and a toothpaste tube, respectively. It further achieved final success rates of 70%, 55%, and 40% on cup insertion/bottle pouring, tong-assisted bread transfer, and slip-modulated peg-in-hole, respectively. These results show that fingertip-aligned visual force representations can be effectively used for force-aware conditioning in VLA-based dexterous hand manipulation.