Exp-Force: Experience-Conditioned Pre-Grasp Force Selection with Vision-Language Models

Siqi Shang; Minchao Huang; Bill Fan; Lillian Chin

cs.ROMar 9, 2026

Exp-Force: Experience-Conditioned Pre-Grasp Force Selection with Vision-Language Models

Authors: Siqi Shang, Minchao Huang, Bill Fan, Lillian Chin

Organizations: The University of Texas at Austin, TX, USA.

Abstract

Accurate pre-contact grasp force selection is critical for safe and reliable robotic manipulation. Adaptive controllers regulate force after contact but still require a reasonable initial estimate. Starting a grasp with too little force requires reactive adjustment, while starting a grasp with too high a force risks damaging fragile objects. This trade-off is particularly challenging for compliant grippers, whose contact mechanics are difficult to model analytically. We propose Exp-Force, an experience-conditioned framework that predicts the minimum feasible grasping force from a single RGB image. The method retrieves a small set of relevant prior grasping experiences and conditions a vision-language model on these examples for in-context inference, without analytic contact models or manually designed heuristics. On 129 object instances, ExpForce achieves a best-case MAE of 0.43 N, reducing error by 72% over zero-shot inference. In real-world tests on 30 unseen objects, it improves appropriate force selection rate from 63% to 87%. These results demonstrate that Exp-Force enables reliable and generalizable pre-grasp force selection by leveraging prior interaction experiences. http://expforcesubmission.github.io/Exp-Force-Website/

Figures & tables

Fig. 1: Overview . Exp-Force uses VLMs to accurately predict appropriate grasp forces using a single object-image, prior to grasp contact. We do this by providing a VLM with a small set of related experiences for in-context learning. When grasping an opened Cheez-It box, a zero-shot VLM overestimates the grasp force by 2.75x, crushing the box, while Exp-Force can lift the box without damaging it.

Fig. 2: Exp-Force architecture. A single target object image $I_{o}$ from the wrist camera is first fed into the descriptor VLM to generate a text description $T_{o}$ of the object’s physical properties related to robot grasping. Together with the object image, the text description is fed into the embedding model to obtain the object embedding $z_{o}$ . The embedding is used to perform a cosine-similarity search against the Experience Pool. The top- $k$ examples are retrieved and supplied to the predictor VLM as in-context examples for estimating the grasping force $\hat{F}_{o}$ . Each example contains the object name, mass, description, image, and ground-truth $F^{\star}$ .

Fig. 3: Evaluation objects, categories, and model performance across categories. (A) Representative objects used in our experiments, categorized by geometry, fragility, and weight. All scale bars are 20cm. (B) Objects weight versus minimum feasible grasping force $F^{\star}$ . Dashed lines indicate bounds for the coefficient of friction $\mu$ that the zero-shot model uses in its force estimation. (C) Comparison of the Mean Absolute Error between zero-shot versus Exp-Force in each of the six object categories, across three vision-language models (GPT 5.2, Gemini-3-Flash, and Gemini-3.1-Pro). For each model, we chose the best $k$ value from our in-context sample efficiency ablation (Sec. IV-D ). The total count of objects within each category is also reported.

Fig. 4: Real-world experiment setup. (A) The compliant tactile two-finger gripper with a wrist-mounted camera used for pre-grasp image capture and force execution. (B) Five selected objects from each of the six categories. All scale bars are 10cm.

Fig. 5: Comparison of force prediction errors across three vision–language models (GPT-5.2, Gemini-3-flash, and Gemini-3.1-pro). Lower values correspond to more accurate force estimation. For the multimodal baseline, we utilize Qwen3-VL-Embedding (denoted as Qwen). For each model, distribution bars summarize the error statistics over five-fold cross-validation, where the thin whiskers denote one standard deviation.

Fig. 6: Effect of the retrieved example size (

k

) on force estimation error. Our evaluation reveals that Exp-Force exhibits remarkable in-context sample efficiency. The plot demonstrates that, across all predictor VLMs, the Mean Absolute Error (MAE) rapidly converges as the model switches from zero-shot inference (

k=0

) to experience-conditioned inference. Performance plateaus early, typically between

k=5

and

k=10

, highlighting the high sample efficiency of Exp-Force. Shaded regions denote the standard deviation across cross-validation folds, illustrating that the Gemini models maintain superior estimation stability and tighter error bounds compared to GPT-5.2.

Fig. 7: Snapshots of representative trials . We compare grasp outcomes of a Pringle chip, a champagne glass, and a banana, using forces generated by both our zero-shot baseline and Exp-Force with Gemini-3-flash and k=7 from the same image. The chip grasp is representative of a typical successful grasp. The Champagne Glass and Banana correspond respectively to typical failure modes - the zero-experience model overestimating, and our method under-estimating. All scale bars are 5cm.

Category	Appr. [%]	Overest. [%]	Insuff. [%]	MAE $\pm$ STD [N]	Appr. [%]	Overest. [%]	Insuff. [%]	MAE $\pm$ STD [N]
	Zero-shot Baseline				Exp-Force (k=7)
Bottles	64	32	4	3.8 $\pm$ 2.9	92 (+44%)	0	8	1.08 $\pm$ 0.76 (-71%)
Cylinders	60	32	8	5.6 $\pm$ 3.3	80 (+33%)	0	20	0.62 $\pm$ 0.73 (-89%)
Cuboids	64	36	0	5.0 $\pm$ 5.0	100 (+56%)	0	0	0.81 $\pm$ 0.92 (-84%)
Fragile–Heavy	56	44	0	3.1 $\pm$ 0.95	84 (+50.0%)	0	16	0.11 $\pm$ 0.10 (-97%)
Fragile–Light	80	20	0	0.47 $\pm$ 0.81	96 (+20.0%)	4	0	0.03 $\pm$ 0.03 (-94%)

TABLE I: Real-world grasping performance using Gemini-3-Flash

Explore similar work

Oct 3, 2026

Robot Learning with Visual Predicted Force

Force-aware manipulation typically relies on specialized force or tactile sensors. We show that force-aware manipulation can instead be achieved through visual force prediction from the deformation of a compliant Fin Ray gripper. Our approach trains two models. First, we train a visual force estimator on calibration data and use it to annotate task demonstrations with force estimates. Second, we train an action--force proposal policy on these force-augmented demonstrations to jointly generate candidate robot actions and their associated forces. At test time, we sample candidate actions and the forces they are expected to produce, then execute the action whose predicted force is closest to a target from the demonstrations. We evaluate our approach on berry picking, empty-can grasping, in-hand reorientation, and plug insertion. Our results show that visual force prediction can guide inference-time action selection for contact-rich manipulation without requiring force or tactile sensors at deployment.

Haonan Chen, Feiyang Wu, Yuxiang Ma +11

Harvard University, Cambridge, MA, USA. · Massachusetts Institute of Technology, Cambridge, MA, USA. · University of Pennsylvania, Philadelphia, PA, USA. +1

Robotic Manipulation Contact-Rich Robotic Manipulation Force-Controlled Robotic Manipulation

Ask PDF

Sep 21, 2026

GraspTune: Tactile-Driven Execution Refinement for Robust Grasping

Visual grasp proposal generation has advanced rapidly, yet converting a selected proposal into a stable physical grasp remains a central execution-stage challenge. This paper introduces GraspTune, a tactile-driven execution-stage refinement framework that starts from a nominal proposal and applies bounded residual TCP motions during approach, contact formation, and final grasp execution. GraspTune learns control-facing contact semantics from local depth, tactile signals, state, and history using state-conditioned expert contact queries and multi-task supervision for contact change, contact risk, and post-close readiness. The representation conditions a diffusion-pretrained residual policy and is aligned with PPO for closed-loop execution. Across more than 60,000 simulated executions over 20 object categories, GraspTune establishes an execution-layer benefit across four proposal generators, raising stable grasp success by +19.22, +9.55, +12.45, and +20.70 percentage points for GraspNet, Contact-GraspNet, AnyGrasp, and VGN. A four-fold held-out category study raises unseen-object execution from 54.58% to 70.33%, showing category-disjoint generalization of contact correction. Across more than 1,000 real-robot trials on a UR5e setup with Xense fingertip sensors, GraspTune raises GraspNet execution from 71.0% to 84.3%, validating direct transfer without realworld policy fine-tuning. Together, these results turn visually plausible proposals into stable physical grasps for downstream contact-rich manipulation. A supplementary video is available at https://youtu.be/kcq7fSLNtzU.

Juntao Li, Xingke Xia, Sichao Liu +1

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · KTH Royal Institute of Technology, Stockholm, Sweden

Robotic Grasping Generalization in Robotic Manipulation Tactile Sensing for Robotic Manipulation

Ask PDF

May 1, 2026

A Model-based Visual Contact Localization and Force Sensing System for Compliant Robotic Grippers

Grasp force estimation can help prevent robots from damaging delicate objects during manipulation and improve learning-based robotic control. Integrating force sensing into deformable grippers negotiates trade-offs in cost, complexity, mechanical robustness, and performance. With the growing integration of RGB-D wrist cameras into robotic systems for control purposes, camera-based techniques are a promising solution for indirect visual force estimation. Current approaches mostly utilize end-to-end deep learning, which can be brittle when generalizing to new scenarios, while existing model-based approaches are unsuited to grasping and modern grasper geometries. To address these challenges, we developed a model-based visual force sensing approach integrating an iterative contact localization with generalization to unseen objects. The system extracts structural key points from wrist camera RGB-D images of deforming fin-ray-shaped soft grippers, and uses these key points to define parameters of an inverse finite element analysis simulation in Simulation Open Framework Architecture. The iterative contact localization sub-system utilizes a deep learning-based online 3D reconstruction and pose estimation pipeline to dynamically update contact location, and is robust to visual occlusion and unseen objects. Our system demonstrated an average root mean square error of 0.23 N and normalized root mean square deviation of 2.11% during the load phase, and 0.48 N and 4.34% over the entire grasping process when interacting with different objects under various conditions, showcasing its potential for real-time model-based indirect force sensing of soft grippers.

Kaiwen Zuo, Shuyuan Yang, Zonghe Chua

Department of Electrical, Computer, and Systems Engineering, Case Western Reserve University, Cleveland, OH 44106, USA

Robotic Grasping Robotic Manipulation Soft Robotics

Ask PDF