Exp-Force: Experience-Conditioned Pre-Grasp Force Selection with Vision-Language Models

Siqi Shang; Minchao Huang; Bill Fan; Lillian Chin

cs.ROMar 9, 2026

Exp-Force: Experience-Conditioned Pre-Grasp Force Selection with Vision-Language Models

Authors: Siqi Shang, Minchao Huang, Bill Fan, Lillian Chin

Organizations: The University of Texas at Austin, TX, USA.

Abstract

Accurate pre-contact grasp force selection is critical for safe and reliable robotic manipulation. Adaptive controllers regulate force after contact but still require a reasonable initial estimate. Starting a grasp with too little force requires reactive adjustment, while starting a grasp with too high a force risks damaging fragile objects. This trade-off is particularly challenging for compliant grippers, whose contact mechanics are difficult to model analytically. We propose Exp-Force, an experience-conditioned framework that predicts the minimum feasible grasping force from a single RGB image. The method retrieves a small set of relevant prior grasping experiences and conditions a vision-language model on these examples for in-context inference, without analytic contact models or manually designed heuristics. On 129 object instances, ExpForce achieves a best-case MAE of 0.43 N, reducing error by 72% over zero-shot inference. In real-world tests on 30 unseen objects, it improves appropriate force selection rate from 63% to 87%. These results demonstrate that Exp-Force enables reliable and generalizable pre-grasp force selection by leveraging prior interaction experiences. http://expforcesubmission.github.io/Exp-Force-Website/

Figures & tables

Fig. 1: Overview . Exp-Force uses VLMs to accurately predict appropriate grasp forces using a single object-image, prior to grasp contact. We do this by providing a VLM with a small set of related experiences for in-context learning. When grasping an opened Cheez-It box, a zero-shot VLM overestimates the grasp force by 2.75x, crushing the box, while Exp-Force can lift the box without damaging it.

Fig. 2: Exp-Force architecture. A single target object image $I_{o}$ from the wrist camera is first fed into the descriptor VLM to generate a text description $T_{o}$ of the object’s physical properties related to robot grasping. Together with the object image, the text description is fed into the embedding model to obtain the object embedding $z_{o}$ . The embedding is used to perform a cosine-similarity search against the Experience Pool. The top- $k$ examples are retrieved and supplied to the predictor VLM as in-context examples for estimating the grasping force $\hat{F}_{o}$ . Each example contains the object name, mass, description, image, and ground-truth $F^{\star}$ .

Fig. 3: Evaluation objects, categories, and model performance across categories. (A) Representative objects used in our experiments, categorized by geometry, fragility, and weight. All scale bars are 20cm. (B) Objects weight versus minimum feasible grasping force $F^{\star}$ . Dashed lines indicate bounds for the coefficient of friction $\mu$ that the zero-shot model uses in its force estimation. (C) Comparison of the Mean Absolute Error between zero-shot versus Exp-Force in each of the six object categories, across three vision-language models (GPT 5.2, Gemini-3-Flash, and Gemini-3.1-Pro). For each model, we chose the best $k$ value from our in-context sample efficiency ablation (Sec. IV-D ). The total count of objects within each category is also reported.

Fig. 4: Real-world experiment setup. (A) The compliant tactile two-finger gripper with a wrist-mounted camera used for pre-grasp image capture and force execution. (B) Five selected objects from each of the six categories. All scale bars are 10cm.

Fig. 5: Comparison of force prediction errors across three vision–language models (GPT-5.2, Gemini-3-flash, and Gemini-3.1-pro). Lower values correspond to more accurate force estimation. For the multimodal baseline, we utilize Qwen3-VL-Embedding (denoted as Qwen). For each model, distribution bars summarize the error statistics over five-fold cross-validation, where the thin whiskers denote one standard deviation.

Fig. 6: Effect of the retrieved example size (

k

) on force estimation error. Our evaluation reveals that Exp-Force exhibits remarkable in-context sample efficiency. The plot demonstrates that, across all predictor VLMs, the Mean Absolute Error (MAE) rapidly converges as the model switches from zero-shot inference (

k=0

) to experience-conditioned inference. Performance plateaus early, typically between

k=5

and

k=10

, highlighting the high sample efficiency of Exp-Force. Shaded regions denote the standard deviation across cross-validation folds, illustrating that the Gemini models maintain superior estimation stability and tighter error bounds compared to GPT-5.2.

Fig. 7: Snapshots of representative trials . We compare grasp outcomes of a Pringle chip, a champagne glass, and a banana, using forces generated by both our zero-shot baseline and Exp-Force with Gemini-3-flash and k=7 from the same image. The chip grasp is representative of a typical successful grasp. The Champagne Glass and Banana correspond respectively to typical failure modes - the zero-experience model overestimating, and our method under-estimating. All scale bars are 5cm.

Category	Appr. [%]	Overest. [%]	Insuff. [%]	MAE $\pm$ STD [N]	Appr. [%]	Overest. [%]	Insuff. [%]	MAE $\pm$ STD [N]
	Zero-shot Baseline				Exp-Force (k=7)
Bottles	64	32	4	3.8 $\pm$ 2.9	92 (+44%)	0	8	1.08 $\pm$ 0.76 (-71%)
Cylinders	60	32	8	5.6 $\pm$ 3.3	80 (+33%)	0	20	0.62 $\pm$ 0.73 (-89%)
Cuboids	64	36	0	5.0 $\pm$ 5.0	100 (+56%)	0	0	0.81 $\pm$ 0.92 (-84%)
Fragile–Heavy	56	44	0	3.1 $\pm$ 0.95	84 (+50.0%)	0	16	0.11 $\pm$ 0.10 (-97%)
Fragile–Light	80	20	0	0.47 $\pm$ 0.81	96 (+20.0%)	4	0	0.03 $\pm$ 0.03 (-94%)

TABLE I: Real-world grasping performance using Gemini-3-Flash

Explore similar work

Robot Learning with Visual Predicted Force
Oct 3, 2026Haonan Chen, Feiyang Wu, Yuxiang Ma +11Robotic Manipulation Contact-Rich Robotic Manipulation
Harvard University, Cambridge, MA, USA. · Massachusetts Institute of Technology, Cambridge, MA, USA. · University of Pennsylvania, Philadelphia, PA, USA. +1
GraspTune: Tactile-Driven Execution Refinement for Robust Grasping
Sep 21, 2026Juntao Li, Xingke Xia, Sichao Liu +1Robotic Grasping Generalization in Robotic Manipulation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · KTH Royal Institute of Technology, Stockholm, Sweden
A Model-based Visual Contact Localization and Force Sensing System for Compliant Robotic Grippers
May 1, 2026Kaiwen Zuo, Shuyuan Yang, Zonghe ChuaRobotic Grasping Robotic Manipulation
Department of Electrical, Computer, and Systems Engineering, Case Western Reserve University, Cleveland, OH 44106, USA

Oct 3, 2026J/K move · Enter open · S save

Robot Learning with Visual Predicted Force

Haonan Chen, Feiyang Wu, Yuxiang Ma, Mustafa Mete+10

Harvard University, Cambridge, MA, USA. · Massachusetts Institute of Technology, Cambridge, MA, USA. · University of Pennsylvania, Philadelphia, PA, USA.+1

Robotic Manipulation Contact-Rich Robotic Manipulation Force-Controlled Robotic Manipulation

Open paper AskPDF arXiv

Force-aware manipulation typically relies on specialized force or tactile sensors. We show that force-aware manipulation can instead be achieved through visual force prediction from the deformation of a compliant Fin Ray gripper. Our approach trains two models. First, we train a visual force estimator on calibration data and use it to annotate task demonstrations with force estimates. Second, we train an action--force proposal policy on these force-augmented demonstrations to jointly generate candidate robot actions and their associated forces. At test time, we sample candidate actions and the forces they are expected to produce, then execute the action whose predicted force is closest to a target from the demonstrations. We evaluate our approach on berry picking, empty-can grasping, in-hand reorientation, and plug insertion. Our results show that visual force prediction can guide inference-time action selection for contact-rich manipulation without requiring force or tactile sensors at deployment.

Abstract

Explore similar work

Robot Learning with Visual Predicted Force

GraspTune: Tactile-Driven Execution Refinement for Robust Grasping

A Model-based Visual Contact Localization and Force Sensing System for Compliant Robotic Grippers