FG-CLTP: Fine-Grained Contrastive Language Tactile Pretraining for Robotic Manipulation

Wenxuan Ma; Chaofan Zhang; Yinghao Cai; Guocai Yao; Shaowei Cui; Shuo Wang

cs.ROMar 11, 2026

FG-CLTP: Fine-Grained Contrastive Language Tactile Pretraining for Robotic Manipulation

Authors: Wenxuan Ma, Chaofan Zhang, Yinghao Cai, Guocai Yao, Shaowei Cui, Shuo Wang

Organizations: Institute of Automation, Chinese Academy of Sciences · Beijing Academy of Artificial Intelligence

Abstract

Recent advancements in integrating tactile sensing into vision-language-action (VLA) models have demonstrated transformative potential for robotic perception. However, existing tactile representations predominantly rely on qualitative descriptors (e.g., texture), neglecting quantitative contact states such as force magnitude, contact geometry, and principal axis orientation, which are indispensable for fine-grained manipulation. To bridge this gap, we propose FG-CLTP, a fine-grained contrastive language tactile pretraining framework. We first introduce a novel dataset comprising over 100k tactile 3D point cloud-language pairs that explicitly capture multidimensional contact states from the sensor's perspective. We then implement a discretized numerical tokenization mechanism to achieve quantitative-semantic alignment, effectively injecting explicit physical metrics into the multimodal feature space. The proposed FG-CLTP model yields a 95.9% classification accuracy and reduces the regression error (MAE) by 52.6% compared to state-of-the-art methods. Furthermore, the integration of 3D point cloud representations establishes a sensor-agnostic foundation with a minimal sim-to-real gap of 3.5%. Building upon this fine-grained representation, we develop a 3D tactile-language-action (3D-TLA) architecture driven by a flow matching policy to enable multimodal reasoning and control. Extensive experiments demonstrate that our framework significantly outperforms strong baselines in contact-rich manipulation tasks, providing a robust and generalizable foundation for tactile-language-action models.

Figures & tables

Fig. 1: Overview of the FG-CLTP Framework. The fine-grained contrastive language-tactile pretraining (FG-CLTP) method aligns 3D tactile point clouds with quantitative contact-state-aware language descriptions. The pretrained encoder is integrated into a flow matching policy (3D-TLA), enabling tactile-based multimodal reasoning and action generation.

Fig. 2: Overview of the Contact3D Dataset. The Contact3D dataset integrates real-world and simulated multi-sensor data, implementing automated collection through pressing, sliding, and rotating primitives, alongside comprehensive contact state annotations for holistic representation.

Dataset	Sensor	Objects	Samples	Marker	Depth	Force	Lang.	Dynamic
VisGel [ 25 ]	GelSight	118	3.0k	✓	$\times$	✓	✓	$\times$
Cloth [ 26 ]	GelSight	153	90k	✓	$\times$	✓	✓	✓
Touch and Go [ 27 ]	GelSight	-	13.9k	✓	✓	✓	✓	✓
SSVTP [ 28 ]	DIGIT	15	4.5k	$\times$	$\times$	$\times$	$\times$	$\times$
TVL [ 5 ]	DIGIT	-	44k	$\times$	$\times$	$\times$	✓	$\times$
Octopi [ 29 ]	GelSight	74	39k	$\times$	$\times$	$\times$	✓	✓

TABLE I: Comparison of different tactile datasets. Our Contact3D dataset provides comprehensive features including depth, force, language and dynamic interactions.

Fig. 3: Architecture of the FG-CLTP. Tactile point clouds, tactile images, and language descriptions are processed through their respective encoders. For text tokenization, discrete numerical tokens are introduced for fine-grained alignment. The original tokens are frozen, while the newly added tokens are learnable. Contrastive learning is employed for semantic alignment within the feature space. Additionally, an explicit physical attribute regression loss is incorporated to enhance the discriminability of specific physical properties.

Fig. 4: Contact State Classification Results.

Method	Depth			Position			Area			Principal Axis			Shear			Macro Avg.
Method	MAE ( $\downarrow$ )	RMSE ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	MAE ( $\downarrow$ )	RMSE ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	MAE ( $\downarrow$ )	RMSE ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	MAE ( $\downarrow$ )	RMSE ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	MAE ( $\downarrow$ )	RMSE ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )	MAE ( $\downarrow$ )	RMSE ( $\downarrow$ )	$R^{2}$ ( $\uparrow$ )
TVL [ 5 ]	$0.268$	$0.340$	$0.499$	$0.346$	$0.453$	$0.492$	$0.121$	$0.176$	$0.858$	$0.494$	$0.601$	$0.043$	$0.373$	$0.478$	$0.286$	$0.320$	$0.410$	$0.436$
AnyTouch [ 6 ]	$0.219$	$0.283$	$0.650$	$0.252$	$0.353$	$0.694$	$0.084$	$0.118$	$0.937$	$0.401$	$0.521$	$0.275$	$0.206$	$0.302$	$0.718$	$0.232$	$0.315$	$0.655$
UniTouch [ 4 ]	$0.192$	$0.250$	$0.729$	$0.246$	$0.346$	$0.703$	$0.075$	$0.109$	$0.946$	$0.346$	$0.468$	$0.423$	$0.240$	$0.343$	$0.630$	$0.220$	$0.303$	$0.686$
CLTP [ 7 ]	$0.093$	$0.120$	$0.938$	$0.124$	$0.162$	$0.935$	$0.070$	$0.098$	$0.956$	$0.249$	$0.373$	$0.626$	$0.225$	$0.323$	$0.676$	$0.152$	$0.215$	$0.826$
FG-CLTP (w/o token)	$0.082$	$0.108$	$0.945$	$0.096$	$0.138$	$0.950$	$0.068$	$0.095$	$0.960$	$0.145$	$0.210$	$0.815$	$0.130$	$0.185$	$0.840$	$0.104$	$0.147$	$0.902$

TABLE II: Comparison across multiple regression targets.

Fig. 5: A case study of generating text descriptions from realistic tactile point clouds.

Domain	Method	Physical Property Tasks							Avg.
Domain	Method	Shape	Texture	Depth	Area	Pos.	Slide	Twist	Avg.
GelStereo 2.0 (Simulation)	TVL	54.6	99.5	62.4	76.7	41.1	85.9	83.5	72.0
	AnyTouch	55.7	99.8	69.6	82.6	63.8	90.7	84.4	78.1
	UniTouch	56.8	99.6	71.9	82.4	63.5	90.3	85.1	78.5
	CLTP	87.7	99.8	95.1	96.2	95.7	89.4	84.4	92.6
	FG-CLTP (Ours)	90.6	99.8	97.6	98.4	97.6	97.6	89.8	95.9

TABLE III: Cross-Sensor Generalization Results.

Fig. 6: Real-world manipulation tasks. (a) Sequential snapshots of the 3D-TLA performing tube insertion, wipe the board, and handwriting. (b) Comparison of end-effector trajectories among 3D-TLA, DP, and ACT. 3D-TLA generates smooth, contact-state-compliant trajectories, whereas the baseline methods suffer from deviations, loss of contact, or distortions. (c) Hardware setup and the collected 3D deformation point cloud.

Task	ACT	DP	3D-TLA (Ours)
Insert Tube	70.0	75.0	85.0
Wipe the Board	65.0	65.0	75.0
Handwriting	45.0	50.0	60.0

TABLE IV: Real-World 3D-TLA Evaluation (Metric: %).

Tactile Sensing for Robotic Manipulation

Tactile sensing for robotic manipulation uses contact signals to help robots perceive forces, prevent slips, and handle objects reliably.

See all 178 papers

ForeTac-VLA: A Forecasting-Based Tactile-Vision-Language-Action Model for Contact-Rich Robotic Manipulation
Sep 17, 2026Zhengyu Tao, Xin Li, Xin WangTactile-Vision-Language-Action Models Contact-Rich Robotic Manipulation
Texas A&M University, College Station, TX 77843, USA
SimpleTouch: Can Vision-Language-Action Models Master Contact-Rich Manipulation Without Tactile Policy Pretraining?
Oct 2, 2026Chen Yang, Linzhe Shi, Changjie Wu +7Tactile-Vision-Language-Action Models Contact-Rich Robotic Manipulation
Tsinghua University · Amap, Alibaba Group · Hong Kong Polytechnic University
TacVLA: Contact-Aware Tactile Fusion for Robust Vision-Language-Action Manipulation
Mar 13, 2026Kaidi Zhang, Heng Zhang, Zhengtong Xu +7Tactile-Vision-Language-Action Models Contact-Rich Robotic Manipulation
Kaidi, Heng, Zhengtong, Zhiyuan, Prince, Xiang, Xiaojing, Yuhao, and Yu are with the Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genova, Italy. · Ph.D. program of national interest in Robotics and Intelligent Machines (DRIM) and Università di Genova, Genoa, Italy. +1

Sep 17, 2026J/K move · Enter open · S save

ForeTac-VLA: A Forecasting-Based Tactile-Vision-Language-Action Model for Contact-Rich Robotic Manipulation

Zhengyu Tao, Xin Li, Xin Wang

Texas A&M University, College Station, TX 77843, USA

Tactile-Vision-Language-Action Models Contact-Rich Robotic Manipulation Vision-Language-Action Models VLMs for Robotics Tactile Sensing for Robotic Manipulation

Open paper AskPDF arXiv

Vision-language-action (VLA) models have demonstrated strong capabilities in robotic manipulation, yet their reliance on visual perception limits robustness in contact-rich environments, where critical physical interaction states may not be visually observable. Existing tactile-enhanced VLA methods improve physical grounding using observed tactile feedback, but most remain largely reactive rather than explicitly modeling how contact may evolve. Therefore, we propose ForeTac-VLA, a forecasting-based tactile-vision-language fusion model that predicts future tactile states to guide action generation. Specifically, ForeTac-VLA encodes recent tactile observations into temporal representations and integrates them with vision-language features through bidirectional cross-attention. Further, a transformer-based forecasting module predicts multi-step future tactile states, enabling the model to reason jointly over observed and anticipated contact. Finally, the fused multimodal representations and predicted future tactile states are fed into the VLA backbone to condition action generation. To stabilize training, a ground-truth-to-prediction curriculum is employed when early forecasts are unreliable. Across four real-world contact-rich manipulation tasks, ForeTac-VLA achieves an average success rate of 95%, outperforming the fine-tuned VLA model by 36.25 percentage points and state-of-the-art tactile-enhanced VLA baselines by over 22 percentage points. ForeTac-VLA also maintains strong performance under low-illumination and visually cluttered conditions. Video demonstrations can be found on https://foretac-vla.github.io/

Abstract

Tactile Sensing for Robotic Manipulation

Explore similar work

ForeTac-VLA: A Forecasting-Based Tactile-Vision-Language-Action Model for Contact-Rich Robotic Manipulation

SimpleTouch: Can Vision-Language-Action Models Master Contact-Rich Manipulation Without Tactile Policy Pretraining?

TacVLA: Contact-Aware Tactile Fusion for Robust Vision-Language-Action Manipulation