The Language of Touch: Translating Vibrations into Text with Dual-Branch Learning

Jin Chen; Yifeng Lin; Chao Zeng; Si Wu; Tiesong Zhao

cs.CVMar 26, 2026

The Language of Touch: Translating Vibrations into Text with Dual-Branch Learning

Authors: Jin Chen, Yifeng Lin, Chao Zeng, Si Wu, Tiesong Zhao

Organizations: the Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information, Fuzhou University, Fuzhou 350108, China · the School of Artificial Intelligence, Hubei University, Wuhan, China · the Key Laboratory of Intelligent Sensing System and Security, Hubei University and Ministry of Education, China · the School of Computer Science and Engineering, South China University of Technology ·

Abstract

The standardization of vibrotactile data by IEEE P1918.1 workgroup has greatly advanced its applications in virtual reality, human-computer interaction and embodied artificial intelligence. Despite these efforts, the semantic interpretation and understanding of vibrotactile signals remain an unresolved challenge. In this paper, we make the first attempt to address vibrotactile captioning, {\it i.e.}, generating natural language descriptions from vibrotactile signals. We propose Vibrotactile Periodic-Aperiodic Captioning (ViPAC), a method designed to handle the intrinsic properties of vibrotactile data, including hybrid periodic-aperiodic structures and the lack of spatial semantics. Specifically, ViPAC employs a dual-branch strategy to disentangle periodic and aperiodic components, combined with a dynamic fusion mechanism that adaptively integrates signal features. It also introduces an orthogonality constraint and weighting regularization to ensure feature complementarity and fusion consistency. Additionally, we construct LMT108-CAP, the first vibrotactile-text paired dataset, using GPT-4o to generate five constrained captions per surface image from the popular LMT-108 dataset. Experiments show that ViPAC significantly outperforms the baseline methods adapted from audio and image captioning, achieving superior lexical fidelity and semantic alignment.

Figures & tables

Fig. 1: Application scenarios of vibrotactile captioning.

Fig. 2: Illustration of the vibrotactile-text dataset generation process. Surface images from the LMT-108 dataset are provided as input to GPT-4o, which generates five textual descriptions per image under predefined linguistic constraints. These descriptions are then paired with the corresponding triaxial acceleration signals collected from the same material surfaces, resulting in the final vibrotactile-text dataset.

Fig. 3: Examples of material surface images and their corresponding three-axis vibrotactile signals. Top: materials with regular textures exhibit strong periodicity. Bottom: irregular surfaces yield noisy, aperiodic signals. This motivates the use of distinct modeling pathways.

Fig. 4: ViPAC takes triaxial acceleration signals as input and applies DFT321 to obtain 1D vibration data. These signals are processed by a dual-branch encoder that separately models periodic and aperiodic components using FAN-based frequency analysis and Transformer+LSTM-based temporal modeling, respectively. The extracted features are dynamically fused based on estimated periodicity scores, and the fused representation is decoded into natural language using a Transformer decoder.

Model	BLEU1	BLEU2	BLEU3	BLEU4	ROUGE-L	METEOR	CIDEr	SPICE	SPIDEr
ACT (DCASE 2021)	0.7204	0.5619	0.4498	0.3510	0.5931	0.2882	0.3107	0.2454	0.2780
Kim et al. (ICASSP 2023)	0.7365	0.5732	0.4278	0.3086	0.5163	0.2597	0.7428	0.1942	0.4685
Recap (ICASSP 2024)	0.7524	0.5891	0.4415	0.3213	0.5278	0.2714	0.7652	0.2126	0.4889
ClipCap (2021)	0.7343	0.5792	0.4667	0.3748	0.5819	0.2985	0.4737	0.2480	0.4959
ViECap (ICCV 2023)	0.7432	0.5764	0.4573	0.3682	0.5867	0.2946	0.6981	0.2413	0.4697
RCMF (TCSVT 2024)	0.7480	0.5900	0.4520	0.3350	0.5800	0.2750	0.7200	0.2260	0.4730

TABLE I: Performance comparison of different captioning models.

Fig. 5: Qualitative comparisons between ViPAC generated captions and five GPT-4o reference descriptions for four representative materials. Matched phrases are highlighted to emphasize semantic consistency. The selected samples—covering regular perforations, fine grids, rough glitter, and irregular bumps—demonstrate ViPAC’s ability to produce accurate and diverse textual descriptions directly from vibrotactile signals.

Fig. 6: Demo interface for caption-based material retrieval. The left panel displays matched material images; the right panel shows vibrotactile files. Images are used only for visualization.

Variant	BLEU1	BLEU2	BLEU3	BLEU4	ROUGE-L	METEOR	CIDEr	SPICE	SPIDEr
Periodic Only	0.6124	0.4893	0.3582	0.2645	0.5012	0.2287	0.4215	0.1683	0.2949
Aperiodic Only	0.6657	0.5328	0.4127	0.3061	0.5237	0.2614	0.5362	0.1947	0.3655
No Fusion	0.6829	0.5535	0.4312	0.3198	0.5379	0.2745	0.5683	0.2054	0.3869
ViPAC (Full)	0.7635	0.6005	0.4782	0.3861	0.6047	0.3094	0.7795	0.2592	0.5194

TABLE II: Ablation study on model components. We evaluate the effects of removing the periodic branch, aperiodic branch, and adaptive fusion module. The full model consistently achieves the highest performance, confirming that both branches and the dynamic weighting mechanism contribute to the effectiveness of ViPAC.

Excluded Category	BLEU1	BLEU2	BLEU3	BLEU4	ROUGE-L	METEOR	CIDEr	SPICE	SPIDEr
G1	0.6325	0.4674	0.3592	0.2841	0.5035	0.2394	0.4892	0.1553	0.3223
G2	0.7152	0.5483	0.4301	0.3456	0.5658	0.2765	0.6518	0.2169	0.4344
G3	0.6950	0.5298	0.4164	0.3340	0.5536	0.2661	0.6142	0.2007	0.4075
G4	0.6159	0.4563	0.3508	0.2756	0.4927	0.2328	0.4671	0.1475	0.3073
G5	0.7524	0.5891	0.4667	0.3748	0.5931	0.2985	0.7100	0.2480	0.4380
G6	0.6437	0.4779	0.3660	0.2901	0.5124	0.2450	0.5094	0.1625	0.3360

TABLE III: Complete evaluation metrics when a specific material category (G1 to G9) is excluded from training. Each row corresponds to a model trained without the respective category.

Input	B1	B2	B3	B4	R-L	MET	CIDEr	SPICE	SPIDEr
X-only	0.6820	0.5220	0.4010	0.3087	0.5450	0.2668	0.5954	0.2130	0.4042
Y-only	0.7090	0.5450	0.4250	0.3370	0.5660	0.2790	0.6512	0.2068	0.4290
Z-only	0.6640	0.5060	0.3890	0.2924	0.5380	0.2583	0.5627	0.2045	0.3836
Mean(X,Y,Z)	0.6850	0.5240	0.4050	0.3127	0.5500	0.2680	0.6031	0.2081	0.4056
DFT321	0.7635	0.6005	0.4782	0.3861	0.6047	0.3094	0.7795	0.2592	0.5194

TABLE IV: Ablation on input representation with the full metric suite. Training three independent models with only one axis (X/Y/Z) yields consistently worse captions than using DFT321 to fuse triaxial signals into one channel.

Tactile Sensing

Tactile sensing measures contact forces, deformation, texture, and motion to help robots perceive and act through physical interaction.

See all 138 papers

HapticLDM: A Diffusion Model for Text-to-Vibrotactile Generation
May 11, 2026Jiahao Xiong, Fei Wang, Anran Xu +4Haptic Feedback Conditional Generative Modeling
Technology Development Center, Guangzhou Shiyuan Electronic Technology Company Limited, Guangzhou 510535, China · School of Automation and Intelligence, Beijing Jiaotong University, Beijing 100044, China · Department of Architecture, University of California, Berkeley, Berkeley, CA 94720, USA +1
VQ-Touch: A Data-Efficient Tactile Generation Framework Across Sensors and Scenarios
Jul 16, 2026Kailin Lyu, Long Xiao, Jianing Zeng +3Tactile Sensing Vector-Quantized Variational Autoencoder
Institute of Automation, Chinese Academy of Sciences
STAR: Sparse Tactile Representation Learning in Vision-Tactile-Language-Action Models for Dexterous Manipulation
Sep 11, 2026Xiangcheng Liu, Tianhao Wu, Le Zheng +6Representation Learning Tactile-Vision-Language-Action Models
Shanghai Innovation Institute · Agibot

May 11, 2026J/K move · Enter open · S save

HapticLDM: A Diffusion Model for Text-to-Vibrotactile Generation

Jiahao Xiong, Fei Wang, Anran Xu, Pinzhi Huang+3

Technology Development Center, Guangzhou Shiyuan Electronic Technology Company Limited, Guangzhou 510535, China · School of Automation and Intelligence, Beijing Jiaotong University, Beijing 100044, China · Department of Architecture, University of California, Berkeley, Berkeley, CA 94720, USA+1

Haptic Feedback Conditional Generative Modeling

Open paper AskPDF arXiv

Text-to-vibration generation converts natural language into haptic feedback, enabling vibration-effect designers to get scenarios-fitted vibrations more efficiently, which shows great potentials in application fields such as metaverse, games, and film to enrich the user experience in interactive scenarios. The core challenge in this field is how to generate accurate, consistent, and complete vibrations according to textual semantics. Very recent autoregressive (AR) approaches (e.g., HapticGen) exhibit limited capacity in fully capturing global dependencies, owing to the inherent sequential nature of their modeling and prevailing data constraints. In this paper, we proposed HapticLDM, the first text-to-vibration generative model built upon Latent Diffusion Models (LDMs). Firstly, with respect to the data, we introduced a text-processing strategy that emphasizes dynamic characteristics to curate high-quality data pairs for fine-grained dynamic modeling. Secondly, HapticLDM incorporates a global denoising mechanism that regulates coherent and stable variations in the temporal envelope. Furthermore, we conduct extensive evaluations, including A/B testing against the state-of-the-art baseline and a user study involving 30 participants. The results demonstrate that our model enhances realism and semantic alignment. Qualitative feedback further indicates that HapticLDM simplifies the haptic design workflow while generating diverse, subtle, and physically precise vibrations.

Abstract

Tactile Sensing

Explore similar work

HapticLDM: A Diffusion Model for Text-to-Vibrotactile Generation

VQ-Touch: A Data-Efficient Tactile Generation Framework Across Sensors and Scenarios

STAR: Sparse Tactile Representation Learning in Vision-Tactile-Language-Action Models for Dexterous Manipulation