cs.CVApr 2, 2026
SaveVitaTouch: Property-Aware Vision-Tactile-Language Model for Robotic Quality Inspection in Manufacturing
Organizations: School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications, Beijing 100876, China. · Zhongguancun Academy, Beijing 100094, China. · School of Automation, Beijing Institute of Technology, Beijing 100081, China. · School of AI, Beijing Institute of Technology, Beijing 100081, China ·
Abstract
Quality inspection in smart manufacturing requires identifying intrinsic material and surface properties beyond visible geometry, yet vision-only methods remain vulnerable to occlusion and reflection. We propose VitaTouch, a property-aware vision-tactile-language model for material-property inference and natural-language attribute description. VitaTouch uses modality-specific encoders and a dual Q-Former to extract language-relevant visual and tactile features, which are compressed into prefix tokens for a large language model. We align each modality with text and explicitly couple vision and touch through contrastive learning. We also construct VitaSet, a multimodal dataset with 186 objects, 52k images, and 5.1k human-verified instruction-answer pairs. VitaTouch achieves the best performance on HCT and the overall TVL benchmark, while remaining competitive on SSVTP. On VitaSet, it reaches 88.89% hardness accuracy, 75.13% roughness accuracy, and 54.81% descriptor recall; the material-description task further achieves a peak semantic similarity of 0.9009. With LoRA-based fine-tuning, VitaTouch attains 100.0%, 96.0%, and 92.0% accuracy for 2-, 3-, and 5-category defect recognition, respectively, and delivers 94.0% closed-loop recognition accuracy and 94.0% end-to-end sorting success in 100 laboratory robotic trials. More details are available at the project page: https://vitatouch.github.io/
Figures & tables
Fig. 1: Overview of VitaTouch. Left: three-stage training pipeline. Stage 1 performs cross-modal alignment via dual Q-Formers with InfoNCE and PTM losses. Stage 2 builds a property-reasoning multimodal model with fused V–T tokens in frozen Vicuna-7B. LoRA-based defect adaptation is then conducted over progressively finer-grained defect label spaces using few-shot labeled samples per category. Right: tactile sensing complements vision for property reasoning and defect recognition under visually ambiguous conditions.
Fig. 2: VitaSet overview (Ours + AnyTouch GelSight-only). Aligned RGB observations and paired GelSight tactile readings across objects, with controlled-vocabulary annotations and dataset statistics under a unified schema.
Fig. 3: VitaTouch model architecture. VitaTouch employs a dual-branch vision-tactile design with modality-specific encoders and Q-Formers. The Q-Formers distill learnable queries into vision-tactile prefix tokens , which are prepended to text embeddings and fed into a frozen Vicuna-7B decoder. Training proceeds in three stages: Stage 1 aligns cross-modal embeddings via frozen encoders; Stage 2 establishes the perception-to-language pathway for property reasoning; Stage 3 adapts the model for few-shot defect recognition using LoRA while freezing the backbone.
| Model | Modalities | Score | ||||
| Vision | Tactile | Language | SSVTP | HCT | TVL | |
| LLaVA-1.5 7B | ✓ | - | ✓ | 3.64 | 3.55 | 3.56 |
| LLaVA-1.5 13B | ✓ | - | ✓ | 3.55 | 3.63 | 3.62 |
| ViP-LLaVA 7B | ✓ | - | ✓ | 2.72 | 3.44 | 3.36 |
| ViP-LLaVA 13B | ✓ | - | ✓ | 4.10 | 3.76 | 3.80 |
| LLaMA-Adapter | ✓ | - | ✓ | 2.56 | 3.08 | 3.02 |
TABLE I: Comparison on the TVL benchmark. VitaTouch achieves the best performance on HCT and TVL, and remains competitive on SSVTP.
Fig. 4: VitaSet validation performance of VitaTouch across training epochs. (a) Multi-task validation trends for hardness accuracy, roughness accuracy, descriptor recall, and mean task score. (b) Comparison of strict exact-match descriptor recall and semantic similarity for the material-property descriptor task. Stars ( ) mark the best epoch for each metric.
| Method | Hardness Acc. | Roughness Acc. | Descriptor Recall | Mean Task Score |
| Vision only (Qwen-2.5-VL) | 73.81% | 61.11% | 32.28% | 55.73% |
| Tactile only (Octopi) | 68.25% | 58.20% | 8.40% | 44.95% |
| VitaTouch (Without Stage 1) | 84.80% | 72.50% | 17.46% | 58.25% |
| VitaTouch (Ours) | 88.89% | 75.13% | 54.81% | 72.95% |
TABLE II: Comparison of the full VitaTouch model, a no-Stage-1 variant, and unimodal settings on VitaSet across tasks.
Fig. 5: Ablation results on the VitaSet dataset across tasks. Each variant removes one key stage from the full model, demonstrating the necessity of explicit alignment and multimodal fusion for robust multi-task property learning.
| Defect Categories | Train Samples per Category | Vision-only (Qwen-2.5-VL) | Tactile-only (Octopi) | VitaTouch (V+T) |
| 2 | 5 | 70.0% | 60.0% | 90.0% |
| 2 | 15 | 76.7% | 66.7% | 96.7% |
| 2 | 50 | 82.0% | 77.0% | 100.0% |
| 3 | 5 | 66.7% | 66.7% | 85.0% |
| 3 | 15 | 75.6% | 73.3% | 88.9% |
| 3 | 50 | 86.7% | 81.3% | 96.0% |
TABLE III: LoRA-based defect adaptation results under different numbers of defect categories and labeled training samples per category. The vision-only and tactile-only baselines are built on Qwen-2.5-VL and Octopi, respectively.
| Metric | Result |
| Number of closed-loop trials | 100 |
| Closed-loop recognition accuracy | 94.0% |
| End-to-end sorting success rate | 94.0% |
| Single-GPU inference latency (p50) | 278 ms/sample |
| Single-GPU inference latency (mean) | 633 ms/sample |
TABLE IV: Quantitative results of the real-robot closed-loop sorting validation.
Fig. 6: Closed-loop robotic inspection and sorting demonstration. A Franka manipulator grasps the object, acquires observations from an external RGB camera and the GelSight Mini tactile sensor during interaction, performs defect-status inference using VitaTouch, and places the object into the corresponding defect or normal container.
| Junyi Zong received the B.Eng. degree in Energy and Power Engineering from Qingdao University of Science and Technology, Qingdao, China, in 2023. He is currently pursuing the Ph.D. degree in Control Science and Engineering at Beijing University of Posts and Telecommunications, Beijing, China. His research interests include multimodal large models, embodied intelligence, and robotic dexterous manipulation. |
| Qingxuan Jia received the B.S. degree in mechanical design and manufacturing from Shandong University of Technology, China, in 1986, the M.S. degree in mechanics in 1991, and the Ph.D. degree in mechanical design and theory in 2005 from Beihang University, China. He is currently a Professor with the School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications, China. His research interests include intelligent robotic technology, robotic manipulation, and system integration. |
| Meixian Shi received the B.Eng. degree in Internet of Things Engineering from Hebei University of Technology, Tianjin, China, in 2025. She is currently pursuing the M.S. degree at the School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications, Beijing, China. Her research interests include vision-language-action models, embodied control, and robotic manipulation. |
| Tong Li received the B.S. degree in engineering mechanics from Beihang University (BUAA), Beijing, China, in 2010, and the Ph.D. degree in mechatronic engineering from Beijing University of Posts and Telecommunications (BUPT), Beijing, China, in 2016. He then worked as a Research Associate with the Aerospace Information Research Institute (AIR), Chinese Academy of Sciences, Beijing, China, until 2020. He is currently an Associate Professor with the School of Intelligent Engineering and Automation, BUPT. His research interests include tactile sensors, intelligent robotic technology, and multi-sensor information fusion. |
| Jiayuan Li received the B.S. degree in control science and engineering from Beijing Institute of Technology, Beijing, China, in 2021, and the M.Eng. degree in computer control and automation from Nanyang Technological University, Singapore, in 2024. He is currently working toward the Ph.D. degree in control science and engineering at Beijing Institute of Technology, Beijing, China. His research interests include large language models, planning, combinatorial optimization, and continual learning. |
| Zihang Lv received the B.Eng. degree in Automation from the School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China, in 2024. He is currently pursuing the M.S. degree in Control Science and Engineering at Beijing University of Posts and Telecommunications, Beijing, China. His research interest is robotic manipulation. |
| Gang Chen received the B.S. degree in mechanical design, manufacture and automation from Beijing Institute of Petrochemical Technology (BIPT), Beijing, China, in 2004 and the Ph.D. degree in mechatronic engineering from Beijing University of Posts and Telecommunications (BUPT), Beijing, China, in 2011. He is currently a Professor with the School of Intelligent Engineering and Automation, BUPT. His research interests include space robotics, motion planning, and control methods. |
| Fang Deng (Fellow, IEEE) received the B.Eng. and Ph.D. degrees in control science and engineering from Beijing Institute of Technology, Beijing, China, in 2004 and 2009, respectively. He is currently a Professor with the School of AI, Beijing Institute of Technology, Beijing, China, and also with Beijing Institute of Technology, Zhuhai, China. He serves as the Executive Vice President of both the Graduate School and the School of AI of Beijing Institute of Technology, and as the Director of the Beijing Key Laboratory of Lightweight Intelligent System. His research interests include swarm intelligence, autonomous sensor networks, and smart wearable devices. |