Can a Robot Read Braille? - Learning to Adapt Contact via Imitation Learning for Tactile Braille Recognition
Authors: Xi Chen, Yunlong Shan, Sihan Chen, Jun Hu, Zhongxuan Li, Shiyao Zhang, Sichao Liu, Zhong Zhao, +2 more
Organizations: College of Mechatronics and Control Engineering, Shenzhen University, Shenzhen, Guangdong, China · School of Advanced Engineering, Great Bay University, Dongguan, Guangdong, China · Jiangsu Key Laboratory of Drug Metabolism and Pharmacokinetics, China Pharmaceutical University, Nanjing, Jiangsu, China · Department of Computer Science, School of Computing and Data Science, The University of Hong Kong, Pokfulam, Hong Kong SAR, China · Tsinghua University, Beijing, China · École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · School of Electrical Engineering, University of Belgrade, Belgrade, Serbia
For people who are blind, touch provides an essen-tial channel for accessing written information through Braille. Bringing a similar capability to robots requires them not only to recognize tactile patterns, but also to actively establish physical contact that makes those patterns readable. Yet existing robotic Braille readers largely focus on recognition after contact, leaving contact establishment itself insufficiently addressed. We present an adaptive-contact framework for robotic tactile Braille reading that assesses contact quality and physically corrects unsuitable contact before recognition and reconstruc-tion. Multi-Head Policy Learning uses expert-guided contact-adjustment demonstrations to jointly learn contact acceptability and pose corrections. During deployment, the robot iteratively evaluates and re-establishes contact, retaining reliable tactile observations for pose-aware fusion and Braille reconstruction. Across 20 physical Braille plates used for learning and eval-uation, the proposed approach achieves 94.0% tactile quality and 88.6% tactile reconstruction on the ten online-evaluation plates. These results demonstrate the importance of actively establishing readable contact, rather than relying solely on recognition under imperfect tactile observations, for reliable robotic Braille reading.
Figures & tables
Fig. 1: Motivation for adaptive contact in robotic tactile Braille reading: humans naturally adjust fingertip contact, while robots must actively establish readable tactile contact.
Fig. 2: Problem formulation for adaptive contact. At trajectory location k , the policy πϕ evaluates the tactile image Ik,t acquired at contact pose uk,t . An unsuitable contact is rejected and the predicted correction Δu^k,t updates the next pose; a suitable tactile image is retained as I^k .
Fig. 3: Overview of the proposed method. (a) Expert contact-adjustment demonstrations are collected and converted into supervision for Multi-Head Policy Learning. (b) The learned policy is deployed in closed loop to improve unsuitable contacts. (c) Reconstruction and decoding fuse retained tactile-image–pose pairs, refine the resulting Braille dots, and decode the text.
Fig. 4: Construction of one grouped expert demonstration. The expert inspects each tactile image Ig,t and changes pg,t and θg,t through UFACTORY Studio until the contact is accepted. Each intermediate tactile image becomes a rejected sample whose pose-correction target points directly to the final accepted end-effector pose ug∗ ; the final tactile image Ig∗ becomes an accepted sample with a zero correction target.
Fig. 5: Experimental platform and end-effector assembly. (a) The xArm7 establishes tactile contact with a Braille plate. (b) Exploded view of the xArm7 end flange, custom 3-D-printed adapter, Xense G1-WS tactile sensor, and Braille plate.
Task
Metric
Value
Quality
F1 (%)
92.8
Error (%)
5.9
Correction
Pos. MAE (mm)
0.20
Roll MAE ( ∘ )
0.27
Pitch MAE ( ∘ )
0.29
Mean R2
0.76
TABLE I: Offline evaluation on held-out Braille plates.
Fig. 6: Representative Braille plates reserved for online evaluation and their ground-truth line transcriptions. (a) English Braille plate with 10 trajectory locations per line. (b) Chinese Braille plate with 13 trajectory locations per line. L1–L4 run from top to bottom; parenthetical English translations are annotations, not decoder outputs.
Fig. 7: Representative acquisition with the Full strategy. (a) Retained tactile images I^1 – I^10 at ten consecutive trajectory locations; orange markers indicate the current trajectory location on the Braille plate. (b) Contact-attempt sequences at trajectory locations k=1 and k=5 . Tactile images I1,1 , I5,1 , and I5,2 are rejected (red), whereas I1,2 and I5,3 are accepted after pose correction (green).
Strategy
Att./loc.
Tactile
Tactile
Line
quality
reconstruction
acc.
(%)
C/E/All (%)
Single-press
1.00
61.4
62.5/65.9/64.0
32.5
Retry-only
2.36
62.9
62.5/66.8/64.4
32.5
Position-only
1.84
81.9
79.6/80.5/80.0
60.0
Full (ours)
1.42
94.0
88.9/88.2/88.6
80.0
TABLE II: Online Braille reading results. C/E/All denote Chinese, English, and pooled tactile reconstruction, respectively.
Blind grasping with a dexterous hand is a crucial manipulation capability. Nevertheless, learning such tactile-only policies for real robots remains challenging due to the tactile sim-to-real gap and the limited expressiveness of sparse tactile signals. To bridge this gap, we propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand. Our approach combines three key components. First, we introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals. Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining. Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy. We evaluate our method on a physical LEAP Hand equipped with distributed tactile sensing across 10 seen and 10 unseen objects. The deployed policy achieves a 27% real-world grasp success rate across all 20 objects, without real-world grasping demonstrations or visual input. Simulation ablations show that layout-aware tactile pretraining improves grasping performance, while sensing-level evaluations confirm that Real2Sim calibration increases the consistency of tactile contact events between simulation and hardware. Together, these results suggest that contact-event calibration, geometry-aware tactile representation learning, and diffusion-based policy aggregation provide an effective path toward tactile-only blind grasping on real dexterous robotic hands. Project page:Dex-Blind-Grasp.github.io.
Shengcheng Luo, Xiyan Huang, Zhe Xu +3
1ShanghaiTech University · 2Beijing Institute for General Artificial Intelligence
Reliable robotic grasping benefits from estimating the evolving physical interaction and selecting a grasp-dependent compression target. Tactile sensors provide direct interaction measurements but require dedicated hardware at deployment. We introduce ZeroTouch, a tactile-supervised framework that predicts dense contact deformation, the instantaneous six-axis wrench, and a grasp-dependent desired compression target from wrist RGB observations, gripper state, and local gravity direction. Tactile measurements are used only as privileged supervision during training and are not required at deployment. On the full validation set, the complete architecture reduces normal-force MAE from 2.017 N for a state-only baseline to 0.531 N. In physical evaluation with 20 trials per condition, ZeroTouch achieves 95% success on an unseen object, 80% in a seen-object/unseen-grasp condition, and 90% under a content/load shift. Under the same evaluation protocol, OpenVLA achieves 25%, 40%, and 55%, while SmolVLA achieves 10%, 25%, and 35%, respectively.
Dmitriy Kosenkov, Daniia Zinniatullina, Miguel Altamirano Cabrera +3
Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology, Moscow, Russian Federation.
Tactile sensing is essential for robots in contact-rich tasks, yet many tactile sensors still provide sparse, low-dimensional signals that do not capture sufficient information for complex robotic perception and interaction. Vision-based tactile sensors (VBTSs) offer a powerful alternative by con-verting contact-induced deformation of a soft interface into im-ages. The image-based formulation gives VBTSs high-resolution, information-rich tactile observations that enable complex robotic tasks. This review surveys the full VBTS pipeline and treats sensing hardware, learning methods, simulation, and datasets as an integrated sensing-and-learning system. We 1) organize representative VBTSs into a hardware taxonomy structured by deformable elastomer design, sensor size and shape, and optical system design to guide future sensor development; 2) present a hierarchical view of learning-based tactile intelligence from low-level signal understanding to task-level policies and foundation models; and 3) examine simulation platforms and tactile datasets as a scaling layer, together with sim-to-real transfer and cross-sensor adaptation for training, benchmarking, and deployment. Finally, we identify open challenges and future directions for VBTSs in robotics. By providing a holistic view of how hardware, AI architectures, simulation, and datasets interact, this review aims to advance tactile intelligence for contact-rich robotic tasks.
Peng Zhou, Jun Hu, Sihan Chen +12
Great Bay University · The University of Hong Kong · Tsinghua University +5