Can a Robot Read Braille? - Learning to Adapt Contact via Imitation Learning for Tactile Braille Recognition

Xi Chen; Yunlong Shan; Sihan Chen; Jun Hu; Zhongxuan Li; Shiyao Zhang; Sichao Liu; Zhong Zhao

cs.ROSep 25, 2026

Can a Robot Read Braille? - Learning to Adapt Contact via Imitation Learning for Tactile Braille Recognition

Authors: Xi Chen, Yunlong Shan, Sihan Chen, Jun Hu, Zhongxuan Li, Shiyao Zhang, Sichao Liu, Zhong Zhao, +2 more

Organizations: College of Mechatronics and Control Engineering, Shenzhen University, Shenzhen, Guangdong, China · School of Advanced Engineering, Great Bay University, Dongguan, Guangdong, China · Jiangsu Key Laboratory of Drug Metabolism and Pharmacokinetics, China Pharmaceutical University, Nanjing, Jiangsu, China · Department of Computer Science, School of Computing and Data Science, The University of Hong Kong, Pokfulam, Hong Kong SAR, China · Tsinghua University, Beijing, China · École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · School of Electrical Engineering, University of Belgrade, Belgrade, Serbia

Abstract

For people who are blind, touch provides an essen-tial channel for accessing written information through Braille. Bringing a similar capability to robots requires them not only to recognize tactile patterns, but also to actively establish physical contact that makes those patterns readable. Yet existing robotic Braille readers largely focus on recognition after contact, leaving contact establishment itself insufficiently addressed. We present an adaptive-contact framework for robotic tactile Braille reading that assesses contact quality and physically corrects unsuitable contact before recognition and reconstruc-tion. Multi-Head Policy Learning uses expert-guided contact-adjustment demonstrations to jointly learn contact acceptability and pose corrections. During deployment, the robot iteratively evaluates and re-establishes contact, retaining reliable tactile observations for pose-aware fusion and Braille reconstruction. Across 20 physical Braille plates used for learning and eval-uation, the proposed approach achieves 94.0% tactile quality and 88.6% tactile reconstruction on the ten online-evaluation plates. These results demonstrate the importance of actively establishing readable contact, rather than relying solely on recognition under imperfect tactile observations, for reliable robotic Braille reading.

Figures & tables

Fig. 1: Motivation for adaptive contact in robotic tactile Braille reading: humans naturally adjust fingertip contact, while robots must actively establish readable tactile contact.

Fig. 2: Problem formulation for adaptive contact. At trajectory location $k$ , the policy $\pi_{\phi}$ evaluates the tactile image $I_{k,t}$ acquired at contact pose $\mathbf{u}_{k,t}$ . An unsuitable contact is rejected and the predicted correction $\Delta\hat{\mathbf{u}}_{k,t}$ updates the next pose; a suitable tactile image is retained as $\hat{I}_{k}$ .

Fig. 3: Overview of the proposed method. (a) Expert contact-adjustment demonstrations are collected and converted into supervision for Multi-Head Policy Learning. (b) The learned policy is deployed in closed loop to improve unsuitable contacts. (c) Reconstruction and decoding fuse retained tactile-image–pose pairs, refine the resulting Braille dots, and decode the text.

Fig. 4: Construction of one grouped expert demonstration. The expert inspects each tactile image $I_{g,t}$ and changes $\mathbf{p}_{g,t}$ and $\boldsymbol{\theta}_{g,t}$ through UFACTORY Studio until the contact is accepted. Each intermediate tactile image becomes a rejected sample whose pose-correction target points directly to the final accepted end-effector pose $\mathbf{u}_{g}^{*}$ ; the final tactile image $I_{g}^{*}$ becomes an accepted sample with a zero correction target.

Fig. 5: Experimental platform and end-effector assembly. (a) The xArm7 establishes tactile contact with a Braille plate. (b) Exploded view of the xArm7 end flange, custom 3-D-printed adapter, Xense G1-WS tactile sensor, and Braille plate.

Task	Metric	Value
Quality	F1 (%)	92.8
Quality	Error (%)	5.9
Correction	Pos. MAE (mm)	0.20
	Roll MAE ( ∘ )	0.27
	Pitch MAE ( ∘ )	0.29
	Mean $R^{2}$	0.76

TABLE I: Offline evaluation on held-out Braille plates.

Fig. 6: Representative Braille plates reserved for online evaluation and their ground-truth line transcriptions. (a) English Braille plate with 10 trajectory locations per line. (b) Chinese Braille plate with 13 trajectory locations per line. L1–L4 run from top to bottom; parenthetical English translations are annotations, not decoder outputs.

Fig. 7: Representative acquisition with the Full strategy. (a) Retained tactile images $\hat{I}_{1}$ – $\hat{I}_{10}$ at ten consecutive trajectory locations; orange markers indicate the current trajectory location on the Braille plate. (b) Contact-attempt sequences at trajectory locations $k=1$ and $k=5$ . Tactile images $I_{1,1}$ , $I_{5,1}$ , and $I_{5,2}$ are rejected (red), whereas $I_{1,2}$ and $I_{5,3}$ are accepted after pose correction (green).

Strategy	Att./loc.	Tactile	Tactile	Line
		quality	reconstruction	acc.
		(%)	C/E/All (%)
Single-press	1.00	61.4	62.5/65.9/64.0	32.5
Retry-only	2.36	62.9	62.5/66.8/64.4	32.5
Position-only	1.84	81.9	79.6/80.5/80.0	60.0
Full (ours)	1.42	94.0	88.9/88.2/88.6	80.0

TABLE II: Online Braille reading results. C/E/All denote Chinese, English, and pooled tactile reconstruction, respectively.

Explore similar work

Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning
Jun 10, 2026Shengcheng Luo, Xiyan Huang, Zhe Xu +3Robotic Grasping Representation Learning
1ShanghaiTech University · 2Beijing Institute for General Artificial Intelligence
ZeroTouch: Tactile-Supervised Visual Contact Estimation for Contact-Rich Manipulation
Sep 18, 2026Dmitriy Kosenkov, Daniia Zinniatullina, Miguel Altamirano Cabrera +3Robotic Grasping Contact-Rich Robotic Manipulation
Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology, Moscow, Russian Federation.
Vision-Based Tactile Intelligence for Robotics: Sensing, Learning, and Embodied Manipulation
Aug 16, 2026Peng Zhou, Jun Hu, Sihan Chen +12Vision-Based Tactile Sensing Tactile Sensing for Robotic Manipulation
Great Bay University · The University of Hong Kong · Tsinghua University +5

Jun 10, 2026, cs.ROJ/K move · Enter open · S save

Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning

Shengcheng Luo, Xiyan Huang, Zhe Xu, Wanlin Li+2

1ShanghaiTech University · 2Beijing Institute for General Artificial Intelligence

Robotic Grasping Representation Learning Visuo-Tactile Representation Learning Sim-to-Real Transfer Tactile Sensing for Robotic Manipulation

Open paper AskPDF arXiv

Blind grasping with a dexterous hand is a crucial manipulation capability. Nevertheless, learning such tactile-only policies for real robots remains challenging due to the tactile sim-to-real gap and the limited expressiveness of sparse tactile signals. To bridge this gap, we propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand. Our approach combines three key components. First, we introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals. Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining. Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy. We evaluate our method on a physical LEAP Hand equipped with distributed tactile sensing across 10 seen and 10 unseen objects. The deployed policy achieves a 27% real-world grasp success rate across all 20 objects, without real-world grasping demonstrations or visual input. Simulation ablations show that layout-aware tactile pretraining improves grasping performance, while sensing-level evaluations confirm that Real2Sim calibration increases the consistency of tactile contact events between simulation and hardware. Together, these results suggest that contact-event calibration, geometry-aware tactile representation learning, and diffusion-based policy aggregation provide an effective path toward tactile-only blind grasping on real dexterous robotic hands. Project page:Dex-Blind-Grasp.github.io.

Abstract

Explore similar work

Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning

ZeroTouch: Tactile-Supervised Visual Contact Estimation for Contact-Rich Manipulation

Vision-Based Tactile Intelligence for Robotics: Sensing, Learning, and Embodied Manipulation