Learning to Read Braille: Benchmarking and Improving Braille Recognition Capabilities of VLMs

Kartik Jangra; Atharva Gawane; Siddharth Pulipaka; Hemachandran Karah; Raj Dabre

cs.CVOct 26, 2026

Learning to Read Braille: Benchmarking and Improving Braille Recognition Capabilities of VLMs

Authors: Kartik Jangra, Atharva Gawane, Siddharth Pulipaka, Hemachandran Karah, Raj Dabre

Organizations: Nilekani Centre at AI4Bharat · Indian Institute of Technology, Madras

Abstract

Despite rapid progress in visual understanding and document transcription, modern vision-language models (VLMs) struggle with a fundamental accessibility task: reading Braille from images. We introduce BrailleBench, a manually curated benchmark of 259 real-world Grade-1 and Grade-2 Braille images. We evaluate proprietary, open-weight, OCR-specialized, and tool-assisted multimodal systems and find that standalone general-purpose models perform poorly at directly transcribing Braille images, whereas tool-assisted systems perform much better. We design a synthetic Braille generation pipeline with diverse layouts and image perturbations. A 2B VLM model trained on our synthetic data reaches 87% exact match on Grade-1 and 61% on Grade-2 BrailleBench, outperforming standalone VLMs and achieving the highest Grade-2 exact match score among the evaluated systems. We further show that high accuracy on synthetic validation data does not guarantee high accuracy on real photographs in our experiments: models trained on prior synthetic data reach up to 96% on their synthetic validation set but at most 3.26% on Grade-1 BrailleBench. Our results identify an accessibility-relevant capability gap in multimodal systems and show that targeted synthetic supervision can improve real-world Braille transcription.

Figures & tables

Figure 1: Overview of our work. (1) BrailleBench contains manually verified real-world Braille images captured on diverse materials and under diverse conditions. (2) Standalone VLMs largely fail to transcribe them, often producing fluent but unrelated text, while tool-assisted harnesses succeed at a high inference cost. (3) Models fine-tuned only on synthetically generated Braille images transcribe real-world Braille with a single model call, without a tool loop. EM denotes exact match; G1 and G2 denote Grade-1 and Grade-2.

Figure 2: Synthetic training data generation pipeline. Text sampled from FineWeb at multiple granularities is translated to Braille with Liblouis, rendered as embossed Braille with randomized dot geometry, spacing, and paper textures, and degraded with augmentations that simulate real-world capture conditions. The resulting image–text pairs are used to fine-tune Qwen3.5-2B and Chandra-OCR-2 without any real-world Braille images.

Model	Grade	CER $\downarrow$	WER $\downarrow$	Exact Match (%) $\uparrow$	chrF $\uparrow$	Norm. Match (%) $\uparrow$	Word F1 (%) $\uparrow$	Edit Similarity (%) $\uparrow$
Standalone General-Purpose VLMs
GPT-5.5	G1	1.147	1.176	0.00	2.10	2.79	5.55	19.32
	G2	0.776	0.882	0.00	11.13	0.00	6.99	23.04
Kimi K2.6	G1	1.276	1.516	0.00	5.68	0.00	0.59	12.82
	G2	0.941	1.077	0.00	5.79	0.00	2.70	13.98
Claude Opus 4.8	G1	0.934	1.157	0.93	8.76	1.40	3.86	18.12

Table 1: Performance comparison of standalone general-purpose VLMs, tool-assisted harnesses, existing Braille systems (an OBR tool and our reproduction of prior synthetic adaptation; Karamolegkou et al., 2025 ), and our synthetic-adapted models on BrailleBench, reported separately for the 215 Grade-1 (G1) and 44 Grade-2 (G2) images. Lower CER and WER indicate better performance, while higher values are preferred for all other metrics. Bold marks the best value per grade.

Figure 3: Models trained only on WMT-Train (Llama-3.2-11B with LoRA; Qwen3.5-2B with full fine-tuning), evaluated on the in-distribution WMT validation set and on Grade-1 BrailleBench (both models reach 0% exact match on Grade-2). Both models score highly on synthetic validation but fail on real photographs. CER and WER use logarithmic axes; values above 1 come from long hallucinated outputs.

Training data	Rank	chrF $\uparrow$	EM (%) $\uparrow$	CER $\downarrow$	WER $\downarrow$
Ours (276k)	8	88.28	72.59	0.4196	1.0704
Ours (276k)	16	88.57	72.20	0.0754	0.1444
Ours (276k)	32	90.10	76.06	0.0759	0.1261

Table 2: Llama-3.2-11B-Vision-Instruct trained with LoRA on our 276k corpus, evaluated on BrailleBench (both grades,

n=259

). The same model trained on WMT-Train transcribes only one of the 259 images exactly (Table 1 ).

Figure 4: Prediction outcomes by grade. Each image falls into exactly one category. Formatting only : differences that disappear after normalization; spacing only : only word spacing differs; partial error : at most half of the normalized reference characters need editing; large error : more than half; no output : empty outputs and refusals. Numbers on bars give exact match (%). Rows above the dotted line are the harnesses, our models, and the strongest standalone model. Exact counts are in Appendix E . For Claude Code Grade-2 (*), categories are recomputed from the predictions using the normalized-match transform in Table 1 , retaining punctuation. Other rows retain the supplied counts under the original looser formatting normalization; their error-type splits are not directly comparable to this recomputed row. Exact match remains raw string equality for all rows.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Grade 1	Grade 2
Images	215	44
Characters: mean	12.06	71.66
Characters: median	9	34.5
Characters: maximum	61	368
Words: mean	2.25	13.89
Words: median	2	6

Appendix

Table 3: BrailleBench statistics by grade. Character and word lengths are measured per reference. Characters include spaces and punctuation; words are whitespace-delimited. Multi-line image counts are not annotated.

Run	Images	G1 / G2 (%)	Words (med.)	Scr. (%)
Real-matched base (FineWeb sample-10BT)	151,096	44.6 / 55.4	4	17.7
Real-matched base (CC-MAIN-2013-20)	20,000	43.2 / 56.8	4	18.3
Add-on: ghost/hollow/strip/close-up, dynamic spacing	65,857	38.4 / 61.6	2	40.1
Add-on: 1–5 words, all scrambled	9,767	50.3 / 49.7	3	99.2
Add-on: 6–25 words, 50% scrambled	9,997	50.0 / 50.0	15	49.9
Earlier generator (sample of 20k)	20,000	100.0 / 0.0	3	0.0

Appendix

Table 4: Composition of the training corpus by generation run. Grade columns are percentages (G1 / G2); “words” is the median number of words per text; “scr.” is the percentage of texts whose letters were scrambled. The earlier-generator part is Grade 1 only.

Granularity	% of images
character	0.0
word	20.6
phrase (2-5 words)	43.6
sentence-length (6-25 words)	34.1
paragraph-length (26+ words)	1.7
numeric expression (overlapping)	13.3

Appendix

Table 5: Text granularity of the training labels (% of images). Word-count bins: 1: 20.6, 2: 17.0, 3: 11.4, 4–5: 15.2, 6–9: 9.5, 10–15: 9.8, 16–25: 14.7, 26+: 1.7 (%).

Figure 5: Left: one text ( we can read , Grade 2) rendered with degradations added cumulatively, from a clean render (L0) to heavy degradation (L6); layout is fixed by the seed, photometric parameters are re-drawn at each level. Right: real Braille photographs ordered from clean to degraded, for comparison.

Stage	Parameter and range	Prob.	Order
Dot size	dot radius = glyph radius $\times$ dot_scale $\times U(.88,1.12)\times U(.90,1.05)$ ; dot_scale $=$ target diameter / pitch / 0.68, clipped to $[0.4,1.7]$	1.0	1 (render)
Dot jitter	centre offset $\pm 2$ px (scaled with font size); dot intensity $U(200,255)$	1.0	1
Height-map blur	Gaussian radius $U(3,6)\times 0.3$ px (scaled); light direction $U(-1.5,1.5)^{2}\times U(.3,1)$ ; relief gain $U(2.5,5)$	1.0	1
Cell spacing	cell pitch / dot pitch $R$ : 60% $\mathcal{N}(2.65,0.15)$ , else $U(2.15,3.3)$ ( $U(2.1,3.8)$ in add-on)	1.0	1
Word spacing	blank advance $b$ (cell pitches): 85% from measured real values ( $-0.28$ calibration, $\times$ log-normal $\sigma{=}0.1$ ; floor $U(.15,.32)$ ), else $U(.32,2.2)$ ; add-on: $U(.3,2.5)$	1.0	1
Line spacing	line pitch $L$ (cell pitches): real value $\times e^{\mathcal{N}(0,0.06)}$ in $[1.45,6]$ ; add-on: $U(1.5,3.5)$	1.0	1

Appendix

Table 6: Augmentations: sampling ranges and per-image probability.

U(a,b)

is uniform,

\mathcal{N}

Gaussian.

\dagger

: add-on runs use the value in parentheses. Camera-optics effects (defocus, motion blur, lens distortion, chromatic aberration, vignetting), vertical flips, and

90^{\circ}

rotations are implemented but disabled (

p=0

Quantity	Sampling	Realised (p5–p95)
Cell pitch $P$ (px)	bank record $\times e^{\mathcal{N}(0,0.08)}$ (10% of samples: $\sigma{=}0.45$ ), clipped to $[5,110]$ ; texts of $\geq$ 16 / $\geq$ 32 cells get a floor of 18 / 22 px	20–64
Cell tightness $R$	see Table 6	2.24–3.53
Word gap $b$	see Table 6	0.20–2.20
Dot diameter / pitch	bank $\times 0.9\,e^{\mathcal{N}(0,0.08)}$	0.33–0.81
$x/y$ squeeze	$e^{\mathcal{N}(0,0.10)}$	0.85–1.18
Margins (cells)	bank $+\mathcal{N}(0,0.06)$ ; every side $\geq 0.08$ (re-drawn in $[0.08,0.23]$ otherwise); no dot is clipped (min. tracked dot-to-border distance 0.9 px)	left 0.11–3.51

Appendix

Table 7: Real-matched layout and the optional rendering styles. Distributions marked “bank” are resampled from measurements on real braille photographs (cell pitch, dot size, margins, line count, line pitch, word gaps;

\approx

3k reliable records); the last column gives the realised 5th–95th percentile in the final corpus (256,717 real-matched images).

Hyperparameter	Value
Epochs	1
Number of GPUs	8
Batch size per GPU	1
Gradient accumulation steps	4,8
Effective batch size	32,64
Learning rate	$2\times 10^{-5}$

Appendix

Table 8: Training hyperparameters used for fine-tuning Qwen3.5-2B and Chandra-OCR-2 on Braille transcription.

Harness	Tool calls	Image views	Image commands	Code run
Claude Code	6.8	3.2	3.5	No
Codex	7.2	4.1	3.7	Yes

Appendix

Table 9: Mean tool use per image in the tool-assisted harnesses. Image views are processed crops read by the model; image commands are image-processing invocations (ImageMagick for Claude Code).

Harness	Images	Tokens	Cost (USD)	USD / image
Claude Code	259	367,395,129	150.82	0.58
Codex	259	180,338,976	132.87	0.51

Appendix

Table 10: Token usage and API-equivalent cost of the tool-assisted harnesses on BrailleBench.

	Grade-1 ( $n=215$ )						Grade-2 ( $n=44$ )
Model	E	F	S	P	L	N	E	F	S	P	L	N
Claude Code (Opus 5.5)	190	14	1	9	1	0	18	0	9	8	8	1
Qwen3.5-2B (ours)	187	6	4	16	2	0	27	5	1	10	1	0
Chandra-OCR-2 (ours)	183	7	1	20	4	0	24	6	0	13	1	0
Codex (GPT-5.6 Sol)	135	20	0	22	6	32	0	0	0	1	42	1
GPT-6.1 Sol	116	8	2	35	24	30	5	1	0	7	9	22

Appendix

Table 11: Error-analysis counts per system on BrailleBench. E: exact match; F: formatting only; S: spacing only; P: partial error; L: large error; N: no output. Claude Code Grade-2 is recomputed from 44 prediction/reference pairs using NFKC, lowercase, and whitespace normalization with punctuation retained; its counts sum to 44. Other rows retain the supplied categorization under the original, looser formatting normalization.

Panel of Figure 6 — Figure 6: Examples from the tool-assisted harnesses on BrailleBench. (a) Claude Code (Opus 5.5) misreads “mother’s” as “men’s”, while Codex generates unrelated characters. (b) Claude Code produces story text that is not in the image, while Codex again produces unrelated characters. (c) On a blurred, low-resolution sign, Claude Code outputs a fluent but wrong phrase, while Codex transcribes it correctly. (d) Claude Code outputs a sentence from a different page of the same book, while Codex produces unrelated characters.

Metric	500k	276k	WMT-Braille
Records	500,000	276,717	100,000
Characters mean/med./max	148.1/150 300	39.1/21 160	82.9/76 200
Words mean/med./max	25.4/25 50	6.9/4 49	13.8/13 43
Width (px) mean/med./range	1536/1857 172–2530	631/493 28–2301	604/600 488–715
Height (px) mean/med./range	706/623 11–2103	279/137 28–2048	242/225 90–708

Appendix

Table 12: Text and image size statistics of our two training corpora (500k, 276k) and the prior work’s WMT-Braille data (WMT-Train and WMT validation together). Statistics cover every record.

	BrailleBench ( $n=259$ )			WMT val. ( $n=1{,}000$ )
Training data	chrF	EM (%)	CER	chrF	EM (%)	CER
276k (short texts)	94.97	82.63	0.0305	87.14	41.70	0.6331
500k (long texts)	87.97	72.97	0.1981	95.25	63.10	0.0345
500k $+$ 15k short (LoRA)	92.41	81.85	0.0640	94.15	70.20	0.0399

Appendix

Table 13: Qwen3.5-2B trained on our two synthetic corpora, and the 500k model after continual LoRA training on 15k samples from the 276k corpus. BrailleBench covers both grades; WMT validation is synthetic and not seen in training. Higher chrF and exact match (EM) and lower CER are better; the best value per column is bold.

Figure 7: Qwen3.5-2B performance during continual LoRA training on samples from our 276k corpus, following pretraining on 500k longer synthetic samples.

Figure 8: Evaluation loss of Llama-3.2-11B-Vision-Instruct during one epoch of LoRA training at ranks 8, 16, and 32, on our 276k corpus (solid) and on WMT-Train (dashed). Lower WMT-Train loss does not mean better transcription: the two datasets differ in text length and difficulty.

BrailleBench ( $n=259$ )
Rank	chrF $\uparrow$	EM (%) $\uparrow$	CER $\downarrow$	WER $\downarrow$
8	88.28	72.59	0.4196	1.0704
16	88.57	72.20	0.0754	0.1444
32	90.10	76.06	0.0759	0.1261
WMT validation ( $n=1{,}000$ )
8	95.27	73.10	0.0595	0.0450

Appendix

Table 14: Transcription scores of Llama-3.2-11B-Vision-Instruct trained with LoRA on our 276k corpus, across LoRA ranks. Best scores within each evaluation set are bold.

Metric	Angelina OBR	Chandra-OCR-2 (ours)	Qwen3.5-2B 500k (ours)	Qwen3.5-2B 276k (ours)	Llama-3.2-11B (WMT-Train)
CER $\downarrow$	0.7030	0.0138	0.1046	0.0218	6.7342
Character accuracy $\uparrow$	0.2970	0.9862	0.8954	0.9782	0.0000
WER $\downarrow$	1.0159	0.0438	0.1713	0.0518	15.6175
Word accuracy $\uparrow$	0.0000	0.9562	0.8287	0.9482	0.0000
Normalized CER $\downarrow$	0.5650	0.0109	0.1017	0.0211	6.6754
Normalized WER $\downarrow$	0.8725	0.0359	0.1633	0.0478	15.6175

Appendix

Table 15: Transcription scores on 142 images from a Grade-1 children’s book that is not part of BrailleBench. Bold indicates the best value for each metric.

\downarrow

: lower is better;

\uparrow

: higher is better. CER: character error rate; WER: word error rate. ChrF is reported on a 0–100 scale; all other scores retain their reported decimal values.

Panel of Table 16 — Table 16: Qualitative comparison of Braille recognition on a Grade-1 Braille children’s book. The Angelina Braille Reader frequently introduces substitutions, omissions, and uncertain characters, whereas our fine-tuned Qwen3.5-2B model exactly recovers the reference text across these examples.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 999 papers

BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models
Aug 27, 2026Jinghan Zhang, Fengran Mo, Zhiyu Chen +3Assistive Technology for Blind and Low-Vision People LLM Evaluation
School of Computing Clemson University Clemson, USA · Department of Computer Science Rochester Institute of Technology Rochester, USA · Amazon.com, Inc. Seattle, USA +1
I'm Sorry, but I Can't Help with Braille: Revealing Accessibility Failures in State-of-the-Art LLMs
May 7, 2026Abdullah AbdullahLLM Evaluation LLM Fine-Tuning
orinu Inc. Hwaseong, Republic of Korea
Reading $\neq$ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models
Mar 9, 2026Heng Zhou, Ao Yu, Li Kang +5Vision-Language Models VLM Evaluation

Aug 27, 2026J/K move · Enter open · S save

BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models

Jinghan Zhang, Fengran Mo, Zhiyu Chen, Xiaoyan Han+2

School of Computing Clemson University Clemson, USA · Department of Computer Science Rochester Institute of Technology Rochester, USA · Amazon.com, Inc. Seattle, USA+1

Assistive Technology for Blind and Low-Vision People LLM Evaluation

Open paper AskPDF arXiv

Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way. However, it is unclear whether existing AI systems are inclusive enough for blind and deafblind users to access the same functionality through Braille, whose indicators, contractions, and digital representations introduce distinct requirements for model comprehension. To this end, we introduce BrailleBench, a benchmark for evaluating LLMs in Braille comprehension from different Criteria. BrailleBench aligns 5,570 instances from five datasets, including mathematics, commonsense, and multi-hop question answering across English and Braille Grades 1 and 2. Different configurations are designed to understand whether the systems can comprehend Braille-authored content, express answers in Braille, and complete end-to-end Braille interaction. To ensure the quality and prevent evaluation bias, the benchmark is built through a deterministic, expert-reviewed pipeline via a self-created Braille Toolkit without using any data instances generated by LLMs. We evaluate six representative LLMs from various aspects. The results reveal a persistent gap between print-English capability and Braille accessibility. Braille understanding and expression are asymmetric, where Grade 2 is especially fragile on the input side compared to Grade 1, and fully Braille requests further reduce performance. The experimental observations provide valuable guidance for the development of future Braille AI systems. All related resources in BrailleBench are publicly available for future research.

Abstract

VLM Evaluation

Explore similar work

BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models

I'm Sorry, but I Can't Help with Braille: Revealing Accessibility Failures in State-of-the-Art LLMs

Reading ≠\neq= Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models

Reading $\neq$ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models