Organizations: School of Computing Clemson University Clemson, USA · Department of Computer Science Rochester Institute of Technology Rochester, USA · Amazon.com, Inc. Seattle, USA · Department of Computer Science Virginia Tech Alexandria, USA
Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way. However, it is unclear whether existing AI systems are inclusive enough for blind and deafblind users to access the same functionality through Braille, whose indicators, contractions, and digital representations introduce distinct requirements for model comprehension. To this end, we introduce BrailleBench, a benchmark for evaluating LLMs in Braille comprehension from different Criteria. BrailleBench aligns 5,570 instances from five datasets, including mathematics, commonsense, and multi-hop question answering across English and Braille Grades 1 and 2. Different configurations are designed to understand whether the systems can comprehend Braille-authored content, express answers in Braille, and complete end-to-end Braille interaction. To ensure the quality and prevent evaluation bias, the benchmark is built through a deterministic, expert-reviewed pipeline via a self-created Braille Toolkit without using any data instances generated by LLMs. We evaluate six representative LLMs from various aspects. The results reveal a persistent gap between print-English capability and Braille accessibility. Braille understanding and expression are asymmetric, where Grade 2 is especially fragile on the input side compared to Grade 1, and fully Braille requests further reduce performance. The experimental observations provide valuable guidance for the development of future Braille AI systems. All related resources in BrailleBench are publicly available for future research.
Figures & tables
(a) Mathematical expression
Print EN
Evaluate the limit as x approaches 0 : limx→0sinx1+x+x2−1−x+x2.
G1
,evaluate the limit as x approaches #j3 "<sqrt"<#a "6 x "6 x‘5#b"> - sqrt"<#a - x "6 x‘5#b">">_/ "<sin"<x">">4
(b) Natural-language question
EN
Who is older, Annie Morton or Terry Richardson?
G1
,who is older1 ,annie ,morton or ,terry ,richardson8
G2
,:o is old}1 ,annie ,morton or ,t}ry ,ri*>dson8
Fig. 1: Two examples to illustrate the challenges in Braille-based interaction. (a) How print mathematics will be conveyed into Grade 1 Braille as a longer sequence of cells and indicators. (b) The same question will be expressed in different sequences in terms of Grade 1 and Grade 2 based on contractions.
Fig. 2: Overview of the BrailleBench construction and evaluation pipeline. Five source datasets are adapted to seven English–Braille interaction protocols, normalized by task type, and deterministically transcribed into UEB Grade 1 and Grade 2 Braille with liblouis ; the aligned Braille ASCII records are then validated, supplemented with Unicode and dot-number representations, and evaluated on LLMs using direction-aware scoring.
Dataset
Task
Answer Format
GSM8K
Grade-school arithmetic
Integer
AIME 2024
Competition mathematics
Integer
CommonsenseQA
Commonsense reasoning
Multi-choice
HotpotQA
Multi-hop QA
Answer span
2WikiMultiHopQA
Multi-hop QA
Answer span
TABLE I: Composition of BrailleBench. Each data sample is aligned with English, Grade 1 Braille, and Grade 2 Braille formats.
Model
GSM8K
AIME 2024
CommonsenseQA
HotpotQA
2Wiki
Avg. Drop
G1
G2
G1
G2
G1
G2
G1
G2
G1
G2
G1
G2
E→E Reference
Opus 4.8 + CoT
97.4
93.3
88.8
42.7
56.1
–
Claude Opus 4.8
97.3
100.0
88.6
40.0
46.7
–
Claude Haiku 4.5
96.2
53.3
81.7
26.4
20.9
–
Llama 3.3 70B
95.1
23.3
80.4
29.7
29.7
–
TABLE II: Main experimental results organized by input–output direction. All Δ annotations continue to use the corresponding model’s standard EN-EN score. Bold denotes the best task score within each directional setting and grade.
Variant
Model-Facing Prompt
Gold
GSM8K
EN-EN
Solve the following math problem step by step. Answer in plain English. Put your final numerical answer within \boxed{}. Problem: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market?
18
G1-EN
The following question is written in Grade 1 (uncontracted) Braille ASCII notation. Solve the following math problem step by step. Answer in plain English. Put your final numerical answer within \boxed{}. Problem: ,janet’s ducks lay #af eggs per day4 ,she eats three for breakfast every morning and bakes muffins for her friends every day with four4 ,she sells the remainder at the farmers’ market daily for ‘s#b per fresh duck egg4 ,how much in dollars does she make every day at the farmers’ market8
18
G2-EN
The following question is written in Grade 2 (contracted) Braille ASCII notation. Solve the following math problem step by step. Answer in plain English. Put your final numerical answer within \boxed{}. Problem: ,janet’s ducks lay #af e7s p} "d4 ,%e eats ?ree = br1kfa/ e morn+ & bakes mu69s = h} frs e "d ) f|r4 ,%e sells ! rema9d} at ! f>m}s’ m>ket daily = ‘s#b p} fre% duck egg4 ,h{ m* 9 doll>s does %e make e "d at ! f>m}s’ m>ket8
18
FULLBR-G1
,solve the following math problem step by step4 ,answer in plain ,english4 ,put your final numerical answer within *boxed<_>4 ,problem3 ,janet’s ducks lay #af eggs per day4 ,she eats three for breakfast every morning and bakes muffins for her friends every day with four4 ,she sells the remainder at the farmers’ market daily for ‘s#b per fresh duck egg4 ,how much in dollars does she make every day at the farmers’ market8
18
FULLBR-G2
,solve ! foll{+ ma? problem /ep by /ep4 ,answ} 9 pla9 ,5gli%4 ,put yr f9al num}ical answ} )9 *box$<_>4 ,problem3 ,janet’s ducks lay #af e7s p} "d4 ,%e eats ?ree = br1kfa/ e morn+ & bakes mu69s = h} frs e "d ) f|r4 ,%e sells ! rema9d} at ! f>m}s’ m>ket daily = ‘s#b p} fre% duck egg4 ,h{ m* 9 doll>s does %e make e "d at ! f>m}s’ m>ket8
18
TABLE III: Representative GSM8K and CommonsenseQA prompt variants. FULLBR-G1 and FULLBR-G2 translate both the instruction and task content into Braille, retain English as the requested answer language, and are evaluated separately from the seven core configurations. Line breaks are collapsed for compact presentation.
Dataset
EN
G1
G2
G2 → E Empty
G2 → G2 Empty
GSM8K
253
978
1,518
12.1
10.4
AIME 2024
1,289
2,214
2,555
20.0
23.3
CommonsenseQA
25
29
519
54.7
56.9
HotpotQA
88
516
910
64.9
72.2
2Wiki
163
588
940
71.6
78.4
TABLE IV: The average response length in non-empty examples for Opus under EN-EN, G1-EN, and G2-EN language settings. The right two columns report the percentage of empty output examples in G2-EN and G2-G2.
Model
GSM8K
AIME
CSQA
Hotpot
2Wiki
Claude Opus 4.8
6.0
2.0
20.5
10.4
5.8
Claude Haiku 4.5
2.2
1.0
17.8
4.3
3.1
Llama 3.3 70B
2.3
0.7
0.7
1.1
1.1
Llama 3.1 8B
1.7
1.2
3.8
4.0
4.2
Qwen3 32B
1.5
1.1
0.8
2.2
1.6
TABLE V: Ratio of average non-empty G2-EN response length to the same model’s EN-EN response length. Values below one indicate that the model produced less text under Braille input rather than externalizing a decode.
Configuration
Reasoning Activation
Reasoning Deactivation
Average Length
EN-EN + CoT
99%
1%
278
EN-EN
57%
43%
163
G1-EN
97%
3%
588
G1-G1
99%
1%
790
Configuration
EM
Sub-EM
F1
EN-EN + CoT
56.1
64.0
65.5
TABLE VI: Response activation and length of Opus on 2WikiMultiHopQA.
Model
GSM8K
AIME
CommQA
HotpotQA
2Wiki
G1-EN / FULLBR-G1
Opus 4.8
89 / 73
90 / 90
86 / 87
43 / 40
53 / 46
Haiku 4.5
34 / 5
3.3 / 6.7
28 / 14
36 / 21
34 / 19
Llama 3.3 70B
9 / 5
0 / 0
11 / 20
18 / 21
22 / 14
Llama 3.1 8B
6 / 7
0 / 0
0 / 0
11 / 12
13 / 20
Qwen3 32B
14 / 4
0 / 0
54 / 27
11 / 16
23 / 30
TABLE VII: Effect of transcribing the task instruction into Braille across all five diagnostic subsets. Each cell reports G1-EN / FULLBR-G1 or G2-EN / FULLBR-G2 on the same 100-example (30-example on AIME 2024 prefix.
Model
GSM8K
AIME
CommQA
HotpotQA
2Wiki
Grade 1: ASCII / Unicode / Dots
Opus 4.8
89/0/0
90/0/0
86/0/0
43/0/0
53/0/0
Haiku 4.5
34/48/1
3.3/10/0
28/19/0
36/24/0
34/29/0
Llama 3.3 70B
9/4/0
0/0/0
11/2/0
18/2/0
22/0/0
Llama 3.1 8B
6/0/1
0/0/0
0/0/0
11/0/0
13/0/0
Qwen3 32B
14/19/0
0/6.7/0
54/23/0
11/5/0
23/6/0
TABLE VIII: Surface-format comparison on the diagnostic subsets. Each cell reports Braille ASCII / Unicode Braille / dot-number notation for the indicated grade.
Model
GSM8K
AIME
CommQA
HotpotQA
2Wiki
Grade 1: no reference / with reference
Opus 4.8
89.0/97.0
90.0/90.0
86.0/82.0
43.0/45.0
53.0/51.0
Haiku 4.5
34.0/73.0
3.3/23.3
28.0/37.0
36.0/39.0
34.0/37.0
Llama 3.3 70B
9.0/46.0
0.0/6.7
11.0/12.0
18.0/19.0
22.0/20.0
Llama 3.1 8B
6.0/11.0
0.0/0.0
0.0/1.0
11.0/8.0
13.0/12.0
Qwen3 32B
14.0/47.0
0.0/6.7
54.0/37.0
11.0/13.0
23.0/24.0
TABLE IX: Ablation on the effectiveness of completed Braille-reference. Each cell compares the same subset without/with the supplied mapping.
Model
Characters
Numbers
G1
G2
Sentences
Grade 1
Opus 4.8
97
97
100
98
40/88
Haiku 4.5
90
33
88
100
34/70
Llama 3.3 70B
92
13
100
98
8/56
Llama 3.1 8B
70
10
75
88
18/46
Qwen3 32B
68
37
100
100
30/72
TABLE X: Complete direct-decoding probe results. All cells report strict EM; sentence cells additionally report punctuation-insensitive EM after the slash. Grade 1 and Grade 2 denote the prompt encoding, while the two word columns denote the type of lexical item being transcribed.
Large Language Models (LLMs) perform strongly on many language tasks, but their capability in structurally constrained, accessibility-critical modalities such as Braille remains unclear. We evaluate state-of-the-art LLMs on bidirectional Korean-Braille translation using a human-annotated dataset. Despite expectations that multilingual, instruction-tuned models can generalize to Braille via text representations, we find consistently poor, unstable outputs and substantial disagreement with human judgments. These results point to missing Braille-aware tokenization and weak alignment between Korean and Braille patterns. In contrast, supervised fine-tuning of a small model (T5-small) on the same data yields large and stable gains over zero-shot and prompted LLM baselines across standard metrics (SacreBLEU, ChrF++, CER, BLEU, ROUGE-L, METEOR, CIDEr). Our findings reveal a systematic limitation of current LLMs and demonstrate the effectiveness of modest task-specific supervision.
Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.
Unsupervised automatic readability assessment (ARA) methods have important practical and research applications (e.g., ensuring medical or educational materials are suitable for their target audiences). In this paper, we propose a new zero-shot prompting methodology for ARA and present the first comprehensive evaluation of using large language models (LLMs) as an unsupervised ARA method by testing 10 diverse open-source LLMs (e.g., different sizes and developers) on 14 diverse datasets (e.g., different text lengths and languages). Our findings show that our proposed prompting methodology outperforms prior methods on 13 of the 14 datasets. Furthermore, we propose LAURAE, which combines LLM and readability formula scores to improve robustness by capturing both contextual and shallow (e.g., sentence length) features of readability. Our evaluation demonstrates that LAURAE robustly outperforms prior methods across languages, text lengths, and amounts of technical language.