Optical character recognition remains critical infrastructure for document digitization, yet state-of-the-art performance is often restricted to well-resourced institutions by prohibitive computational barriers. End-to-end transformer architectures achieve strong accuracy but demand hundreds of GPU hours for domain adaptation, limiting accessibility for practitioners and digital humanities scholars. We present a modular detection-and-correction framework that achieves near-SOTA accuracy with single-GPU training. Our approach decouples lightweight visual character detection (domain-agnostic) from domain-specific linguistic correction using pretrained sequence models including T5, ByT5, and BART. By training the correctors entirely on synthetic noise, we enable annotation-free domain adaptation without requiring labeled target images. Evaluating across modern clean handwriting, cursive script, and historical documents, we identify a critical "Pareto frontier" in architecture selection: T5-Base excels on modern text with standard vocabulary, whereas ByT5-Base dominates on historical documents by reconstructing archaic spellings at the byte level. Our results demonstrate that this decoupled paradigm matches end-to-end transformer accuracy while reducing compute by approximately 95%, establishing a viable, resource-efficient alternative to monolithic OCR architectures.
Figures & tables
Fig. 1 : The Domain Gap. We evaluate across a difficulty spectrum. While token-based models (T5) excel on (a), they struggle with the archaic vocabulary and noise in (c), necessitating the byte-level reconstruction of ByT5.
Fig. 2 : Decoupled detection-and-correction architecture. A detection-based visual module localizes and classifies characters in parallel and is pretrained on large-scale synthetic data, followed by lightweight domain adaptation using weak supervision (CTC loss). A pretrained language model corrector is fine-tuned to repair residual recognition errors and capture domain-specific linguistic patterns. Decoupling visual detection from linguistic correction enables efficient adaptation across writing styles and document domains.
Dataset
Domain
Vocabulary
Visual Noise
CVL
Modern
Standard
Low
IAM
Modern
Standard
High (cursive)
GW
Historical
Archaic
High (degradation)
TABLE I : Dataset characteristics across three handwriting domains.
Method
Architecture
Params
Training Data
Adaptation Cost
W. Acc (%)
CER (%)
MGP-STR †
Multi-Granularity ViT
148M
100M+ Synthetic
100+ GPU hrs
82.30
—
T5-Base (Ours)
Linguistic Adapter
220M
1.5M Synthetic
3 + 0.5 GPU hrs
78.10
1.90
BART-Base (Ours)
Linguistic Adapter
140M
1.5M Synthetic
3 + 0.5 GPU hrs
77.82
1.95
ByT5-Base (Ours)
Linguistic Adapter
582M
1.5M Synthetic
3 + 1.0 GPU hrs
76.45
1.98
TrOCR-Base ‡
End-to-End Transform.
334M
684M Synth Lines
200+ GPU hrs
74.50
3.42
Detector Baseline
Object Detector Only
40M
1M Synthetic
3 GPU hrs
72.84
3.71
TABLE II : CVL (Modern Clean Handwriting) Results.
Method
Architecture
Params
Training Data
Adaptation Cost
CER (%)
DTrOCR †
Decoder-Only Transform.
300M
2B+ Synthetic Lines
> 500 GPU hrs
2.38
TrOCR-Large ‡
End-to-End Transform.
558M
684M Synthetic Lines
300+ GPU hrs
2.89
TrOCR-Base ‡
End-to-End Transform.
334M
684M Synthetic Lines
200+ GPU hrs
3.42
BART-Base (Ours)
Linguistic Adapter
140M
1.5M Synthetic
3 + 0.5 GPU hrs
5.18
T5-Base (Ours)
Linguistic Adapter
220M
1.5M Synthetic
3 + 0.5 GPU hrs
5.40
ByT5-Base (Ours)
Linguistic Adapter
582M
1.5M Synthetic
3 + 1.0 GPU hrs
6.35
TABLE III : IAM (Modern Cursive Handwriting) Results.
Method
Architecture
Evaluation
CER (%)
WER (%)
Kumari et al. (2022)
CNN + BGRU + CTC
Supervised
4.88
14.56
BART-Base
Linguistic Adapter
Annotation-Free
5.20
13.80
ByT5-Base
Byte-Level Adapter
Annotation-Free
5.35
14.20
T5-Base
Token-Level Adapter
Annotation-Free
5.86
15.10
Detector Baseline (Ours)
DINO-DETR Visual
Supervised
3.70
11.20
TABLE IV : George Washington (Historical Handwriting) Results.
Category
Input (Detector Output)
T5 Prediction
Mechanism / Error Type
Success (English)
olider cultivatet plands
older cultivated plants
Phonetic Repair (Fixes multi-token noise)
plys on a qunte
plays on a quote
Contextual Disambiguation (plys → plays)
by Heinz Zamek
by Heinz Zemanek
Knowledge Prior (Restores Named Entity)
Failure (German)
deines Dienstes frey
deines Dienstes frei
Modernization Bias (Archaic y→i )
Zum Augenblicke
Zum Augenblick
Grammatical Flattening (Dative → Nom.)
Todtenglocke schallen
Zeit für mich schallen
Semantic Hallucination (Context Leakage)
TABLE V : Qualitative Error Analysis (CVL Dataset).
Despite impressive reported scores, large vision-language models have seen limited practical uptake in historical automatic text recognition because of their computational cost, dependence on large-scale pretraining, and hallucination. Historical ATR therefore continues to rely largely on compact CRNN line recognizers, which are visually grounded and trainable on modest data. Lightweight recurrence-free recognizers promise the accuracy of larger models with the practical advantages of CRNNs, yet have not been comprehensively evaluated on historical writing. We adapt PP-OCRv6, a recent compact text recognizer without strong language modeling, for historical line recognition and compare it with a conventional CRNN across generalized pretraining, domain-specific training, corpus-level fine-tuning, and manuscript-specific few-shot adaptation on multilingual Latin- and Arabic-script material. While PP-OCRv6 does not consistently outperform the baseline when trained from scratch, heterogeneous pretraining produces markedly better generalization. Comparisons with the Qwen3.5-based Medusa recognizer further show that fine-tuned PP-OCRv6 can outperform a large VLM tailored towards historical Latin-script HTR.
Optical Character Recognition (OCR) is a fundamental task for digitizing information, serving as a critical bridge between visual data and textual understanding. While modern Vision-Language Models (VLM) have achieved high accuracy in this domain, they predominantly rely on autoregressive decoding, which becomes computationally expensive and slow for long documents as it requires a sequential forward pass for every generated token. We identify a key opportunity to overcome this bottleneck: unlike open-ended generation, OCR is a highly deterministic task where the visual input strictly dictates a unique output sequence, theoretically enabling efficient, parallel decoding via diffusion models. However, we show that existing masked diffusion models fail to harness this potential; those introduce structural instabilities that are benign in flexible tasks, like captioning, but catastrophic for the rigid, exact-match requirements of OCR. To bridge this gap, we introduce DODO, the first VLM to utilize block discrete diffusion and unlock its speedup potential for OCR. By decomposing generation into blocks, DODO mitigates the synchronization errors of global diffusion. Empirically, our method achieves near state-of-the-art accuracy while enabling up to 5x faster inference compared to autoregressive baselines.
Sean Man, Gilad Deutch, Roy Ganz +4
Technion - Israel Institute of Technology, Haifa, Israel. · Amazon Web Services.
Large multimodal models (LMMs) have demonstrated strong OCR recognition capabilities, yet remain vulnerable to adversarial visual text that is readable to humans but challenging for models to localize and recognize. Existing OCR benchmarks mainly focus on natural or document-style text, while adversarial OCR evaluations remain limited in scale, task coverage, or region-aware evaluation. In this paper, we formulate adversarial OCR as a grounded OCR perception task and introduce AdvSpot, the first benchmark for grounded adversarial OCR evaluation. AdvSpot comprises 390 images with region-level annotations, spanning 5 primary categories and 13 fine-grained adversarial OCR types. To address this challenge, we propose ArmorOCR, a two-stage training framework for robust adversarial OCR perception. ArmorOCR first acquires missing adversarial OCR perception from privileged transformed observations through On-Policy Self-Distillation (OPSD), and then refines grounded OCR perception through Group Relative Policy Optimization (GRPO) with task-conditioned rewards for localization, recognition, full spotting, and visual question answering (VQA). Experiments on our AdvSpot, other adversarial OCR benchmarks, and general OCR benchmarks demonstrate that ArmorOCR consistently improves adversarial OCR perception while preserving competitive general OCR capability.
Linhan Cao, Siyuan Li, Jun Lan +8
Ant Group · Shanghai Jiao Tong University · East China Normal University