While state-of-the-art Handwritten Text Recognition (HTR) models perform well on standard benchmarks, they frequently struggle with writers exhibiting highly specific styles that are underrepresented in the training data. To handle unseen and atypical writers, writer adaptation techniques personalize HTR models to individual handwriting styles. Leading writer adaptation methods require either offline fine-tuning or parameter updates at inference time, both involving gradient computation and backpropagation, which increase computational costs and demand careful hyperparameter tuning. In this work, we propose a novel context-driven HTR framework3 inspired by multimodal in-context learning, enabling inference-time writer adaptation using only a few examples from the target writer without any parameter updates. We further demonstrate the impact of context length, design a compact 8M-parameter CNN-Transformer that enables few-shot in-context adaptation, and show that combining context-driven and standard OCR training strategies leads to complementary improvements. Experiments on IAM and RIMES validate our approach with Character Error Rates of 3.92% and 2.34%, respectively, surpassing all writer-independent HTR models without requiring any parameter updates at inference time.
Figures & tables
Figure 1: Qualitative examples of inter-writer variation in the IAM dataset, highlighting writer-specific characteristics in letter formation.
Figure 2: By integrating context lines Xc and their corresponding labels Yc from the same author as the query X , our context-driven model effectively captures the writer’s stylistic characteristics. Leveraging these few in-context examples significantly reduces errors caused by writer-specific variations.
Figure 3: Illustration of our Context-driven architecture, structured around three core components: (1) a Context-Aware Tokenizer (2) a CNN encoder and (3) a Transformer decoder
Figure 4: Confidence-based fusion of Context-Driven and Standard OCR predictions. ⟨ooc⟩ prediction is represented by the symbol ’*’
Figure 5: ⟨out-of-context⟩ token rate vs. number of context lines for IAM and RIMES.
IAM
RIMES
K
TER (%)
F1 ⟨ ooc ⟩ (%)
TER (%)
F1 ⟨ ooc ⟩ (%)
1
6.30
92.21
5.13
92.60
3
4.99
88.20
3.15
93.14
5
4.73
85.34
2.91
91.18
7
4.69
83.26
2.87
89.92
9
4.64
82.61
2.92
88.69
Table 1: Analysis of the Context-Driven model as a function of K , the maximum number of context lines, on IAM and RIMES. TER : Token Error Rate; F1 ⟨ ooc ⟩ : F1-score of the ⟨ ooc ⟩ token prediction.
Figure 6: Distribution of writers according to which model achieves the lowest Character Error Rate (CER) on fully context-covered lines. “Equal” denotes identical CER for both models. Maximum of 9 context lines.
Figure 7: Qualitative samples showing improvements obtained with context-driven few-shot adaptation over a baseline OCR model.
IAM
RIMES
K
CER (%)
WER (%)
CER (%)
WER (%)
0
4.22
14.10
2.91
8.67
1
4.20
14.60
2.83
9.91
3
4.00
13.96
2.47
8.86
5
3.96
14.03
2.41
8.38
7
3.94
13.93
2.35
8.08
Table 2: Effect of the number of context lines K on fusion performance on IAM and RIMES. The row at K=0 corresponds to the standard OCR model used as baseline, without any fusion.
Method
Architecture
IAM
RIMES
Param. updates
CER
WER
CER
WER
Without test-time adaptation
SPAN [ 9 ]
FCN
5.45%
19.83%
3.81%
13.8%
No
Decoupled AN [ 38 ]
CNN-RNN + Attn
6.4%
19.6%
2.7%
8.9%
No
Kang et al. [ 19 ]
CNN-Transf.
4.67%
15.45%
–
–
No
Retsinas et al. [ 22 ]
CNN-BiLSTM
4.62%
15.89%
2.75%
9.93%
No
Table 3: Line-level comparison of handwritten text recognition (HTR) methods on the IAM and RIMES datasets. “–” indicates values not reported in the original publication; ∗ denotes paragraph-level evaluation. Ours-baseline refers to our model without any writer-specific adaptation. MAE denotes a Masked Autoencoder-based image reconstruction auxiliary decoder.
Handwritten Text Recognition (HTR) models trained on synthetic handwriting often struggle to generalize to real text, and existing adaptation methods still require real samples from the target domain. In this work, we tackle the fully zero-shot synthetic-to-real generalization setting, where no real data from the target language is available. Our approach learns how model parameters change when moving from synthetic to real handwriting in one or more source languages and transfers this learned correction to new target languages. When using multiple sources, we rely on linguistic similarity to weigh their contrubition when combining them. Experiments across five languages and six architectures show consistent improvements over synthetic-only baselines and reveal that the transferred corrections benefit even languages unrelated to the sources.
Carlos Garrido-Munoz, Aniello Panariello, Silvia Cascianelli +4
University of Alicante, Spain · University of Modena and Reggio Emilia, Italy
Handwritten Text Recognition (HTR) systems have become an indispensable tool for the digitization of historical documents. Not only do they cut down time and cost, but they also allow democratizing access and processing of their contents by generating their transcripts. However, literature in HTR currently focuses mostly on specialized models that require large amounts of annotated samples to achieve satisfactory performance. We explore the use of In-Context Learning with pre-trained Vision-Language Models (VLMs) to create a transcription pipeline without updating the model's parameters. We then evaluate this pipeline across multiple collections and models, and demonstrate that general-purpose VLMs can be effectively taught how to transcribe handwritten text from images. To assess how our observations may translate to practical applications, we evaluate the performance in a Cross-Domain (CD) scenario, where context examples are drawn from a different collection than the query image. Results in both the controlled In-Domain (ID) scenario and the realistic CD scenario follow the same patterns. First, as context size grows, the error range is expected to narrow towards the average performance. Thus, larger context sizes sacrifice the performance of the oracle-best sampling for lower expected error rates. The results obtained show that, without any parameter updates, this methodology has strong potential to compete with traditional HTR in the presence of domain shift. Moreover, we show and argue that some context samplings work better than others and suggest more effort should be put into finding an ideal sampling method in future work.
Handwritten mathematical expression recognition (HMER) refers to the task of recognizing and converting handwritten mathematics into a parsable markup language, usually LaTeX. No current state-of-the-art-competitive system adjusts to the way a specific person writes, and the domain gap between training images (usually digital or perfectly binarized) and images physically taken with a camera used in inference has received fairly little attention for this specific problem. We characterize this domain gap through fragmentation and stroke-width analyses of the images as well as introduce a writer-adaptive fine-tuning pipeline to MFH-CoMER in an attempt to address it. We further introduce sample author-specific datasets, consisting of five handwriting category subsets from two authors, and evaluate using a McNemar's test and permutation tests adapted to limited data. Results suggest an increase in expression recognition rate and a decrease in CER for digital handwriting but more varied results for physical handwriting, with the model struggling for handwriting articles that the base model can already evaluate well. Nonetheless, adapted models were found to have improved results for four out of five subsets. Statistical testing results suggest a consistency in improvement for two of the tested author-specific subsets. Our results point towards the potential feasibility of writer adaptation for the HMER task.
Paulo Grane Gabriel Silva, Lorenz Bernard Marqueses, Joel Ilao
Department of Computer Technology De La Salle University Manila, Philippines