While state-of-the-art Handwritten Text Recognition (HTR) models perform well on standard benchmarks, they frequently struggle with writers exhibiting highly specific styles that are underrepresented in the training data. To handle unseen and atypical writers, writer adaptation techniques personalize HTR models to individual handwriting styles. Leading writer adaptation methods require either offline fine-tuning or parameter updates at inference time, both involving gradient computation and backpropagation, which increase computational costs and demand careful hyperparameter tuning. In this work, we propose a novel context-driven HTR framework3 inspired by multimodal in-context learning, enabling inference-time writer adaptation using only a few examples from the target writer without any parameter updates. We further demonstrate the impact of context length, design a compact 8M-parameter CNN-Transformer that enables few-shot in-context adaptation, and show that combining context-driven and standard OCR training strategies leads to complementary improvements. Experiments on IAM and RIMES validate our approach with Character Error Rates of 3.92% and 2.34%, respectively, surpassing all writer-independent HTR models without requiring any parameter updates at inference time.
Figures & tables
Figure 1: Qualitative examples of inter-writer variation in the IAM dataset, highlighting writer-specific characteristics in letter formation.
Figure 2: By integrating context lines Xc and their corresponding labels Yc from the same author as the query X , our context-driven model effectively captures the writer’s stylistic characteristics. Leveraging these few in-context examples significantly reduces errors caused by writer-specific variations.
Figure 3: Illustration of our Context-driven architecture, structured around three core components: (1) a Context-Aware Tokenizer (2) a CNN encoder and (3) a Transformer decoder
Figure 4: Confidence-based fusion of Context-Driven and Standard OCR predictions. ⟨ooc⟩ prediction is represented by the symbol ’*’
Figure 5: ⟨out-of-context⟩ token rate vs. number of context lines for IAM and RIMES.
IAM
RIMES
K
TER (%)
F1 ⟨ ooc ⟩ (%)
TER (%)
F1 ⟨ ooc ⟩ (%)
1
6.30
92.21
5.13
92.60
3
4.99
88.20
3.15
93.14
5
4.73
85.34
2.91
91.18
7
4.69
83.26
2.87
89.92
9
4.64
82.61
2.92
88.69
Table 1: Analysis of the Context-Driven model as a function of K , the maximum number of context lines, on IAM and RIMES. TER : Token Error Rate; F1 ⟨ ooc ⟩ : F1-score of the ⟨ ooc ⟩ token prediction.
Figure 6: Distribution of writers according to which model achieves the lowest Character Error Rate (CER) on fully context-covered lines. “Equal” denotes identical CER for both models. Maximum of 9 context lines.
Figure 7: Qualitative samples showing improvements obtained with context-driven few-shot adaptation over a baseline OCR model.
IAM
RIMES
K
CER (%)
WER (%)
CER (%)
WER (%)
0
4.22
14.10
2.91
8.67
1
4.20
14.60
2.83
9.91
3
4.00
13.96
2.47
8.86
5
3.96
14.03
2.41
8.38
7
3.94
13.93
2.35
8.08
Table 2: Effect of the number of context lines K on fusion performance on IAM and RIMES. The row at K=0 corresponds to the standard OCR model used as baseline, without any fusion.
Method
Architecture
IAM
RIMES
Param. updates
CER
WER
CER
WER
Without test-time adaptation
SPAN [ 9 ]
FCN
5.45%
19.83%
3.81%
13.8%
No
Decoupled AN [ 38 ]
CNN-RNN + Attn
6.4%
19.6%
2.7%
8.9%
No
Kang et al. [ 19 ]
CNN-Transf.
4.67%
15.45%
–
–
No
Retsinas et al. [ 22 ]
CNN-BiLSTM
4.62%
15.89%
2.75%
9.93%
No
Table 3: Line-level comparison of handwritten text recognition (HTR) methods on the IAM and RIMES datasets. “–” indicates values not reported in the original publication; ∗ denotes paragraph-level evaluation. Ours-baseline refers to our model without any writer-specific adaptation. MAE denotes a Masked Autoencoder-based image reconstruction auxiliary decoder.