Transformer-based encoder-decoder networks have recently achieved impressive results in handwritten text recognition, partly thanks to their auto-regressive decoder which implicitly learns a language model. However, such networks suffer from a large performance drop when evaluated on a target corpus whose language distribution is shifted from the source text seen during training. To retain recognition accuracy despite this language shift, we propose an external n-gram injection (NGI) for dynamic adaptation of the network's language modeling at inference time. Our method allows switching to an n-gram language model estimated on a corpus close to the target distribution, therefore mitigating bias without any extra training on target image-text pairs. We opt for an early injection of the n-gram into the transformer decoder so that the network learns to fully leverage text-only data at the low additional cost of n-gram inference. Experiments on three handwritten datasets demonstrate that the proposed NGI significantly reduces the performance gap between source and target corpora.
Figures & tables
Figure 1: n -gram injection (NGI) into the auto-regressive decoder of the word attention network (WAN). An n -gram distribution vector SNGI is made noisy by function ϕ , projected by f then summed with standard embeddings of previously output characters ξ(c) and positional encoding P to form the new decoder input X . Switching to an appropriate n -gram at test time reduces the error gap on target corpora.
Dataset
Split
Size
5 -gram PPL
Source
Target
Source dev.
Target NG
Dev.
Test
NG
Test
Source
Target
Target
IAM
Lexicon
69,595
6,507
6,511
7,187
4.6
14.5
4.3
k -means
69,595
6,730
6,511
6,964
4.6
26.2
3.8
RIMES
Lexicon
51,908
5,022
4,857
5,191
3.1
8.3
2.4
k -means
51,908
5,670
4,857
4,543
3.1
66.4
2.4
Table 1: Size and perplexity of the five word-level HTR dataset splits. Dev. denotes the source development set, NG the target n -gram set.
Figure 2: Scans of names (left) and surnames (right) from the N2S dataset.
Model
IAM
RIMES
Lexicon split
k -means split
Lexicon split
k -means split
Source
Target
Source
Target
Source
Target
Source
Target
TrOCR [ 22 ]
6.3
12.9
6.7
21.4
3.3
12.9
3.6
33.3
DAN [ 7 ]
6.0
15.0
6.1
26.1
3.7
16.8
3.9
40.2
SaLT [ 25 ]
15.5
25.1
15.5
29.8
10.7
25.9
12.3
38.6
WAN 3 3 3 For a fair comparison, here the WAN is trained without TFE to match the training strategy of other models.
6.9
15.8
7.6
23.4
4.7
18.1
4.3
29.9
Table 2: CER(%) of transformer-based models trained on the four custom dataset splits from IAM and RIMES. All four models suffer from a significant performance gap between source and target subsets, including the WAN without NGI.
Model
Source
Target
TrOCR [ 22 ]
2.2
7.1
DAN [ 7 ]
2.1
5.3
SaLT [ 25 ]
4.0
13.4
WAN 3
2.5
6.3
Table 3: CER(%) of transformer-based models trained on the name to surname (N2S) dataset. As on IAM and RIMES, all four models suffer from a performance gap between source and target corpora, including the WAN without NGI.
Model
IAM
RIMES
Lexicon split
k -means split
Lexicon split
k -means split
Source
Target
Source
Target
Source
Target
Source
Target
WAN
6.0
13.2
6.8
18.0
3.4
12.2
4.0
25.4
WAN+NGI
6.2
11.8
6.8
10.1
3.5
8.8
3.6
19.2
WAN+LM
5.9
10.6
6.8
9.7
3.3
6.6
3.7
13.8
WAN+NGI+LM
6.5
11.5
6.9
8.4
3.4
5.1
3.7
10.7
Table 4: CER(%) of the proposed NGI compared to the WAN alone on the four custom dataset splits from IAM and RIMES. NGI consistently reduces the performance gap from source to target subsets. A post-processing LM, denoted by +LM, further reduces the gap. We highlight in bold the best results on target test sets as they are our main objective.
Model
Source
Target
WAN
2.2
5.0
WAN+NGI
2.4
4.5
Table 5: CER(%) of the proposed NGI compared to the WAN alone on the name to surname (N2S) dataset. As on IAM and RIMES, NGI reduces the drop on the target subset. We highlight in bold the best result on the target test set as it is our main objective.
Model
IAM
RIMES
Source
Target
Source
Target
WAN+NGI (5-gram)
6.8
10.1
3.6
19.2
- Noise ϕ
6.6
10.6
3.7
19.2
- TFE
10.3
16.5
5.9
20.8
3-gram
6.8
18.1
3.2
26.2
2-gram
6.7
19.3
3.3
25.5
Table 6: Ablation study of our proposal (WAN+NGI) by removing specific components of NGI and reducing the n -gram order on the k -means splits from IAM and RIMES. We report CER(%) when removing noise or the TFE training strategy from NGI, and lowering n to {3,2} for NGI. Ablations are not cumulative.