Zero-Shot Synthetic-to-Real Handwritten Text Recognition via Task Analogies
Organizations: University of Alicante, Spain · University of Modena and Reggio Emilia, Italy
Abstract
Handwritten Text Recognition (HTR) models trained on synthetic handwriting often struggle to generalize to real text, and existing adaptation methods still require real samples from the target domain. In this work, we tackle the fully zero-shot synthetic-to-real generalization setting, where no real data from the target language is available. Our approach learns how model parameters change when moving from synthetic to real handwriting in one or more source languages and transfers this learned correction to new target languages. When using multiple sources, we rely on linguistic similarity to weigh their contrubition when combining them. Experiments across five languages and six architectures show consistent improvements over synthetic-only baselines and reveal that the transferred corrections benefit even languages unrelated to the sources.
Figures & tables
| TrOCR (S) [ 32 ] | TrOCR (B) [ 32 ] | TrOCR (L) [ 32 ] | CRNN [ 50 ] | VAN [ 13 ] | HTR-VT [ 33 ] | |||||||||||||
| CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | |||||||
| In-domain | 5.0 | 16.7 | 4.4 | 14.8 | 4.3 | 14.1 | 4.2 | 16.0 | 5.4 | 21.8 | 5.3 | 21.8 | ||||||
| Baseline | 39.7 (+0.0) | 72.4 (+0.0) | 37.7 (+0.0) | 69.1 (+0.0) | 36.2 (+0.0) | 67.8 (+0.0) | 38.8 (+0.0) | 75.6 (+0.0) | 35.2 (+0.0) | 76.3 (+0.0) | 41.7 (+0.0) | 81.8 (+0.0) | ||||||
| Single analogy | ||||||||||||||||||
| 37.9\text{{\color[rgb]{0,0.6,0}\scriptsize(-1.8)}} | 72.8\text{{\color[rgb]{1,0,0}\scriptsize(+0.4)}} | 36.9\text{{\color[rgb]{0,0.6,0}\scriptsize(-0.8)}} | 70.9\text{{\color[rgb]{1,0,0}\scriptsize(+1.8)}} | 35.3\text{{\color[rgb]{0,0.6,0}\scriptsize(-0.9)}} | 73.1\text{{\color[rgb]{1,0,0}\scriptsize(+5.3)}} | 35.9\text{{\color[rgb]{0,0.6,0}\scriptsize(-2.9)}} | 73.3\text{{\color[rgb]{0,0.6,0}\scriptsize(-2.2)}} | 34.5\text{{\color[rgb]{0,0.6,0}\scriptsize(-0.7)}} | 78.0\text{{\color[rgb]{1,0,0}\scriptsize(+1.7)}} | 40.6\text{{\color[rgb]{0,0.6,0}\scriptsize(-1.1)}} | 83.7\text{{\color[rgb]{1,0,0}\scriptsize(+1.9)}} | |||||||
| 35.0\text{{\color[rgb]{0,0.6,0}\scriptsize(-4.7)}} | 68.2\text{{\color[rgb]{0,0.6,0}\scriptsize(-4.2)}} | 32.9\text{{\color[rgb]{0,0.6,0}\scriptsize(-4.8)}} | 64.6\text{{\color[rgb]{0,0.6,0}\scriptsize(-4.5)}} | 32.4\text{{\color[rgb]{0,0.6,0}\scriptsize(-3.8)}} | 64.5\text{{\color[rgb]{0,0.6,0}\scriptsize(-3.3)}} | 35.4\text{{\color[rgb]{0,0.6,0}\scriptsize(-3.4)}} | 70.0\text{{\color[rgb]{0,0.6,0}\scriptsize(-5.6)}} | 31.7\text{{\color[rgb]{0,0.6,0}\scriptsize(-3.5)}} | 72.1\text{{\color[rgb]{0,0.6,0}\scriptsize(-4.2)}} | 36.6\text{{\color[rgb]{0,0.6,0}\scriptsize(-5.1)}} | 77.1\text{{\color[rgb]{0,0.6,0}\scriptsize(-4.7)}} | |||||||
| Model | CER oracle | CER heldout | WER oracle | WER heldout |
| TrOCR(S) | 37.4\text{{\color[rgb]{1,0,0}\scriptsize(+1.2)}} | 70.0\text{{\color[rgb]{1,0,0}\scriptsize(+2.1)}} | ||
| TrOCR(B) | 33.3\text{{\color[rgb]{1,0,0}\scriptsize(+1.4)}} | 65.6\text{{\color[rgb]{1,0,0}\scriptsize(+2.9)}} | ||
| TrOCR(L) | 32.8\text{{\color[rgb]{1,0,0}\scriptsize(+1.5)}} | 66.4\text{{\color[rgb]{1,0,0}\scriptsize(+4.1)}} | ||
| CRNN | 37.0\text{{\color[rgb]{1,0,0}\scriptsize(+0.4)}} | 71.9\text{{\color[rgb]{1,0,0}\scriptsize(+0.8)}} | ||
| VAN | 31.4\text{{\color[rgb]{1,0,0}\scriptsize(+1.0)}} | 72.1\text{{\color[rgb]{1,0,0}\scriptsize(+2.0)}} | ||
| HTR-ViT | 35.7\text{{\color[rgb]{1,0,0}\scriptsize(+1.1)}} | 76.6\text{{\color[rgb]{1,0,0}\scriptsize(+2.2)}} |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Lang. | Period | No. writ | Train | Val | Test | |
| IAM [ 39 ] | En | 1999 | 657 | 6.4k | 976 | 2.9k | 79 |
| Rimes [ 29 ] | Fr | 2011 | 1.3k | 10k | 1.1k | 778 | 100 |
| LAM [ 12 ] | It | 17-18th c. | 1 | 20k | 2.5k | 3.5k | 89 |
| Rodrigo [ 58 ] | Sp | 1545 | 1 | 20k | 1k | 5k | 115 |
| ICFHR 2016 [ 55 ] | De | 15-19th c. | unk. | 8.2k | 1k | 1k | 91 |
| Model | Type | Align. | Vocab | Params | Input size |
| CRNN [ 50 ] | CRNN | CTC | 95 | 9.6M | 128 1024 |
| VAN [ 13 ] | FCN | CTC | 95 | 2.7M | 64 1024 |
| HTR-VT [ 33 ] | R 18 +ViT | CTC | 95 | 53.5M | 64 512 |
| TrOCR(S) [ 32 ] | ViT+T.Dec | AR | 8192 | 33M | 384 384 |
| TrOCR(B) [ 32 ] | ViT+T.Dec | AR | 8192 | 299M | 384 384 |
| TrOCR(L) [ 32 ] | ViT+T.Dec | AR | 8192 | 523M | 384 384 |
| IAM (en) | Rimes (fr) | LAM (it) | Rodrigo (es) | AVG. | ||||||||||||||
| CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | |||||||
| In-domain | 7.7 | 21.8 | 4.6 | 12.5 | 4.1 | 13.2 | 2.1 | 10.7 | 6.3 | 25.2 | 5.0 | 16.7 | ||||||
| Baseline | 16.4 (+0.0) | 39.7 (+0.0) | 24.1 (+0.0) | 59.1 (+0.0) | 46.9 (+0.0) | 78.9 (+0.0) | 30.3 (+0.0) | 76.6 (+0.0) | 80.7 (+0.0) | 107.8 (+0.0) | 39.7 (+0.0) | 72.4 (+0.0) | ||||||
| Single analogy | ||||||||||||||||||
| 16.3 (-0.1) | 40.6 (+0.9) | 29.0 (+4.9) | 66.5 (+7.4) | 36.4 (-10.5) | 72.0 (-6.9) | 23.8 (-6.5) | 70.3 (-6.3) | 84.0 (+3.3) | 114.8 (+7.0) | 37.9 (-1.8) | 72.8 (+0.4) | |||||||
| 12.4 (-4.0) | 33.4 (-6.3) | 24.8 (+0.7) | 61.5 (+2.4) | 33.8 (-13.1) | 68.2 (-10.7) | 23.8 (-6.5) | 70.3 (-6.3) | 80.3 (-0.4) | 107.6 (-0.2) | 35.0 (-4.7) | 68.2 (-4.2) | |||||||
| IAM (en) | Rimes (fr) | LAM (it) | Rodrigo (es) | AVG. | ||||||||||||||
| CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | |||||||
| In-domain | 6.3 | 17.7 | 3.8 | 9.6 | 3.5 | 11.0 | 2.2 | 10.7 | 6.4 | 24.8 | 4.4 | 14.8 | ||||||
| Baseline | 14.8 (+0.0) | 37.7 (+0.0) | 22.2 (+0.0) | 51.8 (+0.0) | 40.3 (+0.0) | 75.8 (+0.0) | 31.6 (+0.0) | 75.7 (+0.0) | 79.6 (+0.0) | 104.3 (+0.0) | 37.7 (+0.0) | 69.1 (+0.0) | ||||||
| Single analogy | ||||||||||||||||||
| 19.5 (+4.7) | 45.6 (+7.9) | 27.1 (+4.9) | 61.6 (+9.8) | 32.4 (-7.9) | 67.4 (-8.4) | 26.5 (-5.1) | 72.4 (-3.3) | 79.0 (-0.6) | 107.5 (+3.2) | 36.9 (-0.8) | 70.9 (+1.8) | |||||||
| 11.5 (-3.3) | 31.8 (-5.9) | 19.2 (-3.0) | 49.8 (-2.0) | 32.2 (-8.1) | 69.2 (-6.6) | 22.5 (-9.1) | 67.2 (-8.5) | 79.2 (-0.4) | 105.1 (+0.8) | 32.9 (-4.8) | 64.6 (-4.5) | |||||||
| IAM (en) | Rimes (fr) | LAM (it) | Rodrigo (es) | AVG. | ||||||||||||||
| CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | |||||||
| In-domain | 6.7 | 19.3 | 3.9 | 9.6 | 3.7 | 11.9 | 1.8 | 9.0 | 5.2 | 20.5 | 4.3 | 14.1 | ||||||
| Baseline | 14.0 (+0.0) | 35.0 (+0.0) | 19.1 (+0.0) | 48.3 (+0.0) | 40.2 (+0.0) | 73.7 (+0.0) | 30.2 (+0.0) | 76.3 (+0.0) | 77.3 (+0.0) | 105.7 (+0.0) | 36.2 (+0.0) | 67.8 (+0.0) | ||||||
| Single analogy | ||||||||||||||||||
| 17.0 (+3.0) | 42.7 (+7.7) | 26.1 (+7.0) | 63.0 (+14.7) | 33.6 (-6.6) | 71.7 (-2.0) | 26.3 (-3.9) | 76.0 (-0.3) | 73.6 (-3.7) | 112.1 (+6.4) | 35.3 (-0.9) | 73.1 (+5.3) | |||||||
| 12.2 (-1.8) | 33.3 (-1.7) | 20.0 (+0.9) | 53.6 (+5.3) | 23.6 (-16.6) | 56.5 (-17.2) | 27.3 (-2.9) | 72.8 (-3.5) | 79.1 (+1.8) | 106.4 (+0.7) | 32.4 (-3.8) | 64.5 (-3.3) | |||||||
| IAM (en) | Rimes (fr) | LAM (it) | Rodrigo (es) | AVG. | ||||||||||||||
| CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | |||||||
| In-domain | 6.2 | 22.0 | 4.1 | 13.8 | 3.5 | 12.5 | 1.0 | 5.5 | 6.3 | 26.1 | 4.2 | 16.0 | ||||||
| Baseline | 17.0 (+0.0) | 48.4 (+0.0) | 23.0 (+0.0) | 63.3 (+0.0) | 37.3 (+0.0) | 78.2 (+0.0) | 32.9 (+0.0) | 87.0 (+0.0) | 83.8 (+0.0) | 100.9 (+0.0) | 38.8 (+0.0) | 75.6 (+0.0) | ||||||
| Single analogy | ||||||||||||||||||
| 18.3 (+1.3) | 50.4 (+2.0) | 23.0 (-.0) | 63.3 (-.0) | 38.2 (+0.9) | 73.7 (-4.5) | 21.3 (-11.6) | 76.7 (-10.3) | 78.8 (-5.0) | 102.5 (+1.6) | 35.9 (-2.9) | 73.3 (-2.3) | |||||||
| 15.3 (-1.7) | 44.7 (-3.7) | 23.0 (-.0) | 64.3 (+1.0) | 32.3 (-5.0) | 71.7 (-6.5) | 21.7 (-11.2) | 77.2 (-9.8) | 84.5 (+0.7) | 100.8 (-0.1) | 35.4 (-3.4) | 71.7 (-3.9) | |||||||
| IAM (en) | Rimes (fr) | LAM (it) | Rodrigo (es) | AVG. | ||||||||||||||
| CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | |||||||
| In-domain | 6.7 | 25.5 | 5.5 | 20.4 | 3.7 | 13.5 | 1.7 | 11.2 | 9.4 | 38.5 | 5.4 | 21.8 | ||||||
| Baseline | 17.6 (+0.0) | 52.6 (+0.0) | 19.9 (+0.0) | 62.9 (+0.0) | 33.6 (+0.0) | 79.3 (+0.0) | 26.8 (+0.0) | 84.5 (+0.0) | 78.2 (+0.0) | 102.0 (+0.0) | 35.2 (+0.0) | 76.3 (+0.0) | ||||||
| Single analogy | ||||||||||||||||||
| 22.6 (+5.0) | 62.6 (+10.0) | 27.2 (+7.3) | 74.1 (+11.2) | 28.7 (-4.9) | 74.0 (-5.3) | 20.5 (-6.3) | 75.1 (-9.4) | 73.4 (-4.8) | 104.3 (+2.3) | 34.5 (-0.7) | 78.0 (+1.7) | |||||||
| 14.1 (-3.5) | 45.2 (-7.4) | 20.5 (+0.6) | 65.0 (+2.1) | 27.6 (-6.0) | 73.3 (-6.0) | 19.9 (-6.9) | 75.4 (-9.1) | 76.3 (-1.9) | 101.6 (-0.4) | 31.7 (-3.5) | 72.1 (-4.2) | |||||||
| IAM (en) | Rimes (fr) | LAM (it) | Rodrigo (es) | AVG. | ||||||||||||||
| CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | |||||||
| In-domain | 7.9 | 29.6 | 5.5 | 20.4 | 3.6 | 13.3 | 2.0 | 12.9 | 7.5 | 33.0 | 5.3 | 21.8 | ||||||
| Baseline | 21.9 (+0.0) | 62.3 (+0.0) | 31.2 (+0.0) | 77.2 (+0.0) | 45.4 (+0.0) | 89.5 (+0.0) | 26.2 (+0.0) | 79.4 (+0.0) | 84.0 (+0.0) | 100.4 (+0.0) | 41.7 (+0.0) | 81.8 (+0.0) | ||||||
| Single analogy | ||||||||||||||||||
| 28.4 (+6.5) | 73.0 (+10.7) | 38.3 (+7.1) | 86.9 (+9.7) | 34.4 (-11.0) | 79.7 (-9.8) | 23.1 (-3.1) | 78.0 (-1.4) | 78.8 (-5.2) | 101.1 (+0.7) | 40.6 (-1.1) | 83.7 (+1.9) | |||||||
| 14.9 (-7.0) | 49.1 (-13.2) | 27.0 (-4.2) | 75.5 (-1.7) | 36.9 (-8.5) | 84.2 (-5.3) | 21.5 (-4.7) | 76.6 (-2.8) | 82.9 (-1.1) | 100.3 (-0.1) | 36.6 (-5.1) | 77.1 (-4.7) | |||||||
| IAM (en) | Rimes (fr) | LAM (it) | Rodrigo (es) | AVG. | ||||||||||||||
| CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | CER | WER | |||||||
| TrOCR (S) | ||||||||||||||||||
| In-domain | 7.7 | 21.8 | 4.6 | 12.5 | 4.1 | 13.2 | 2.1 | 10.7 | 6.3 | 25.2 | 5.0 | 16.7 | ||||||
| Baseline | 13.3 (+0.0) | 33.1 (+0.0) | 13.1 (+0.0) | 29.9 (+0.0) | 37.5 (+0.0) | 66.4 (+0.0) | 18.0 (+0.0) | 48.5 (+0.0) | 70.0 (+0.0) | 99.0 (+0.0) | 30.4 (+0.0) | 55.4 (+0.0) | ||||||
| Single analogy | ||||||||||||||||||
| 12.9 (-0.4) | 31.9 (-1.2) | 12.7 (-0.4) | 29.0 (-0.9) | 23.7 (-13.8) | 50.0 (-16.4) | 11.8 (-6.2) | 35.4 (-13.1) | 66.9 (-3.1) | 96.5 (-2.5) | 25.6 (-4.8) | 48.6 (-6.8) | |||||||