State-of-the-art handwritten text recognition (HTR) systems commonly use Transformers, whose growing key-value (KV) cache makes decoding slow and memory-intensive. We introduce DRetHTR, a decoder-only model built on Retentive Networks (RetNet). Compared to an equally sized decoder-only Transformer baseline, DRetHTR delivers 1.6-1.9x faster inference with 38-42% less memory usage, without loss of accuracy. By replacing softmax attention with softmax-free retention and injecting multi-scale sequential priors, DRetHTR avoids a growing KV cache: decoding is linear in output length in both time and memory. To recover the local-to-global inductive bias of attention, we propose layer-wise gamma scaling, which progressively enlarges the effective retention horizon in deeper layers. This encourages early layers to model short-range dependencies and later layers to capture broader context, mitigating the flexibility gap introduced by removing softmax. Consequently, DRetHTR achieves best reported test character error rates of 2.26% (IAM-A, en), 1.81% (RIMES, fr), and 3.46% (Bentham, en), and is competitive on READ-2016 (de) with 4.21%. This demonstrates that decoder-only RetNet enables Transformer-level HTR accuracy with substantially improved decoding speed and memory efficiency.
Figures & tables
Figure 1: Decoder-only RetNet architecture that fuses the image and text in the Decoder
Architectures
Parallelized training
Inference Cost
Long-Seq Memory
Performance
Transformer
✓
O(N)
O(N2)
✓✓
RetNet
✓
O(1)
O(N)
✓✓
RNN
✗
O(1)
O(N)
✓
Table 1: Comparison of model architectures by parallelization, inference cost, memory complexity, and performance.
Form
Total Computations
Asymptotic Complexity
Vanilla Form
2n2d+n2−1+n(d−1)
O(n2d)
Key-Value Cached
2dn+2(n−1)
O(nd)
Recurrent Form
2d2+d−1
O(d2)
Table 2: Computational costs of different inference forms. RetNet’s recurrent form is more effective than the KV-cached Transformer when model dimension d is less than sequence length n (derivation in Supplementary Materials, Appendix B. Derivation of the Asymptotic Complexity for Each Inference Form ).
Figure 2: Illustration of the DRetHTR training and inference processes: (a) training; (b) inference. ARMF mixes softmax (image) + retention (text) without breaking recurrence.
Figure 4: Examples of the six augmentations applied to a handwriting image.
DRetHTR SMALL
RetHTR SMALL
Dataset
CER ↓
WER ↓
CER ↓
WER ↓
IAM
4.49
13.05
4.45
12.93
IAM + Synth
2.97
1 9.13
3.01
1 9.20
Table 3: Comparison of CER and WER between DRetHTR SMALL and RetHTR SMALL on different datasets.
Model
Parameters
Memory ↓ (beam=6)
Time ↓ (beam=6)
DRetHTR SMALL
1 73M
1 9GB
1 52.8s
RetHTR SMALL
136M
11GB
170.4s
DTrHTR SMALL
1 73M
15GB
1 64.5s
TrHTR SMALL
133M
16GB
189.4s
Table 4: Model performance comparison showcasing memory and time usage for beam-search decoding on the IAM test set (Aachen split). We refer to our Transformer baselines as TrHTR (encoder–decoder) and DTrHTR (decoder-only), which are matched reimplementations of RetHTR/DRetHTR and distinct from TrOCR/DTrOCR. Times correspond to a single pass over the full IAM test set with beam size 6 and batch size 128 on a single NVIDIA A100 80 GB GPU.
CER (%) ↓
CNN
IAM
IAM + Synth
ResNet50
14.5
4.57
ShallowCNN
30.0
4.23
EfficientNetV2-S
4.49
2.97
Table 5: Comparison of CER (%) on IAM and IAM + Synth datasets for different CNN architectures.
Table 6: Ablation of γ scheduling on IAM (Aachen split) with an EfficientNetV2-S image encoder. All CER values are measured on the test set of Aachen split. Layer-wise γ scaling (last row) matches the Transformer’s CER (4.49 %) while retaining RetNet’s softmax-free token interactions, enabling linear-time, linear-memory decoding. See Fig. 5(a) for the local → global progression that motivates the schedule.
Figure 5: Local-to-global progression and its retention mimic. (a) In the Transformer baseline (DTrHTR), attention naturally shifts from local to broader context with depth. (b) By assigning smaller γ to early decoder layers and larger γ to deeper layers, retention reproduces a similar progression without softmax for text–text interactions (cf. Table 6 ).
Sequential Priors for Image
CER ↓
Priors only on text
4.49
+ RetNet’s temporal decay on image
4.41
+ Layer-wise gamma scaling on image
4.35
Table 7: Effect of Sequential Priors on image tokens. RetNet’s temporal decay is applied bidirectionally, and layer-wise gamma scaling is added to it.
Tokenization
CER (%) ↓
WER (%) ↓
Zero-shot Performance
Character-Level
11.72
42.09
BPE ∣V∣=500
1 9.12
19.87
Fine-tuned on IAM-A
Character-Level
2.26
6.55
BPE ∣V∣=500
2.36
4.81
Table 8: Comparison of Character Error Rate (CER) and Word Error Rate (WER) of DRetHTR BASE for different tokenization methods in both zero-shot and fine-tuned scenarios on the IAM test set (Aachen split). BPE uses a vocabulary size 500. BPE improves zero-shot via shorter sequences; character-level wins after fine-tuning due to precise character mapping in HTR.
Figure 6: The three hardest IAM lines for DRetHTR (highest CER), excluding faulty labels.
Method
IAM
RIMES
READ-2016
Bentham
DRetHTR BASE (ours)
2.26
1.81
4.21
3.46
DTrOCR [ 13 ]
2.38
–
–
–
Diaz et al. [ 18 ]
2.75
1.99
–
–
Retsinas et al. [ 33 ]
4.22
2.70
–
–
Wang et al. [ 42 ]
6.40
2.70
–
–
Coquenet et al. [ 8 ]
4.45
1.91
3.59
–
Table 9: Test CER (%) on four benchmarks. DRetHTR is pre-trained on the English synthetic dataset and fine-tuned separately on IAM, RIMES, READ-2016, and Bentham, and evaluated on the corresponding official test sets (IAM Aachen split, RIMES test set, READ-2016 test set, and Bentham test set).
Model
Params
Tokenization
Total Tokens
Inference Time ↓
Memory ↓
CER ↓
DRetHTR BASE
107M
Character Level
442,832
123s
22.14GB
2.26
DTrHTR BASE
107M
Character Level
442,832
233s
36.25GB
2.35
DRetHTR SMALL
1 73M
Character Level
442,832
1 67s
14.5GB
2.97
DTrHTR SMALL
1 73M
Character Level
442,832
110s
25.1GB
2.98
DRetHTR SMALL
1 74M
BPE ∣V∣=500
1 49,574
1 44s
14.9GB
3.01
DTrHTR SMALL
1 74M
BPE ∣V∣=500
1 49,574
1 69s
23.9GB
3.01
Table 10: Comparative inference efficiency of RetNet, Transformer, and TrOCR models on the IAM test set (Aachen split). All times and memory values are measured for decoding the full IAM test set with the specified tokenization and beam size, using batch size 128 on a single NVIDIA A100 80 GB GPU.
Model
Params
Encoder
Decoder
Train
LM
CER ↓
DAN [ 42 ]
≈ 2.18M
23Conv+GRU
GRU, L1/d256
IAM
No
6.40
Wick et al. [ 44 ]
≈ 22M
3Conv + Tx, L3/h8/d512
Bidi Tx, L3/h8/d512
IAM
No
5.67
Michael et al. [ 25 ]
≈ 1.8M
3Conv+3BLSTM, d256
Attn-LSTM, L1/d256
IAM
No
4.87
HTR-VT [ 22 ]
53.5M
ResNet18 + Tx-Enc, L4/h6/d768
CTC
IAM
No
4.7
Kang et al. [ 19 ]
100M
Tx w/ResNet50, L4/h8/d1024
Tx, L4/h8/d1024
Syn+IAM
No
4.67
VAN [ 8 ]
2.7M
18 VGG-style Conv
1 × LSTM d256 + CTC
IAM
No
4.45
Table 11: Comparison of HTR models: parameters, architecture, training data, external language model (LM) usage, and CER on the IAM test set (Aachen split). Abbrev.: Syn = synthetic data; Int. = internal data; Tx = Transformer; EN-V2 = EfficientNetV2. For architectures, L / h / d denote the number of blocks (layers), attention heads per block, and embedding dimension (e.g., L24,h16,d1024 means 24 layers, 16 heads, 1024-dim embeddings).
Decoding Method
Memory per Layer
Number of Elements
Recurrent (RetNet)
Bd2/H
0.49 M
KV-Cached Transformer
2BNd
2.88 M
Table 12: Dominant per-layer state size for beam-search decoding. d is the model dimension, H the number of heads, N the decoded length, and B the beam size. Example element counts use d=768 , H=12 , N≈94 , and B=10 (DTrHTR BASE ). RetNet’s state is fixed-size in N , while KV caching scales linearly with N .
Figure 7: Beam-size scaling on the IAM Aachen test set: end-to-end inference time (left), peak memory (middle), and test CER (right) for DRetHTR vs. DTrHTR. DRetHTR scales more favorably with beam size because of fixed-size recurrent states, whereas KV-cached decoding incurs beam-dependent cache growth and cache reindexing after beam pruning. CER generally improves with larger beams up to B=5 , after which it fluctuates slightly while decoding cost continues to grow with B .
Handwritten Text Recognition (HTR) is computationally imbalanced in two ways: most image pixels are background, and many width-axis sequence positions are blank-dominated. This creates a mismatch for Spiking Neural Networks (SNNs): handwriting is observed as a static image, whereas spiking computation unfolds over timesteps. We propose Spike-HTR, a hybrid spiking recognizer that controls both the number of spiking steps and the number of width positions processed by the deep sequence mixer. To make a static image suitable for short-horizon spiking inference, InkCoder converts it into a coarse-to-fine input stream, where early steps cover broad stroke regions and later steps emphasize sharper stroke details. To reduce sequence computation, a CTC-guided length reducer keeps likely character or uncertain positions and compresses long blank-dominated stretches before deep mixing. With T=2, Spike-HTR trains only on target data, decodes without language models or lexicons, and reaches validation/test CERs of 3.5/5.4, 2.3/2.5, and 4.2/3.9 on IAM, LAM, and READ2016. Codes are available at https://github.com/QomolangmaH/SpikeHTR.
Xiubo Liang, Jinxing Han, Yuke Li +3
School of Software Technology, Zhejiang University, Ningbo, China
Fine-tuning transformer-based handwritten text recognition (HTR) models on medieval manuscripts is challenging because these models are pre-trained on modern text and must adapt to a very different visual domain. This paper studies how three controllable fine-tuning choices (contrast normalization, data augmentation, and layer freezing) affect recognition accuracy when adapting TrOCR to small historical datasets. We run controlled experiments on a 13th-century Italian manuscript (I-CT 91 "Cortonese") and replicate the same experimental grid on the public READ-16 benchmark as robustness evidence. On Cortonese, our best configuration achieves 8.03% character error rate (CER). Statistical comparisons across 13 configurations show that freezing up to three encoder layers or six decoder layers does not significantly harm accuracy, while deeper freezing becomes progressively detrimental. Removing contrast normalization (CLAHE) yields 7.84% CER, comparable to a domain-specialized baseline, suggesting strong optimization can reduce reliance on image preprocessing. Cross-dataset validation on READ-16 shows that decoder freezing thresholds transfer more robustly than encoder thresholds, and combined freezing strategies require dataset-specific re-validation. Finally, we use Grad-CAM gradient attributions and decoder cross-attention maps to diagnose error patterns and failure modes revealed by the ablations. Source code is available at https://github.com/LaudareProject/TrOCR-analysis
Transformers process tokens in parallel but are temporally shallow: at position t, each layer attends to key-value pairs computed based on the previous layer, yielding a depth capped by the number of layers. Recurrent models offer unbounded temporal depth but suffer from optimization instability and historically underutilize modern accelerators. We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations, yielding layerwise recurrent memory while preserving standard autoregressive decoding cost. We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-token recurrent updates under mild assumptions, while avoiding optimization instability. Naively, prefill/training appears bandwidth-bound with effective arithmetic intensity near 1 because keys and values are revealed sequentially; we give an exact tiling-based algorithm that preserves the mathematical computation while reducing HBM traffic from Θ(N2) to Θ(NlogN), increasing effective arithmetic intensity to Θ(N/logN) for sequence length N. On 150M and 300M parameter C4 pretraining, Recurrent Transformers improve cross-entropy over a parameter-matched Transformer baseline and achieve the improvement with fewer layers (fixed parameters), suggesting that recurrence can trade depth for width, thus reducing KV cache memory footprint and inference latency.