DRetHTR: Linear-Time Decoder-Only Retentive Network for Handwritten Text Recognition

Changhun Kim; Martin Mayr; Thomas Gorges; Fei Wu; Mathias Seuret; Andreas Maier; Vincent Christlein

cs.CVFeb 19, 2026

DRetHTR: Linear-Time Decoder-Only Retentive Network for Handwritten Text Recognition

Authors: Changhun Kim, Martin Mayr, Thomas Gorges, Fei Wu, Mathias Seuret, Andreas Maier, Vincent Christlein

Organizations: Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany

Abstract

State-of-the-art handwritten text recognition (HTR) systems commonly use Transformers, whose growing key-value (KV) cache makes decoding slow and memory-intensive. We introduce DRetHTR, a decoder-only model built on Retentive Networks (RetNet). Compared to an equally sized decoder-only Transformer baseline, DRetHTR delivers 1.6-1.9x faster inference with 38-42% less memory usage, without loss of accuracy. By replacing softmax attention with softmax-free retention and injecting multi-scale sequential priors, DRetHTR avoids a growing KV cache: decoding is linear in output length in both time and memory. To recover the local-to-global inductive bias of attention, we propose layer-wise gamma scaling, which progressively enlarges the effective retention horizon in deeper layers. This encourages early layers to model short-range dependencies and later layers to capture broader context, mitigating the flexibility gap introduced by removing softmax. Consequently, DRetHTR achieves best reported test character error rates of 2.26% (IAM-A, en), 1.81% (RIMES, fr), and 3.46% (Bentham, en), and is competitive on READ-2016 (de) with 4.21%. This demonstrates that decoder-only RetNet enables Transformer-level HTR accuracy with substantially improved decoding speed and memory efficiency.

Figures & tables

Figure 1: Decoder-only RetNet architecture that fuses the image and text in the Decoder

Architectures	Parallelized training	Inference Cost	Long-Seq Memory	Performance
Transformer	✓	$O(N)$	$O(N^{2})$	✓✓
RetNet	✓	$O(1)$	$O(N)$	✓✓
RNN	✗	$O(1)$	$O(N)$	✓

Table 1: Comparison of model architectures by parallelization, inference cost, memory complexity, and performance.

Form	Total Computations	Asymptotic Complexity
Vanilla Form	$2n^{2}d+n^{2}-1+n(d-1)$	$\mathcal{O}(n^{2}d)$
Key-Value Cached	$2dn+2(n-1)$	$\mathcal{O}(nd)$
Recurrent Form	$2d^{2}+d-1$	$\mathcal{O}(d^{2})$

Table 2: Computational costs of different inference forms. RetNet’s recurrent form is more effective than the KV-cached Transformer when model dimension

d

is less than sequence length

n

(derivation in Supplementary Materials, Appendix B. Derivation of the Asymptotic Complexity for Each Inference Form ).

Panel of Figure 2 — Figure 2: Illustration of the DRetHTR training and inference processes: (a) training; (b) inference. ARMF mixes softmax (image) + retention (text) without breaking recurrence.

Figure 4: Examples of the six augmentations applied to a handwriting image.

	DRetHTR SMALL		RetHTR SMALL
Dataset	CER $\downarrow$	WER $\downarrow$	CER $\downarrow$	WER $\downarrow$
IAM	4.49	13.05	4.45	12.93
IAM + Synth	2.97	1 9.13	3.01	1 9.20

Table 3: Comparison of CER and WER between DRetHTR SMALL and RetHTR SMALL on different datasets.

Model	Parameters	Memory $\downarrow$ (beam=6)	Time $\downarrow$ (beam=6)
DRetHTR SMALL	1 73M	1 9GB	1 52.8s
RetHTR SMALL	136M	11GB	170.4s
DTrHTR SMALL	1 73M	15GB	1 64.5s
TrHTR SMALL	133M	16GB	189.4s

Table 4: Model performance comparison showcasing memory and time usage for beam-search decoding on the IAM test set (Aachen split). We refer to our Transformer baselines as TrHTR (encoder–decoder) and DTrHTR (decoder-only), which are matched reimplementations of RetHTR/DRetHTR and distinct from TrOCR/DTrOCR. Times correspond to a single pass over the full IAM test set with beam size 6 and batch size 128 on a single NVIDIA A100 80 GB GPU.

	CER (%) $\downarrow$
CNN	IAM	IAM + Synth
ResNet50	14.5	4.57
ShallowCNN	30.0	4.23
EfficientNetV2-S	4.49	2.97

Table 5: Comparison of CER (%) on IAM and IAM + Synth datasets for different CNN architectures.

Strategy	Gamma Value	CER $\downarrow$
Original	$\gamma=1-e^{\text{linspace}(\log 1/32,\log 1/512,H)}$	4.65
+ Data dependent gating	$\gamma=\operatorname{sigmoid}(XW_{\gamma})^{1/\tau}$	4.66
+ Small Gamma Only	$\gamma=1-\gamma_{\text{subtractor}}-e^{\text{linspace}(\log 1/32,\log 1/512,H)}$	4.60
+ Head-wise Gamma Scaling	$\gamma_{h}=\left(1-\gamma_{\text{subtractor }}-\frac{1}{32}\right)+\frac{h-1}{H-1}\gamma_{\text{subtractor }}$	4.51
+ Layer-wise Gamma Scaling	$\gamma_{l}=1-\gamma_{\text{subtractor}}\!\left(1-\frac{l}{L-1}\right)-e^{\text{linspace}(\log(1/32),\log(1/512),H)}$	4.49
Transformer with Softmax	–	4.49

Table 6: Ablation of

\gamma

scheduling on IAM (Aachen split) with an EfficientNetV2-S image encoder. All CER values are measured on the test set of Aachen split. Layer-wise

\gamma

scaling (last row) matches the Transformer’s CER (4.49 %) while retaining RetNet’s softmax-free token interactions, enabling linear-time, linear-memory decoding. See Fig. 5(a) for the local

\rightarrow

global progression that motivates the schedule.

Panel of Figure 5 — Figure 5: Local-to-global progression and its retention mimic. (a) In the Transformer baseline (DTrHTR), attention naturally shifts from local to broader context with depth. (b) By assigning smaller $\gamma$ to early decoder layers and larger $\gamma$ to deeper layers, retention reproduces a similar progression without softmax for text–text interactions (cf. Table 6 ).

Sequential Priors for Image	CER $\downarrow$
Priors only on text	4.49
+ RetNet’s temporal decay on image	4.41
+ Layer-wise gamma scaling on image	4.35

Table 7: Effect of Sequential Priors on image tokens. RetNet’s temporal decay is applied bidirectionally, and layer-wise gamma scaling is added to it.

Tokenization	CER (%) $\downarrow$	WER (%) $\downarrow$
Zero-shot Performance
Character-Level	11.72	42.09
BPE $\|\mathcal{V}\|=500$	1 9.12	19.87
Fine-tuned on IAM-A
Character-Level	2.26	6.55
BPE $\|\mathcal{V}\|=500$	2.36	4.81

Table 8: Comparison of Character Error Rate (CER) and Word Error Rate (WER) of DRetHTR BASE for different tokenization methods in both zero-shot and fine-tuned scenarios on the IAM test set (Aachen split). BPE uses a vocabulary size 500. BPE improves zero-shot via shorter sequences; character-level wins after fine-tuning due to precise character mapping in HTR.

Figure 6: The three hardest IAM lines for DRetHTR (highest CER), excluding faulty labels.

Method	IAM	RIMES	READ-2016	Bentham
DRetHTR BASE (ours)	2.26	1.81	4.21	3.46
DTrOCR [ 13 ]	2.38	–	–	–
Diaz et al. [ 18 ]	2.75	1.99	–	–
Retsinas et al. [ 33 ]	4.22	2.70	–	–
Wang et al. [ 42 ]	6.40	2.70	–	–
Coquenet et al. [ 8 ]	4.45	1.91	3.59	–

Table 9: Test CER (%) on four benchmarks. DRetHTR is pre-trained on the English synthetic dataset and fine-tuned separately on IAM, RIMES, READ-2016, and Bentham, and evaluated on the corresponding official test sets (IAM Aachen split, RIMES test set, READ-2016 test set, and Bentham test set).

Model	Params	Tokenization	Total Tokens	Inference Time $\downarrow$	Memory $\downarrow$	CER $\downarrow$
DRetHTR ${}_{\text{BASE}}$	107M	Character Level	442,832	123s	22.14GB	2.26
DTrHTR ${}_{\text{BASE}}$	107M	Character Level	442,832	233s	36.25GB	2.35
DRetHTR ${}_{\text{SMALL}}$	1 73M	Character Level	442,832	1 67s	14.5GB	2.97
DTrHTR ${}_{\text{SMALL}}$	1 73M	Character Level	442,832	110s	25.1GB	2.98
DRetHTR ${}_{\text{SMALL}}$	1 74M	BPE $\|\mathcal{V}\|=500$	1 49,574	1 44s	14.9GB	3.01
DTrHTR ${}_{\text{SMALL}}$	1 74M	BPE $\|\mathcal{V}\|=500$	1 49,574	1 69s	23.9GB	3.01

Table 10: Comparative inference efficiency of RetNet, Transformer, and TrOCR models on the IAM test set (Aachen split). All times and memory values are measured for decoding the full IAM test set with the specified tokenization and beam size, using batch size 128 on a single NVIDIA A100 80 GB GPU.

Model	Params	Encoder	Decoder	Train	LM	CER $\downarrow$
DAN [ 42 ]	$\approx$ 2.18M	23Conv+GRU	GRU, L1/d256	IAM	No	6.40
Wick et al. [ 44 ]	$\approx$ 22M	3Conv + Tx, L3/h8/d512	Bidi Tx, L3/h8/d512	IAM	No	5.67
Michael et al. [ 25 ]	$\approx$ 1.8M	3Conv+3BLSTM, d256	Attn-LSTM, L1/d256	IAM	No	4.87
HTR-VT [ 22 ]	53.5M	ResNet18 + Tx-Enc, L4/h6/d768	CTC	IAM	No	4.7
Kang et al. [ 19 ]	100M	Tx w/ResNet50, L4/h8/d1024	Tx, L4/h8/d1024	Syn+IAM	No	4.67
VAN [ 8 ]	2.7M	18 VGG-style Conv	1 $\times$ LSTM d256 + CTC	IAM	No	4.45

Table 11: Comparison of HTR models: parameters, architecture, training data, external language model (LM) usage, and CER on the IAM test set (Aachen split). Abbrev.: Syn = synthetic data; Int. = internal data; Tx = Transformer; EN-V2 = EfficientNetV2. For architectures,

L

h

d

denote the number of blocks (layers), attention heads per block, and embedding dimension (e.g.,

L24,h16,d1024

means 24 layers, 16 heads, 1024-dim embeddings).

Decoding Method	Memory per Layer	Number of Elements
Recurrent (RetNet)	$B\,d^{2}/H$	$0.49$ M
KV-Cached Transformer	$2\,B\,N\,d$	$2.88$ M

Table 12: Dominant per-layer state size for beam-search decoding.

d

is the model dimension,

H

the number of heads,

N

the decoded length, and

B

the beam size. Example element counts use

d{=}768

H{=}12

N{\approx}94

, and

B{=}10

(DTrHTR

{}_{\text{BASE}}

). RetNet’s state is fixed-size in

N

, while KV caching scales linearly with

N

Figure 7: Beam-size scaling on the IAM Aachen test set: end-to-end inference time (left), peak memory (middle), and test CER (right) for DRetHTR vs. DTrHTR. DRetHTR scales more favorably with beam size because of fixed-size recurrent states, whereas KV-cached decoding incurs beam-dependent cache growth and cache reindexing after beam pruning. CER generally improves with larger beams up to $B=5$ , after which it fluctuates slightly while decoding cost continues to grow with $B$ .

Memory-Efficient Inference

Memory-Efficient Inference reduces the memory needed to run neural networks while balancing accuracy, latency, throughput, and energy use.

See all 168 papers

Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition
Aug 3, 2026Xiubo Liang, Jinxing Han, Yuke Li +3Efficient Neural Network Inference Spiking Transformers
School of Software Technology, Zhejiang University, Ningbo, China
TrOCR for Medieval HTR: A Systematic Ablation Study with Cross-Dataset Validation
Jun 23, 2026Sachin Sharma, Michele Flammini, Federico SimonettaFine-Tuning VLM Adaptation
GSSI – Gran Sasso Science Institute L’Aquila, Italy
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Autoregressive Language Modeling Self-Attention
Harvard University

Aug 3, 2026J/K move · Enter open · S save

Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition

Xiubo Liang, Jinxing Han, Yuke Li, Haoqi Zhu+2

School of Software Technology, Zhejiang University, Ningbo, China

Efficient Neural Network Inference Spiking Transformers Spiking Neural Networks

Open paper AskPDF arXiv

Handwritten Text Recognition (HTR) is computationally imbalanced in two ways: most image pixels are background, and many width-axis sequence positions are blank-dominated. This creates a mismatch for Spiking Neural Networks (SNNs): handwriting is observed as a static image, whereas spiking computation unfolds over timesteps. We propose Spike-HTR, a hybrid spiking recognizer that controls both the number of spiking steps and the number of width positions processed by the deep sequence mixer. To make a static image suitable for short-horizon spiking inference, InkCoder converts it into a coarse-to-fine input stream, where early steps cover broad stroke regions and later steps emphasize sharper stroke details. To reduce sequence computation, a CTC-guided length reducer keeps likely character or uncertain positions and compresses long blank-dominated stretches before deep mixing. With

T{=}2

, Spike-HTR trains only on target data, decodes without language models or lexicons, and reaches validation/test CERs of 3.5/5.4, 2.3/2.5, and 4.2/3.9 on IAM, LAM, and READ2016. Codes are available at https://github.com/QomolangmaH/SpikeHTR.

Abstract

Memory-Efficient Inference

Explore similar work

Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition

TrOCR for Medieval HTR: A Systematic Ablation Study with Cross-Dataset Validation

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding