What Not to Erase: Saliency-Constrained Augmentation for Low-Resource Handwritten Character Recognition

Paraskevi Platanou; John Pavlopoulos

cs.CVOct 19, 2026

What Not to Erase: Saliency-Constrained Augmentation for Low-Resource Handwritten Character Recognition

Authors: Paraskevi Platanou, John Pavlopoulos

Organizations: Athens University of Economics and Business Archimedes, Athena Research Center, Greece

Abstract

Optical Character Recognition (OCR) and handwritten character recognition (HCR) in low-resource historical material collections often rely on data augmentation, yet augmentation strategies are typically evaluated almost exclusively through predictive accuracy. This leaves open whether a model's improved performance is accompanied by saliency stability. This question is important in low-resource historical HCR, where recognition is performed on variable and partially degraded letterforms and two models with similar predictive performance may differ in how consistently their saliency patterns respond to such variation. We therefore study saliency-guided augmentation as an explanation-aware strategy for low-resource HCR. Rather than treating saliency maps only as post-hoc interpretations, we use them to define augmentation variants that perturb, preserve, or constrain visually informative regions of character images. We compare classical document-style transformations, random lacunae, and several saliency-constrained lacuna variants. To evaluate the effect of augmentation beyond accuracy, we introduce the Explainability Consistency Index (ECI), a metric that quantifies the stability of saliency patterns under controlled perturbations. We find that saliency-constrained augmentation can improve recognition over classical augmentation, but no single saliency strategy is consistently best. The preferred form of the constraint varies with both architecture and corpus, indicating that saliency is better treated as a mechanism for controlling augmentation than as a fixed augmentation strategy. In some settings, saliency-constrained training improves explanation consistency even when predictive performance is similar. This shows that predictive metrics alone can obscure differences in model sensitivity under perturbation, motivating explanation consistency as a complementary evaluation dimension.

Figures & tables

ID	Classical aug.	Lacuna placement	Saliency role
A0	No	None	None
A1	Yes	None	None
A2	No	Low-saliency foreground	Target low-saliency regions
A3	No	Random foreground	None
A4	Yes	Low-saliency foreground	Target low-saliency regions
A5	Yes	Random foreground	None

Table 1: Augmentation conditions A0–A11. Classical augmentation includes affine transformations, blur, brightness/contrast changes, and Gaussian noise. All lacuna locations are restricted to the character foreground.

Dataset	Classes	Available	LR Train	Val	Test
Hell-Char	24	13014	240	2139	2643
HPGTR-Char	22	2292	220	22	22

Table 2: Dataset statistics. “Available” denotes the character images available in the corresponding dataset release before the low-resource experimental subsampling used here. For HPGTR-Char, 1,996 alphabetic records with available image files remain after the initial eligibility filtering; the final experiment retains 22 sufficiently supported classes. Hell-Char uses a document-grouped split by TM papyrus identifier, whereas HPGTR-Char uses a balanced low-resource split. Validation and test images are never augmented.

Architecture	LR	Epochs	Batch	Initialization
ResNet18	$10^{-4}$	20	16	ImageNet
ViT-B/16	$5{\times}10^{-5}$	15	8	ImageNet
ConvNeXt-Tiny	$5{\times}10^{-5}$	15	8	ImageNet

Table 3: Architecture-specific training settings. All models trained with AdamW, cross-entropy loss, weight decay

10^{-4}

, early stopping patience (5 epochs), checkpoint selection by val. macro-F1. Experiments repeated (seeds 13, 42, 123).

Dataset	Architecture	Best	Macro-F1 $\pm$ SEM	Acc.	Bal. Acc.	$\Delta$ A1	ECI top	ECI corr
Hell-Char	ResNet18	A8	$0.564\pm 0.011$	0.586	0.601	+0.024	0.182	0.283
Hell-Char	ViT-B/16	A9	$0.609\pm 0.004$	0.629	0.639	+0.008	0.190	0.288
Hell-Char	ConvNeXt-Tiny	A8	$0.634\pm 0.002$	0.653	0.673	+0.026	0.276	0.475
HPGTR-Char	ResNet18	A4	$0.687\pm 0.018$	0.758	0.758	+0.061	0.157	0.217
HPGTR-Char	ViT-B/16	A9	$0.821\pm 0.003$	0.864	0.864	+0.139	0.176	0.262
HPGTR-Char	ConvNeXt-Tiny	A9	$0.770\pm 0.003$	0.818	0.818	+0.018	0.266	0.446

Table 4: Best augmentation condition for each dataset and architecture, selected by mean macro-F1 over three random seeds. Macro-F1 is reported as mean

\pm

standard error of the mean (SEM).

\Delta

A1 denotes the macro-F1 difference between the best method and classical augmentation. ECI scores are reported for the selected method. A saliency-constrained condition obtains the highest mean macro-F1 in all six primary dataset–architecture settings, although no Holm-corrected pairwise comparison reaches significance with three seeds.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Classes	Shots	Train	Val	Test
Hell-Char	24	10	240	2139	2643
HPGTR-Char	22	10	220	22	22
Corpus Litterarum	20	4	80	200	400

Appendix

Table 5: Experimental splits before augmentation. Shots denotes original training images per class. Validation and test images are never augmented.

Architecture	LR	Epochs	Batch
ResNet18	$10^{-4}$	20	16
ViT-B/16	$5{\times}10^{-5}$	15	8
ConvNeXt-Tiny	$5{\times}10^{-5}$	15	8

Appendix

Table 6: Training settings. All models use ImageNet initialization; checkpoints are selected by validation macro-F1.

Dataset	Architecture	Sel.	Test F1 $\pm$ SEM	Acc.	$\Delta$ A1	$\Delta$ A5	ECI top	ECI corr
Hell-Char	ResNet18	A8	$0.554\pm 0.007$	0.578	+0.009	+0.005	0.176	0.270
Hell-Char	ViT-B/16	A9	$0.606\pm 0.005$	0.623	+0.019	+0.004	0.188	0.283
Hell-Char	ConvNeXt-Tiny	A8	$0.622\pm 0.006$	0.641	+0.009	-0.0002	0.276	0.475
HPGTR-Char	ResNet18	A7	$0.683\pm 0.023$	0.750	+0.034	+0.007	0.159	0.219
HPGTR-Char	ViT-B/16	A4	$0.736\pm 0.012$	0.786	+0.017	+0.017	0.169	0.245
HPGTR-Char	ConvNeXt-Tiny	A9	$0.780\pm 0.016$	0.827	+0.005	-0.004	0.269	0.452

Appendix

Table 7: Validation-selected conditions evaluated on test data over ten seeds.

\Delta

A1 and

\Delta

A5 are test macro-F1 differences from classical augmentation and classical+random foreground lacunae. ECI is reported for the selected condition.

Data Augmentation

Data augmentation expands or transforms training examples to improve learning, data efficiency, and robustness to distribution shifts.

See all 249 papers

Mar 3, 2026

An Effective Data Augmentation Method by Asking Questions about Scene Text Images

Scene text recognition (STR) and handwritten text recognition (HTR) face significant challenges in accurately transcribing textual content from images into machine-readable formats. Conventional OCR models often predict transcriptions directly, which limits detailed reasoning about text structure. We propose a VQA-inspired data augmentation framework that strengthens OCR training through structured question-answering tasks. For each image-text pair, we generate natural-language questions probing character-level attributes such as presence, position, and frequency, with answers derived from ground-truth text. These auxiliary tasks encourage finer-grained reasoning, and the OCR model aligns visual features with textual queries to jointly reason over images and questions. Experiments on WordArt and Esposalles datasets show consistent improvements over baseline models, with significant reductions in both CER and WER. Our code is publicly available at https://github.com/xuyaooo/DataAugOCR.

Xu Yao, Lei Kang

Computer Vision Center, Barcelona, Spain

Optical Character Recognition Scene Text Recognition Data Augmentation

Ask PDF

Aug 23, 2026

Does a Modern-Handwriting Warm-Up Help Historical Arabic OCR? A Reproducible, Compute-Matched Evaluation on Muharaf and KHATT

Whether an intermediate stage of modern Arabic handwriting helps or hurts historical Arabic HTR is usually decided from one implementation and one comparison, too thin a basis for a claim either way. We test stability by running the same nominal ablation four times, letting the base checkpoint, encoder-freezing strategy, epoch budget, precision, and learning-rate schedule vary as they naturally did during development, while holding the normalization, scorer, and interval estimation fixed. Each run compares intermediate training on modern handwriting (KHATT) then fine-tuning on historical manuscripts (Muharaf) against fine-tuning on Muharaf directly. Across the four runs the estimated effect swings from -17.64 to +14.52 CER points and reverses sign. The two extremes are exactly the two runs with an identifiable confound (a fivefold lower learning rate in one; a checkpoint of undisclosed provenance in the other); the two clean runs land at -0.25 and +0.94, i.e. no effect. A tight interval from one implementation says nothing about the next. We then run a compute-matched experiment with identical budgets over three seeds: KHATT warm-up is +2.42 CER points worse than a matched same-domain control (95% interval [+0.60, +4.25]); the part of that gap specific to the handwriting domain is only about 0.6 points a small negative effect under this configuration, not a universal result. We release a SaudiHeritage-OCR package with the normalizer, interval scorer, a verified KHATT decoder, experimental manifests, VLM baselines, and an edition-alignment protocol, so the result can be checked independently. The Al-Mahd inscription line is held strictly out and is not offered as a benchmark.

Sumaih Almarshad, Maram Alamri, Dona Aloraini +4

Dal Research Team, Riyadh, Saudi Arabia

Optical Character Recognition Historical Document Processing Cross-Domain Transfer Learning

Ask PDF

Jun 17, 2026

Fuzzy-Geometric Branch-Point Modeling for Structure-Aware Augmentation of Handwritten Chinese Characters

Data scarcity and structural distortion significantly limit handwriting recognition in high-security authentication. Existing augmentation methods often cause topological and morphological damage, particularly when processing complex Chinese characters where stroke intersections, ligatures, and sharp turns render traditional branch-point detection unreliable. To address this, this paper proposes a fuzzy geometry-driven structure-aware (FGSA) augmentation framework. We model branch points as fuzzy sets within the skeleton space, constructing a continuous branch-point membership field by integrating topological neighborhood evidence with direction field divergence. This membership field is adaptively optimized via an unsupervised surrogate objective, enabling robust stroke decoupling without manual annotation. Finally, kinematically-aligned samples are synthesized through parameterized cubic Bézier reconstruction and multi-strategy perturbations, ensuring a balance between structural fidelity and sample diversity. Moreover, we establish LZUSig, a large-scale, highly challenging dataset specifically dedicated to fine-grained structural degradation in Chinese handwritten signatures. Extensive experiments on CASIA-HWDB1.1, ChiSig, and LZUSig demonstrate that FGSA significantly reduces the word-level error rate (

Δ

WER), achieving optimal recognition gains over the compared baselines. More importantly, it strikes a robust trade-off among task gain, structural fidelity, and discriminative feature preservation, offering a highly controllable solution for handwriting augmentation.

Dongbin Jiao, Yibo Lyu, Qiulu Wei +3

School of Information Science and Engineering, Lanzhou University, Lanzhou, 730000, P. R. China · Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, 518055, P. R. China

Synthetic Data Augmentation Data Augmentation Optical Character Recognition

Ask PDF

Abstract

Data Augmentation

Explore similar work

An Effective Data Augmentation Method by Asking Questions about Scene Text Images

Does a Modern-Handwriting Warm-Up Help Historical Arabic OCR? A Reproducible, Compute-Matched Evaluation on Muharaf and KHATT

Fuzzy-Geometric Branch-Point Modeling for Structure-Aware Augmentation of Handwritten Chinese Characters