Despite the rapid advancement of Vision-Language Models (VLMs), their linguistic reach remains largely confined to high-resource languages, leaving the majority of the world's 7,000+ living languages on the wrong side of a growing digital divide. This disparity is especially pronounced in Optical Character Recognition (OCR), where low-resource scripts lack the massive datasets required for traditional scaling laws. We investigate OCR adaptation in extreme data-scarce regimes (<10K real and <250K synthetic images), demonstrating that conventional fine-tuning strategies often reach a performance ceiling. Our key finding reveals a structural inefficiency in language-specific adaptation: while higher layers of specialized models diverge to capture unique script nuances, the lower layers learn redundant, highly similar features. Motivated by this observation, we propose PSMC (Pre-train, Specialize, Merge, and Co-train), a data-efficient framework that capitalizes on a cross-script "transfer effect". Our approach first derives language-specific experts from a high-resource base model, then employs task arithmetic to fuse these experts into a unified, high-performance multilingual back- bone. Extensive evaluation across 10 Indian scripts (supporting 20+ languages) shows that PSMC achieves a ~2% average improvement in Word Recognition Rate (WRR) over individual specialist models without increasing parameter count. Our results indicate that joint training in the merged latent space facilitates a constructive knowledge transfer that benefits all constituent scripts, providing a scalable pathway for inclusive VLM development. Source code and datasets will be released post publication.
Figures & tables
Figure 1: To quantitatively assess the similarity of learned representations across language-specific models, we employ Centered Kernel Alignment (CKA) [ 11 ] , a representation similarity metric that is invariant to orthogonal transformations and isotropic scaling. We consider a base model trained on the full English dataset and further finetuned on 250K samples (10K real, 240K synthetic) of each script. We choose 25 scene and 25 printed text images per language and pass them through each model, extracting features at each encoder layer. We then perform a CKA analysis at each layer (more details in the experiments section). The plot shows the average similarity between layers at the same index, averaged across all images and model pairs. Observe that the representations in the lower layers are very similar, while language specific features are learned in the deeper layers of each encoder. This motivates us to co-train across languages, allowing the encoder to jointly learn the early script-agnostic robust representations from varied samples while accommodating script-specific features in the final encoder layers.
Script
Languages
Devanagari
Hindi, Marathi, Sindhi, Kashmiri, Dogri,
Nepali, Bodo, Maithili, Sanskrit, Konkani
Bengali-Assamese
Bengali, Assamese, Santali, Manipuri
Odia
Odia
Gujarati
Gujarati
Kannada
Kannada
Table 1: Indic Languages and Scripts
Dataset
Languages
Train Size
T est Size
IndicSTR-Roadside (Real, Scene) [ 17 ]
Bengali, Gujarati 1 1 1 We found that the Gujarati set had several Hindi samples, mislabelled GT and samples with no visible text. We present results after omitting these. , Hindi,
175K
50K
Kannada, Malayalam, Marathi
Oriya, Punjabi, Tamil, Telugu
IndicSTR12 (Real, Scene) [ 14 ]
Assamese, Bengali, Odia
20K
7K
Marathi, Hindi, Kannada
Urdu, Telugu, Malayalam
Table 2: Public Datasets Used
Figure 2: PSMC Framework
Figure 3: A comprehensive visualization to highlight the feature similarity of layer-wise encoder features across the individual specialist models: Following the same experimental set up as Figure 1 Heatmap (X,Y) depicts the averaged pairwise CKA feature similarity across all the 24 encoder layers between Model X and Y for input images from language X. Note the high similarity spreads in the upper left corners of all the heatmaps demonstrating the similar high-level, script-agnostic visual cues learnt in the early layers across all specialists.
Figure 4: Validation loss for the task arithmetic initialization vs the simple pretrained initialization
Language
Baseline [ 16 ]
Skyline
Ind-FT
Co-FT
PSMC
WRR
NED
WRR
NED
WRR
NED
WRR
NED
WRR
NED
Assamese
96.5
99
98.38
99.6
97.16
99.21
97.52
99.37
97.94
99.52
Bengali
97.9
99.4
98.45
99.55
98.27
99.49
98.55
99.63
98.71
99.64
Gujarati
93.9
96.5
97.12
99
94.68
98.15
96.1
98.61
96.29
98.72
Hindi
96.3
98.2
98.51
99.58
97.82
99.29
98.15
99.43
98.24
99.46
Kannada
90.7
97.7
97.69
99.43
95.12
98.62
97.01
99.3
97.14
99.38
Table 3: WRR and NED scores on the Mozhi Dataset
Language
Baseline [ 17 ]
Skyline
Ind-FT
Co-FT
PSMC
WRR
NED
WRR
NED
WRR
NED
WRR
NED
WRR
NED
Bengali
85.34
92.75
90.9
96.55
89.08
96.15
90.22
96.39
90.78
96.73
Gujarati
81.91
88.12
84.65
94.92
80.49
93.25
81.03
93.35
82.28
93.82
Hindi
87.24
95.01
88.88
95.83
87.5
95.75
84.26
89.31
88.92
94.59
Kannada
79.27
87.64
93.18
97.76
91.92
97.5
93.04
97.69
92.88
97.84
Malayalam
80.31
89.42
91.58
97.82
90.82
97.48
91.88
98.05
92.44
98.17
Table 4: WRR and NED Scores on the IndicSTR-Roadside Dataset
Figure 5: Drop in WRR with respect to the Skyline WRR metrics for (left) Printed Text, and (right) Scene Text.
Figure 6: Mistakes in individually finetuned model vs final model
Optical character recognition (OCR) has advanced rapidly with deep learning and multimodal models, yet most methods focus on well-resourced scripts such as Latin and Chinese. Ethnic minority languages remain underexplored due to complex writing systems, scarce annotations, and diverse historical and modern forms, making generalization in low-resource or zero-shot settings challenging. To address these challenges, we present OmniOCR, a universal framework for ethnic minority scripts. OmniOCR introduces Dynamic Low-Rank Adaptation (Dynamic LoRA) to allocate model capacity across layers and scripts, enabling effective adaptation while preserving knowledge.A sparsity regularization prunes redundant updates, ensuring compact and efficient adaptation without extra inference cost. Evaluations on TibetanMNIST, Shui, ancient Yi, and Dongba show that OmniOCR outperforms zero-shot foundation models and standard post training, achieving state-of-the-art accuracy with superior parameter efficiency, and compared with the state-of-the-art baseline models, it improves accuracy by 39%-66% on these four datasets. Code: https://github.com/AIGeeksGroup/OmniOCR.
Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.
Ran Zhang, Miryam de Lhoneux, Wessel Poelman
LAGOM·NLP, Department of Computer Science, KU Leuven
Current multilingual evaluations for Vision-Language Models (VLMs) assume a one-to-one mapping between language and orthography, overlooking billions of users of multi-script languages. We introduce PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts: Gurmukhi, Shahmukhi, and Roman. Evaluating 10 state-of-the-art VLMs, we expose a substantial and systematic Script Gap. Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%. Crucially, visual input boosts absolute performance uniformly yet does not close the orthographic gap. Furthermore, cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation. Supported by McNemar tests across all script pairs, our findings demonstrate that current "multilingual" VLMs are not truly multi-script. We propose the Script Consistency Rate (SCR), which falls as low as 24.8% on our benchmark, as a mandatory metric for script-agnostic evaluation to ensure equitable AI access. Data and code are available at: https://github.com/prabhjotschugh/Not-Truly-Multilingual-PuMVR.