Despite the rapid advancement of Vision-Language Models (VLMs), their linguistic reach remains largely confined to high-resource languages, leaving the majority of the world's 7,000+ living languages on the wrong side of a growing digital divide. This disparity is especially pronounced in Optical Character Recognition (OCR), where low-resource scripts lack the massive datasets required for traditional scaling laws. We investigate OCR adaptation in extreme data-scarce regimes (<10K real and <250K synthetic images), demonstrating that conventional fine-tuning strategies often reach a performance ceiling. Our key finding reveals a structural inefficiency in language-specific adaptation: while higher layers of specialized models diverge to capture unique script nuances, the lower layers learn redundant, highly similar features. Motivated by this observation, we propose PSMC (Pre-train, Specialize, Merge, and Co-train), a data-efficient framework that capitalizes on a cross-script "transfer effect". Our approach first derives language-specific experts from a high-resource base model, then employs task arithmetic to fuse these experts into a unified, high-performance multilingual back- bone. Extensive evaluation across 10 Indian scripts (supporting 20+ languages) shows that PSMC achieves a ~2% average improvement in Word Recognition Rate (WRR) over individual specialist models without increasing parameter count. Our results indicate that joint training in the merged latent space facilitates a constructive knowledge transfer that benefits all constituent scripts, providing a scalable pathway for inclusive VLM development. Source code and datasets will be released post publication.
Figures & tables
Figure 1: To quantitatively assess the similarity of learned representations across language-specific models, we employ Centered Kernel Alignment (CKA) [ 11 ] , a representation similarity metric that is invariant to orthogonal transformations and isotropic scaling. We consider a base model trained on the full English dataset and further finetuned on 250K samples (10K real, 240K synthetic) of each script. We choose 25 scene and 25 printed text images per language and pass them through each model, extracting features at each encoder layer. We then perform a CKA analysis at each layer (more details in the experiments section). The plot shows the average similarity between layers at the same index, averaged across all images and model pairs. Observe that the representations in the lower layers are very similar, while language specific features are learned in the deeper layers of each encoder. This motivates us to co-train across languages, allowing the encoder to jointly learn the early script-agnostic robust representations from varied samples while accommodating script-specific features in the final encoder layers.
Script
Languages
Devanagari
Hindi, Marathi, Sindhi, Kashmiri, Dogri,
Nepali, Bodo, Maithili, Sanskrit, Konkani
Bengali-Assamese
Bengali, Assamese, Santali, Manipuri
Odia
Odia
Gujarati
Gujarati
Kannada
Kannada
Table 1: Indic Languages and Scripts
Dataset
Languages
Train Size
T est Size
IndicSTR-Roadside (Real, Scene) [ 17 ]
Bengali, Gujarati 1 1 1 We found that the Gujarati set had several Hindi samples, mislabelled GT and samples with no visible text. We present results after omitting these. , Hindi,
175K
50K
Kannada, Malayalam, Marathi
Oriya, Punjabi, Tamil, Telugu
IndicSTR12 (Real, Scene) [ 14 ]
Assamese, Bengali, Odia
20K
7K
Marathi, Hindi, Kannada
Urdu, Telugu, Malayalam
Table 2: Public Datasets Used
Figure 2: PSMC Framework
Figure 3: A comprehensive visualization to highlight the feature similarity of layer-wise encoder features across the individual specialist models: Following the same experimental set up as Figure 1 Heatmap (X,Y) depicts the averaged pairwise CKA feature similarity across all the 24 encoder layers between Model X and Y for input images from language X. Note the high similarity spreads in the upper left corners of all the heatmaps demonstrating the similar high-level, script-agnostic visual cues learnt in the early layers across all specialists.
Figure 4: Validation loss for the task arithmetic initialization vs the simple pretrained initialization
Language
Baseline [ 16 ]
Skyline
Ind-FT
Co-FT
PSMC
WRR
NED
WRR
NED
WRR
NED
WRR
NED
WRR
NED
Assamese
96.5
99
98.38
99.6
97.16
99.21
97.52
99.37
97.94
99.52
Bengali
97.9
99.4
98.45
99.55
98.27
99.49
98.55
99.63
98.71
99.64
Gujarati
93.9
96.5
97.12
99
94.68
98.15
96.1
98.61
96.29
98.72
Hindi
96.3
98.2
98.51
99.58
97.82
99.29
98.15
99.43
98.24
99.46
Kannada
90.7
97.7
97.69
99.43
95.12
98.62
97.01
99.3
97.14
99.38
Table 3: WRR and NED scores on the Mozhi Dataset
Language
Baseline [ 17 ]
Skyline
Ind-FT
Co-FT
PSMC
WRR
NED
WRR
NED
WRR
NED
WRR
NED
WRR
NED
Bengali
85.34
92.75
90.9
96.55
89.08
96.15
90.22
96.39
90.78
96.73
Gujarati
81.91
88.12
84.65
94.92
80.49
93.25
81.03
93.35
82.28
93.82
Hindi
87.24
95.01
88.88
95.83
87.5
95.75
84.26
89.31
88.92
94.59
Kannada
79.27
87.64
93.18
97.76
91.92
97.5
93.04
97.69
92.88
97.84
Malayalam
80.31
89.42
91.58
97.82
90.82
97.48
91.88
98.05
92.44
98.17
Table 4: WRR and NED Scores on the IndicSTR-Roadside Dataset
Figure 5: Drop in WRR with respect to the Skyline WRR metrics for (left) Printed Text, and (right) Scene Text.
Figure 6: Mistakes in individually finetuned model vs final model