Impact of Iterative Fine-Tuning on Transcription Accuracy in Complex Historical Sanskrit Manuscripts

Kartik Chincholikar; Kaushik Gopalan; Mihir Hasabnis

cs.CVAug 19, 2026

Impact of Iterative Fine-Tuning on Transcription Accuracy in Complex Historical Sanskrit Manuscripts

Authors: Kartik Chincholikar, Kaushik Gopalan, Mihir Hasabnis

Organizations: Centre for Inter-disciplinary Artificial Intelligence (CAI), FLAME University, Pune, India

Abstract

Digitizing the text from handwritten historical manuscripts is required to make them easily accessible, preservable, and to enable historical scholars to study them in new ways. Historical manuscripts, however, often exhibit complex heterogeneous layouts and non-standard appearance due to period-specific writing styles, page textures, camera noise, and other nuisance factors, making them difficult to perform OCR on. To tackle this challenge, we introduce a local traditional OCR pipeline, which can be iteratively fine-tuned on the target manuscript at the layout-level and the appearance-level. By adapting to the target manuscript distribution, the proposed Traditional OCR pipeline makes better predictions on subsequent pages, causing iterative reduction in human annotation effort, which is expensive and time-consuming as it requires historical domain expertise. Using this pipeline, we digitize text from three complex historical Sanskrit manuscripts and introduce a dataset with granular layout-level annotations, along with Unicode annotations in the standard PAGE-XML format. We demonstrate quantitative gains due to iterative fine-tuning of the proposed traditional OCR pipeline, and also benchmark the performance of leading Multi-Modal Large Language Models on the introduced Dataset. Code and dataset are available at: https://github.com/flame-cai/gnn-synthetic-layout-historical/.

Figures & tables

Panel of Figure 1 — Figure 1: The manuscripts. Samples of pages from the three manuscripts digitized in this case study. The manuscripts exhibit non-standard layout variations , including dense marginalia, circular text, interlinear commentary, irregular text-line orientation, as well as non-standard appearance variations such as ink bleed-through, faded ink, paper degradation, scribe-specific handwriting styles, and period-specific orthographic conventions.

Manuscript	Pages	Text-lines per page			Grapheme clusters per page
		Min.	Max.	Mean	Min.	Max.	Mean
Moderate Layout	15	12	28	21.07	285	525	392.27
Dense Layout	7	15	78	44.43	285	1098	691.86
Circular Layout	9	10	65	28.33	164	434	258.89

Table 1: Dataset statistics

Panel of Figure 2 — Figure 2: Rich granular annotations at the Layout-level and Appearance-level. The dataset is also exported in the standard PAGE-XML format.

Panel of Figure 3 — Figure 3: Iterative Fine-tuning Pipeline. The frozen U-Net CRAFT [ 8 ] detects character locations in the original image (a) as a heatmap (b). Next, the GNN predicts the layout graph (c) where characters (or grapheme clusters) belonging to the same text-line are connected. In the next step, text-line images are prepared using the GNN predictions, and curved text (if any) is unwrapped and straightened (d). Finally, the CNN–BiLSTM–CTC takes in the text-line images as input, and predicts its text contents as Unicode text (e). In both (c) and (e), the colors denote differences between the predictions and the ground-truth, and hence illustrate the scope of improvement from iterative fine-tuning of the GNN and CNN–BiLSTM–CTC. Orange denotes missing nodes, edges, or Unicode characters. Blue denotes extra nodes, edges, or Unicode characters. Maroon denotes modified text, and Black denotes correct nodes, edges, and unicode text.

Manuscript	Multi-Modal LLM	TextEdit $\downarrow$	Page-CER $\downarrow$
Moderate Layout	Claude Sonnet 5	$0.60\,[0.45,\,0.76]$	$0.64\,[0.50,\,0.80]$
	Gemini 3.5 Flash	0.26 [0.18, 0.37]	0.35 [0.29, 0.44]
	GPT 5.6 Terra	$0.71\,[0.70,\,0.73]$	$0.74\,[0.73,\,0.76]$
	Sarvam Vision	$0.33\,[0.31,\,0.36]$	$0.39\,[0.36,\,0.41]$ *
Dense Layout	Claude Sonnet 5	$0.80\,[0.60,\,1.00]$	$0.85\,[0.65,\,1.00]$
	Gemini 3.5 Flash	0.30 [0.27, 0.34]	0.44 [0.38, 0.47]

Table 2: Benchmarking the performance of Multi-Modal Large Language Models using the metrics TextEdit and Page-CER. For Sarvam Vision, * marks Page-CER computed by treating the model’s emitted HTML text-line order as reading order, as Sarvam does not provide text-line level bounding polygons. Each entry reports the metric value followed by its 95% confidence interval. Lower values are better.

Panel of Figure 4 — Figure 4: We fine-tune the traditional pipeline up to three pages (across the same 5-fold train-test split as described in Sec. 5 ) using three ablations: one where only the GNN is fine-tuned at layout-level, one where only the CNN-BiLSTM-CTC OCR recognition model is fine-tuned at the appearance-level, and one where both are fine-tuned. We also report a fourth ablation where a human annotator manually corrects the layout of the test set pages at inference time, and an iteratively fine-tuned CNN-BiLSTM-CTC OCR recognition model is used to predict the text content from the text-line images. This fourth ablation is meant to mimic the expected working conditions of the Traditional Pipeline Annotation Tool, where manual layout correction takes a few minutes at most.

Method	Model	API endpoint
gemini_e2e	gemini-3.5-flash	POST https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash:generateContent
openai_e2e	gpt-5.6-terra	POST https://api.openai.com/v1/responses
claude_e2e	claude-sonnet-5	POST https://api.anthropic.com/v1/messages
sarvam_e2e	sarvam-vision $\dagger$	POST https://api.sarvam.ai/doc-digitization/job/v1 $\ddagger$ and, relative to that base: POST /upload-files POST /{job_id}/start GET T /{job_id}/status POST /{job_id}/download-files

Table 4: The four end-to-end VLM baselines: the exact model identifier requested and the HTTP endpoint it is requested from. Gemini, OpenAI, and Claude receive the resized page image followed by the same VLM_END_TO_END_PROMPT and return JSON; Sarvam receives the image alone and returns HTML, hence its different call shape.

Optical Character Recognition

Optical character recognition converts text in images into machine-readable text, from printed pages and handwriting to signs and symbols.

See all 200 papers

TongGuOCR: A Layout-Aware and Token-Augmented OCR MLLM for Chinese Historical Documents
Aug 8, 2026Zhongheng Zhou, Yi Sun, Huiguo He +6Multimodal Large Language Models Optical Character Recognition
School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China · Huawei Technologies Co., Ltd., Dongguan 511700, China
A Free Lunch? Adapting PP-OCRv6 for Historical Text Recognition
Sep 17, 2026Benjamin KiesslingOptical Character Recognition Historical Document Processing
ALMAnaCH
UniLipi: A Unified Multi-Script OCR for Historical Indic Manuscripts
Aug 28, 2026Tathagata Ghosh, Sai Madhusudan Gunda, Simran Singh Sandral +1Optical Character Recognition Historical Document Analysis
International Institute of Information Technology Hyderabad, INDIA · GITAM

Aug 8, 2026J/K move · Enter open · S save

TongGuOCR: A Layout-Aware and Token-Augmented OCR MLLM for Chinese Historical Documents

Zhongheng Zhou, Yi Sun, Huiguo He, Yuyi Zhang+5

School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China · Huawei Technologies Co., Ltd., Dongguan 511700, China

Multimodal Large Language Models Optical Character Recognition Historical Document Processing

Open paper AskPDF arXiv

Chinese historical documents preserve valuable cultural heritage, but many collections remain accessible only as scanned page images, preventing full-text retrieval, collation, and computational analysis. Optical character recognition (OCR) can bridge this gap, but accurate transcription remains challenging because historical documents often contain complex layouts, rare characters, and nontrivial reading orders. We propose TongGuOCR, a layout-aware and token-augmented multimodal large language model (MLLM) for OCR of Chinese historical documents. First, a Layout-Aware Preprocessing module constructs and refines locally coherent recognition blocks to preserve local context while reducing interference across regions. Second, a Token-Augmented Recognition module augments the transcription target at two complementary levels: character-level vocabulary expansion gives each rare glyph a direct one-token representation and shortens its decoding path, while line-to-line transition modeling injects discrete spatial displacement tokens that guide the decoder along complex reading paths without requiring precise coordinates. Experiments on two Chinese historical document OCR benchmarks show that TongGuOCR outperforms representative traditional task-specific OCR models, general-purpose MLLMs, and OCR-oriented MLLMs. On the more challenging M5HisDoc benchmark, TongGuOCR achieves 93.76 AR and reduces NED from 10.43 to 6.15 and RO-ED from 7.53 to 3.49 relative to the best competing score for each metric. An online demo is available at https://jzzh2004.github.io/TongGuOCR.

Abstract

Optical Character Recognition

Explore similar work

TongGuOCR: A Layout-Aware and Token-Augmented OCR MLLM for Chinese Historical Documents

A Free Lunch? Adapting PP-OCRv6 for Historical Text Recognition

UniLipi: A Unified Multi-Script OCR for Historical Indic Manuscripts