PalmLeaf-VQA: A Multi-Script Visual Question Answering Benchmark for Historical Palm-Leaf Manuscript Understanding Across Diverse Regions

Nimol Thuon; Jun Du; Panhapin Theang

cs.CVSep 14, 2026

PalmLeaf-VQA: A Multi-Script Visual Question Answering Benchmark for Historical Palm-Leaf Manuscript Understanding Across Diverse Regions

Authors: Nimol Thuon, Jun Du, Panhapin Theang

Organizations: University of Science and Technology of China · Université Paris Cité

Abstract

Historical manuscripts remain largely absent from modern vision-language benchmarks, leaving open how well multimodal large language models (MLLMs) handle culturally diverse, degraded, and non-Latin document images. We introduce PalmLeaf-VQA, a multi-script visual question answering benchmark for historical palm-leaf manuscript understanding across South and Southeast Asian traditions. PalmLeaf-VQA contains 923 curated manuscript images and 7,384 question--answer pairs from eight collection groups: Balinese, Grantha, Jathakam, Kambaramayanam, Kannada, Khmer, Sundanese, and Tamil. Unlike recognition-oriented resources, the benchmark targets manuscript-aware visual reasoning over preservation-relevant cues, including physical condition, line structure, material and coating, binding holes, margins, symbols, drawings, and localized visual artifacts. We evaluate recent proprietary and open-weight MLLMs under open-answer and constrained-answer prompting and provide fine-grained analysis across collections, question categories, and task types. The strongest evaluated model reaches only 58.00% exact-match accuracy on the held-out test split, revealing substantial limitations in current MLLMs for rare-script, degraded-layout, and preservation-oriented document understanding. PalmLeaf-VQA provides a standardized benchmark for advancing culturally grounded and layout-aware multimodal document analysis.

Figures & tables

Figure 1: Overview of PalmLeaf-VQA. (a) Representative palm-leaf manuscript collections from South Asia (SA) and Southeast Asia (SEA), with example visual question-answer pairs. (b) The benchmark covers eight manuscript-aware VQA categories, including script/collection, symbols and drawings, visible line count, physical condition, material and coating, binding and holes, scribe and margins, and other physical features. (c) Questions are grouped into VQA types, including classification, counting, layout reasoning, binary recognition, and attribute recognition.

Benchmark	Coverage	Hist.	Low-R	Layout	VQA	Pres.
VQA [ 1 ]	Natural images	–	–	–	$\checkmark$	–
TextVQA [ 37 ]	Scene text	–	–	$\circ$	$\checkmark$	–
DocVQA [ 28 ]	Scanned docs	–	–	$\checkmark$	$\checkmark$	–
InfographicVQA [ 27 ]	Infographics	–	–	$\checkmark$	$\checkmark$	–
ChartQA [ 26 ]	Charts	–	–	$\checkmark$	$\checkmark$	–
DUDE [ 46 ]	Multi-domain docs	–	–	$\checkmark$	$\checkmark$	–

Table 1: Compact comparison of benchmark scope. The table does not rank dataset quality; it highlights whether each benchmark explicitly targets historical materials (Hist.), rare or low-resource scripts/languages (Low-R), layout reasoning, visual question answering (VQA), and material or preservation cues (Pres.).

\checkmark

: primary focus;

\circ

: partial or indirect coverage.

Collection	Train	Val	Test	Images	QA
Balinese	60	10	30	100	800
Grantha	42	7	21	70	560
Jathakam	59	10	30	99	792
Kambaramayanam	48	8	24	80	640
Kannada	154	26	77	257	2,056
Khmer	93	15	47	155	1,240

Table 2: PalmLeaf-VQA dataset statistics by collection group. Splits are image-disjoint. Each image has eight question-answer pairs.

Figure 2: Task-type and collection distribution in PalmLeaf-VQA. The nested donut summarizes 7,384 question–answer pairs from 923 manuscript images. The inner ring aggregates the eight equally represented question categories into five VQA task types, while the outer ring decomposes each task-type segment across the eight collection groups.

		Constrained-answer			Free-form/open
Type	Model	EM	F1	W1	EM	F1	W1
Open-weight	Qwen3-VL-8B	44.02	54.73	71.22	21.49	21.80	68.60
	Qwen2.5-VL-7B	31.03	37.41	61.87	13.62	13.95	71.92
	Qwen2.5-VL-3B	35.21	41.71	56.29	15.15	15.61	54.33
	InternVL3.5-8B	44.96	54.48	62.05	24.78	25.02	61.51
	InternVL3.5-4B	46.00	54.93	59.71	24.96	25.12	65.65

Table 3: Full model results on PalmLeaf-VQA under constrained-answer and free-form/open prompting. Values are percentages on the held-out test split. Type distinguishes open-weight models from proprietary systems. EM denotes exact-match accuracy, F1 denotes mean answer F1, and W1 denotes within-one accuracy for line-count questions. Bold marks the best result within each access type. FL denotes Flash-Lite.

Model	Bal	Gra	Jat	Kam	Kan	Khm	Sun	Tam	Avg.
Qwen3-VL-8B	47.9	38.7	38.3	47.9	41.6	42.6	43.8	54.8	44.0
Qwen2.5-VL-7B	30.0	29.2	46.2	28.6	25.0	32.2	29.2	34.7	31.0
Qwen2.5-VL-3B	27.5	41.1	33.3	38.0	34.9	43.4	27.8	31.0	35.2
InternVL3.5-8B	50.8	67.3	37.9	48.4	38.1	46.8	43.8	43.1	45.0
InternVL3.5-4B	49.2	50.0	42.9	49.0	47.7	37.0	39.6	54.0	46.0
Llama-3.1-8B	34.2	40.5	36.2	32.8	18.7	37.5	24.3	45.6	31.6

Table 4: Collection-level exact-match accuracy under constrained-answer prompting. Values are percentages on the held-out test split. Avg. matches constrained EM in Table 3 . Column abbreviations: Balinese (Bal), Grantha (Gra), Jathakam (Jat), Kambaramayanam (Kam), Kannada (Kan), Khmer (Khm), Sundanese (Sun), and Tamil (Tam).

Evaluator	Questions	EM	F1	W1
Human participants	256	87.50	92.40	96.88
Gemini 3.5 Flash	256	58.59	67.07	93.75
ChatGPT 5.5	256	59.77	69.89	90.63

Table 5: Human and model performance on a balanced 256-question subset. Human EM and F1 are averaged over 768 responses, while human W1 is computed over 96 responses to the 32 line-count questions. Model EM and F1 are computed over 256 questions, while model W1 is computed over the corresponding 32 line-count questions. All evaluators use the constrained-answer setting. Values are percentages.

Panel of Figure 3 — Figure 3: Performance under different prompting and answer settings. (a) Exact-match accuracy under open-answer and constrained-answer prompting. (b) Performance by answer type, including single-answer EM, multi-answer F1, and exact line-count accuracy.

Figure 4: Question-category performance under constrained-answer prompting. Exact-match accuracy (%) is reported on the held-out PalmLeaf-VQA test split across the eight question categories. Rows correspond to question categories, and columns correspond to five selected high-performing models. Color intensity indicates accuracy, showing stronger performance on material and coating and binding and holes, while localized physical-feature and physical-condition reasoning remain challenging.

Question category	Primary answer format	Output rule
Script Name	Single label	One canonical collection-group label
Physical Condition	Single label	One canonical condition label
Line Count	Count	One non-negative integer
Symbols & Drawings	Single label	Standardized yes / no answer
Material & Coating	Single label	One canonical material or coating label
Binding & Holes	Count	One non-negative integer for visible binding holes

Table 6: Answer formats used in PalmLeaf-VQA. Formats match the released annotation/answer_vocabularies.json file.

Field	Type	Description
sample_id	String	Unique question–answer identifier used by the benchmark
image_name	String	Image filename in the corresponding collection folder
script_name	String	One of the eight PalmLeaf-VQA collection groups
question_id	String/Integer	Question index or identifier for the image
task_family	String	Higher-level task family for analysis
category	String	Manuscript-aware question category

Table 7: PalmLeaf-VQA QA annotation fields. The table follows the released annotation/qa_schema.json . Source provenance, license status, redistributability, and checksums are retained in separate metadata and manifest files.

Component	Configuration
Local GPUs	$2\times$ NVIDIA A800 80GB PCIe
Aggregate GPU memory	160GB
Driver / CUDA report	NVIDIA driver 580.126.09; CUDA 13.0
Local precision	bfloat16 when supported
Batch size	1 image-question group per forward pass
API execution	Proprietary models evaluated remotely through APIs

Table 8: Hardware and software environment used for evaluation. Open-weight inference used local A800 GPUs. Proprietary API models did not use local GPU memory for model execution.

Model	Access	Prompt modes	Default decoding	Execution
Gemini 3.5 Flash	Proprietary API	Open, constrained	temperature 0; top- $p$ 1; max 1024 tokens	Remote API
ChatGPT 5.5	Proprietary API	Open, constrained	JSON schema; max 1200 tokens	Remote API
Gemini 3.1 Flash-Lite	Proprietary API	Open, constrained	temperature 0; top- $p$ 1; max 256 tokens	Remote API
Qwen3-VL-8B-Instruct	Open-weight	Open, constrained	deterministic; max 256 tokens	Local A800 GPUs
Qwen2.5-VL-7B-Instruct	Open-weight	Open, constrained	deterministic; max 256 tokens	Local A800 GPUs
Qwen2.5-VL-3B-Instruct	Open-weight	Open, constrained	deterministic; max 256 tokens	Local A800 GPUs

Table 9: Model and inference configuration summary. The evaluated model list and decoding defaults match configs/models.yaml . The two Llama models are text-only controls and do not receive manuscript images.

Raw output	Normalized output	Rule
BALINESE.	balinese	Case and punctuation normalization
yes, it is visible	yes	Binary-answer canonicalization
twelve	12	Documented numerical mapping
label_b, label_a	{label_a,label_b}	Unordered multi-label set
label_a, label_a	{label_a}	Duplicate-label removal

Table 10: Illustrative normalization examples. The released annotation/normalization_rules.yaml and eval/normalize.py define the exact normalization behavior used for scoring.

Evaluator	Questions	Responses	EM	F1	W1
Human participants	256	768	87.50	92.40	96.88
Gemini 3.5 Flash	256	256	58.59	67.07	93.75
ChatGPT 5.5	256	256	59.77	69.89	90.63

Table 11: Human and model performance on the balanced comparison subset. Human EM and F1 are averaged over 768 individual responses, while human W1 is computed over 96 responses to the 32 visible-line-count questions. Model EM and F1 are evaluated over 256 questions, and model W1 is evaluated over the corresponding 32 visible-line-count questions. Values are percentages.

Figure 5: Aggregate distribution of incorrect predictions across PalmLeaf-VQA question categories. The chart pools 30,897 incorrect row-level predictions from 20 available model–prompt runs. Counts and percentages indicate each category’s contribution to the total number of exact-match failures. Summary-only runs without row-level prediction files are excluded. Visible-line-count failures are determined using exact-count correctness rather than W1.

Figure 6: Qualitative comparison of open-answer and constrained-answer predictions. Each card includes the manuscript image, question category, question, reference answer, model prediction, and exact-match status. Green denotes reference-consistent outputs, whereas red denotes incorrect, incomplete, unsupported, or non-canonical responses. The examples illustrate correct canonical predictions, line-count errors, localized physical-feature failures, and open-answer output drift.

Historical Document Analysis

Historical document analysis uses AI to transcribe, interpret, organize, and retrieve information from archival texts and images.

See all 52 papers

Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation
Jun 15, 2026Prabhjot Singh, Bhushan Pawar, Madhu Reddiboina +1VLM Evaluation Multilingual Language Models
RediMinds Inc., USA · The University of Texas at Austin, USA · Indian AI Research Organization (IAIRO), India
SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asia
Mar 16, 2026Pengfei Yue, Xingran Zhao, Juntao Chen +6Multilingual VLM Evaluation Document Parsing
Xiamen University, China · Shopee, China · Tongji University, China
VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark
Aug 19, 2025Vy Tuong Dang, An Vo, Emilio Villa-Cueva +4VLM Evaluation Multimodal Reasoning
KAIST · MBZUAI

Jun 15, 2026J/K move · Enter open · S save

Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation

Prabhjot Singh, Bhushan Pawar, Madhu Reddiboina, Rajvee Sheth

RediMinds Inc., USA · The University of Texas at Austin, USA · Indian AI Research Organization (IAIRO), India

VLM Evaluation Multilingual Language Models Multimodal Reasoning Multilingual VLM Evaluation VLM Reasoning Multimodal Understanding

Open paper AskPDF arXiv

Current multilingual evaluations for Vision-Language Models (VLMs) assume a one-to-one mapping between language and orthography, overlooking billions of users of multi-script languages. We introduce PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts: Gurmukhi, Shahmukhi, and Roman. Evaluating 10 state-of-the-art VLMs, we expose a substantial and systematic Script Gap. Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%. Crucially, visual input boosts absolute performance uniformly yet does not close the orthographic gap. Furthermore, cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation. Supported by McNemar tests across all script pairs, our findings demonstrate that current "multilingual" VLMs are not truly multi-script. We propose the Script Consistency Rate (SCR), which falls as low as 24.8% on our benchmark, as a mandatory metric for script-agnostic evaluation to ensure equitable AI access. Data and code are available at: https://github.com/prabhjotschugh/Not-Truly-Multilingual-PuMVR.

Abstract

Historical Document Analysis

Explore similar work

Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation

SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asia

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark