LongHisDoc: A Comprehensive Benchmark for Chinese Long Historical Document Understanding

Zeyu Shan; Yongxin Shi; Shihang Wu; Peirong Zhang; Yuanyuan Fang; Zhaopeng Wang; Lianwen Jin

cs.CLOct 10, 2026

LongHisDoc: A Comprehensive Benchmark for Chinese Long Historical Document Understanding

Authors: Zeyu Shan, Yongxin Shi, Shihang Wu, Peirong Zhang, Yuanyuan Fang, Zhaopeng Wang, Lianwen Jin

Organizations: School of Electronic and Information Engineering, South China University of Technology, Guangzhou, 510641, China · College of Literature and Journalism, Sichuan University, Chengdu, 610065, China

Abstract

Chinese historical documents are invaluable cultural treasures, but their complexity poses significant comprehension challenges for non-specialists. Despite the success of Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) in modern document tasks, their capabilities in Long (full-book) Historical Document understanding remain largely unexplored. To bridge this gap, we introduce LongHisDoc, the first benchmark tailored for long historical document understanding. LongHisDoc comprises 1,012 expert-annotated QA pairs derived from 101 PDF documents, with an average of 85.1 pages and 42,776.4 tokens per document. It spans 10 categories and features four QA types: single-page, cross-page, multi-hop, and unanswerable. Answering these questions requires integrating multimodal evidence from text, layout, and visual elements. Our evaluation reveals that the top-performing LVLM and LLM achieve only 62.9% and 69.9% accuracy, respectively, highlighting a substantial performance gap. Furthermore, we assess various modal retrievers and the end-to-end RAG performance to provide references for Retrieval-Augmented Generation (RAG) systems. We believe that LongHisDoc serves as a comprehensive benchmark to catalyze future research in full-book historical document understanding. The benchmark and evaluation code are available at https://huggingface.co/datasets/qweq12433454/LongHisDoc and https://github.com/SCUT-DLVCLab/LongHisDoc.

Figures & tables

Figure 1: LongHisDoc assesses the capabilities of LLMs and LVLMs in understanding long-context historical documents. The experiments (bottom-right) reveal that most LVLMs face difficulties, showing significantly lower performance compared to LLMs fed with Optical Character Recognition (OCR)-parsed historical documents. Zoom in for better view.

Benchmarks	Venue & Year	Document		Question type			Answer Evidence	Doc Type	HisDoc
Benchmarks	Venue & Year	Avg. Pages	Avg. Tokens	Cross-Page	Multi-Hop	Unans	Source	Num	Rel.
DocVQA [ 9 ]	WACV 2021	1.0	151.5	$\times$	$\times$	$\times$	T/L/V	-	$\times$
ChartVQA [ 10 ]	ACL 2022	1.0	236.9	$\times$	$\times$	$\times$	V	-	$\times$
MCS-Bench [ 7 ]	ACL 2025	1.0	314.2	$\times$	$\times$	$\times$	T/L/V	-	$\checkmark$
AncientDoc [ 8 ]	ArXiv 2025	1.0	283.2	$\times$	$\times$	$\times$	T/L/V	14	$\checkmark$
InfoVQA [ 11 ]	WACV 2022	1.2	288.0	$\times$	$\times$	$\times$	L/V	-	$\times$

Table 1: Comparison between LongHisDoc and previous document understanding datasets. T/L/V :text/layout/visual. HisDoc Rel. : historical document relevance.

Figure 2: The data construction pipeline of LongHisDoc consists of four stages: (1) Documents Collection ; (2) Q&A Annotation ; (3) Quality Control ; (4) Data Standardization .

Models	Size	Evidence Page				Answer Format						Evidence Source			AVG
Models	Size	SIN	CRO	MUL	UNA	STR	INT	OPT	LIST	BOOL	NONE	TXT	LAY	VIS	AVG
OCR Results + Large Language Models (LLMs)
Open-Source Models
LLaVA-V1.6-Mistral [ 16 ]	7B	17.0	12.2	14.4	2.0	12.1	11.9	36.1	13.0	49.1	2.0	17.5	12.6	16.7	14.2
Llama-3.2-Vision-Instruct [ 17 ]	11B	47.7	35.0	28.9	14.3	39.0	41.5	58.3	36.4	66.7	14.3	49.7	35.7	24.1	38.8
InternLM3-Instruct [ 18 ]	8B	49.7	38.6	29.8	20.4	41.4	39.8	63.9	41.4	64.9	20.4	50.0	40.1	26.1	41.3

Table 3: Generalized accuracy scores (%) on LongHisDoc. There are four types of evidence pages: single-page (SIN), cross-page (CRO), multi-hop (MUL), and unanswerable (UNA). There are six types of answer formats: String (STR), Integer (INT), Option (OPT), List (LIST), Boolean (BOOL), and None (NONE). There are three types of evidence sources: text (TXT), layout (LAY), and visual (VIS). (Think) refers to using the model’s thinking mode. The highest scores among models in each section are highlighted in green .

Figure 4: Multi-faceted analysis on various document types, evidence pages and evidence sources. Zoom in for better view.

Panel of Figure 5 — Figure 5: Visualization of test results. Left : Incorrect response attributed to OCR parsing failure; Right : Complex layouts of historical documents with interlinear annotations pose challenges to model understanding. Zoom in for better view.

Panel of Figure 6 — Figure 6: Visualization comparison between Text-Input and Image-Input . Left : Text-input models fail due to the loss of visual elements in OCR outputs. Right : Image-input models fail due to image-quality degradation.

Figure 7: Comparison among different input lengths on LongHisDoc benchmark.

OCR Models	size	Generation Models	Evidence Page				Answer Format						Evidence Source			AVG
OCR Models	size	Generation Models	SIN	CRO	MUL	UNA	Str	Int	Enum	LIST	Bool	None	TXT	LAY	VIS	AVG
DeepSeek-OCR-2 [ 35 ]	3B	Qwen3-VL-8B	38.8	29.6	31.3	35.7	30.0	37.5	58.3	36.4	68.4	35.7	40.8	30.1	30.4	35.5
		Qwen3-32B	38.8	33.2	35.1	38.8	32.2	33.3	63.9	35.6	77.2	38.8	43.9	33.4	17.2	37.1
		Qwen3-VL-32B	41.8	32.6	29.3	44.9	35.5	32.5	63.9	31.5	64.9	44.9	46.4	31.2	19.3	38.2
		Gemini-3.1-Pro	53.3	44.8	42.5	59.2	45.7	42.5	69.4	52.7	80.7	59.2	55.2	46.6	33.2	50.4
dots.ocr [ 36 ]	1.7B	Qwen3-VL-8B	58.5	53.5	38.3	29.6	53.2	45.0	66.7	50.1	75.4	29.6	59.4	51.8	33.8	51.3

Table 4: Performance of generation models on LongHisDoc using outputs from different OCR models.

Character Location(%) $\uparrow$	Character Annotation(%) $\uparrow$	Character Arrangement $\downarrow$	Text Punctuation(%) $\uparrow$
98.57	98.43	1.92	89.34

Table 5: Quality evaluation of automated OCR annotations against human-verified ground truth.

Table 12

Method	Size	SIN		CRO		MUL		TXT	LAY	VIS	AVG
Method	Size	top 1	top 5	top 2	top 5	top 5	top 25	top 5			top 5
Text-based Method
SBERT [ 39 ]	0.02B	11.0	32.1	9.4	20.4	19.1	55.6	26.3	25.1	38.5	27.1
BGE-large-en-v1.5 [ 40 ]	0.3B	19.6	44.7	19.9	34.4	23.0	55.9	40.3	35.2	41.3	38.4
BM25 [ 41 ]	-	39.7	62.1	36.2	52.8	26.0	54.9	58.7	50.7	37.8	53.3
BGE-small-zh-v1.5 [ 40 ]	0.02B	39.4	69.8	36.2	53.8	34.3	70.7	61.8	56.9	60.1	59.7

Table 8: Retriever performance on LongHisDoc using top-k Recall metrics.

Retriever	Size	Gen Model	Evidence Page				Answer Format						Evidence Source			AVG
Retriever	Size	Gen Model	SIN	CRO	MUL	UNA	STR	INT	OPT	LIST	BOOL	NONE	TXT	LAY	VIS	AVG
Text-based RAG
No-Retrieval(baseline)	-	DeepSeek-V3.2 [ 23 ]	68.7	64.7	46.3	40.8	62.1	56.7	77.8	65.1	80.7	40.8	69.5	64.7	32.2	61.4
No-Retrieval(baseline)	-	Gemini-3.1-Pro [ 3 ]	73.7	73.0	63.0	54.1	69.3	69.2	80.6	72.6	91.2	54.1	76.3	73.9	40.5	69.9
SBERT [ 39 ]	0.02B	DeepSeek-V3.2 [ 23 ]	32.5 (-36.2)	26.3 (-38.4)	16.3 (-30.0)	65.3 (+24.5)	27.6 (-34.5)	14.4 (-42.3)	47.2 (-30.6)	31.5 (-33.6)	42.1 (-38.6)	65.3 (+24.5)	32.0 (-37.5)	27.3 (-37.4)	13.4 (-18.8)	31.7 (-29.7)
SBERT [ 39 ]	0.02B	Gemini-3.1-Pro [ 3 ]	30.2 (-43.5)	23.6 (-49.4)	20.6 (-42.4)	83.7 (+29.6)	26.4 (-42.9)	12.7 (-56.5)	55.6 (-25.0)	22.7 (-49.9)	54.4 (-36.8)	83.7 (+29.6)	30.5 (-45.8)	25.5 (-48.4)	16.2 (-24.3)	32.5 (-37.4)

Table 9: End-to-end RAG evaluation on LongHisDoc using Top-5 retrieved results from different text-based and visual-based retrievers with state-of-the-art generation models. Zoom in for better view.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Historical Document Title	Publication Dynasty	Author’s Dynasty
Buddhism
《二经同卷2卷.佛说波斯匿王太后崩尘岔身经1卷.西晋.释法炬译.须摩提女经1卷.吴.释支谦译.美国加利福尼亚大学伯克利分校藏宋刊本>古籍/》	宋 (Song)	西晋/三国·吴 (Western Jin / Three Kingdoms - Wu)
《佛母出生三法藏般若波罗蜜多经.存卷第20.卷第25.宋.释施护等译.美国加利福尼亚大学伯克利分校藏元至元南山普宁寺刊本》	元 (Yuan)	宋 (Song)
《佛说佛名经二》	明 (Ming)	北魏/元魏 (Northern Wei)
《佛说仁王护国般若波罗蜜经2卷.后秦.释鸠摩罗什译.美国加利福尼亚大学伯克利分校藏明万历20年（1592）刊本》	明 (Ming)	后秦 (Later Qin)
《佛说大自在天子因地经.1卷.佛说如意摩尼陀罗经.1卷.宋释施护译.赵城广胜寺金版大藏经之一.金刊本》	金 (Jin)	宋 (Song)

Appendix

Table 10: Details of Historical Documents in LongHisDoc. Zoom in for better view.

Panel of Figure 8 — Figure 8: More distribution characteristics of LongHisDoc. Top Left : Absolute position of answer evidences (the page index). Top Right : Relative position (the page index / document page number). Bottom Left : Evidence page number of each question. Bottom Right : Document textual token number.

Model	Size	Context Window	Access	API Version	Website
LLaVA-V1.6-Mistral [ 16 ]	7B	32K	Weights	-	https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b
Llama-3.2-Vision-Instruct [ 17 ]	11B	128K	Weights	-	https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct
Qwen2.5-Instruct [ 60 ]	7B	128K	Weights	-	https://huggingface.co/Qwen/Qwen2.5-7B-Instruct
Qwen2.5-Instruct [ 60 ]	14B	128K	Weights	-	https://huggingface.co/Qwen/Qwen2.5-14B-Instruct
Qwen2.5-Instruct [ 60 ]	32B	128K	Weights	-	https://huggingface.co/Qwen/Qwen2.5-32B-Instruct
mPlug-DocOwl2 [ 30 ]	7B	8K	Weights	-	https://huggingface.co/mPLUG/DocOwl2

Appendix

Table 11: Details of all evaluated LLMs and LVLMs. Zoom in for better view.

Retriever	Size	Context Window	Access	Base Model	Website
SBERT [ 39 ]	0.02B	512	Weights	-	https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
BGE-large-en-v1.5 [ 40 ]	0.3B	512	Weights	-	https://huggingface.co/BAAI/bge-large-en-v1.5
BM25 [ 41 ]	-	-	Program	-	https://github.com/dorianbrown/rank_bm25
BGE-small-zh-v1.5 [ 40 ]	0.02B	512	Weights	-	https://huggingface.co/BAAI/bge-small-zh-v1.5
Conan-embedding-v1 [ 42 ]	0.3B	512	Weights	-	https://huggingface.co/TencentBAC/Conan-embedding-v1
NV-Embed-v2 [ 43 ]	7B	32K	Weights	Mistral-7B-v0.1 [ 63 ]	https://huggingface.co/nvidia/NV-Embed-v2

Appendix

Table 12: Details of all evaluated retrievers. Zoom in for better view.

Figure 9: Prompt for response generation.

Model	LLM Extract(%)	Human Extract(%)	LLM Extract Var(%)
GPT-5.2	41.0	41.3(+0.3)	0.04
Gemini-3.1-Pro	62.9	63.1(+0.2)	0.09
Claude-4.6-Opus	46.0	46.1(+0.1)	0.13
Qwen3.5-Plus	54.1	53.9(-0.2)	0.14

Appendix

Table 13: Reliability Validation of the Automated Answer Extractor.

Task Type
		Gemini-3.1-Pro+IMG		Qwen3-VL-32B+IMG		Gemini-3.1-Pro+OCR		DeepSeek-v3.2+OCR
Subgroup Category	Size ( $n$ )	Score	95%CI	Score	95%CI	Score	95%CI	Score	95%CI
Single Page	546	66.9%	[63.2%, 70.5%]	53.9%	[49.9%, 57.6%]	73.7%	[70.2%, 77.2%]	68.7%	[65.0%, 72.2%]
Cross Pages	196	61.5%	[55.5%, 67.7%]	36.2%	[30.0%, 42.4%]	73.0%	[67.3%, 78.3%]	64.7%	[58.4%, 70.6%]
Multi-Hop	172	49.3%	[42.0%, 56.1%]	44.3%	[37.4%, 51.4%]	63.0%	[56.0%, 69.6%]	46.3%	[38.9%, 53.3%]
Unanswerable	98	67.4%	[58.2%, 76.5%]	40.8%	[30.6%, 51.0%]	54.1%	[43.9%, 64.3%]	40.8%	[30.6%, 51.0%]

Appendix

Table 14: Subgroup uncertainty estimates (95% bootstrap CI,

N=1,000

) across evaluated frontier models on LongHisDoc .

Models	Size	Evidence Page				Answer Format						Evidence Source			AVG
Models	Size	SIN	CRO	MUL	UNA	STR	INT	OPT	LIST	BOOL	NONE	TXT	LAY	VIS	AVG
OCR Results + Large Language Models (LLMs)
Qwen3 [ 19 ]	0.6B	21.2	13.8	18.5	26.5	15.7	17.7	47.2	14.4	49.1	26.5	20.3	18.5	15.8	19.8
Qwen3 (Think) [ 19 ]	0.6B	22.8	13.3	20.7	22.5	17.4	16.8	50.0	13.7	54.4	22.5	21.8	19.1	18.5	20.6
Qwen3 [ 19 ]	1.7B	36.4	25.1	23.2	11.2	28.9	29.4	44.4	26.6	64.9	11.2	36.3	27.8	22.7	29.5
Qwen3 (Think) [ 19 ]	1.7B	39.1	26.3	26.2	28.6	30.1	29.4	61.1	32.2	68.4	28.6	39.5	30.0	22.9	33.4

Appendix

Table 15: More model evaluation results.

Figure 10: Prompt for answer extraction (Format: String and List). Zoom in for better view.

Figure 11: Prompt for answer extraction (Format: Option and Bool). Zoom in for better view.

Figure 12: Prompt for answer extraction (Format: Integer and None). Zoom in for better view.

Figure 13: Selection rules of images input.

Figure 14: Text-input case 1. Evidence source: "Layout". Answer format: "List". Key information in the image is highlighted with green boxes, while incorrect information is marked in red. Zoom in for better view.

Figure 15: Text-input case 2. Evidence source: "Visual". Answer format: "String". Key information in the image is highlighted with green boxes, while incorrect information is marked in red. Zoom in for better view.

Figure 16: Text-input case 3. Evidence source: "Visual". Answer format: "List". Key information in the image is highlighted with green boxes, while incorrect information is marked in red. Zoom in for better view.

Figure 17: Image-input case 1. Evidence source: "Text". Answer format: "List". Key information in the images is highlighted with green boxes, while incorrect information is marked in red. Zoom in for better view.

Figure 18: Image-input case 2. Evidence source: "Layout". Answer format: "String". Key information in the images is highlighted with green boxes, while incorrect information is marked in red. Zoom in for better view.

Figure 19: Image-input case 3. Evidence source: "Layout". Answer format: "List". Key information in the images is highlighted with green boxes, while incorrect information is marked in red. Zoom in for better view.

Figure 20: Top 2~50 recall rate of retrievers on LongHisDoc.

Document Understanding

Document understanding enables machines to interpret text, layout, tables, and images to extract information and answer questions about documents.

See all 93 papers

Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination
Apr 27, 2026Lirong Gao, Zeqing Wang, Yuyan Cai +6LLM Evaluation Question Answering
State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group · Zhejiang University
XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding
Jul 21, 2026Hongchen Wei, Yuanzhe Wang, Bei Liu +9Long-Context Retrieval LLM Evaluation
1Wuhan University · 2Microsoft
DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
May 9, 2026Xiang Feng, Jiawei Zhou, Zhangfeng Huang +6Multimodal Understanding LLM Answer Verification
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China · Alibaba Group, Hangzhou, China · Independent Researcher +1

Apr 27, 2026J/K move · Enter open · S save

Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination

Lirong Gao, Zeqing Wang, Yuyan Cai, Jiayi Deng+5

State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group · Zhejiang University

LLM Evaluation Question Answering

Open paper AskPDF arXiv

While Large Language Models (LLMs) have increasingly assisted in historical tasks such as text processing, their capacity for professional-level historical reasoning remains underexplored. Existing benchmarks primarily assess basic knowledge breadth or lexical understanding, failing to capture the higher-order skills, such as evidentiary reasoning,that are central to historical research. To fill this gap, we introduce ProHist-Bench, a novel benchmark anchored in the Chinese Imperial Examination (Keju) system, a comprehensive microcosm of East Asian political, social, and intellectual history spanning over 1,300 years. Developed through deep interdisciplinary collaboration, ProHist-Bench features 400 challenging, expert-curated questions across eight dynasties, accompanied by 10,891 fine-grained evaluation rubrics. Through a rigorous evaluation of 18 LLMs, we reveal a significant proficiency gap: even state-of-the-art LLMs struggle with complex historical research questions. We hope ProHist-Bench will facilitate the development of domain-specific reasoning LLMs, advance computational historical research, and further uncover the untapped potential of LLMs. We release ProHist-Bench at https://github.com/inclusionAI/ABench/tree/main/ProHist-Bench.

Abstract

Document Understanding

Explore similar work

Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination

XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding

DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding