Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation

Jianing Zhang; Runan Li; Honglin Pang; Ding Xia; Zhou Zhu; Qian Zhang; Chuntao Li; Xi Yang

cs.CVApr 8, 2026

Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation

Authors: Jianing Zhang, Runan Li, Honglin Pang, Ding Xia, Zhou Zhu, Qian Zhang, Chuntao Li, Xi Yang

Organizations: College of Software, Jilin University · School of Artificial Intelligence, Jilin University · Graduate School of Information Science and Technology, The University of Tokyo · School of Archaeology, Jilin University ·

Abstract

Deciphering ancient Chinese Oracle Bone Script (OBS) is a challenging task that offers insights into the beliefs, systems, and culture of the ancient era. Existing approaches treat decipherment as a closed-set image recognition problem, which fails to bridge the ``interpretation gap'': while individual characters are often unique and rare, they are composed of a limited set of recurring, pictographic components that carry transferable semantic meanings. To leverage this structural logic, we propose an agent-driven Vision-Language Model (VLM) framework that integrates a VLM for precise visual grounding with an LLM-based agent to automate a reasoning chain of component identification, graph-based knowledge retrieval, and relationship inference for linguistically accurate interpretation. To support this, we also introduce OB-Radix, an expert-annotated dataset providing structural and semantic data absent from prior corpora, comprising 1,022 character images (934 unique characters) and 1,853 fine-grained component images across 478 distinct components with verified explanations. By evaluating our system across three benchmarks of different tasks, we demonstrate that our framework yields more detailed and precise decipherments compared to baseline methods.

Figures & tables

Figure 1: Oracle Bone Script (OBS), a pictographic writing system of semantic components.

Figure 2: Comparison of our proposed framework and baselines. We design an agentic RAG framework to integrate component-level knowledge for structured semantic augmentation of OBS.

Figure 3: Our annotation of an oracle character at the component level.

Panel of Figure 4 — Figure 4: Detailed pipeline of our approach: (a) Component Identification Module identifies radical components from input OBS images; (b) Agent-Driven Graph Knowledge Retrieval retrieves relevant information from our constructed knowledge graph; (c) Component Relationship Inference uses VLMs to determine the structural relationships among components; (d) Interpretation Generation produces comprehensive semantic interpretations of oracle characters.

Metric	ACC $\uparrow$
Top-1	0.7795
Top-3	0.8855
Top-5	0.9157

Table 1: OBS component retrieval results.

Category	Model	ACC $\uparrow$	BERT $\uparrow$	Mover $\uparrow$	ROUGE-1 $\uparrow$	LLM-Judge $\uparrow$
Baseline	GPT	0.364	0.497	0.310	0.007	0.237
	Claude	0.475	0.495	0.324	0.009	0.225
	GLM	0.447	0.519	0.293	0.010	0.113
	Qwen	0.350	0.503	0.318	0.012	0.165
Ours	GPT	0.563 +0.199	0.670 +0.173	0.472 +0.161	0.199 +0.192	0.435 +0.198
	Claude	0.551 +0.075	0.648 +0.152	0.490 +0.166	0.221 +0.212	0.412 +0.187

Table 2: OBS component relationship inference results.

Figure 5: Reasoning examples for component relationship inference. Ground truth shows expert interpretations.

Category	Model	BERT $\uparrow$	Mover $\uparrow$	ROUGE-1 $\uparrow$	LLM-Judge $\uparrow$
Baseline	GPT	0.633	0.393	0.227	0.102
	Claude	0.614	0.365	0.232	0.052
	GLM	0.634	0.338	0.275	0.042
	Qwen	0.636	0.362	0.264	0.076
Ours	GPT	0.727 +0.094	0.475 +0.082	0.321 +0.094	0.558 +0.456
	Claude	0.716 +0.102	0.474 +0.109	0.335 +0.103	0.382 +0.330

Table 3: OBS interpretation generation results.

Model	Retrieval	BERT $\uparrow$	Mover $\uparrow$	ROUGE-1 $\uparrow$	LLM-Judge $\uparrow$
GPT	✓	0.727	0.475	0.321	0.558
GPT		0.717 -0.010	0.469 -0.006	0.305 -0.016	0.542 -0.016
Claude	✓	0.716	0.474	0.335	0.382
Claude		0.699 -0.017	0.451 -0.023	0.286 -0.049	0.372 -0.010
GLM	✓	0.706	0.453	0.337	0.212
GLM		0.687 -0.019	0.415 -0.038	0.283 -0.054	0.180 -0.032

Table 4: OBS relationship inference results.

Figure 6: Comparison of approach outputs. Character displays the original Oracle bone characters; Ground truth provides the ground truth interpretations; Multi-agent output shows our multi-agent approach using Graph RAG; Baseline output presents results from the baseline approach.

Retriever	Reasoner	BERT $\uparrow$	Mover $\uparrow$	ROUGE-1 $\uparrow$	LLM-Judge $\uparrow$
Qwen3-VL	DeepSeek-R1	0.760	0.507	0.431	0.531
	GPT-5	0.705	0.445	0.296	0.645
	Qwen3	0.734	0.470	0.361	0.494
GPT-5	DeepSeek-R1	0.733	0.476	0.402	0.413
	GPT-5	0.713	0.458	0.310	0.657
	Qwen3	0.729	0.454	0.366	0.464

Table 5: Performance comparison of multi-agent configuration.

Category	Model	BERT $\uparrow$	Mover $\uparrow$	ROUGE-1 $\uparrow$	LLM-Judge $\uparrow$
Baseline	GPT	0.152	-0.123	0.111	0.098
	Claude	0.136	-0.136	0.107	0.070
	GLM	0.036	-0.173	0.075	0.051
	Qwen	0.159	-0.126	0.117	0.068
Ours	GPT	0.272 +0.120	0.034 +0.157	0.201 +0.090	0.797 +0.699
	Claude	0.318 +0.182	0.071 +0.207	0.233 +0.126	0.752 +0.682

Table 6: Results of interpretations conducted in English.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Two images of oracle bone characters segmented by LabelMe.

Figure 9: The left side shows the baseline outputs, while the right side shows our results.

Model	Top-1@ACC	Top-5@ACC	Top-10@ACC
GPT-5	5.13% — 2	5.13% — 2	5.13% — 2
Claude Opus 4.1	0.00% — 0	2.56% — 1	5.13% — 2
GLM-4.5V	2.56% — 1	2.56% — 1	2.56% — 1
Qwen3-VL-235B-A22B	2.56% — 1	2.56% — 1	5.13% — 2

Appendix

Table 7: Variant character search (39 samples).

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

AlphaOracle: Oracle bone script decipherment via human-workflow-inspired deep learning
Jul 20, 2026Yuliang Liu, Haisu Guan, Pengjie Wang +11
School of Software Engineering, Huazhong University of Science and Technology, Wuhan 430074, China · Key Laboratory of Oracle Bone Inscriptions Information Processing, Anyang Normal University, Anyang 455002, China · School of Electronic Information Engineering, South China University of Technology, Guangzhou 510641, China
Decoding Ancient Oracle Bone Script via Generative Dictionary Retrieval
Apr 1, 2026Yin Wu, Gangjian Zhang, Jiayu Chen +4
Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
Beyond Single Character: Evaluating MLLMs for Sentence-Level Oracle Bone Inscription Understanding
Jun 30, 2026Ziqi Li, Zijian Chen, Tingzhu Chen +1VLM Evaluation Document Understanding
Nanjing University of Science and Technology, Nanjing 210094, China · Shanghai Jiao Tong University, Shanghai 200240, China · Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China

Jul 20, 2026J/K move · Enter open · S save

AlphaOracle: Oracle bone script decipherment via human-workflow-inspired deep learning

Yuliang Liu, Haisu Guan, Pengjie Wang, Xinyu Wang+10

School of Software Engineering, Huazhong University of Science and Technology, Wuhan 430074, China · Key Laboratory of Oracle Bone Inscriptions Information Processing, Anyang Normal University, Anyang 455002, China · School of Electronic Information Engineering, South China University of Technology, Guangzhou 510641, China

Open paper AskPDF arXiv

Approximately 3,000 of the 4,500 oracle bone script (OBS) characters remain undeciphered due to fragmentary inscriptions and sparse evidence. Current AI approaches fail to replicate expert workflows that integrate form analysis, contextual semantics, and philological reasoning. We introduce AlphaOracle, a human-workflow-inspired framework that systematizes OBS decipherment using the largest digitized corpus to date. Its multi-stage pipeline comprises: (i) rubbing parsing; (ii) radical-based morphological analysis with diachronic modeling; (iii) contextual retrieval with semantic alignment; and (iv) philological validation against classical sources. Each stage generates explicit, confidence-weighted evidence chains, culminating in interpretable reports for scholarly verification. Across multiple test characters, AlphaOracle's readings strongly agreed with expert interpretations. In a study of 86 domain specialists, it reduced analysis time by 64% and 79% of participants rated it highly useful. Notably, AlphaOracle resolves the character "Lao" as a toponymic or clan designation, offering concrete revisions to Shang administrative and social interpretations. These results suggest that computational methods aligned with philological practice can facilitate OBS research and provide a conceptual reference for studies of other undeciphered scripts.

Abstract

VLM Reasoning

Explore similar work

AlphaOracle: Oracle bone script decipherment via human-workflow-inspired deep learning

Decoding Ancient Oracle Bone Script via Generative Dictionary Retrieval

Beyond Single Character: Evaluating MLLMs for Sentence-Level Oracle Bone Inscription Understanding