OralAgent: Integrating Reasoning, Tools, and Knowledge for Interactive Dental Image Analysis
Authors: Jing Hao, Siyuan Dai, Yongxin Zhang, Yuci Liang, Jiamin Wu, Jiahao Bao, Yuxuan Fan, Zanting Ye, +8 more
Organizations: Faculty of Dentistry, the University of Hongkong, Hong Kong SAR, China · Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, USA · Shenzhen University, China · Department of Craniomaxillofacial Surgery, Shanghai Ninth People’s Hospital, China · the Nanyang technological University, Singapore · School of Biomedical Engineering, Southern Medical University, China · Singapore University of Technology and Design, Singapore · the University of Auckland, new zealand · Shanghai Artificial Intelligence Laboratory , China
Dental image analysis plays a pivotal role in supporting accurate diagnosis and treatment planning in oral healthcare. Although recent advances have produced dental AI models for specific tasks and individual imaging modalities, their isolated designs limit practical use in real-world clinical workflows. In this paper, we present OralAgent, the first dental-specialized AI agent that unifies multimodal reasoning, tool-based decision-making, and knowledge-grounded retrieval within an end-to-end automated framework. It integrates 22 visual analysis tools and 368 widely-used classical dental textbooks, enabling autonomous reasoning, planning, tool use, knowledge retrieval, and multi-step workflow execution. Furthermore, we introduce OralCorpus, a large-scale, high-quality bilingual textual resource containing 134.8M tokens curated for dental retrieval-augmented generation (RAG). To evaluate models' multidisciplinary dental knowledge, we construct OralQA-ZH, a Chinese multiple-choice question benchmark consisting of 798 items across eleven oral subspecialties. Extensive experiments demonstrate that OralAgent achieves state-of-the-art performance on the MMOral-Uni, MMOral-OPG, and OralQA-ZH benchmarks, highlighting its effectiveness, interpretability, and adaptability in real-world clinical settings. The code and models are publicly available at https://github.com/isjinghao/OralAgent.
Figures & tables
Fig. 1 : The interaction paradigm between the user and our OralAgent. Users submit instructions to the OralAgent, which then begins reasoning based on the user’s intent and initiates an observation–thought–action loop until it produces a final response. OralAgent produces interpretable and reliable responses by generating visual explanations and by providing knowledge reference sources.
Fig. 2 : Overview of the OralAgent system. (a) The system detects user intent and automatically classifies any uploaded image modality to enable accurate tool selection and invocation. The identified intent is converted into a structured instruction for the core orchestrator. (b) The multi‑functional dental toolkit integrates tools that cover six imaging modalities and more than fifty diagnostic functions. (c) The dental knowledge‑based RAG module integrates 368 textbooks and also supports user‑provided private knowledge bases processed through offline embedding. During online retrieval, it selects the top-k knowledge chunks relevant to the user’s intent together with their sources. (d) The memory module stores the complete interaction history and system state as necessary context for reasoning. (e) OralAgent generates responses that include tool execution details, visualized image analysis, and retrieved knowledge with reference.
TABLE I : OralAgent integrates 22 tools across six dental imaging modalities, spanning six tasks, including classification, object detection, segmentation, keypoint detection, report generation, and visual QA. For reference, the table lists each tool’s supported functions, source, and performance.
Fig. 3 : Distribution of the proposed OralCorpus. Left: language composition; Right: dental specialty composition
Category
Abbrev.
Count
Endodontics
Endo
133
Periodontology
Perio
89
Oral and Maxillofacial Surgery
OMFS
180
Prosthodontics
Prosth
168
Orthodontics
Ortho
24
Oral Mucosal Diseases
OMD
54
TABLE II : Overview of Category Distribution in the OralQA-ZH Benchmark.
Fig. 4 : Examples and Distribution of the Proposed OralQA‑ZH Benchmark. (a) Two example QA pairs from the OralQA‑ZH Benchmark. All QA items are in multiple‑choice format, and each question may have more than one correct option. (b) Distribution of knowledge categories in the dental domain. (c) English translations of these two Chinses QA pairs for readability.
Fig. 5 : Tool use distribution of OralAgent on the MMOral‑Uni benchmark, where the system makes an average of 3.67 tool calls per case.
Model
II
PA
CE
PI
TP
IV
Overall
Loc
Dx-I
Dx-R
Proprietary MLLMs
GPT-5 [ 33 ]
44.60
45.24
25.16
31.43
41.27
40.52
80.67
56.00
36.42
o3 [ 34 ]
45.00
53.49
28.19
37.48
28.60
37.52
75.33
50.00
38.70
Grok-4 [ 35 ]
11.80
35.71
20.27
30.89
22.60
34.73
72.67
23.00
28.47
Doubao-1.5-vision-pro-32k [ 36 ]
26.90
43.33
16.97
40.48
18.20
28.49
68.00
45.00
30.67
TABLE III : Results on the MMOral-Uni for various existing MLLMs across five dental imaging modalities. The abbreviations are defined as follows: II - Intraoral Image, PA - Periapical Radiograph, CE - Cephalometric Radiograph, PI - Pathological Image, TP - Treatment Planning, IV - Intraoral Video. The best-performing model in each category is highlighted in-bold , while the second-best is underlined .
Model
Open‐ended VQA
Teeth
Patho
His
Jaw
Summ
Report
Overall
General-purpose MLLMs
GPT-5 [ 33 ]
39.77
29.32
44.05
78.56
40.12
28.20
42.42
GPT-4V [ 57 ]
31.46
23.79
39.51
69.81
34.29
43.70
39.38
Gemini-2.5-Flash [ 37 ]
28.04
24.77
31.90
47.81
12.98
16.70
27.84
Qwen-Max-VL [ 58 ]
2.10
4.47
7.06
11.62
7.98
5.50
5.29
TABLE IV : Performance on the MMOral-OPG benchmark for various MLLMs on open-ended VQA tasks. The best-performing model is highlighted in-bold , while the second-best is underlined .
Model
Endo
Perio
OMFS
Prosth
Ortho
OMD
PedDent
OMFR
PrevDent
OralEpi
OMFP
Overall
Individual LLM Baselines
GPT-5.4 [ 62 ]
74.44
89.89
76.11
66.07
37.50
90.74
82.14
58.33
63.41
75.86
82.50
74.69
GPT-5-nano [ 33 ]
63.91
80.90
60.00
51.19
25.00
77.78
60.71
50.00
58.54
72.41
80.00
62.53
Gemini-3.1-pro-thinking [ 37 ]
55.64
71.91
69.44
31.55
0.00
90.74
71.43
58.33
19.51
3.45
12.50
50.88
Kimi-k2.5 [ 63 ]
62.41
65.17
59.44
57.74
29.17
75.93
82.14
75.00
68.29
34.48
70.00
61.53
MiniMax-M2.7-Guan [ 64 ]
57.89
62.92
56.11
50.00
25.00
59.26
53.57
33.33
34.15
48.28
57.50
53.38
TABLE V : Performance comparison of various individual LLM on the OralQA-ZH benchmark, as well as the improvements obtained when each LLM serves as the core orchestrator in OralAgent and performs RAG over the OralCorpus knowledge base. The use of OralAgent significantly improves performance. The best-performing model in each category is highlighted in-bold , while the second-best is underlined .
Model
Intent
Modality
II
PA
CE
PI
TP
IV
Overall
Loc
Dx-I
Dx-R
OralAgent (Ours)
✓
✓
71.20
55.65
58.22
50.06
67.23
59.27
78.67
65.00
57.70
w/o Intent
×
✓
68.50
53.01
59.10
48.78
65.20
59.71
74.67
45.00
56.66
w/o Modality
✓
×
61.80
51.78
48.42
47.74
64.67
57.47
73.33
56.00
52.76
w/o Intent & Modality
×
×
59.90
49.47
46.46
47.35
65.63
57.18
67.33
47.00
51.50
TABLE VI : Ablation Study of the Input Instruction Comprehension Module in OralAgent: Prior Intent Recognition and Image Modality Classification. The best-performing model in each category is highlighted in-bold , while the second-best is underlined .
Fig. 6 : Case study on MMOral-Uni and MMOral-OPG benchmarks, respectively. (a) A case about an intraoral image from the MMOral-Uni benchmark. Note that the visualization is created through manual post-processing for readability only. Our OralAgent accurately localizes the carious region by invoking tools, and it can directly generate visualized outputs that support intuitive user interpretation. (b) A case about a panoramic x-ray from the MMOral-OPG benchmark. OralAgent performs a comprehensive assessment using four dental-specialized tools and successfully identifies the full condition of tooth 46. Besides, OralAgent provides traceable references throughout multi-turn interactions, further ensuring its reliability.
Fig. 7 : Case study on the OralQA-ZH benchmark. (a) Original example in Chinese. OralAgent can automatically invoke RAG tools to retrieve knowledge relevant to the question, generate the correct answer, and provide precise supporting references. (b) English translation for readability.
Department of Computer Science and Software Engineering, College of Information Technology, United Arab Emirates University, Al Ain, 15551, UAE · Department of Oral and Craniofacial Health Sciences, College of Dental Medicine, University of Sharjah, Sharjah, UAE · Weill Cornell Medicine-Qatar, Doha, Education City, Qatar +1
Department of Orthodontics, Beijing Stomatological Hospital, School of Stomatology, Capital Medical University, Fanjiacun Road #9, Fengtai District, Beijing 100070, China · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN, USA · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China +1