Large language models answering questions over multi-page documents are expected to cite the supporting pages, yet supplied citations are sometimes inaccurate, and current evaluations score citations at generation time or against text passages: no existing benchmark evaluates whether a system can verify and correct a page-level citation already attached to an answer. We propose AtomCite, an agentic framework that parses an answer into claims, checks each claim against the image of its cited page, and applies a deterministic repair policy. To evaluate it, we introduce DocCite, to our knowledge the first benchmark for systems that verify and correct page-level citations in document images. Built on MP-DocVQA and DUDE, it combines 928 validated injected instances with 2,468 candidate natural errors harvested from frontier- and efficiency-tier models, of which a two-annotator audit confirms 1,909 as genuine errors. Primary labels are assigned deterministically, not by LLM judges, with the human audit as a separate validation layer. Across three model families (Gemini, Claude, and GPT), AtomCite reaches around 93% binary verification accuracy on the injected benchmark, significantly outperforming every OCR-only condition, including a compute-matched control, and exceeding every prior text-based baseline given the same OCR text. Its repair policy lifts citation precision on the injected mix from a constructed 34% to 87-90% while retaining over 90% of correct claims. AtomCite also transfers: with frozen prompts and zero training, it raises the hallucination-detection scores of two open 7-8B models on five public benchmarks above the same models prompted as direct judges. Finally, the audit shows that noise in automatic labels biases measured verifier accuracy and can reverse system rankings, so evaluations relying only on synthetic or automatic labels risk mismeasuring verification capability.
Figures & tables
Figure 1: The AtomCite pipeline on a four-claim answer, the five stages grouped into parse, verify, and correct. Each parsed (claim, cited page) pair is checked by text and image sub-agents on the cited page only; window pages contribute correction proposals, never verdicts. The example covers all four actions: a true claim cited to the wrong page is repointed ( fix-page ), a verified claim kept, a claim supported nowhere removed, and a hard text–image conflict ( inconsistent ) kept and flagged, never deleted.
Gemini
Claude
GPT
Condition
Acc
Sup. R
Acc
Sup. R
Acc
Sup. R
(A1) Tesseract flat
84.5
59.7
84.8
63.5
83.7
58.5
(A2) Tesseract layout
85.2
62.3
84.2
63.8
85.2
61.9
(A3) Tesseract matched
86.2
62.6
86.7
71.7
87.8
69.2
(A1 ′ ) MinerU flat
87.7
69.8
88.4
70.8
89.0
73.6
(A2 ′ ) MinerU layout
88.5
72.0
89.1
74.5
88.1
71.4
Table 1: DocCite-Syn conditions ladder ( n=928 ; %). Acc = binary cited-page support accuracy (primary metric); Sup. R = recall on truth-supported instances. (C) is claim-level AtomCite (its decomposition variant is ablated with the results). Bold = per-family best among image-grounded conditions; the two (B1) Sup. R cells above (C) reflect lenient acceptance, not sharper discrimination (see Architecture vs. Prompting); paired differences with CIs in Table 2 . The matched control (A3) reaches its score with near-zero wrong-location recall.
Difference
Gemini
Claude
GPT
Modality axis: what does the image channel contribute?
(C) − (A1 ′ )
+5.9 [3.8,8.0]
+4.5 [2.4,6.5]
+3.9 [2.0,5.8]
(C) − (A2 ′ )
+5.1 [3.2,7.1]
+3.7 [1.8,5.6]
+4.8 [2.8,6.8]
(C) − (A3 ′ )
+4.9 [3.0,6.9]
+2.3 [0.3,4.2]
+3.9 [2.1,5.8]
Prompting axis: does staged prompting close the gap?
(C) − (B1)
+5.9 [3.4,8.5]
+11.2 [8.7,13.8]
+4.8 [2.6,6.9]
Table 2: Key paired differences in binary accuracy (pp; doc-cluster bootstrap, B=104 , 95% CIs). Condition codes as in Table 1 ; the modality rows use the stronger MinerU engine (corpus-engine gaps are larger). All 18 differences are Holm-significant (CIs per-comparison).
System
Tier
Acc
Sup. R
CiteEval-Auto (Gem/Cl/GPT)
T2
89.0/88.2/88.9
—
HHEM
T1
83.9
64.5
MiniCheck
T1
82.9
57.5
LettuceDetect
T1
81.7
52.5
AlignScore
T1
76.0
33.0
DeBERTa-NLI
T1
72.8
81.4
Table 3: Text-Baseline Tier on DocCite-Syn ( n=928 ; %; detector operating point 0.5 ). T1 = specialized detectors run as intended; T2 = adapted protocols. Same-family paired differences, (C) − CiteEval-Auto: +4.5[2.0,7.1] Gemini, +4.9[2.2,7.7] Claude, +4.1[1.7,6.7] GPT, all significant.
Gemini
Claude
GPT
Condition
Det
FA
Acc
Det
FA
Acc
Det
FA
Acc
AtomCite (C)
97.7
5.5
97.1
97.3
5.4
96.8
98.6
12.3
96.5
Flat OCR (A1)
99.7
43.8
91.4
99.8
48.2
90.7
99.7
49.3
90.3
Seq. chain (B3)
91.1
16.9
89.6
87.2
10.0
87.7
96.3
12.4
94.6
Table 4: DocCite-Nat against audit-validated labels (%; best per column in bold). Partial support counts as an error. Det = detection of the 1,909 audit-validated errors; FA = false-alarm rate on the 450 human-confirmed supported claims; Acc = overall accuracy on the 2,359 audited instances. Rates are over the audited candidate pool, not all supplied citations. Balanced accuracy: C 96.1 / 96.0 / 93.1 , A1 78.0 / 75.8 / 75.2 , B3 87.1 / 88.6 / 91.9 (Gemini/Claude/GPT).
System
Precision
Retention
Coverage
Fix-reach
AtomCite (Gemini)
34.3 → 89.9
91.8
95.4
68.9
AtomCite (Claude)
34.3 → 87.7
91.8
94.1
72.9
AtomCite (GPT)
34.3 → 87.0
92.5
94.1
67.7
CiteFix (T3, reimpl.)
34.3 → 50.3
100.0 †
54.9
80.5
RARR (T2, adapted)
34.3 → 36.8
94.0
15.7
0.0
Table 5: Correction on DocCite-Syn (conservative mode; %). Precision = citation precision before → after correction; Retention = correct claims kept; Coverage = error instances acted on; Fix-reach = exact gold-page recovery on window-reachable wrong-location citations. † CiteFix is replacement-only (cannot remove): its 100% retention is uninformative.
Qwen2.5-7B
Llama-3-8B
Benchmark (metric)
direct
AtomCite
direct
AtomCite
RAGTruth (F1)
49.5
71.2
56.4
66.2
FaithBench (bAcc)
50.8
57.0
52.8
56.9
HalluMix (acc)
76.3
80.0
58.0
59.7
TofuEval (bAcc)
67.7
69.1
54.3
64.2
VeriGray resp. (F1)
29.8
44.6
39.7
54.7
Table 6: Transfer to five public hallucination benchmarks (%), under official test sets and metrics ( n=2,700 / 750 / 6,500 / 534 / 329 ): F1 = response-level F1, bAcc = balanced accuracy, acc = accuracy. The direct arm is a same-backbone single-call judge; AtomCite is the framework in text mode.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Generator
AB claims
Cand. errors
Prev. (%)
Gemini frontier
954
426
44.7
Gemini efficiency
1,076
556
51.7
Claude frontier
1,037
590
56.9
Claude efficiency
1,098
601
54.7
GPT frontier
284
134
47.2
GPT efficiency
139
85
61.2
Appendix
Table A1: Layer-1 candidate-error prevalence by generator (AB = answer-bearing cited claims). Candidate status is the Layer-1 localization label; the two-annotator human audit validates 77.4% of pooled candidates as genuine errors (partial support counted as error).
Condition
Binary acc. (%)
$/item
Calls/item
(C) AtomCite
93.7
0.024 ∗
2.8
(C-atom)
91.9
0.030
3.5
(B3) seq. chain
87.5
0.029
3.5
(A3) Tesseract matched
86.2
0.028
4.0
(B2) checklist
88.1
0.020
1.0
(B1) direct
87.9
0.018
1.0
Appendix
Table A2: Cost-effectiveness on DocCite-Syn (Gemini backbone; list prices, uncached calls, per-item ledger averages). ∗ (C) measured on its two cache-disabled repeats (run-to-run variance paragraph); its original run shared cached grounding calls with (C-atom) on single-atom claims. Call/token structure is identical across families; absolute prices differ by provider list price.
Condition
Bin
Sup. R
Gap to (C)
95% CI
Gem.
(A1) flat
87.7
69.8
+5.9
[3.8,8.0]
(A2) layout
88.5
72.0
+5.1
[3.2,7.1]
(A3) matched
88.7
72.6
+4.9
[3.0,6.9]
Cl.
(A1) flat
88.4
70.8
+4.5
[2.4,6.5]
(A2) layout
89.1
74.5
+3.7
[1.8,5.6]
(A3) matched
90.5
78.6
+2.3
[0.3,4.2]
Appendix
Table A3: Full MinerU 2.5 replication of the A ladder ( n=928 ; %). Gap = paired difference (C) − rung, doc-cluster bootstrap, B=104 ; all nine are Holm-significant ( α=0.05 , family of nine). Against the corpus-engine ladder, the stronger engine lifts every rung (supported recall by +8 – 15 pp) without closing the image gap in any family.
Condition
Binary
Sup. R
WL R
Gemini
(A1) Tesseract flat
84.5
59.7
56.9
(A2) Tesseract layout
85.2
62.3
57.7
(A3) Tesseract matched
86.2
62.6
0 0.6
(B1) direct
87.9
81.4
59.9
(B2) structured
88.1
80.8
55.8
(B3) seq. matched
87.5
75.2
58.6
Appendix
Table A4: Full metric breakdown of the conditions ladder ( n=928 ; %). WL R = recall on truth-wrong-location instances. The compute-matched OCR control (A3) reaches its binary score with no localization ability: its wrong-location recall collapses to ≤1.4 in every family — its binary gains come entirely from coarse supported/unsupported calls. Wrong-location recognition is the capability the correction policy builds on.
Window
Bin
Prec.
Ret.
Reach
All
Pages
±1
91.9
91.8
85.8
69.2
45.3
1.77
±2 (default)
91.9
89.9
86.5
65.3
45.3
3.33
±3
91.9
88.6
86.5
62.9
45.9
3.77
±4
91.9
88.0
86.2
47.1
36.5
4.00
±5
91.9
86.9
87.7
38.5
35.6
4.00
Appendix
Table A5: Correction-window sweep (Gemini, n=928 ; %; run under the decomposition-variant configuration — verification is window-invariant in both configurations, and the correction trade-off direction carries over). Bin = verification binary accuracy (invariant by design); Prec. = post-correction citation precision; Reach = exact recovery on wrong-location citations reachable within each window; All = exact recovery over all wrong-location citations; Pages = mean effective window size after document-boundary and evidence-cap truncation. Marginal re-run token cost rises monotonically with width (from ∼ 2.2k/item at ±1 to ∼ 4.2k at ±5 ).
Input: answer A ; document D (pages: OCR text + image)
1:
C←\textscParse(A)⊳ (claim, cited page) pairs
2:
for each (x,p)∈C in parallel:
3:
E←\textscEvidence(D,p)⊳ cited page +±2 window
4:
vtxt←\textscGround(x,OCR(p))⊳ text channel
vimg←\textscGround(x,Img(p))⊳ image channel
5:
v←\textscMerge(vtxt,vimg)⊳ hard → inc. ; soft → image wins
Multimodal Large Language Models (MLLMs) have significantly advanced document understanding, yet current Doc-VQA evaluations score only the final answer and leave the supporting evidence unchecked. This answer-only approach masks a critical failure mode: a model can land on the correct answer while grounding it in the wrong passage---a critical risk in high-stakes domains like law, finance, and medicine, where every conclusion must be traceable to a specific source region. To address this, we introduce CiteVQA, a benchmark that requires models to return element-level bounding-box citations alongside each answer, evaluating both jointly. CiteVQA comprises 1,897 questions across 711 PDFs spanning seven domains and two languages, averaging 40.6 pages per document. To ensure fidelity and scalability, the ground-truth citations are generated by an automated pipeline---which identifies crucial evidence via masking ablation and enforces multi-stage quality control. At the core of our evaluation is Strict Attributed Accuracy (SAA), which credits a prediction only when the answer and the cited region are both correct. Auditing 20 MLLMs reveals a pervasive Attribution Hallucination: models frequently produce the right answer while citing the wrong region. The strongest system (Gemini-3.1-Pro-Preview) achieves an SAA of only 76.0, and the strongest open-source MLLM reaches just 22.5. Ultimately, towards trustworthy document intelligence, CiteVQA exposes a reliability gap that answer-only evaluations overlook, providing the instrumentation needed to close it. Our repository is available at https://github.com/opendatalab/CiteVQA.
Dongsheng Ma, Jiayu Li, Zhengren Wang +9
Peking University · Shanghai Artificial Intelligence Laboratory
Large language models can answer questions correctly and still hallucinate their citations. Most citation systems address this by retrieving documents at inference time. Retrieval adds latency and depends on external infrastructure. Retrieval noise can also affect the answer. We study whether LLMs can cite documents from their (continual) pretraining without test-time retrieval. We introduce CitePretrainBench to evaluate this setting. It includes Wikipedia, Common Crawl, and arXiv as real-world corpora and also includes novel, unseen documents. The benchmark includes short-form citation tasks for a single fact and long-form tasks for multiple facts. Continual-pretraining first binds factual knowledge to a persistent identifier for each document. Instruction tuning then teaches the model to put these identifiers in its answers as citations. Passive Indexing appends an identifier at the end of each document. Synthetic data used in Active Indexing restate facts in diverse, compositional forms. We train two mappings together. These mappings are source→fact and fact→source. The first direction involves the model generating content given a cited source. In the second, it learns to attribute its own answers when facts are paraphrased or combined. On Qwen-2.5-7B and 3B, Active Indexing improves citation precision over Passive Indexing on every evaluated citation task, with gains of up to 30.2 percentage points. In the scaling ablation, citation performance continues to improve as we add augmented data, including at 16 times the original token count. Combining internal and external citations also improves robustness to retrieval noise.
Scientific research relies on citation integrity, yet large language models (LLMs) have introduced a critical risk: fabricated references that appear plausible but correspond to no real publications. As manual verification becomes infeasible and existing automated tools remain fragile, we introduce CiteAudit, a comprehensive benchmark and detection framework for hallucinated citations. We design a multi-agent verification pipeline that decomposes citation checking into metadata extraction, memory lookup, web-based retrieval, and final judgment. To evaluate this, we construct a large-scale, human-validated dataset spanning diverse domains and hallucination types. Experiments demonstrate that our framework achieves superior verification performance over state-of-the-art LLMs and commercial baselines. Our work provides the necessary infrastructure to audit citations at scale and safeguard the trustworthiness of scholarly discourse. Code is available at https://github.com/shiiiikw/CiteAudit.