BibTeX Citation Errors in Scientific Publishing Agents: Evaluation and Mitigation

Delip Rao; Chris Callison-Burch

cs.DLApr 3, 2026

BibTeX Citation Errors in Scientific Publishing Agents: Evaluation and Mitigation

Authors: Delip Rao, Chris Callison-Burch

Organizations: University of Pennsylvania

Abstract

Large language models with web search are increasingly used in scientific publishing agents, yet they produce BibTeX entries with pervasive field-level errors stemming from omission, partial corruption, substitution, and hallucination. We construct a benchmark of 931 papers across four domains and three citation tiers---popular, low-citation, and recent post-cutoff---with version-aware ground truth. Three search-enabled frontier models (GPT-5, Claude Sonnet-4.6, Gemini-3 Flash) generate approximately 23,000 field-level observations. Overall accuracy is 83.6%, but only 50.9% of entries are fully correct; accuracy drops 27.7 pp from popular to recent papers, revealing heavy reliance on parametric memory even when search is available. Co-occurrence analysis identifies two failure modes: wholesale entry substitution and isolated field error. We present clibib, an open-source tool for deterministic BibTeX retrieval, as a mitigation mechanism. Two-stage integration raises accuracy to 91.5% (+8.0 pp) and fully correct entries to 78.3%, with a 0.8% regression rate. Separating search from revision yields larger gains and lower regression than single-stage tool loops (0.8% vs. 4.8%), demonstrating that integration architecture matters independently of model/tool capability. We release clibib and an accompanying agent skill under the MIT License to improve citation accuracy in increasingly automated scientific workflows.

Figures & tables

Model	API model ID	Search mechanism	Cutoff
GPT-5	gpt-5-2025-08-07	web_search (Responses API)	Oct 2024
Claude Sonnet 4.6	claude-sonnet-4-6	web_search_20250305 (beta)	Aug 2025
Gemini 3 Flash	gemini-3-flash-preview	Google Search grounding	Jan 2025

Table 1: Model configurations for BibTeX generation. All models use single-turn prompting with search tools force-enabled and a maximum output length of 8,192 tokens. GPT-5 does not expose a temperature parameter; Claude Sonnet 4.6 and Gemini 3 Flash use temperature 1.0.

Domain	Tier	$n$	Med. cit.	Years	DOI
AI	Popular	96	1,437	2010–2022	70%
	Low-citation	90	1	2015–2024	82%
	Recent	49	22	2025–2026	100%
Medicine	Popular	100	3,567	2010–2022	100%
	Low-citation	93	0	2015–2024	100%
	Recent	45	0	2026	100%

Table 2: Paper statistics by domain and citation tier after quality filtering. Med. cit. is the median cited_by_count . DOI column reports coverage rate.

Field	GPT-5	Claude 4.6	Gemini 3 Fl.	All
entry_type	90.3	86.1	96.9	91.1
author	93.5	86.0	93.9	91.1
year	87.4	83.1	94.4	88.3
venue	84.8	83.4	94.7	87.6
volume	74.8	82.4	92.6	83.3
title	78.5	78.1	85.3	80.6

Table 3: Per-field accuracy (% C of evaluable fields) by model. Fields ordered by overall accuracy. Entry_key excluded (always X).

Figure 1: Field-level accuracy by citation tier and model with 95% bootstrap confidence intervals. Popular papers achieve 92.7%; recent (post-cutoff) papers drop to 65.0%.

Figure 2: Conditional co-error probability matrix. Cell $(i,j)$ shows $P(\text{field }j\text{ is wrong}\mid\text{field }i\text{ is wrong})$ . Entry type, author, and year errors strongly predict venue and DOI errors (0.74–0.91), consistent with wholesale entry substitution.

Model	Baseline	Single	$\boldsymbol{\Delta}$	Two-st.	$\boldsymbol{\Delta}$
GPT-5	81.4	86.7	+5.3	92.5	+11.1
Claude Sonnet 4.6	80.4	84.0	+3.6	87.2	+6.8
Gemini 3 Flash	88.9	—		94.9	+6.0
All	83.6	—		91.5	+8.0

Table 4: Field-level accuracy (%) by model and integration strategy.

\Delta

= gain over baseline (pp). Gemini lacks single-stage support.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

partial_match	different_paper	Label
met	any	P
unmet	met	S
unmet	unmet	F
unmet	cannot_assess	F
cannot_assess	met	S
cannot_assess	unmet	F

Appendix

Table 5: Verdict mapping from binary criteria to field labels.

Mode	Field	Generated	Ground truth	Label
Wholesale substitution (GPT-5, medicine, low-citation)
	title	Clinical implications of intravesical recurrence after radical nephroureterectomy…	Impact of relapse site on oncological outcomes after radical nephroureterectomy…	S
	venue	International Journal of Urology	Journal of Clinical Oncology	S
	volume	23	34	F
	number	5	2_suppl	F
	pages	378–384	426–426	F

Appendix

Table 6: Representative examples of the two failure modes. Wholesale substitution : GPT-5 returns a coherent but wrong paper by the same first author, same year, same disease—a plausible near-miss. Isolated field error : Gemini 3 Flash identifies the correct paper but fabricates a page range offset by exactly 9 pages.

Domain	Venues
AI	NeurIPS, ICML, ICLR, ACL, EMNLP,
	CVPR, AAAI, NAACL, JMLR, TACL
Medicine	NEJM, The Lancet, JAMA, BMJ,
	Nature Medicine, Ann. Internal Med.,
	PLOS Medicine, Lancet Digital Health,
	J. Clinical Oncology, Circulation

Appendix

Table 7: Venues per domain. Venues marked with

\dagger

require topic filtering (quantum computing or materials chemistry subsets of broader journals).

Figure 3: Paper counts by domain and citation tier after quality filtering. Each domain initially targeted 100 popular, 100 low-citation, and 50 recent papers; final counts reflect removal of 66 non-research items during the data quality audit.

Figure 4: Distribution of cited-by counts across citation tiers (log scale). Popular papers have median citations of 500–3,567; low-citation papers cluster near 0–7; recent papers are predominantly uncited. The clear separation confirms that the tiers probe distinct regions of model familiarity.

Domain	$n$	Proc.	Journal	arXiv	Multi-ver.	Total ver.
AI	247	218	29	5	5 (2.0%)	252
Medicine	250	0	250	1	1 (0.4%)	251
MatSci	250	0	250	16	16 (6.4%)	266
QuantComp	250	0	250	112	112 (44.8%)	362
Total	997	218	779	134	134 (13.4%)	1,131

Appendix

Table 8: Version landscape of the 997 papers. Columns report the number of papers with at least one version of the given type. Multi-ver. counts papers with more than one fetchable version (i.e., both a published venue version and an arXiv preprint). Total ver. is the sum of all fetchable versions across papers.

Figure 5: Version composition of papers by domain. Each bar shows the proportion of papers in each version combination. Quantum Computing has the highest multi-version rate (44.8%), driven by the physics preprint culture. AI is proceedings-dominated (86.2%), while Medicine and Materials Science are almost entirely journal-only.

Figure 6: BibTeX retrieval outcomes by domain. Quantum Computing and Materials Science achieve

>

95% success. Medicine’s lower rate reflects paywalled journal articles that Zotero cannot resolve, with many title-mismatch failures on fallback queries.

Figure 7: Canonical field coverage by domain (percentage of papers with a resolved value). Core fields (DOI, title, author, year) exceed 80% in all domains except Medicine. Booktitle is concentrated in AI (conference proceedings); journal dominates the other three domains. Medicine has the lowest coverage due to its lower BibTeX retrieval rate.

Figure 8: Per-field accuracy by model. Gemini 3 Flash leads on all nine fields; GPT-5 and Claude are competitive on author and pages.

Figure 9: Per-domain accuracy by model. Medicine is the hardest domain; materials science and quantum computing are easiest.

Figure 10: Accuracy conditioned on search invocation, by model. GPT-5 achieves 85.1% without search on 49 entries (predominantly popular papers). Gemini and Claude without search produce all-Missing fields (0% accuracy); these 22 and 1 entries, respectively, are included in all analyses.

Field	Popular	Low-cit.	Recent	$\boldsymbol{\Delta}$
Entry type	99.4 [99.0, 99.7]	92.0 [90.2, 93.9]	72.4 [68.4, 76.3]	$-$ 27.0
Author	97.3 [95.8, 98.6]	90.4 [88.0, 92.6]	79.8 [75.6, 83.8]	$-$ 17.5
Title	90.6 [88.0, 93.0]	74.4 [70.4, 78.2]	71.6 [66.5, 76.4]	$-$ 19.0
Year	98.7 [97.9, 99.4]	88.3 [85.8, 90.6]	67.0 [62.5, 71.6]	$-$ 31.7
Venue	97.4 [96.0, 98.6]	88.0 [85.6, 90.4]	67.0 [62.3, 72.1]	$-$ 30.4
Volume	89.8 [87.1, 92.2]	82.5 [79.8, 85.3]	57.2 [50.6, 63.4]	$-$ 32.6

Appendix

Table 9: Per-field accuracy (% C) by citation tier, pooled across models. 95% bootstrap CIs in brackets.

\Delta

is the popular-to-recent drop in percentage points.

Field	$n$	Agreement (%)	$\kappa$
year	56	100.0	1.000
number	27	92.6	0.816
author	37	89.2	0.798
pages	65	81.5	0.674
title	157	96.8	0.536
volume	29	79.3	0.461

Appendix

Table 10: Per-field inter-annotator agreement between the LLM judge and human annotator on 521 Stage 2 fields. Fields are ordered by descending

\kappa

\dagger

marks fields with systematic disagreement patterns (see text).

Figure 11: Error-type distribution per field (all models combined). Each row shows the fraction of evaluable fields receiving each label. Author, entry type, and year have the highest correct rates; number and DOI are most error-prone.

Figure 12: Error composition by model (excluding correct and not-applicable fields). Missing is the dominant error type for all models. Claude has the highest missing rate; GPT-5 has the highest fabrication rate.

Matching criterion	GPT-5	Claude Sonnet 4.6	Gemini 3 Flash	All
First-author last name	93.1 [91.3, 94.6]	85.6 [83.4, 87.8]	93.4 [91.7, 95.1]	90.7 [89.6, 91.7]
All last names (set)	82.3 [79.8, 84.6]	76.6 [73.9, 79.2]	80.0 [77.4, 82.6]	79.6 [78.1, 81.1]
All last names (ordered)	82.3 [79.8, 84.6]	76.4 [73.6, 79.1]	79.9 [77.3, 82.5]	79.5 [78.0, 81.0]
$\Delta$ (first-author $\to$ set)	$-$ 10.8	$-$ 9.0	$-$ 13.4	$-$ 11.1

Appendix

Table 11: Author-field accuracy under increasingly strict matching criteria. The baseline evaluation matches only the first author’s last name; stricter criteria require all authors’ last names to match. The 11.1 pp drop from first-author to set matching reflects truncated author lists and name-normalization artifacts rather than wholesale substitution (see text). Brackets denote 95% bootstrap confidence intervals.

Tier	Model	Field-level	Fully correct
Popular	GPT-5	91.7 [90.3, 92.9]	60.3 [55.7, 65.4]
	Claude 4.6	93.4 [92.1, 94.6]	68.2 [63.6, 72.5]
	Gemini 3 Fl.	93.1 [91.6, 94.4]	64.1 [59.3, 68.7]
	Pooled	92.7 [91.6, 93.7]	—
Low-citation	GPT-5	81.4 [78.6, 84.1]	42.7 [37.2, 47.8]
	Claude 4.6	77.8 [74.3, 81.3]	49.0 [43.8, 54.2]

Appendix

Table 12: Accuracy by citation tier and model. Field-level: % of evaluable fields labeled C. Fully correct: % of entries with all evaluable fields correct. 95% bootstrap CIs in brackets (2,000 cluster-resampled iterations).

Figure 13: Overall accuracy (percentage of evaluable fields labeled C) before and after single-stage clibib augmentation (GPT-5 and Claude only). Both models improve: GPT-5 by +5.3 pp and Claude by +3.6 pp.

Figure 14: Per-field accuracy before (baseline) and after single-stage clibib -augmented generation (GPT-5 and Claude only). Annotations show the net change in percentage points. Number (+19.1 pp), volume (+9.0 pp), and pages (+7.9 pp) show the largest gains.

Error type	Baseline errors	Corrected	Rate (%)
Missing (M)	1,877	924	49.2
Partial (P)	564	268	47.5
Fabricated (F)	315	57	18.1
Substituted (S)	188	28	14.9
All errors	2,944	1,277	43.4

Appendix

Table 13: Correction rates by original error type (GPT-5 and Claude only). Correction = baseline error that becomes C with single-stage clibib augmentation.

Figure 15: Correction rate by original error type, per model (single-stage clibib , GPT-5 and Claude only). Missing fields are corrected most often (49.2% overall). GPT-5 shows the highest correction rates for missing and partial errors.

Condition	Lookups	Success (%)	Not found (%)
Overall	2,617	88.9	11.1
GPT-5	1,432	89.7	10.3
Claude Sonnet 4.6	1,185	87.9	12.1
AI	864	81.8	18.2
Medicine	642	87.5	12.5
Mat. Sci.	577	93.4	6.6

Appendix

Table 14: clibib lookup coverage by model, domain, and citation tier (single-stage, GPT-5 and Claude only). “Lookups” counts individual clibib_lookup calls; “Success” is the fraction that returned a valid BibTeX entry; “Not found” is the fraction with no match in any backend database.

Figure 16: Per-error-type correction rates for single-stage vs. two-stage integration, by model. GPT-5 shows a consistent two-stage advantage across all error types; Claude shows a reversal for missing fields (

\Delta=-10.4

pp), where single-stage corrects more effectively.

Figure 17: Concordance and discordance between single-stage and two-stage integration on 15,402 evaluable fields (GPT-5 and Claude). Two-stage wins 2.4

\times

more often than single-stage when the strategies disagree.

Figure 18: Label transition matrix between single-stage (rows) and two-stage (columns) integration, aggregated over GPT-5 and Claude. The dominant off-diagonal transitions involve missing (M) labels: 630 fields are corrected from M to C by two-stage; 380 regress from C to M. Log-scale coloring accommodates the C $\to$ C cell (12,661).

Figure 19: Field-level accuracy by model and integration strategy (baseline, single-stage clibib , two-stage clibib ). Single-stage results are shown for GPT-5 and Claude only; Gemini lacks single-stage support. Two-stage integration improves all three models over baseline.

Figure 20: Per-field accuracy before (baseline) and after two-stage clibib revision, averaged across all three models. Number (+22.1 pp), doi (+14.9 pp), and pages (+13.0 pp) show the largest gains.

Figure 21: Two-stage correction rate by original error type, per model. GPT-5 shows the highest correction rates for partial and missing errors; Gemini corrects partial errors most effectively (71.4%).

Citation Verification

Citation verification checks whether cited sources exist, are accurately represented, and actually support the claims attached to them.

See all 48 papers

HALLMARK: How to Diagnose LLM Citation Verifiers and When to Deploy Them
Jul 20, 2026Kayoon Kim, Wieland Brendel, Patrik ReizingerLLM Evaluation Citation Verification
Max Planck Institute for Intelligent Systems, ELLIS Institute Tübingen
Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection
May 9, 2026Mingzhe Li, Zhiqiang Lin, Shiqing MaHallucination in Language Models Citation Verification
University of Massachusetts Amherst · The Ohio State University
Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences
Jul 1, 2026Mark Russinovich, Ram Shankar Siva Kumar, Ahmed SalemHallucination in Language Models Citation Verification
*Microsoft Azure · §Microsoft

Jul 20, 2026J/K move · Enter open · S save

HALLMARK: How to Diagnose LLM Citation Verifiers and When to Deploy Them

Kayoon Kim, Wieland Brendel, Patrik Reizinger

Max Planck Institute for Intelligent Systems, ELLIS Institute Tübingen

LLM Evaluation Citation Verification LLM Hallucination Detection

Open paper AskPDF arXiv

GPTZero found 53 NeurIPS 2025 accepted papers with hallucinated citations, and venues now screen submissions with automated reference checkers. Rule- and Large Language Model (LLM)-based verifiers are emerging, but many are released without a shared evaluation and each flags a different kind of error, so it is unclear which to trust or when to deploy them. We address this with HALLMARK (Hallucination benchmark): 2,526 BibTeX entries spanning 14 hallucination types, with per-type and diagnostic sub-test labels and further temporal, cross-domain, and real-world evaluation splits. We evaluate zero-shot and tool-augmented LLMs, a DOI-lookup baseline, and our own co-designed rule-based verifier, bibtex-updater. Verifiers can miss hallucinations and wrongly flag correct papers, so every flag still needs human review. The false positive rate (FPR) determines the review effort and thus limits deployment. The tested models exhibit an order-of-magnitude spread in FPR; agentic lookups buy recall but inflate FPR, and neither smaller models nor prompt optimization reach the frontier's operating point. A two-stage cascade with bibtex-updater avoids that inflation, providing an upper bound, given the co-design. These rankings, however, depend on what counts as a hallucination: re-scoring the types that describe a real work cited inaccurately changes which verifier ranks first, and venues have not agreed on a definition. We call for a community definition and release per-type labels and a re-scoring protocol so that any agreed definition can be scored against every verifier.

Abstract

Citation Verification

Explore similar work

HALLMARK: How to Diagnose LLM Citation Verifiers and When to Deploy Them

Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection

Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences