cs.IRMar 10, 2026

Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learning

Authors: Huidong Wu, Haojia Xiang, Jingtong Gao, Xiangyu Zhao, Dengsheng Wu, Jianping Li

Organizations: Chinese Academy of Sciences & City University of Hong Kong · City University of Hong Kong · Shenzhen University & Beijing Dongbi Data Technology · University of Chinese Academy of Sciences

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scholarly web is a vast network of knowledge connected by citations. However, this system is increasingly compromised by miscitation, where references do not support or even contradict the claims they are cited for. Current miscitation detection methods, which primarily rely on semantic similarity or network anomalies, struggle to capture the nuanced relationship between a citation's context and its place in the wider network. While large language models (LLMs) offer powerful capabilities in semantic reasoning for this task, their deployment is hindered by hallucination risks and high computational costs. In this work, we introduce LLM-Augmented Graph Learning-based Miscitation Detector (LAGMiD), a novel framework that leverages LLMs for deep semantic reasoning over citation graphs and distills this knowledge into graph neural networks (GNNs) for efficient and scalable miscitation detection. Specifically, LAGMiD introduces an evidence-chain reasoning mechanism, which uses chain-of-thought prompting, to perform multi-hop citation tracing and assess semantic fidelity. To reduce LLM inference costs, we design a knowledge distillation method aligning GNN embeddings with intermediate LLM reasoning states. A collaborative learning strategy further routes complex cases to the LLM while optimizing the GNN for structure-based generalization. Experiments on three real-world benchmarks show that LAGMiD achieves state-of-the-art miscitation detection with significantly reduced inference cost.

Figures & tables

Figure 1. Illustrations of traditional miscitation detection methods. (a) Detection via anomalous citation patterns. (b) Detection via low semantic similarity. Illustrations of traditional miscitation detection methods.
Figure 2. The overview of the proposed LAGMiD framework. Overview Framework
Figure 3. The t-SNE visualizations of citation embeddings from the RED, comparing representations before and after knowledge distillation across different GNN layers. Blue circles denote valid citations, while red triangles represent miscitations. The solid line indicates the decision boundary. TSNE
Figure 4. Efficiency comparison of LAGMiD and LLM-based models: Training and inference runtime (in minutes). ruintime
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

DatasetAUC
K=1K=1K=2K=2K=3K=3K=4K=4
RED0.83620.85430.84700.8215
S2ORC0.72060.76800.74500.7312
Appendix
Table 5. Performance analysis of evidence chain length KK .
DatasetAUC
m=1m=1m=5m=5m=10m=10m=20m=20
RED0.81860.85430.85600.8420
S2ORC0.72000.74800.75900.7460
Appendix
Table 6. Performance analysis of evidence chain selection parameter mm (using Qwen3-8B).
Figure 5. Performance analysis of distillation temperature δ\delta across datasets. temperature analysis
Figure 6. Case study of a 2-hop evidence-chain reasoning process with Chain-of-Thought. A case of Evidence-Chain Reasoning
More in this topic

LLM Reasoning with Graphs

Large language model reasoning with graphs uses structured relationships to support multi-step inference, evidence retrieval, and planning.

See all 210 papers

Explore similar work

CardsList
Feb 26, 2026

CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era

Scientific research relies on citation integrity, yet large language models (LLMs) have introduced a critical risk: fabricated references that appear plausible but correspond to no real publications. As manual verification becomes infeasible and existing automated tools remain fragile, we introduce CiteAudit, a comprehensive benchmark and detection framework for hallucinated citations. We design a multi-agent verification pipeline that decomposes citation checking into metadata extraction, memory lookup, web-based retrieval, and final judgment. To evaluate this, we construct a large-scale, human-validated dataset spanning diverse domains and hallucination types. Experiments demonstrate that our framework achieves superior verification performance over state-of-the-art LLMs and commercial baselines. Our work provides the necessary infrastructure to audit citations at scale and safeguard the trustworthiness of scholarly discourse. Code is available at https://github.com/shiiiikw/CiteAudit.

Kaiwen Shi, Weixiang Sun, Zheyuan Zhang +3

University of Notre Dame · Lehigh University

Citation VerificationLLM Hallucination DetectionLLM Auditing
AskPDF
May 26, 2026

CiteCheck: Retrieval-Grounded Detection of LLM Citation Hallucinations in Scientific Text

Large language models (LLMs) are increasingly used to generate scientific reports, but they can produce references that appear plausible while containing corrupted metadata or pointing to papers that do not exist. We introduce CiteCheck, a hybrid framework for citation hallucination detection that verifies whether a citation corresponds to a real scholarly work and whether its metadata is faithful to that work. CiteCheck retrieves candidate publications from external scholarly sources, compares the citation against the retrieved candidate using a structured LLM verifier, and maps verifier scores into three labels: Exact, Minor, and Major. We also construct a 982-citation physics benchmark with controlled corruptions that capture both subtle metadata drift and fully fabricated references. On the held-out test set, CiteCheck achieves 88.7 macro-F1 and 88.9% accuracy, outperforming GPT, Claude, and Gemini baselines, including web-search and few-shot variants. These results show that reliable citation verification benefits from combining scholarly retrieval, structured LLM-based comparison, and calibrated decision rules.

Khashayar Khajavi, Shaghayegh Sadeghi, Rise Adhikari +1

FirstPrinciples · School of Computing Science, Simon Fraser University

Citation VerificationAutomated Fact-CheckingLLM Hallucination Detection
AskPDF
May 9, 2026

Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection

Large language models are increasingly used in scientific writing, yet they can fabricate citation-shaped references that appear plausible but fail bibliographic verification. Existing detectors often reduce verification to binary found/not-found decisions and rely on brittle parsing or incomplete retrieval, offering little field-level signal to auditors. We reframe citation hallucination detection as taxonomy-aligned field-level adjudication and introduce a 12-code taxonomy spanning Real, Potential, and Hallucinated citations. Based on this taxonomy, we build CiteTracer, a cascading multi-agent detector that extracts structured citations from PDF and BibTeX, retrieves evidence through cache lookup, URL fetch, scholar connectors, and web search, applies deterministic field matching, and routes ambiguous cases to class-specialist judgers. We release a benchmark of 2,450 synthetic citations built from real seeds with controlled LLM mutations, paired with 957 real-world fabricated citations drawn from desk-rejected submissions at ICLR 2026 and an anonymous conference. CiteTracer reaches 97.1% accuracy on the synthetic benchmark, with class-level F1 scores of 97.0, 95.8, and 98.5 for Real, Potential, and Hallucinated, respectively, and detects 97.1% of fabrications on the real-world set without abstaining. Code: https://github.com/aaFrostnova/CiteTracer.

Mingzhe Li, Zhiqiang Lin, Shiqing Ma

University of Massachusetts Amherst · The Ohio State University

Hallucination in Language ModelsCitation VerificationLLM Hallucination DetectionCitation Hallucination
AskPDF