NLPCC 2026 Task 10: Citation-Level Faithfulness Verification with DeBERTa Ensembles and Class-Wise Calibration

Yanling Li; Zirui Li; Mingyu Wan

cs.CLSep 19, 2026

NLPCC 2026 Task 10: Citation-Level Faithfulness Verification with DeBERTa Ensembles and Class-Wise Calibration

Authors: Yanling Li, Zirui Li, Mingyu Wan

Organizations: The Hong Kong Polytechnic University, Hong Kong SAR, China

Abstract

This paper presents our system for Track 2 of the NLPCC 2026 Shared Task 10 on citation-level faithfulness in AI-assisted scientific reporting. Given an atomic scientific claim and the structured full text of its cited paper, the task requires both a four-way relation label and up to three evidence paragraph identifiers. The label head ensembles a paragraph-aware cross-encoder with a document-level DeBERTa-large classifier, followed by class-wise decision calibration. Probability-level fusion is motivated by an out-of-fold tendency to over-predict Topical Match. The evidence head combines paragraph scores from top-20 and top-30 joint models with BM25 scores. The system runs fully offline without external retrieval or LLM prompting. On the final leaderboard, our system achieved 82.9898 overall (89.5491 Macro-F1 and 76.4305 Joint@3), ranking second in Track 2. Ablations and error analysis show that model complementarity and calibration drive the label gains. Gold-evidence inference changes label Macro-F1 negligibly, whereas evidence ranking remains important for Joint@3.

Figures & tables

Label	Operational meaning
Supported	The cited paper directly supports the claim.
Overstate	The claim extends or strengthens what the paper establishes.
Topical Match	The paper is topically related but does not provide the claimed evidence.
Irrelevant	The paper does not substantively concern the claim.

Table 1: Track 2 citation-faithfulness labels.

Figure 1: FULL2 offline inference pipeline. The top-20 and top-30 joint systems are separately trained five-fold ensembles; the label-only large system is another five-fold ensemble.

Component	Configuration
Joint models	microsoft/deberta-v3-base ; $K=20/30$ ; max length 384; lr $2\times 10^{-5}$ ; 1 epoch; batch 1; accumulation 4
Large model	microsoft/deberta-v3-large ; top-10; max length 512; lr $1\times 10^{-5}$ ; 2 epochs; batch 1; accumulation 8
Optimization	AdamW; weight decay 0.01; linear schedule; 6% warmup; gradient clipping 1.0; seed 42
Cross-validation	Two fixed label-stratified instance-level files: top-20/large 434/432/433/432/431; top-30 433/433/432/432/432; seed 42; neither grouped by cited paper
Hardware	Single GPU (Google Colab A100 or T4, depending on availability)
Inference	Fully offline; local checkpoints and local BM25 only

Table 2: Main training and inference configuration.

Configuration	Backbone	OOF Macro-F1
Context injected, top-8	DeBERTa-v3-base	83.91
Pure BM25, top-10	DeBERTa-v3-base	87.95
Pure BM25, top-10	DeBERTa-v3-large	89.40

Table 3: Input and model-scale ablations on five-fold OOF predictions.

Variant	Phase 1 score
L30, no calibration	79.39
Mild calibration	80.03
Topical-only calibration	80.32
Full calibration	80.78
FULL2: $[1.4,0.9,0.65,1.0]$	81.28

Table 4: Effect of class-wise calibration on the Phase 1 Track 2 score.

BM25 weight $\alpha$	OOF Joint@3
0.00	89.13
0.03	89.27
0.05	89.22
0.10	89.08
0.20	88.90

Table 5: BM25 evidence weight on OOF predictions.

	BM25 candidate retrieval				Top-30 OOF prediction
Group	H@20	H@30	R@20	R@30	E-H@3	J3
Overall	98.15	99.10	91.56	95.82	85.62	81.96
Supported	97.67	98.74	90.96	95.55	78.91	75.12
Overstate	98.40	99.47	89.43	94.64	92.00	88.00
Topical Match	99.20	99.73	95.35	97.74	97.61	85.64
Irrelevant	–	–	–	–	–	90.84

Table 6: Retrieval ceiling and label-conditioned top-30 OOF evidence performance.

Evaluation	Macro-F1	Joint@3	Track 2 score
Phase 1	86.9902	75.5603	81.2753
Phase 2	92.1080	77.3006	84.7043
Final	89.5491	76.4305	82.9898

Table 7: Official Track 2 results for our system. Final values average the two phases.

Gold	Sup.	Over.	Top.	Irr.
Supported	968	11	19	31
Overstate	6	352	12	5
Topical Match	12	21	326	17
Irrelevant	10	14	32	326

Table 8: Raw DeBERTa-large OOF confusion matrix (rows: gold; columns: prediction).

ID and decision	Claim cue and diagnosis
001517 / Topical $\rightarrow$ Supported / (corrected)	The claim accurately describes NLF’s two-stage transformer. Gold P4 explicitly states that one stage aggregates along epipolar lines and the next along reference views. Calibration recovers direct support.
000099 / Topical $\rightarrow$ Overstate / (corrected)	The source reports selected simulation and real-robot tasks, whereas the claim generalizes to diverse real-world domains and uses the cue “consistently”; the gold label is Overstate.
000211 / Supported $\rightarrow$ Supported / (still wrong)	The claim transfers BYOL’s online/target-network mechanism to a modality-specific masked student-teacher system. Strong local lexical overlap makes the prediction highly confident, so a global coefficient cannot repair the compositional mismatch.
001105 / Supported $\rightarrow$ Supported / (still wrong)	The claim redefines software parameter-efficient adapters as hardware deployment modules. Terminology and cited adapter names create a misleading topical signal; detecting the category error requires finer semantic comparison.

Table 9: Representative OOF decisions and diagnoses.

Citation Verification

Citation verification checks whether cited sources exist, are accurately represented, and actually support the claims attached to them.

See all 49 papers

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation
May 26, 2026Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari +1Selective Prediction Citation Verification
FirstPrinciples · School of Computing Science, Simon Fraser University
Abstention vs. Hallucination: Benchmarking LLM Source Attribution for Scientific Citations
May 3, 2024Deepa Tilwani, Yash Saxena, Seyedali Mohammadi +5Citation Verification RAG Evaluation
Artificial Intelligence Institute, University of South Carolina, Columbia, South Carolina, USA · University of Maryland, Baltimore County, Baltimore, Maryland, USA · IAIRO, India +1
HNR-DAC: Hard-Negative Reranking and Distribution-Aligned Classification for Scientific Claim Verification
Aug 7, 2026Zhenchao Wang, Xin Chen, Luoxi Zhang +2Claim Verification Distribution Shift
Southern University of Science and Technology, Shenzhen, China · Institute of Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

May 26, 2026J/K move · Enter open · S save

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation

Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari, Alexander Tessier

FirstPrinciples · School of Computing Science, Simon Fraser University

Selective Prediction Citation Verification Evidence Retrieval Scientific Claim Verification

Open paper AskPDF arXiv

Misalignment between claims and their cited evidence is a common failure mode in reports generated by large language models, limiting their reliability in scientific and other high-stakes settings. We present DeepSciVerify, a two-stage pipeline for scientific claim-citation verification that combines abstract-level reasoning with selective escalation to passage-level evidence. The system first verifies claims using the abstract and defers uncertain cases, retrieving and analyzing full-text passages only when necessary. This design leverages complementary behaviors across LLMs, as some models are more conservative while others are more decisive under uncertainty. On the SCitance benchmark, DeepSciVerify achieves 86.7 Micro-F1, outperforming strong abstract-only baselines by +4.5 points while resolving 67% of instances without full-text retrieval. These results suggest that selective evidence escalation improves both accuracy and efficiency in claim-citation verification.

Abstract

Citation Verification

Explore similar work

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation

Abstention vs. Hallucination: Benchmarking LLM Source Attribution for Scientific Citations

HNR-DAC: Hard-Negative Reranking and Distribution-Aligned Classification for Scientific Claim Verification