ScienceJudge: Pattern Perturbation Augmentation for Sentence-Level Scientific Claim Verification
Organizations: Institute of Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
Abstract
Sentence-level scientific claim verification determines whether each sentence in an AI-generated scientific paragraph is supported by a given multimodal evidence bundle. Fine-grained reporting errors often preserve fluent and plausible statements while introducing subtle evidence-claim deviations, making them difficult to detect under sparse and imbalanced supervision. We propose Pattern Perturbation Augmentation (PPA), which models reporting errors as label-specific faithfulness breaking transformations and induces reusable error patterns from labeled examples. PPA applies these patterns to supported claims to construct controlled hard negatives and expands their coverage through pattern-level, operation-level, and cross-domain augmentation. Based on these augmented views, we develop ScienceJudge, which integrates PPA-trained vision-language predictors and instruction-guided LLM predictors through class-weighted ensemble inference. On the local development set, ScienceJudge improves the average score from 0.8124 to 0.8692, achieving a Macro-F1 of 0.8466 and a Paragraph Exact Match score of 0.8919. Our submitted system ranks first on the official NLPCC 2026 Task 10 Track 1 leaderboard with a final score of 0.6890.
Figures & tables
| Data setting | Sup. | Entity | Scope | Contra. | Causal | Para. |
|---|---|---|---|---|---|---|
| Base | 14,818 | 581 | 543 | 193 | 251 | 3,000 |
| Pattern PPA | 4,950 | 540 | 510 | 300 | 290 | 800 |
| Operation PPA | 2,113 | 200 | 100 | 100 | 100 | 500 |
| Operation PPA(scaled) | 10,849 | 1,000 | 500 | 500 | 500 | 2,500 |
| Domain PPA | 3,070 | 180 | 180 | 110 | 170 | 800 |
| Predictor Metric | Label-wise F1 | Overall | ||||||
|---|---|---|---|---|---|---|---|---|
| Sup. | Entity | Scope | Contra. | Causal | Macro-F1 | PEM | Avg. | |
| Text-only Surrogate | ||||||||
| Qwen3-8B-Instruct | 0.7963 | 0.1754 | 0.0339 | 0.0696 | 0.1096 | 0.2370 | 0.2523 | 0.2446 |
| + Base Training | 0.9784 | 0.7586 | 0.7833 | 0.5366 | 0.7692 | 0.7652 | 0.8078 | 0.7865 |
| Our PPA-Based Individual Model | ||||||||
| Qwen3-VL-8B-Instruct | 0.8098 | 0.2872 | 0.0000 | 0.1154 | 0.0909 | 0.2606 | 0.2643 | 0.2625 |
| Rank | System | Phase 1 | Phase 2 | Avg. | ||||
|---|---|---|---|---|---|---|---|---|
| Macro-F1 | PEM | Avg. | Macro-F1 | PEM | Avg. | |||
| 1 | ScienceJudge (Ours) | 0.5659 | 0.4710 | 0.5185 | 0.9053 | 0.8136 | 0.8595 | 0.6890 |
| 2 | System 1 | 0.5754 | 0.4369 | 0.5061 | 0.9056 | 0.8337 | 0.8697 | 0.6879 |
| 3 | System 2 | 0.4996 | 0.3823 | 0.4409 | 0.7523 | 0.6854 | 0.7188 | 0.5799 |
Claim Verification
Claim verification assesses whether statements are supported, contradicted, or unresolved by relevant evidence.
See all 61 papers