Overview of the NTCIR-19 SciClaimEval Task: Cross-modal Claim Verification in Scientific Papers
Authors: Xanh Ho, André Greiner-Petter, Florian Boudin, Yun-Ang Wu, Sunisth Kumar, Tian Cheng Xia, Hiba Akbi, Tim Schopf, Akiko Aizawa
Organizations: University of Göttingen, Germany · Inria, LS2N, Nantes Université, France · NII LLMC, Japan · The University of Tokyo, Japan · University of Bologna, Italy · Sorbonne University, France · NII, Japan
This paper presents an overview of SciClaimEval, a pilot task at NTCIR-19 on verifying scientific claims against the tables and figures of the papers in which they appear. Our dataset pairs authentic claims from 180 papers in machine learning, natural language processing, and medicine with contrastive versions of their tables and figures, one supporting the claim and one minimally modified so that it does not. The task comprises two subtasks: deciding whether a piece of evidence supports a claim, and selecting which of two evidence versions does. Twenty-two teams registered, the highest number among NTCIR-19 tasks, and 14 submitted 177 runs. Most systems built on multimodal large language models, relying on prompting and multi-stage inference rather than task-specific fine-tuning. On the first subtask, the best systems reached 98.1% pair accuracy, but all six top-ranked systems relied on the contrastive pairs; the best system that judged each claim-evidence pair on its own scored 24.3 points lower. Open-weight models, embedded in carefully designed pipelines, matched proprietary ones. Yet verifying a single claim against its evidence, the setting that matters in practice, remains an open problem.
Figures & tables
Figure 1. Our dataset construction pipeline consists of three main steps: data preparation, automatic claim-evidence extraction, and human annotation. The human annotation process involves two tasks: claim-evidence verification and evidence modification. After collecting all samples, we design two subtasks in our dataset: claim-label prediction and claim-evidence prediction.
Figure 2. An example from our dataset involves creating an unsupported claim by adding spurious data points to the figure. The context provides the necessary information to understand the plot in the evidence.
Property
Val
Test
All Data
Labels
#Supported
389
474
863
#Refuted
338
432
770
Modality Type
Table
469
518
987
Figure
258
388
646
Table 1. Dataset statistics of SciClaimEval.
Figure 3. Distribution of all figure types in test and dev split.
Team ID
Group name
Country
Subtasks
# Runs
SIB
SIB Text Mining
Switzerland
1, 2
8
bit-ias9
Bonn-Juelich Informatics
Germany
1, 2
44
tabr25
Black Socks
Australia
1, 2
12
ECS-UoS
University of Southampton
UK
1, 2
2
ditlab
Denso IT Laboratory
Japan
1, 2
5
trace29
SEBIS TUM team
Germany
1, 2
2
Table 2. Participating teams, the subtasks they submitted runs to, and their number of runs.
Team
P
R
Macro-F1
Acc.
Pair Acc.
SciTrue †
98.0
98.0
98.0
98.0
98.1
Bonn-Juelich Informatics †
97.2
97.3
97.2
97.2
98.1
Black Socks †
94.1
93.7
93.8
93.8
90.0
Pro²Crast †
87.2
87.2
87.2
87.2
86.8
University of Southampton †
85.8
85.9
85.8
85.9
85.9
Denso IT Laboratory †
74.1
74.1
74.1
74.1
75.9
Table 3. Best run of each team on Subtask 1 (PNG evidence), ranked by pair accuracy. † indicates runs that use the paired structure of the test data.
Team
P
R
Macro-F1
Acc.
Pair Acc.
SciTrue
98.1
98.1
98.1
98.1
98.4
Black Socks
93.1
93.2
93.2
93.2
88.7
Pro²Crast
83.6
83.7
83.6
83.7
83.6
TakasuLab
85.0
84.9
84.9
85.0
71.3
SciNex
78.3
78.1
77.8
77.8
59.7
CAISALab
65.7
64.7
64.4
65.2
32.6
Table 4. Subtask 1 results (JSON evidence).
Team
P
R
Macro-F1
Acc.
Pair Acc.
SciTrue
98.1
98.1
98.1
98.1
98.4
Bonn-Juelich Informatics
96.7
96.7
96.7
96.7
97.7
ProvideQ
89.9
90.0
90.0
90.0
82.4
Table 5. Subtask 1 results (TeX/HTML evidence).
Team
Accuracy
SciTrue
98.4
Bonn-Juelich Informatics
98.1
University of Southampton
96.3
TakasuLab
94.0
ProvideQ
91.7
SIB Text Mining
90.5
Table 6. Best run of each team on Subtask 2 results (PNG evidence).
Team
Accuracy
TakasuLab
93.5
SciNex
89.3
Black Socks
84.5
LMU & TUM NLP
83.6
VerifyAI
66.7
Pro²Crast
66.7
Table 7. Subtask 2 results (JSON evidence).
Team
Accuracy
Bonn-Juelich Informatics
97.7
Table 8. Subtask 2 results (TeX/HTML evidence).
Figure 4. Accuracy on Subtask 1 across evidence modification categories. The red and green percentages indicate performance drops and gains, respectively, compared to the overall average (black triangle marker). The brown and turquoise triangle markers indicate the average accuracy for tables and figures, respectively.
Figure 5. Accuracy on Subtask 1 across figure modification categories. The red and green percentages indicate performance drops and gains, respectively, compared to the average accuracy across all figures (black triangle marker).
Figure 6. Accuracy on Subtask 1 across table modification categories. The red and green percentages indicate performance drops and gains, respectively, compared to the average accuracy across all tables (black triangle marker).
Figure 7. Accuracy on Subtask 2 across evidence modification categories. The red and green percentages indicate performance drops and gains, respectively, compared to the overall average accuracy (black triangle marker). The brown and turquoise triangle markers indicate the average accuracy for tables and figures, respectively.
Figure 8. Best-run accuracy on Subtask 1 across figure types. Single refers to a single plot, which may combine multiple chart types (e.g., a bar chart with error bars). Multiple refers to multiple unlabeled plots, Subplots refers to multiple labeled plots (e.g., (a), (b)), and Multiple Subplots refers to a combination of labeled and unlabeled plots. Italicized categories (middle part) represent chart types across all figure structures (Single, Multiple, Subplots, and Multiple Subplots). The bottom category, Combination , refers to figures containing two or more chart types, regardless of their structure.
Figure 9. Label prediction bias across all runs in Subtask 1.
Figure 10. Evidence selection bias across all runs in Subtask 2.
Figure 11. Full evaluation results for each submitted run and claim–evidence sample. Red and gray markers indicate incorrect and correct predictions, respectively. Queries are sorted by consensus from hardest (left, most runs predicted incorrectly) to easiest (right, most runs predicted correctly), with figure samples on the left and table samples on the right. The rightmost gray labels show the average accuracy of each run. At the bottom, P(correct) represents the fraction of runs with correct predictions for each sample, while Agreement indicates the agreement among all runs on the predicted label. Majority correct is marked red if more than half of the runs predicted incorrectly and gray otherwise.
Figure 12. Similarity (Cohen’s κ ) between each team’s best-performing run, with runs sorted by accuracy.
Figure 13. Reported performance gains from exploiting pair information. Not all teams reported these gains, and some reported results only on the development (dev) split. SciTrue reported multiple gains depending on the underlying LLM.