Large language models (LLMs) are widely used to answer factual questions, but their answers can contain errors and must be fact-checked claim by claim. Yet the resulting score depends on three design decisions: which questions the claims answer, under what conditions each claim is judged, and which sources count as evidence. No existing benchmark specifies all three. We introduce JSFactCheckBench, a Japanese benchmark that specifies all three, with 1,913 claims from LLM answers to questions about misleading information that circulated on Japanese social media, each annotated for factuality at a reference time and for importance to the question, using evidence from preselected Web sources. Across six LLMs, all verify claims directly answering the question less accurately than supplementary claims, a gap that aggregate scores obscure. Scores also drop when models must locate the relevant content in longer or multiple sources and, for all four open models, when the surrounding question and response are added. These weaknesses are visible only because the benchmark records the conditions behind each label. A benchmark that leaves them implicit cannot reveal them.
Figures & tables
(1) Claim collection
(2) Judgment conditions
(3) Evidence sources
Benchmark
Lang.
LLM responses
Real-world origin
Reference time
Human importance
Open web
Source control
FEVER ( 2018 )
En
–
–
–
–
–
✓
TruthfulQA ( 2021 )
En
△
–
–
–
✓
–
CHEF ( 2022 )
Zh
–
✓
△
–
✓
△
FactDrill ( 2022 )
Multi
–
✓
△
–
✓
–
FActScore ( 2023 )
En
✓
–
–
–
–
✓
Table 1: Comparison of construction. ✓: satisfied; △ : partially satisfied; – : not satisfied. LLM responses : factuality or hallucination annotations of LLM-generated responses at claim, segment, or span level ( △ : answer-level labels or perturbation conditions only). Real-world origin : provenance of the source questions or claims ( △ : mixed provenance). VeriTaS and JAD-AFC include truth-altering claim variants. Reference time : a recorded reference time explicitly anchoring gold factuality judgments ( △ : temporal metadata or constraints only). Human importance : manually annotated claim importance that is incorporated into the benchmark’s evaluation ( △ : annotated but not used in evaluation, or automatically assigned). Open web : Web-derived evidence or reference sources not restricted to a fixed reference corpus. Source control : predefined restrictions on eligible evidence sources (✓: fixed reference sources or an allowlist; △ : blocklist-based restrictions). AVeriTeC excludes only the originating fact-checking article and merely flags unreliable sites.
Figure 1: Overview of the dataset construction pipeline.
Question 1
Is it free to call an ambulance in Japan?
Response
Calling an ambulance in Japan normally incurs a fee. However, ⋯ to request an ambulance, call 119. ⋯
Claim
Calling an ambulance in Japan normally incurs a fee. ✗ Refuted Main Source ID: d61417498a , Time: 2026-02-15 Evidence span: In Japan, ambulance users currently bear no charge and can use the service free of cost. ⋯
Question 2
Will the ¥1.1 million gift tax exemption be abolished starting in 2024?
Response
No decision to abolish the annual gift tax exclusion of ¥1.1 million in 2024 has been announced. The tax-free threshold for gift tax is set under Japanese tax legislation. ⋯
Claim
No decision to abolish the ¥1.1 million gift tax exemption in 2024 has been announced. ✓ Supported Main Source ID: b3655be520 , Time: 2026-02-15 Evidence span: ⋯ it was decided that the annual gift tax system would remain in place. ⋯
Table 2: Example instances from JSFactCheckBench. The Japanese text is translated into English.
Main
Sub
Irrel.
All
Supported
250
625
54
929 (48.6%)
Refuted
70
172
8
250 (13.1%)
NotEnoughInfo
140
345
74
559 (29.2%)
Unevaluable
23
108
44
175 0 (9.1%)
All
483
1,250
180
1,913
(25.2%)
(65.3%)
(9.4%)
Table 3: Label distribution of factuality and importance. Percentages are relative to all 1,913 claims; the 20 claims labeled Unclear are excluded from the dataset.
Topic domain
N
Topic domain
N
Medicine and health
24
Science and technology
4
Nuclear energy
12
Justice and law
4
Economy and industry
12
Society and gender
4
Culture and history
8
Politics
3
Government services
7
International affairs
2
Media and broadcasting
5
Table 4: Distribution of question domains.
Geographic scope
N
%
Japan
40
47.1
Universal/non-geographic
24
28.2
Foreign country
14
16.5
International/global
7
8.2
Table 5: Distribution of geographic scope.
Input given to the model
Macro-F1
Claim
Time
Q & R
Evidence
Llama-3.1-Sw.
Qwen3-Sw.
LLM-jp-4
Mistral-Nemo-Ja
GPT-5.1
Sonnet 5
(a) Closed book: no evidence supplied
✓
✗
✗
None
0.444
0.463
0.485
0.490
0.549
0.600
✓
✓
✗
None
0.418
0.437
0.474
0.489
0.557
0.593
✓
✓
✓
None
0.265
0.327
0.307
0.256
0.544
0.586
(b) Source given: the cited page is supplied
Table 6: Factuality classification results. See Section 3.2 for the input settings. Best score per row in bold.
Model
Acc (all)
Accuracy by importance
Main
Sub
Irrel.
Llama-3.1-Sw.
0.648
0.563
0.687
0.615
Qwen3-Sw.
0.741
0.697
0.754
0.785
LLM-jp-4
0.741
0.710
0.746
0.807
Mistral-Nemo-Ja
0.699
0.648
0.725
0.644
GPT-5.1
0.741
0.707
0.747
0.807
Table 7: Accuracy by claim importance (Claim + Evidence Full). Acc (all) is accuracy over all claims; the importance columns are accuracy within each group. We report accuracy rather than the macro-F1 of the main results because the Irrelevant group contains only 8 Refuted claims, too few for a macro-averaged score.
Figure 2: Distribution of models used for response generation and the number of responses in the dataset.
Figure 3: Prompt for search-query generation. The Japanese original is translated into English; slashes mark the spaces separating keywords.
Label
Definition
Decision criterion and examples (Q: What is the capital of Japan? )
Main
A claim that directly answers the question, or that states the core or a part of the answer. It need not be a complete or correct answer, only a direct response to the question.
Shown on its own, the claim stands as an answer to the question, whether full or partial. It is Tokyo. / I do not know.
Sub
A claim that supplies information related to the subject of the question or to the answer, but does not itself contain the specific value or name being asked for.
On its own it does not answer the question; it mentions the topic of the question but not the value asked for. It became the capital in 1869.
Irrelevant
A claim with no direct relation to the question or to the content of the answer. It contributes neither to the correctness nor to the understanding of the answer.
The claim is unrelated to the topic or purpose of the question, and fits neither Main nor Sub. Osaka is famous for its food culture.
Appendix
Table 9: Criteria for the importance axis. Examples are translated from the Japanese guidelines.
Subtype
Definition
Example
Relative evaluation only
Uses relative terms such as many , few , sufficient , or excessive without stating the standard or the object of comparison, so that the assessment depends on the reader’s values.
The Japanese government allocates a large budget to scholarship programs for international students. With a figure supplied ( a budget of JPY 22.2 billion ), the claim is verifiable and is judged on the factuality axis.
Hypothetical conditional
States a prospect or a generality conditional on an event, without asserting that the event occurred.
If such an incident really did occur, the truth will come to light.
Metalinguistic statement
States the speaker’s own position, capability, constraints, or willingness to respond.
I cannot provide detailed information about specific incidents or individuals.
Value or normative statement
Asserts what should be done or what is desirable, important, or recommended.
The government should be more transparent.
Appendix
Table 10: Subtypes of the Unevaluable label. Examples are translated from the Japanese guidelines.
JPY
USD
Total cost
2,302,300
14,632
Cost per annotation
393
2.50
Cost per claim
1,180
7.50
Annotators
3
Claims
1,952
Individual annotations
5,856
Appendix
Table 11: Cost and effort of annotation. Cost per claim covers the three independent annotations of that claim. USD amounts use the rate of JPY 157 per USD.
Domain category
Candidates
Selected
Core
7,931
539
go.jp
2,408
205
co.jp
2,758
172
or.jp
1,160
75
lg.jp
949
54
ac.jp
656
33
Appendix
Table 12: Domain categories of candidate and selected evidence set. Numbers indicate the counts of unique links.
Figure 4: Number of claims per response.
Question stance
N
Temporal stability
N
Neutral
71
Static
46
Reported
9
Long-term dynamic
18
Leading
5
Short-term dynamic
21
Appendix
Table 13: Distribution of question stance and temporal stability.
Code
Name
Definition and inclusion/exclusion criteria
D1
Medicine and health
Diseases, treatments, pharmaceuticals, vaccines, nutrition and food components, and physiology. Excludes the management or corporate history of pharmaceutical companies (D8) and the operation of health-care systems (D4).
D2
Science and technology
Natural phenomena such as earthquakes, weather, and astronomy, and scientific, technical, or infrastructure topics such as chemistry, biology, communications, and energy facilities. Includes events, policies, and facilities in these fields, not only scientific propositions. Nuclear energy and radiation take D3 priority; questions centered on health effects take D1.
D3
Nuclear energy
Nuclear power plants, radioactive materials, treated water, and nuclear fuel. If human health effects are central, assign D1; if materials, facilities, or regulation are central, assign D3.
D4
Government services
Taxation, social security, benefits, disaster administration, government organizations, and public expenditure. Excludes legal interpretation itself (D5) and party politics (D6).
D5
Justice and law
Courts, police, and other justice or law-enforcement institutions.
D6
Politics
Political parties, politicians, elections, and policy disputes. Excludes the content of public systems themselves (D4).
Appendix
Table 14: Annotation scheme for Axis 1: Question domains.
Code
Name
Definition and inclusion/exclusion criteria
G1
Japan
Institutions, events, or people in Japan are the referent.
G2
Foreign country
Domestic affairs within a specific foreign country are the referent.
G3
International or global
The claim has geographic referents but is not restricted to one country, including international organizations, cross-border events, global distributions, and global companies or platforms.
G0
Universal or non-geographic
The proposition has no geographic referent, as in natural laws, general biomedical facts, or normative propositions. Universal validity alone does not imply G3; the distinction depends on whether the proposition refers to a particular place or institution.
Appendix
Table 15: Annotation scheme for Axis 2: Geographic scope.
Code
Name
Definition and inclusion/exclusion criteria
Q1
Neutral
Presents a proposition and asks whether it is true without marking the speaker’s prior commitment, as in ordinary yes/no questions.
Q2
Reported
Marks the proposition as information attributed to another source, using expressions such as “apparently,” “is it true that,” “I heard that,” or “there is a story that.”
Q3
Leading
Presupposes the proposition and seeks agreement, using forms equivalent to “isn’t it?,” “right?,” or “wouldn’t you say?”
Appendix
Table 16: Annotation scheme for Axis 3: Question stance.
Code
Name
Definition and inclusion/exclusion criteria
T1
Static
The conclusion does not change in any substantial way, as with past events, established findings, definitions, and norms.
T2
Long-term dynamic
The conclusion may change over a period of more than a year to a decade or so, as with laws and institutions, official standards, and scientific findings that are still accumulating.
T3
Short-term dynamic
The conclusion may change within days to a year, as with current conditions, ongoing cases, private-sector specifications, and future plans.
Appendix
Table 17: Annotation scheme for Axis 4: Temporal stability.
Figure 5: Prompt template for the Claim only setting (no evidence). Angle brackets mark the slots filled per claim.
Figure 6: Prompt template for the Claim + Time + Q&R + Reference setting. Angle brackets mark the slots filled per claim.
Figure 7: Prompt template for the Claim + Time + Q&R + Pages setting. Angle brackets mark the slots filled per claim.
Evaluating the factual correctness of large language models (LLMs) is vital for many applications. But are our evaluation tools themselves trustworthy? Despite the rise of factuality-based metrics, their sensitivity and reliability remain underexplored. This paper introduces a meta-evaluation framework that systematically tests these metrics using controlled corruptions of gold standard answers. Our method generates ranked outputs with known degrees of degradation to probe how metrics capture nuanced changes in truthfulness. Our experiments reveal that pipeline-based methods, such as the RAGAS's factual correctness metric, better track degradation than LLM-as-judge approaches. We also propose a new variant of the factual correctness metric that provides a competitive and cost-efficient.
As automated fact-checking scales on social media, large language model (LLM) verdict scores can look stronger than warranted. One reason is that evaluations mix in information that was not knowable at claim time. Two channels are easy to conflate: outcomes memorized in pre-training and retrieved evidence published after the claim. Yet standard benchmarks rarely separate the two. In this study we measure both channels on AVeriTeC and QuanTemp++ by reconstructing point-in-time evidence conditions and probing for outcome information encoded in model representations. We find substantial evidence of parametric leakage, that can be hidden by the aggregate accuracy, while a simple representation bottleneck reduces this future leakage more efficiently than a mutual-information-based training penalty. We also find that allowing post-claim evidence inflates zero-shot accuracy by 6.3 points in AVeriTeC while the effect is negligible in QuanTemp++, where retrieval provides little post-claim evidence. These results show that misinformation benchmarks can overstate fact-checking performance when they do not account for what information was actually available at claim time.
Kuan-Hua Wu Lu, Yohanes Andre Setiawan
Georgia Institute of Technology Atlanta, Georgia, USA
Grounded claim factuality checking is important for large language model (LLM) applications such as retrieval-augmented generation, as it helps users assess the correctness of generated outputs. Existing metrics using entailment classifiers require dataset-specific threshold tuning, while LLM-based approaches often use direct prompting, which underutilises the reasoning capabilities of LLMs. We address this by formulating grounded claim factuality checking as a true/false reading comprehension task and prompting LLMs with explicit test-taking strategies for efficient reasoning. Our method reduces token usage by over 80% compared to unguided open-ended reasoning, and achieves competitive performance to more expensive alternatives across two factuality benchmarks, setting a new state of the art on one. To further reduce inference cost, we train small language models (SLMs) to replace LLMs in the checking pipeline. Using supervised fine-tuning (SFT) and a self-revision mechanism, the SLMs learn to improve their factuality judgements. Experimental results show that the resulting SLMs perform on par with strong baselines, combining low inference costs with generating supporting rationales to support interpretability. Code and datasets will be released upon acceptance.
Yuxuan Ye, Raul Santos-Rodriguez, Edwin Simpson
Intelligent Systems Laboratory University of Bristol United Kingdom