Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation

Nazanin Jafari; James Allan; Mohit Iyyer

cs.CLApr 3, 2026

Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation

Authors: Nazanin Jafari, James Allan, Mohit Iyyer

Organizations: UMass Amherst · University of Maryland

Abstract

Evaluating the factuality of long-form output generated by large language models (LLMs) remains challenging, particularly when responses are open-ended and contain many fine-grained factual statements. Existing evaluation methods primarily focus on precision: they decompose a response into atomic claims and verify each claim against external knowledge sources such as Wikipedia. However, this overlooks an equally important dimension of factuality: recall, whether the generated response covers the relevant facts that should be included. We propose a comprehensive factuality evaluation framework that jointly measures precision and recall. Our method leverages external knowledge sources to construct reference facts and determine whether they are captured in generated text. We further introduce an importance-aware weighting scheme based on relevance and salience. Our analysis reveals that current LLMs perform substantially better on precision than on recall, suggesting that factual incompleteness remains a major limitation of long-form generation and that models are generally better at covering highly important facts than the full set of relevant facts.

Figures & tables

LLMs	Bio			LongFact			LongForm
	$Prec$	$Rec$	$F1$	$Prec$	$Rec$	$F1$	$Prec$	$Rec$	$F1$
Llama-3.1-8B-Inst	8.0	5.0	4.0	35.9	30.3	28.2	22.7	5.3	5.3
Llama-3.1-70B-Inst	14.0	9.9	8.3	34.4	31.8	27.4	27.7	8.5	8.9
Qwen2.5-7B-Instruct	6.3	3.0	3.0	36.5	34.7	30.7	20.1	4.4	4.2
Mistral-7B-instruct-v0.3	9.9	4.8	4.8	35.5	37.9	32.2	20.7	6.2	6.0
GPT4o-mini	13.3	9.1	8.2	38.6	42.6	36.8	33.4	7.2	8.6

Table 1: Factuality of LLMs on three data domains. Recall is based on unweighted factual statements

LLMs	Bio (11.4)			LongFact (7.4)			LongForm (10.2)
	$Prec$	$Rec$	$\rho$	$Prec$	$Rec$	$\rho$	$Prec$	$Rec$	$\rho$
Llama-3.1-8B-Inst	8.0	5.0	3.2	35.9	30.3	4.1	22.7	5.3	1.8
Llama-3.1-70B-Inst	14.0	9.9	3.3	34.4	31.8	4.7	27.7	8.5	2.1
Qwen2.5-7B-Instruct	6.3	3.0	2.8	36.5	34.7	5.3	20.1	4.4	2.0
Mistral-7B-instruct-v0.3	9.9	4.8	2.5	35.5	37.9	4.6	20.7	6.2	1.9
GPT4o-mini	13.3	9.1	2.5	38.6	42.6	5.2	33.4	7.2	1.4

Table 2: Tradeoff between precision and recall based on generated claims over factual statements. Average number of facts is provided in parentheses for each dataset.

\rho

measures the ratio of number of generated claims over number of fact sets in each dataset.

Figure 1: Percentage of claims labeled as supported, not supported, or contradicted across models and datasets.

Figure 2: Recall comparison for fact reference sets formed using combined importance scoring ( $\alpha=\beta=1$ ) versus relevance-only( $\alpha=1,\beta=0$ ) and salience-only scoring ( $\alpha=0,\beta=1$ ). The first column in each panel reports recall for the combined score ( Co ), while the next two columns show differences relative to relevance-only ( $\Delta$ (Co-Sal) ) and salience-only ( $\Delta$ (Co-Rel) ) rankings.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Instructions for Fact generation You need to extract as many facts from the given “context” as possible. Output each fact in bullet-point format. Each of these facts should be generated directly from the “context”, should be objective and factual, so avoid opinion-based sentences. Each fact should add new information and avoid redundancy. Each fact should be self-contained and make sense on its own. Choose facts that provide new insights or something unusual about the topic. Avoid general statements that apply to many things. A good fact should be specific and unique and contribute to the essential understanding of the topic. Here is an example: Context: Adam Jared Brody (born December 15, 1979) is an American actor. His breakout role was as Seth Cohen on the Fox television series The O.C. (2003–2007). For his performance as Noah in the Netflix romantic comedy series Nobody Wants This (2024), he earned a nomination for the Golden Globe Award for Best Actor in a Television Series (Musical/Comedy) and won the Critics’ Choice Television Award for Best Actor in a Comedy Series. Output: Facts: • Adam Jared Brody was born on December 15, 1979. • Adam Brody’s breakout role was as Seth Cohen on the Fox television series The O.C. (2003–2007). • He earned a nomination for the Golden Globe Award for Best Actor in a Television Series (Musical/Comedy) for his performance as Noah in the Netflix romantic comedy series Nobody Wants This (2024). • Brody won the Critics’ Choice Television Award for Best Actor in a Comedy Series for his role in Nobody Wants This.

Appendix

Table 3: Prompt used for fact generation.

Instructions for fact coverage verification You are checking whether a given fact is COVERED by a set of claim sentences (an answer). Important: • The fact is assumed to be TRUE. Do not question or evaluate its truth. • Decide whether the fact is stated or clearly implied by the claim sentences. Definitions: • The fact is COVERED if the claim sentences clearly state or entail the fact. That means: if a careful reader had only these claim sentences and nothing else, they would be confident that the fact is true. • The fact is NOT_COVERED if the claim sentences do not provide enough information to guarantee the fact. It is NOT_COVERED even if the fact seems plausible based on outside knowledge. Notes: • You may use multiple claim sentences together to decide if the fact is covered. • Do not use any outside knowledge beyond the claim sentences. • Be strict: if the claim sentences are compatible with the fact but do not actually say or entail it, label it NOT_COVERED . Fact: {fact} Claim sentences (numbered, 1-based indices): {claims_block} Your tasks: • Decide whether the fact is COVERED or NOT_COVERED by the claim sentences above. • If and only if the fact is COVERED , list all claim IDs (1-based indices) that directly help cover/entail the fact. If NOT_COVERED , use an empty list. Output strictly as JSON, with no extra text: { "label": "COVERED" | "NOT_COVERED", "evidence_claim_ids": [<int>, ...] }

Appendix

Table 4: Prompt used for fact coverage verification.

Instructions for relevance and salience scoring You are scoring each statement based on its relevance and salience to the given query. Task:

Given a query and a list of sentences about that query, assign each sentence:

- a RELEVANCE rating from 1 to 5

- a SALIENCE rating from 1 to 5 Definitions:

- Relevance (1–5): how directly this sentence helps answer the query.

1 = completely unrelated

2 = weakly related

Appendix

Table 5: Prompt used for relevance and salience scoring.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration
Apr 13, 2026Xin Liu, Lu WangLLM Hallucination Mitigation Language Model Calibration
Computer Science and Engineering, University of Michigan, Ann Arbor, MI, USA
Only Say What You Know: Calibration-Aware Generation for Long-Form Factuality
May 3, 2026Wen Luo, Guangyue Peng, Liang Wang +7Long-Form Document Generation LLM Hallucination Mitigation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University Research Asia · MSRA · Microsoft
Not All Claims Are Equally Risky: FACTOR for Adaptive Verification in Factual Long-Form Generation
Jun 21, 2026Areeba Hassan, Arooj Kausar, Syeda Kisaa Fatima +2LLM Answer Verification Long-Form Document Generation
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan

Apr 13, 2026J/K move · Enter open · S save

Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration

Xin Liu, Lu Wang

Computer Science and Engineering, University of Michigan, Ann Arbor, MI, USA

LLM Hallucination Mitigation Language Model Calibration LLM Uncertainty Estimation

Open paper AskPDF arXiv

Large language models (LLMs) often hallucinate in long-form generation. Existing approaches mainly improve factuality through post-hoc revision or reinforcement learning (RL) with correctness-based rewards, but they do not teach the model to estimate which parts of its generation are reliable. As a result, models may still state incorrect claims confidently in their responses. Recent advances in reasoning have significantly improved LLM performance, and have been leveraged to estimate confidence by incorporating calibration into RL objectives. However, existing approaches remain limited to a single scalar confidence for the entire response, which is insufficient for long-form generation where uncertainty varies across individual claims. To mitigate this problem, we propose CURE, a framework that improves long-form factuality by teaching LLMs to reason about uncertainty at the claim level. We first introduce a Claim-Aware Reasoning Protocol, which structures outputs into atomic claims paired with explicit confidence estimates. We then develop a multi-stage training pipeline that aligns model confidence with claims' correctness and then optimizes on factuality. The resulting calibrated confidence further enables selective prediction, allowing the model to abstain from uncertain claims at inference time. Experiments on four long-form factuality benchmarks show that CURE consistently improves factual accuracy over competitive supervised and RL baselines, while maintaining factual recall. In particular, it improves claim-level accuracy by up to 39.9% on Biography generation. These gains are accompanied by improved calibration, as reflected by a 16.0% increase in AUROC on FactBench.

Abstract

LLM Evaluation

Explore similar work

Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration

Only Say What You Know: Calibration-Aware Generation for Long-Form Factuality

Not All Claims Are Equally Risky: FACTOR for Adaptive Verification in Factual Long-Form Generation