LLMs are widely used, yet they remain prone to factual errors that erode trust and limit adoption in high-risk settings. One approach to mitigate this risk is to equip models with uncertainty estimation mechanisms that abstain when confidence is low. However, this binary "all-or-nothing" approach is excessively restrictive in long-form settings, often discarding valuable information. We introduce Selective Abstraction (SA), a framework that enables LLMs to trade specificity for reliability by selectively reducing the detail of uncertain content. We formalize SA through selective risk and coverage, and propose Atom-wise Selective Abstraction, a claim-level instantiation that decomposes responses into atomic claims and replaces uncertain atoms with higher-confidence, less specific abstractions. We develop a novel end-to-end pipeline for evaluating long-form generation that instantiates risk as factual correctness and measures coverage as retained information. Our comprehensive empirical evaluation across five LLMs and two long-form factual benchmarks shows that atom-wise SA consistently improves the risk-coverage trade-off, outperforming seven baselines, including claim removal, self-revision, sampling-based aggregation, and chain-of-verification. It improves AURC by up to 35.61% over claim removal, demonstrating that reducing specificity boosts reliability while preserving substantial information. We further provide a risk-guided procedure for selecting a confidence threshold that satisfies a target risk.
Figures & tables
Figure 1: Left: Example abstraction sequence for atom-wise selective abstraction (SA). Increasing the confidence threshold replaces low-confidence atoms with less specific, more reliable abstractions. Right: Example risk-coverage curve comparing atom-wise SA to baselines. Model: gpt-oss-120b, dataset: FactScore.
Figure 2: An overview of the Selective Abstraction framework. The generated text is decomposed into atoms, and low-confidence atoms are replaced with confident abstractions. The confidence threshold in this example is 85 % .
Figure 3: Qualitative example of atom-wise Selective Abstraction at two confidence thresholds: a higher threshold reduces detail but improves reliability, while a lower threshold retains detail at higher risk. Red text signifies incorrect claims. Generating model: gpt-oss-120b, dataset: FactScore. See Appendix A for additional examples.
FactScore
LongFact
Model
Redaction
Atom-wise SA
Improvement
Redaction
Atom-wise SA
Improvement
AURC ↓ ( ×100 )
AURC ↓ ( ×100 )
↑ (%)
AURC ↓ ( ×100 )
AURC ↓ ( ×100 )
↑ (%)
gpt-oss-120b
61.30
44.30
27.73
36.04
28.47
21.02
gpt-oss-20b
73.45
61.59
16.14
42.60
36.39
14.59
qwen3-235b-t
48.30
38.62
20.04
27.85
22.89
17.83
qwen3-30b-t
60.98
50.99
16.39
32.24
28.70
10.97
Table 1: AURC comparison between atom-wise selective abstraction (ours) and redaction. For each model, we report AURC (×100) on FactScore and LongFact, and the relative improvement over redaction. Atom-wise SA consistently reduces AURC across all models and both datasets, yielding substantial gains on average.
FactScore Δ Risk (%)
LongFact Δ Risk(%)
Model
Inline
Self-Revision
Inline
Self-Revision
gpt-oss-120b
5.33
15.71
13.90
11.25
gpt-oss-20b
10.13
14.56
12.58
11.21
qwen3-235b-t
20.31
12.10
11.96
14.80
qwen3-30b-t
15.81
16.40
6.63
10.10
qwen3-30b-i
13.05
5.48
7.96
11.65
Table 2: Risk gap at matched coverage between atom-wise SA and the Inline and Self-Revision baselines. For each baseline point, we find the point on the atom-wise SA RC curve with the same coverage, and report the risk difference. Positive values mean lower risk for atom-wise SA. Across both datasets, atom-wise SA consistently achieves substantially lower risk at matched coverage than Inline and Self-Revision.
FactScore
LongFact
Model
Avg #Atoms
Avg #Abs./Atom
Avg #Atoms
Avg #Abs./Atom
gpt-oss-120b
68.8
5.6
73.9
5.3
gpt-oss-20b
57.0
3.9
61.0
3.7
qwen3-235b-t
51.2
5.0
65.4
4.9
qwen3-30b-t
32.9
3.6
63.4
3.4
qwen3-30b-i
43.9
4.2
60.0
4.5
Table 3: Response statistics. For each model and dataset, we report the average number of atomic claims per response before abstraction ( Avg #Atoms ) and the average number of abstraction candidates per atomic claim ( Avg #Abs./Atom ), counting the abstraction levels proposed by the model per claim.
Figure 4: Risk-guided threshold selection on FactScore (left) and LongFact (right), averaged across models (600 runs per dataset). Blue points show mean risk at the selected thresholds, bands show one standard deviation, and red X marks the target risk.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8Figure 9
Figure 5: Risk–coverage curves obtained by instantiating κ with different logprob-based confidence scores on FactScore using gpt-oss-120b. Legend is sorted by AURC in ascending order.
Figure 6: Risk–coverage curves on FactScore with gpt-oss-120b, comparing verbalized confidence, P(True) [ 20 ] , and the best-performing (lowest AURC) logprob-based baseline (log likelihood).
AURC
AUROC
Method
( ×100 ) ↓
(atoms) ↑
(all) ↑
Verbal confidence
44.30
0.74
0.74
Best logprob-based
51.94
0.69
0.70
P(true)
69.42
0.35
0.52
Appendix
Table 4: Comparison of confidence scoring functions. We report AURC ( ×100 ) and AUROC, on gpt-oss-120b on the FactScore benchmark, evaluated at the atom level and on the full set of abstractions across all levels.
Figure 7: Risk-coverage curves for atom-wise selective abstraction (ours) compared to redaction, Inline, and Self-Revision across six LLMs evaluated on 36 FactScore entities . Lower curves indicate improved trade-offs between factual risk and retained coverage.
Figure 8: Risk-coverage curves for atom-wise selective abstraction (ours) compared to redaction, Inline, and Self-Revision across six LLMs evaluated on 76 LongFact prompts . Lower curves indicate improved trade-offs between factual risk and retained coverage.
Figure 9: Risk-coverage curves on 36 FactScore entities comparing atom-wise selective abstraction (ours) compared to redaction, Inline, and Self-Revision for gpt-oss-120b (left) and gpt-oss-20b (right). We also report thresholded Inline, and Self-Revision evaluated at three thresholds, θ∈{10,50,90} (shown in transparent markers, size increasing with θ ). Introducing θ into these baselines does not yield a meaningful trade-off for these baselines: the thresholded points cluster closely and show no consistent trend of reducing risk at the cost of coverage.
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
Large language models (LLMs) often hallucinate in long-form generation. Existing approaches mainly improve factuality through post-hoc revision or reinforcement learning (RL) with correctness-based rewards, but they do not teach the model to estimate which parts of its generation are reliable. As a result, models may still state incorrect claims confidently in their responses. Recent advances in reasoning have significantly improved LLM performance, and have been leveraged to estimate confidence by incorporating calibration into RL objectives. However, existing approaches remain limited to a single scalar confidence for the entire response, which is insufficient for long-form generation where uncertainty varies across individual claims. To mitigate this problem, we propose CURE, a framework that improves long-form factuality by teaching LLMs to reason about uncertainty at the claim level. We first introduce a Claim-Aware Reasoning Protocol, which structures outputs into atomic claims paired with explicit confidence estimates. We then develop a multi-stage training pipeline that aligns model confidence with claims' correctness and then optimizes on factuality. The resulting calibrated confidence further enables selective prediction, allowing the model to abstain from uncertain claims at inference time. Experiments on four long-form factuality benchmarks show that CURE consistently improves factual accuracy over competitive supervised and RL baselines, while maintaining factual recall. In particular, it improves claim-level accuracy by up to 39.9% on Biography generation. These gains are accompanied by improved calibration, as reflected by a 16.0% increase in AUROC on FactBench.
Xin Liu, Lu Wang
Computer Science and Engineering, University of Michigan, Ann Arbor, MI, USA
Large Language Models (LLMs) have demonstrated impressive capabilities in long-form generation, yet their application is hindered by the hallucination problem. While Uncertainty Quantification (UQ) is essential for assessing reliability, the complex structure makes reliable aggregation across heterogeneous themes difficult, in addition, existing methods often overlook the nuance of neutral information and suffer from the high computational cost of fine-grained decomposition. To address these challenges, we propose AGSC (Adaptive Granularity and GMM-based Semantic Clustering), a UQ framework tailored for long-form generation. AGSC first uses NLI neutral probabilities as triggers to distinguish irrelevance from uncertainty, reducing unnecessary computation. It then applies Gaussian Mixture Model (GMM) soft clustering to model latent semantic themes and assign topic-aware weights for downstream aggregation. Experiments on BIO and LongFact show that AGSC achieves state-of-the-art correlation with factuality while reducing inference time by about 60% compared to full atomic decomposition.
Uncertainty quantification has emerged as an effective approach to closed-book hallucination detection for LLMs, but existing methods are largely designed for short-form outputs and do not generalize well to long-form generation. We introduce a taxonomy for fine-grained uncertainty quantification in long-form LLM outputs that distinguishes methods by design choices at three stages: response decomposition, unit-level scoring, and response-level aggregation. We formalize several families of consistency-based black-box scorers, providing generalizations and extensions of existing methods. We also introduce FactScore-STEM-Geo, a new 400-question long-form QA dataset spanning four categories across STEM and Geography. In our experiments across multiple LLMs and datasets, we find 1) claim-response entailment consistently performs better or on par with more complex claim-level scorers, 2) claim-level scoring generally yields better results than sentence-level scoring, and 3) uncertainty-aware decoding is highly effective for improving the factuality of long-form outputs. Our framework clarifies relationships between prior methods, enables apples-to-apples comparisons, and provides practical guidance for selecting components for fine-grained UQ.