When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generation
Organizations: Technion · NVIDIA
Abstract
LLMs are widely used, yet they remain prone to factual errors that erode trust and limit adoption in high-risk settings. One approach to mitigate this risk is to equip models with uncertainty estimation mechanisms that abstain when confidence is low. However, this binary "all-or-nothing" approach is excessively restrictive in long-form settings, often discarding valuable information. We introduce Selective Abstraction (SA), a framework that enables LLMs to trade specificity for reliability by selectively reducing the detail of uncertain content. We formalize SA through selective risk and coverage, and propose Atom-wise Selective Abstraction, a claim-level instantiation that decomposes responses into atomic claims and replaces uncertain atoms with higher-confidence, less specific abstractions. We develop a novel end-to-end pipeline for evaluating long-form generation that instantiates risk as factual correctness and measures coverage as retained information. Our comprehensive empirical evaluation across five LLMs and two long-form factual benchmarks shows that atom-wise SA consistently improves the risk-coverage trade-off, outperforming seven baselines, including claim removal, self-revision, sampling-based aggregation, and chain-of-verification. It improves AURC by up to 35.61% over claim removal, demonstrating that reducing specificity boosts reliability while preserving substantial information. We further provide a risk-guided procedure for selecting a confidence threshold that satisfies a target risk.
Figures & tables
| FactScore | LongFact | |||||
| Model | Redaction | Atom-wise SA | Improvement | Redaction | Atom-wise SA | Improvement |
| AURC ( ) | AURC ( ) | (%) | AURC ( ) | AURC ( ) | (%) | |
| gpt-oss-120b | ||||||
| gpt-oss-20b | ||||||
| qwen3-235b-t | ||||||
| qwen3-30b-t |
| FactScore Risk (%) | LongFact Risk(%) | |||
|---|---|---|---|---|
| Model | Inline | Self-Revision | Inline | Self-Revision |
| gpt-oss-120b | 5.33 | 15.71 | 13.90 | 11.25 |
| gpt-oss-20b | 10.13 | 14.56 | 12.58 | 11.21 |
| qwen3-235b-t | 20.31 | 12.10 | 11.96 | 14.80 |
| qwen3-30b-t | 15.81 | 16.40 | 6.63 | 10.10 |
| qwen3-30b-i | 13.05 | 5.48 | 7.96 | 11.65 |
| FactScore | LongFact | |||
|---|---|---|---|---|
| Model | Avg #Atoms | Avg #Abs./Atom | Avg #Atoms | Avg #Abs./Atom |
| gpt-oss-120b | 68.8 | 5.6 | 73.9 | 5.3 |
| gpt-oss-20b | 57.0 | 3.9 | 61.0 | 3.7 |
| qwen3-235b-t | 51.2 | 5.0 | 65.4 | 4.9 |
| qwen3-30b-t | 32.9 | 3.6 | 63.4 | 3.4 |
| qwen3-30b-i | 43.9 | 4.2 | 60.0 | 4.5 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| AURC | AUROC | ||
|---|---|---|---|
| Method | ( ) | (atoms) | (all) |
| Verbal confidence | |||
| Best logprob-based | |||
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 854 papers