UtilityBench: Measuring Language Model Utility When Errors Have Different Costs

Qifeng Lei; Zhiyong Yang; Zitai Wang; Shilong Bao; Linchao Zhang; Qianqian Xu; Qingming Huang

cs.LGOct 19, 2026

UtilityBench: Measuring Language Model Utility When Errors Have Different Costs

Authors: Qifeng Lei, Zhiyong Yang, Zitai Wang, Shilong Bao, Linchao Zhang, Qianqian Xu, Qingming Huang

Organizations: School of Computer Science and Technology, University of Chinese Academy of Sciences · Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, CAS · China Electronics Technology Group Corporation, Artificial Intelligence Institute of CETC

Abstract

As language models are increasingly deployed in real-world scenarios with important outcomes, the impact of their mistakes often extends far beyond what correctness can capture. In domains like medicine, a single error can lead to serious consequences and put patient safety at risk. However, mainstream evaluation protocols still treat all errors as equal, overlooking the spectrum of real-world costs. To address this, we propose a utility-based evaluation paradigm that explicitly incorporates the varying consequences of errors. Guided by this principle, we develop UtilityBench, a large-scale benchmark of 15,000 instances with verifiable answers, quantifying error severity using Best-Worst Scaling to elicit meaningful, data-driven cost estimates. These quantified utilities shape when a model should answer or abstain, guiding it to make decisions that align with the true importance of each scenario. As another merit, we introduce UtilityAUC, a metric that measures the area under the utility curve determined by a model's response ordering, thereby enabling utility preference alignment beyond error frequency. Our theoretical analysis demonstrates that UtilityAUC can distinguish between models where standard accuracy and AURC fall short. Empirically, evaluating 20 state-of-the-art models, we find that greater accuracy or more advanced architectures do not reliably translate to higher UtilityAUC, highlighting a gap between problem-solving ability and responsible decision-making in language model evaluation.

Figures & tables

Panel of Figure 2 — Figure 2: Consequence costs in UtilityBench . (a) Overall distribution of the sample-specific costs $L_{i}$ . (b) Cost composition across domains. Both panels group $\log(L_{i}/2)$ into five equal-width bands for display, while metric computation uses the continuous $L_{i}$ .

Figure 3: Construction of the utility curve. At coverage

\tau

, the

\tau n

highest-scoring model responses are preserved and the remainder are replaced by abstentions;

\tau_{1}

and

\tau_{2}

illustrate the resulting response–refusal partitions.

Family	Model	UtilityAUC $\uparrow$	Ans. Acc $\uparrow$	Ref. Acc $\uparrow$	Answer Rate
GPT	5	0.7346	60.83	79.60	53.17
	5.1	0.7167	67.60	74.13	60.73
	5.4 Mini	0.6352	59.79	59.49	65.56
	5.6 Luna	0.6991	61.32	78.53	54.93
	5.6 Sol	0.7298	74.97	62.44	67.44
Gemini	3 Flash	0.8132	71.91	86.55	51.87

Table 1: Comparison of 20 language models on the full UtilityBench under the shared Refusal Hint, binary scores, and

C_{R}=2

. Ans. Acc is the fraction of answerable samples receiving a correct substantive answer. Ref. Acc is the fraction of refusal-required samples receiving a refusal. Answer Rate is the fraction of samples receiving a substantive answer. The three rates are reported in percent (%). Darker cells indicate larger values within each metric column. Bold marks the best value in a column and underline marks the second best. Detailed setup can be found in Appendix E.1 .

Figure 4: Domain-level utility contributions of Gemini 3.6 Flash and GPT-5.6 Sol. Positive bars show utility gained from correct answers, while negative bars show utility lost through incorrect answers and refusals. The answerable domains are ordered by increasing average consequence cost, following Figure 2 (b). AbstentionBench contains refusal-required samples.

Model	Thinking	UtilityAUC $\uparrow$	Ans. Acc $\uparrow$	Ref. Acc $\uparrow$	Answer Rate
GPT-5.6 Luna	Off	0.6991	61.32	78.53	54.93
	Low	0.7316 (+0.0325)	75.47 (+14.15)	65.97 (-12.56)	64.73 (+9.81)
	Medium	0.7294 (+0.0302)	75.88 (+14.56)	63.48 (-15.05)	66.09 (+11.17)
	High	0.7317 (+0.0325)	76.59 (+15.27)	62.52 (-16.01)	66.77 (+11.85)
DeepSeek V4 Flash (Preview)	Off	0.7273	57.76	85.59	49.21
	On	0.7293 (+0.0020)	75.85 (+18.09)	68.39 (-17.20)	63.60 (+14.39)

Table 2: Thinking-mode comparison under the shared Refusal Hint. The evaluation setting and reported metrics are the same as in Table 1 . Parenthetical changes are relative to the corresponding Off baseline within each model.

Model	Add. Prompt	UtilityAUC $\uparrow$	Ans. Acc $\uparrow$	Ref. Acc $\uparrow$	Answer Rate
DeepSeek V4 Flash (Preview)	N/A	0.6055	63.47	43.61	78.11
	Refusal Hint	0.7273 (+0.1218)	57.76 (-5.71)	85.59 (+41.98)	49.21 (-28.90)
	Con. Guided	0.7493 (+0.1438)	62.51 (-0.96)	79.79 (+36.18)	57.23 (-20.88)
Qwen3.7 Plus	N/A	0.6628	73.47	54.92	72.29
	Refusal Hint	0.7678 (+0.1050)	67.79 (-5.68)	84.88 (+29.96)	53.51 (-18.78)
	Con. Guided	0.6663 (+0.0035)	73.71 (+0.24)	63.35 (+8.43)	67.71 (-4.58)

Table 4: Prompting strategy comparison. Four models respond with no additional prompt (N/A), a Refusal Hint, or a Consequence-Guided instruction. These prompts are given in Appendix E.1 . The evaluation setting and reported metrics are the same as in Table 1 . Parenthetical changes are relative to the N/A condition within each model.

Panel of Figure 5 — Figure 5: Comparison of human and LLM consequence rankings on 1,500 records. (a) For visualization, the continuous human and LLM rankings are each divided into five equal-sized consequence groups, from 1 the lowest to 5 the highest. Cells report the overlap between groups. The reported Spearman $\rho$ is computed over the complete rankings of all 1,500 records before this grouping. (b,c) Human–LLM Spearman $\rho$ under increasing numbers of human judgments in (b) and LLM judgments in (c). Each bar reports the average across repeated random selections of judgments.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Count
AbstentionBench ( Kirichenko et al., 2025 )	7,500
SuperGPQA ( M-A-P Team et al., 2025 )	1,350
LegalBench ( Guha et al., 2023 )	1,000
BigCodeBench ( Zhuo et al., 2025 )	550
MMLU-Pro ( Wang et al., 2024 )	500
GSM8K ( Cobbe et al., 2021 )	500

Appendix

Table 5: Source composition of the 15,000-record UtilityBench . The answerable half combines 17 public benchmarks, and the refusal-required half comes from AbstentionBench.

Model	Provider	Temp./top- $p$
GPT-5	OpenCode Zen	–/–
GPT-5.1	OpenCode Zen	–/–
GPT-5.4 Mini	OpenCode Zen	–/–
GPT-5.6 Luna	OpenCode Zen	–/–
GPT-5.6 Sol	OpenCode Zen	–/–
Gemini 3 Flash	OpenCode Zen	–/–

Appendix

Table 6: Model providers and principal generation parameters for the 20-model comparison. A dash means that the provider default is used.

Figure 6: Model rankings over a wider shadow-price range, using the same responses as Table 1 . Each column orders all 20 models by UtilityAUC at one value of $C_{R}$ , with the highest score at the top. Names on the left identify the ranking at $C_{R}=0$ . The outlined column marks the default $C_{R}=2$ .

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring
Oct 5, 2025Thomas F BurnsLLM Evaluation Language Model Generation Evaluation
Aleph Alpha Research
Aligning Language Model Benchmarks with Pairwise Preferences
Feb 2, 2026Marco Gutierrez, Xinyi Leng, Hannah Cyberey +3LLM Evaluation Learning to Rank
School of Data Science, University of Virginia · Imperial College London · Thomson Reuters Foundational Research +1
What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models
May 3, 2026Ranit Karmakar, Jayita ChatterjeeLLM Evaluation Language Model Calibration
Harvard University, Boston, MA 02115 · Boston, MA 02134

Oct 5, 2025J/K move · Enter open · S save

When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring

Thomas F Burns

Aleph Alpha Research

LLM Evaluation Language Model Generation Evaluation LLM Reliability LLM Hallucination Detection LLM Uncertainty Estimation LLM Abstention

Open paper AskPDF arXiv

Standard language model evaluation assigns scores to single predicted answers, rewarding high-confidence responses regardless of how residual probability mass is distributed over alternative options. This creates a systematic pressure toward overconfident guessing: under accuracy-based schemes, a model maximises its expected score by always committing to an answer rather than abstaining, even when its uncertainty is high. While penalty-based approaches partially address this by raising the confidence threshold for strategic guessing, they still treat all sub-threshold responses identically, ignoring a fundamental distinction in how models can express uncertainty - for example between hedging toward incorrect answers versus hedging toward "I don't know" responses. This paper introduces a novel evaluation metric to solve this problem of not considering a model's entire probability distribution over answer choices. The metric naturally distinguishes between harmful overconfidence in wrong answers and uncertainty expressed through abstention, providing scores in an interpretable default range. Through theoretical analysis and illustrative examples, the metric is shown to offer a more nuanced and aligned evaluation paradigm that incentivises models to express genuine uncertainty rather than guessing. Adapting 12 existing evaluation benchmarks to the metric's variants and measuring performance on six language models shows that for half of the tested benchmarks scores are negative across all tested models, indicating significant tendencies towards hallucination.

Abstract

LLM Evaluation

Explore similar work

When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring

Aligning Language Model Benchmarks with Pairwise Preferences

What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models