cs.CLOct 6, 2026

Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices

Authors: Jinhyeok Kim, Hye-Young Jung

Organizations: Applied Artificial Intelligence Hanyang University, Republic of Korea · Mathematical Data Science Hanyang University, Republic of Korea

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multiple-choice question answering (MCQA) is commonly used to evaluate large language models under the assumption that one of the provided options is correct, typically using answer-selection accuracy. However, in real deployments, users or retrieval systems may provide invalid option sets in which none of the listed choices is correct, and selecting one of them may incur downstream cost. We study this setting as penalty-framed no-valid-option MCQA. Using the mathematics subset of MMLU-Pro, we remove the labeled correct option, allow models to either choose a remaining option or output ABSTAIN, and penalize invalid forced-choice responses. We further introduce correct-conditioned analysis, evaluating abstention only on instances that the model originally answered correctly. Experiments show that high MCQA accuracy does not fully guarantee abstention reliability: even under explicit no-valid-option-aware instructions and penalty-based scoring, models still produce invalid forced-choice responses for a subset of originally correct instances. These results show that penalty-framed no-valid-option MCQA reveals an aspect of model reliability not captured by standard answer-selection accuracy.

Figures & tables

Figure 1: Original MCQA accuracy and no-valid-option abstention rate. AR is reported under the main setting: CoT prompting, no-valid-option-aware instruction, invalid forced-choice penalty −1-1 , robust parsing. Values are 3-run means on the filtered MMLU-Pro Mathematics subset (1,118 instances).
DatasetModelAcc ↑\uparrowAR ↑\uparrowIFR ↓\downarrowCAR ↑\uparrowCIFR ↓\downarrow
MMLU-Pro MathGPT-5-mini97.585.614.487.212.8
GPT-5-nano94.181.118.984.515.5
GPT-4.1-mini95.369.031.072.427.6
GPT-4.1-nano78.147.852.254.545.5
Gemini-3.0-Flash96.290.19.990.99.1
Gemini-3.1-Flash-Lite96.583.816.286.413.6
Table 1: Main post-exclusion results under CoT prompting, robust parsing, and the no-valid-option-aware instruction with invalid forced-choice penalty −1-1 . AR/IFR are computed over all no-valid-option instances, while CAR/CIFR are computed on each model’s stable-correct subset for the corresponding dataset. Subset sizes are reported in Appendix F , Table 6 . All values are 3-run means reported as percentages; standard deviations are reported in Appendix C , Table 5 .
 Scoring-rule variationAbstention-instruction variation
Model−0.5-0.5−1-1−2-2Penalty-onlyUncertainty-awareNo-valid-option-aware
GPT-5-mini13.1 ± 0.4\pm\,0.412.8 ± 0.7\pm\,0.713.2 ± 0.8\pm\,0.823.5 ± 0.3\pm\,0.321.5 ± 1.1\pm\,1.112.8 ± 0.7\pm\,0.7
GPT-5-nano15.2 ± 0.4\pm\,0.415.5 ± 1.0\pm\,1.014.8 ± 0.6\pm\,0.623.4 ± 0.8\pm\,0.820.8 ± 1.2\pm\,1.215.5 ± 1.0\pm\,1.0
GPT-4.1-mini27.8 ± 0.5\pm\,0.527.6 ± 0.6\pm\,0.626.6 ± 0.2\pm\,0.241.1 ± 0.8\pm\,0.848.2 ± 0.1\pm\,0.127.6 ± 0.6\pm\,0.6
GPT-4.1-nano46.8 ± 0.7\pm\,0.745.5 ± 0.6\pm\,0.646.4 ± 0.5\pm\,0.575.7 ± 0.6\pm\,0.665.0 ± 0.4\pm\,0.445.5 ± 0.6\pm\,0.6
Gemini-3.0-Flash10.2 ± 0.3\pm\,0.39.1 ± 0.6\pm\,0.69.9 ± 1.1\pm\,1.131.2 ± 0.5\pm\,0.532.5 ± 1.4\pm\,1.49.1 ± 0.6\pm\,0.6
Table 2: Stable-correct CIFR on the filtered MMLU-Pro Mathematics subset ( N=1,118N=1{,}118 ) under scoring-rule and instruction ablations. In the scoring-rule block, the no-valid-option-aware instruction is fixed and only the invalid forced-choice penalty varies. In the instruction block, +4+4 / −1-1 / 00 scoring is fixed and only the abstention instruction varies. CIFR values are reported as percentages in mean ±\pm standard deviation format over three independent runs. Lower CIFR is better.
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Construction of penalty-framed no-valid-option MCQA. The labeled correct option is removed from an original MCQA instance. The desired behavior is to output Abstain ; selecting any remaining option is treated as an invalid forced-choice response and incurs a penalty.
Model name in paperAPI model identifierProviderRuntime setting
GPT-5-minigpt-5-mini-2025-08-07OpenAIreasoning_effort=low
GPT-5-nanogpt-5-nano-2025-08-07OpenAIreasoning_effort=low
GPT-4.1-minigpt-4.1-mini-2025-04-14OpenAItemperature=0.0
GPT-4.1-nanogpt-4.1-nano-2025-04-14OpenAItemperature=0.0
Gemini-3.0-Flashgemini-3-flash-previewGoogle Vertex AIthinking_level=low
Gemini-3.1-Flash-Litegemini-3.1-flash-liteGoogle Vertex AIthinking_level=low
Appendix
Table 3: Model identifiers and runtime settings used in our experiments. The six core models are used in the prompting, scoring-rule, and parser-sensitivity analyses. GPT-5, Claude Sonnet 5, and Qwen3-8B are additionally evaluated under the main setting. Gemini 2.5-family models are reported only as supplementary diagnostics because of weaker completion and parsing reliability under the current evaluation pipeline.
Exclusion reasonMMLU-Pro MathMulti-domain MMLU
Label error4726
Alternative correct option7234
Total excluded11960
Appendix
Table 4: Exclusion counts by reason. Alternative correct options are options that could remain valid after removal of the labeled correct option.
DatasetModelSetNNAcc ↑\uparrowAR ↑\uparrowIFR ↓\downarrowCAR ↑\uparrowCIFR ↓\downarrow
 GPT-5-miniUnfiltered123792.3 ± 0.6\pm\,0.679.8 ± 0.6\pm\,0.620.2 ± 0.6\pm\,0.684.3 ± 0.6\pm\,0.615.7 ± 0.6\pm\,0.6
  Filtered111897.5 ± 0.5\pm\,0.585.6 ± 0.7\pm\,0.714.4 ± 0.7\pm\,0.787.2 ± 0.7\pm\,0.712.8 ± 0.7\pm\,0.7
 GPT-5-nanoUnfiltered123789.6 ± 0.4\pm\,0.476.4 ± 0.8\pm\,0.823.6 ± 0.8\pm\,0.881.9 ± 0.9\pm\,0.918.1 ± 0.9\pm\,0.9
  Filtered111894.1 ± 0.4\pm\,0.481.1 ± 1.0\pm\,1.018.9 ± 1.0\pm\,1.084.5 ± 1.0\pm\,1.015.5 ± 1.0\pm\,1.0
 GPT-4.1-miniUnfiltered123791.3 ± 0.2\pm\,0.264.3 ± 0.6\pm\,0.635.7 ± 0.6\pm\,0.669.3 ± 0.5\pm\,0.530.7 ± 0.5\pm\,0.5
  Filtered111895.3 ± 0.3\pm\,0.369.0 ± 0.6\pm\,0.631.0 ± 0.6\pm\,0.672.4 ± 0.6\pm\,0.627.6 ± 0.6\pm\,0.6
Appendix
Table 5: Unfiltered and filtered results on MMLU-Pro Mathematics and multi-domain MMLU under CoT prompting, the no-valid-option-aware instruction, invalid forced-choice penalty −1-1 , and robust parsing. NN is the number of dataset instances; Acc is computed over completed original MCQA responses in each run. AR/IFR are computed over evaluation-eligible no-valid-option responses, and CAR/CIFR over evaluation-eligible responses within each model’s stable-correct subset. Values are percentages reported as mean ±\pm standard deviation over three independent runs.
DatasetModelStable-correct countStable-correct (%)
MMLU-Pro MathGPT-5-mini106895.5
GPT-5-nano100489.8
GPT-4.1-mini103292.3
GPT-4.1-nano72664.9
Gemini-3.0-Flash102591.7
Gemini-3.1-Flash-Lite104693.6
Appendix
Table 6: Model-specific stable-correct subset sizes under CoT prompting. An instance is included if the model answers the original MCQA instance correctly in all three runs. Percentages use the corresponding filtered dataset size: 1,118 for MMLU-Pro Mathematics and 340 for multi-domain MMLU.
ModelAcc ↑\uparrowAR ↑\uparrowIFR ↓\downarrowCAR ↑\uparrowCIFR ↓\downarrow
GPT-5-mini97.3 ± 0.1\pm\,0.184.5 ± 1.3\pm\,1.315.5 ± 1.3\pm\,1.386.6 ± 1.1\pm\,1.113.4 ± 1.1\pm\,1.1
GPT-5-nano93.5 ± 0.2\pm\,0.279.9 ± 0.7\pm\,0.720.1 ± 0.7\pm\,0.784.0 ± 0.5\pm\,0.516.0 ± 0.5\pm\,0.5
GPT-4.1-mini47.7 ± 0.2\pm\,0.241.6 ± 0.2\pm\,0.258.4 ± 0.2\pm\,0.241.6 ± 0.4\pm\,0.458.4 ± 0.4\pm\,0.4
GPT-4.1-nano28.0 ± 0.2\pm\,0.236.4 ± 0.6\pm\,0.663.6 ± 0.6\pm\,0.638.2 ± 0.9\pm\,0.961.8 ± 0.9\pm\,0.9
Gemini-3.0-Flash93.9 ± 0.3\pm\,0.385.5 ± 0.3\pm\,0.314.5 ± 0.3\pm\,0.387.9 ± 0.4\pm\,0.412.1 ± 0.4\pm\,0.4
Gemini-3.1-Flash-Lite55.4 ± 1.1\pm\,1.118.4 ± 0.9\pm\,0.981.6 ± 0.9\pm\,0.930.5 ± 1.5\pm\,1.569.5 ± 1.5\pm\,1.5
Appendix
Table 7: Direct prompting results on the filtered MMLU-Pro Mathematics subset ( N=1,118N=1{,}118 ) under the no-valid-option-aware instruction and invalid forced-choice penalty −1-1 . Values are mean ±\pm standard deviation over three independent runs.
ModelPenaltyAR ↑\uparrowIFR ↓\downarrowCAR ↑\uparrowCIFR ↓\downarrow
GPT-5-mini−0.5-0.585.5 ± 0.6\pm\,0.614.5 ± 0.6\pm\,0.686.9 ± 0.4\pm\,0.413.1 ± 0.4\pm\,0.4
 −1-185.6 ± 0.7\pm\,0.714.4 ± 0.7\pm\,0.787.2 ± 0.7\pm\,0.712.8 ± 0.7\pm\,0.7
 −2-285.1 ± 0.8\pm\,0.814.9 ± 0.8\pm\,0.886.8 ± 0.8\pm\,0.813.2 ± 0.8\pm\,0.8
GPT-5-nano−0.5-0.581.4 ± 0.4\pm\,0.418.6 ± 0.4\pm\,0.484.8 ± 0.4\pm\,0.415.2 ± 0.4\pm\,0.4
 −1-181.1 ± 1.0\pm\,1.018.9 ± 1.0\pm\,1.084.5 ± 1.0\pm\,1.015.5 ± 1.0\pm\,1.0
 −2-281.8 ± 0.7\pm\,0.718.2 ± 0.7\pm\,0.785.2 ± 0.6\pm\,0.614.8 ± 0.6\pm\,0.6
Appendix
Table 8: Full scoring-rule ablation results on the filtered MMLU-Pro Mathematics subset ( N=1,118N=1{,}118 ) under CoT prompting and the no-valid-option-aware instruction. Only the invalid forced-choice penalty is varied.
ModelInstructionAR ↑\uparrowIFR ↓\downarrowCAR ↑\uparrowCIFR ↓\downarrow
GPT-5-miniPenalty-only75.2 ± 0.4\pm\,0.424.8 ± 0.4\pm\,0.476.5 ± 0.3\pm\,0.323.5 ± 0.3\pm\,0.3
 Uncertainty-aware76.8 ± 1.1\pm\,1.123.2 ± 1.1\pm\,1.178.5 ± 1.1\pm\,1.121.5 ± 1.1\pm\,1.1
 No-valid-option-aware85.6 ± 0.7\pm\,0.714.4 ± 0.7\pm\,0.787.2 ± 0.7\pm\,0.712.8 ± 0.7\pm\,0.7
GPT-5-nanoPenalty-only73.4 ± 0.5\pm\,0.526.6 ± 0.5\pm\,0.576.6 ± 0.8\pm\,0.823.4 ± 0.8\pm\,0.8
 Uncertainty-aware76.1 ± 1.3\pm\,1.323.9 ± 1.3\pm\,1.379.2 ± 1.2\pm\,1.220.8 ± 1.2\pm\,1.2
 No-valid-option-aware81.1 ± 1.0\pm\,1.018.9 ± 1.0\pm\,1.084.5 ± 1.0\pm\,1.015.5 ± 1.0\pm\,1.0
Appendix
Table 9: Full instruction ablation results on the filtered MMLU-Pro Mathematics subset ( N=1,118N=1{,}118 ) under CoT prompting and the +4/−1/0+4/-1/0 scoring rule. Only the abstention instruction is varied.
ModelParserAcc ↑\uparrowAR ↑\uparrowIFR ↓\downarrowCAR ↑\uparrowCIFR ↓\downarrow
GPT-5-miniRobust97.5 ± 0.5\pm\,0.585.6 ± 0.7\pm\,0.714.4 ± 0.7\pm\,0.787.2 ± 0.7\pm\,0.712.8 ± 0.7\pm\,0.7
 Strict97.5 ± 0.5\pm\,0.585.6 ± 0.7\pm\,0.714.4 ± 0.7\pm\,0.787.2 ± 0.7\pm\,0.712.8 ± 0.7\pm\,0.7
 Stored97.5 ± 0.5\pm\,0.585.6 ± 0.7\pm\,0.714.4 ± 0.7\pm\,0.787.2 ± 0.7\pm\,0.712.8 ± 0.7\pm\,0.7
GPT-5-nanoRobust94.1 ± 0.4\pm\,0.481.1 ± 1.0\pm\,1.018.9 ± 1.0\pm\,1.084.5 ± 1.0\pm\,1.015.5 ± 1.0\pm\,1.0
 Strict94.1 ± 0.4\pm\,0.481.1 ± 1.0\pm\,1.018.9 ± 1.0\pm\,1.084.5 ± 1.0\pm\,1.015.5 ± 1.0\pm\,1.0
 Stored94.1 ± 0.4\pm\,0.481.1 ± 1.0\pm\,1.018.9 ± 1.0\pm\,1.084.5 ± 1.0\pm\,1.015.5 ± 1.0\pm\,1.0
Appendix
Table 10: Parser-mode sensitivity on the filtered MMLU-Pro Mathematics subset ( N=1,118N=1{,}118 ) under the main CoT setting: no-valid-option-aware instruction and invalid forced-choice penalty −1-1 .
ModelConditionIncomplete rate (%)Invalid-format rate (%)Eval-eligible rate (%)
GPT-5-miniOriginal MCQA0.00 ± 0.00\pm\,0.000.33 ± 0.05\pm\,0.0599.67 ± 0.05\pm\,0.05
 No-valid-option0.00 ± 0.00\pm\,0.000.12 ± 0.05\pm\,0.0599.88 ± 0.05\pm\,0.05
GPT-5-nanoOriginal MCQA0.00 ± 0.00\pm\,0.001.28 ± 0.26\pm\,0.2698.72 ± 0.26\pm\,0.26
 No-valid-option0.00 ± 0.00\pm\,0.000.15 ± 0.10\pm\,0.1099.85 ± 0.10\pm\,0.10
GPT-4.1-miniOriginal MCQA0.45 ± 0.09\pm\,0.090.21 ± 0.14\pm\,0.1499.34 ± 0.19\pm\,0.19
 No-valid-option1.28 ± 0.44\pm\,0.440.15 ± 0.14\pm\,0.1498.57 ± 0.32\pm\,0.32
Appendix
Table 11: Response-quality diagnostics on the filtered MMLU-Pro Mathematics subset ( N=1,118N=1{,}118 ) under CoT prompting and robust parsing. The no-valid-option condition uses the no-valid-option-aware instruction and invalid forced-choice penalty −1-1 . Values are percentages reported as mean ±\pm standard deviation over three independent runs. Incomplete, invalid-format, and evaluation-eligible rates are all computed over the full set of 1,1181{,}118 instances in each run. These three mutually exclusive response states sum to 100%100\% before rounding. Gemini 2.5-family models below the dividing line are supplementary.
ModelConditionIncomplete rate (%)Invalid-format rate (%)Eval-eligible rate (%)
GPT-5-miniOriginal MCQA0.00 ± 0.00\pm\,0.000.20 ± 0.17\pm\,0.1799.80 ± 0.17\pm\,0.17
 No-valid-option0.00 ± 0.00\pm\,0.000.00 ± 0.00\pm\,0.00100.00 ± 0.00\pm\,0.00
Qwen3-8BOriginal MCQA0.00 ± 0.00\pm\,0.0013.43 ± 0.61\pm\,0.6186.57 ± 0.61\pm\,0.61
 No-valid-option0.10 ± 0.17\pm\,0.174.41 ± 1.35\pm\,1.3595.49 ± 1.22\pm\,1.22
Appendix
Table 12: Response-quality diagnostics on the filtered multi-domain MMLU subset ( N=340N=340 ) under CoT prompting and robust parsing. The no-valid-option condition uses the no-valid-option-aware instruction and invalid forced-choice penalty −1-1 . Values are percentages reported as mean ±\pm standard deviation over three independent runs. Incomplete, invalid-format, and evaluation-eligible rates are all computed over the full set of 340340 instances in each run. These three mutually exclusive response states sum to 100%100\% before rounding.
A. 1.234F. 1.618
B. 1.414G. 1.789
C. 2.000H. 1.000
D. 1.581I. 3.141
E. 2.718J. 2.345
Appendix
Figure 4: Paired responses for the infinite-series example. In the original condition, the model selects the labeled correct option, D (1.581). After that option is removed, the model still computes 1+1/(e−1)≈1.581981+1/(e-1)\approx 1.58198 but explicitly selects the closest remaining option, E (1.618), instead of ABSTAIN .
A. 6F. 3
B. 4G. 2
C. 5H. 1
D. 10I. 9
E. 7J. 8
Appendix
Figure 5: Paired responses for the Ramsey example. In the original condition, the model selects the labeled correct option, F (3). After that option is removed, the model still identifies R(3,2)=3R(3,2)=3 but invokes the related result R(3,3)=6R(3,3)=6 and selects A (6), instead of ABSTAIN .
WorkAnswer absent by constructionRejection expressionAbstention utility separationInstance-level correctness conditioning
Tam et al. (2025) , None of the Above, Less of the RightYesIn-option NOTANoNo
Madhusudhan et al. (2025) , Do LLMs Know When to NOT Answer?YesIn-option IDK/NOTANoNo
Góral et al. (2025) , Wait, that’s not an optionYesFree-form rejection, detected post hocNoNo
Wang et al. (2025) , LLMs May Perform MCQA by Selecting the Least Incorrect OptionYesIn-option NOTA and free-form no-answerNoCorrectness across re-ordered answer options
Wang et al. (2026) , Are LLM Decisions Faithful to Verbal Confidence? (RiskEval)NoFormal abstention actionYesNo
OursYesFormal out-of-option ABSTAINYesStable correctness across repeated runs
Appendix
Table 13: Comparison of related evaluation designs in terms of answer absence, rejection expression, utility separation between abstention and invalid choices, and instance-level correctness conditioning.

Explore similar work

CardsList

  1. Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

    Aug 30, 2026Nishant Balepur, Paiheng Xu, Wei Ai +3Multiple-Choice QuestionsScoring

    University of Maryland · New York University · Nanyang Technological University

  2. Two Axes of LLM Abstention: Answer Correctness and Question Answerability

    Jul 9, 2026Benedikt J. WagnerAnswerInstruction-Tuned Models

    City St George’s, University of London

  3. When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA

    Sep 8, 2026Manikandan Ravikiran, Siddharth VohraMultiple-Choice QuestionsCorrect

    Indian Institute of Technology Mandi, India · Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA