cs.CLSep 24, 2026

Likelihood Ranking doesn't Scale Like Prompting in LLMs

Authors: Alessandro Bondielli, Lucia Passaro, Davide Bacciu, Alessandro Lenci

Organizations: CoLingLab, Department of Philology, Literature and Linguistics, University of Pisa · Department of Computer Science, University of Pisa

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM evaluation is commonly performed either by prompting models to produce answers or by scoring candidate outputs with likelihood-based metrics. In multiple-choice QA, however, standard likelihood-based scoring is still conditioned on the question and answer set, and can therefore leverage the same task-conditioned answer-selection interface used in prompting. We study a complementary protocol based on likelihood ranking of declarative statements constructed from the same question--answer pairs. Across 95 decoder-only models, ranging from 0.1B to 104B parameters, and 10 MCQA datasets, we find a systematic divergence between declarative-statement likelihood ranking and prompted answering. Statement-likelihood accuracy remains comparatively stable across scale, whereas prompted answering improves sharply with scale and instruction-tuning. These results suggest that likelihood preferences over controlled declarative alternatives and task-conditioned answer selection probe distinct aspects of model behavior, and should not be treated as interchangeable.

Figures & tables

Figure 1: Overview of the experimental setting and evaluation. The design enables a comparison between prompting-based evaluation and declarative-statement likelihood ranking.
Dataset (HF Name)# QuestionAvg. Options ( ±\pm Std.)
TIGER-Lab/MMLU-Pro10,6329.45 ±\pm 1.50
allenai/ai2_arc7,6984.00 ±\pm 0.07
allenai/openbookqa11,8364.00 ±\pm 0.00
allenai/qasc8,6858.00 ±\pm 0.00
allenai/sciq13,6274.00 ±\pm 0.00
maveriq/bigbenchhard4,5394.73 ±\pm 3.09
Table 1: Statistics of multiple-choice datasets used in our evaluation, same order as the table: Wang et al. (2024c) ; Clark et al. (2018) ; Mihaylov et al. (2018) ; Khot et al. (2020) ; Welbl et al. (2017) ; Suzgun et al. (2023) ; Talmor et al. (2019) ; Lin et al. (2022) ; Science (2025) ; Sakai et al. (2024) .
Figure 2: Accuracies under prompting (APX) and declarative statement likelihood (APS) settings, across model scales and families. APS shows weak scaling and limited gains from instruction-tuning; APX improves sharply with scale, and larger models outperform APS.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Number of examples and errors in declarative statements generation for each dataset.
Figure 4: Distribution of BERTScore F1 for each statement and question–answer pair.
Figure 5: Distribution of Mean BLEU scores per question.
Figure 6: Distribution of answers in each dataset.
Figure 7: Prompting-based (APX) and standard likelihood-based MCQA (Likelihood-MCQA) accuracy across model scales and families. Regardless of the presence of Instruction fine-tuning, likelihood-based MCQA and APX move in the same general direction, with comparable steepness.
Family# ModelsBaseInstructSize Range (B)
Qwen2.514770.5–72
Qwen312660.6–31
Llama-39541–71
Gemma-310550.3–27
Gemma-26332–27
OLMo-28441–32
Appendix
Table 2: Overview of the model families evaluated in this work, reporting the number of base and instruction-tuned variants and their parameter scale.
Figure 8: Prompting-based and declarative-statement likelihood accuracies, for instruction-tuned and base models, as a function of model scale, for each dataset.
Figure 9: Average PPL score assigned to the correct answer by each model, divided by dataset.
Figure 10: Average delta between second lowest and lowest PPL score for declarative statements vs model size.

Explore similar work

CardsList

  1. ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice QA

    May 7, 2026Inder Preet, Shuxin Lin, Dhaval PatelMultiple-Choice QuestionsLarge Reasoning Models

    IBM, Dublin · IBM, New York

  2. LLMs Can Better Capture Human Judgments--With the Right Prompts

    Jun 10, 2026Danica Dillion, Chen Cecilia Liu, Baihui Wang +5Human JudgmentMoral Reasoning

    Complexity Science Hub, Vienna · The Ohio State University · University of Cambridge +2

  3. Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation

    Aug 30, 2026Nishant Balepur, Paiheng Xu, Wei Ai +3Multiple-Choice QuestionsScoring

    University of Maryland · New York University · Nanyang Technological University