Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domains

Andrew Katz

cs.CLMar 15, 2026

Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domains

Authors: Andrew Katz

Organizations: Department of Engineering Education, Virginia Tech

Abstract

The minimal pairs paradigm of comparing model probabilities for contrasting completions has proven useful for evaluating linguistic knowledge in language models, yet its application has largely been confined to binary grammaticality judgments over syntactic phenomena. Additionally, standard prompting-based evaluation requires expensive text generation, may elicit post-hoc rationalizations rather than model judgments, and discards information about model uncertainty. We address both limitations by extending surprisal-based evaluation from binary grammaticality contrasts to ordinal-scaled classification and scoring tasks across multiple domains. Rather than asking models to generate answers, we measure the information-theoretic "surprise" (negative log probability) they assign to each position on rating scales (e.g., 1-5 or 1-9), yielding full surprisal curves that reveal both the model's preferred response and its uncertainty via entropy. We explore this framework across four domains: social-ecological-technological systems classification, causal statement identification (binary and scaled), figurative language detection, and deductive qualitative coding. Across these domains, surprisal curves produce interpretable classification signals with clear minima near expected ordinal scale positions, and entropy over the completion tended to distinguish genuinely ambiguous items from easier items.

Figures & tables

Domain	Task Type	Scale
SETS Classification	Score Discovery	1-9
Causal Statement	Scaled Classif.	1-5, 1-9
Figurative Lang.	Multi-format	Binary + Scales
Deductive Coding	Code Applic.	1-5, 1-9

Table 1: Summary of experimental domains and task types.

Model	Soc.	Ecol.	Tech.	Mean
3B-Instruct	2.92	2.99	2.94	2.95
7B-Instruct	2.21	1.51	1.78	1.83
14B-Instruct	1.86	1.17	1.33	1.45
14B	2.04	0.98	1.27	1.43

Table 2: SETS classification MAE by model and dimension (

n=90

entities). Lower is better. All models are Qwen2.5.

Figure 1: Surprisal curves for “bug” (insect meaning) on the social, ecological, and technological dimensions with moderate context (“The garden bug pollinated flowers while feeding on nectar”). The 14B models correctly identify high ecological scores.

Figure 2: Surprisal curves for “bug” (software meaning) on the social, ecological, and technological dimensions with moderate context (“The software bug caused the application to crash unexpectedly”). The 14B models correctly identify high technological scores.

Figure 3: Surprisal curves for “virus” (computer meaning) across three context levels: (a) minimal context, where models interpret “virus” as biological and assign high ecological scores; (b) moderate context, where 14B models correctly shift to high technological scores; (c) rich context describing malware behavior, indicating context-dependent disambiguation. The 3B model never adjusts its technological score regardless of context.

Figure 4: Binary classification surprisal for clear cases: (a) the causal statement “The heavy rain caused widespread flooding in the city,” where upward-sloping lines indicate lower surprisal for “True”; (b) the non-causal statement “The meeting was scheduled for 3 PM,” where downward-sloping lines indicate lower surprisal for “False.” Each panel shows one model with three context levels.

Figure 5: Binary classification surprisal for ambiguous cases: (a) the indirect causal statement “If you heat water to 100 degrees Celsius, it will boil,” where lines slope upward but less steeply than clear cases; (b) the correlational statement “Students who study more tend to get better grades,” where mixed slopes reflect genuine ambiguity. Compare with Figure 4 .

Model	Full	Minimal	None	Mean
3B-Instruct	68.0	47.3	44.7	53.3
7B-Instruct	75.7	72.7	69.3	72.6
14B-Instruct	78.0	77.0	76.7	77.2
14B	76.0	68.3	77.0	73.8

Table 3: Causal binary classification accuracy (%) by model and context level (

n=100

). Accuracy averaged across response formats. All models are Qwen2.5.

Model	Full/CS	Full/BC	Min./CS	Min./BC	None/CS	None/BC	Mean
3B-Instruct	82.0	82.0	79.0	85.0	81.0	74.0	80.5
7B-Instruct	86.0	83.0	84.0	83.0	87.0	78.0	83.5
14B-Instruct	90.0	91.0	91.0	88.0	88.0	86.0	89.0
14B	88.0	87.0	89.0	86.0	92.0	85.0	87.8

Table 4: Causal scaled directional accuracy (%) by model, context level, and framing (

n=100

). CS = causal strength, BC = bipolar causality. All models are from the Qwen2.5 family.

Figure 6: Paired metaphor intensity surprisal curves (5-point scale) for Pair 7. Left: the figurative statement “The words hung in the air between them…afraid of what might happen if they acknowledged the truth” shows decreasing surprisal toward high intensity. Right: the literal counterpart “The banner hung in the air between them…afraid they might tear the delicate fabric” shows minimum surprisal at low intensity. Both statements share key phrases but differ in figurative status.

5-point scale
Model	Minimal	Full	Mean
3B-Instruct	80.0	53.3	66.7
7B-Instruct	80.0	80.0	80.0
14B-Instruct	70.0	63.3	66.7
14B	96.7	93.3	95.0
9-point scale

Table 5: Figurative language paired scale discrimination rate (%) by model, scale, and context (

n=30

pairs). Rate = percentage of pairs where the figurative statement’s minimum-surprisal position exceeds the literal counterpart’s. All models are Qwen2.5.

Code	Definition
Back to normal with no changes/admin	Frustration with administration expecting return to pre-pandemic conditions without implementing improvements
Delays, setbacks in careers/research	Career advancement delays, research disruptions, publication setbacks, or professional development impacts
Don’t want to work in office	Preference to continue remote work rather than returning to office-based work
Equity	Concerns about fairness, equal treatment, disparities, or social justice issues related to pandemic recovery
Family, personal priorities, childcare	Statements about family responsibilities, childcare arrangements, or balancing personal priorities with work
Financial/support	Financial concerns, economic impacts, funding issues, or need for institutional support

Table 6: Codebook for Deductive Coding of Faculty Pandemic Survey Responses (13 codes used in evaluation)

Figure 7: Surprisal-based coding for the code “not over, less hope of ever recovering” applied to a text about the pandemic not ending. Models correctly find minimum surprisal at scores 3–4 (applicable).

Figure 8: Surprisal-based coding for the code “family, personal priorities, childcare” applied to a text about pandemic media coverage. Models correctly find minimum surprisal at score 1 (not applicable).

Figure 9: Surprisal-based coding for the code “not over, less hope of ever recovering” applied to a text about striving for a new normal. Models show varied patterns, with larger models finding minimum surprisal at moderate applicability scores.

Model	No Pers.	Qual. Res.	Mean
3B-Instruct	65.0	70.0	67.5
7B-Instruct	67.5	65.0	66.3
14B-Instruct	75.0	75.0	75.0
14B	67.5	67.5	67.5

Table 7: Deductive coding accuracy (%) by model and persona (

n=40

, threshold

\geq 3

). All models are Qwen2.5.

Model	No Pers.	Qual. Res.	Mean
3B-Instruct	65.0	64.7	64.9
7B-Instruct	60.6	56.2	58.4
14B-Instruct	72.2	72.2	72.2
14B	66.7	64.9	65.8

Table 8: Deductive coding F1 score (%) by model and persona (

n=40

). All models are Qwen2.5.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Surprisal curves for the causal statement “The heavy rainfall caused widespread flooding in the city” across all model, context, and scale combinations. Each panel shows two scale framings (causal strength and bipolar causality). Monotonically decreasing curves indicate consistent assignment of high causal ratings.

Figure 11: Surprisal curves for the correlational statement “Students who study more tend to get better grades” across all model, context, and scale combinations. Compared to the causal statement in Figure 10 , the curves are flatter and more parabolic, reflecting model uncertainty about the causal content.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring
Oct 5, 2025Thomas F BurnsLLM Evaluation Language Model Generation Evaluation
Aleph Alpha Research
Beyond Model Bias: Cross-Entropy Based Rankings in Literary and Academic Texts Remain Stable across Small Language Models
Oct 12, 2026Ankit Bhattacharjee, Indrasabarna Hari, Krityapriya Bhaumik +2Language Modeling Computational Literary Studies
Indian Institute of Technology Kharagpur, India · Asutosh College, University of Calcutta, India
surprisal is Not a Theory
Jul 22, 2026Andrés Buxó-Lugo, Aniello De Santo, Morgan Grobol +2Surprisal Theory Language Modeling
aUniversity at Buffalo, Buffalo, NY, USA · bUniversity of Utah, Salt Lake City, UT, USA · cUniversité Paris Nanterre, Paris, France +1

Oct 5, 2025J/K move · Enter open · S save

When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring

Thomas F Burns

Aleph Alpha Research

LLM Evaluation Language Model Generation Evaluation LLM Reliability LLM Hallucination Detection LLM Uncertainty Estimation LLM Abstention

Open paper AskPDF arXiv

Standard language model evaluation assigns scores to single predicted answers, rewarding high-confidence responses regardless of how residual probability mass is distributed over alternative options. This creates a systematic pressure toward overconfident guessing: under accuracy-based schemes, a model maximises its expected score by always committing to an answer rather than abstaining, even when its uncertainty is high. While penalty-based approaches partially address this by raising the confidence threshold for strategic guessing, they still treat all sub-threshold responses identically, ignoring a fundamental distinction in how models can express uncertainty - for example between hedging toward incorrect answers versus hedging toward "I don't know" responses. This paper introduces a novel evaluation metric to solve this problem of not considering a model's entire probability distribution over answer choices. The metric naturally distinguishes between harmful overconfidence in wrong answers and uncertainty expressed through abstention, providing scores in an interpretable default range. Through theoretical analysis and illustrative examples, the metric is shown to offer a more nuanced and aligned evaluation paradigm that incentivises models to express genuine uncertainty rather than guessing. Adapting 12 existing evaluation benchmarks to the metric's variants and measuring performance on six language models shows that for half of the tested benchmarks scores are negative across all tested models, indicating significant tendencies towards hallucination.

Abstract

LLM Evaluation

Explore similar work

When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring

Beyond Model Bias: Cross-Entropy Based Rankings in Literary and Academic Texts Remain Stable across Small Language Models

surprisal is Not a Theory