cs.CLFeb 17, 2026

What Is Missing: Interpretable Ratings for Large Language Model Outputs

Authors: Nicholas Stranges, Yimin Yang

Organizations: Department of Electrical and Computer Engineering, Western University, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Current Large Language Model (LLM) preference learning methods such as Proximal Policy Optimization and Direct Preference Optimization learn from direct rankings or numerical ratings of model outputs, these rankings are subjective, and a single numerical rating chosen directly by a judge is a poor proxy for the quality of natural language, we introduce the What Is Missing (WIM) rating system to produce rankings from natural-language feedback, WIM integrates into existing training pipelines, can be combined with other rating techniques, and can be used as input to any preference learning method without changing the learning algorithm, to compute a WIM rating, a human or LLM judge writes feedback describing what the model output is missing, we embed the output and the feedback with a sentence embedding model and compute the cosine similarity between the resulting vectors, we empirically observe that, compared to discrete numerical ratings, WIM yields fewer ties and larger rating deltas, which improves the availability of a learning signal in pairwise preference data, we use interpretable in the following limited sense: for each scalar rating, we can inspect the judge's missing-information text that produced it, enabling qualitative debugging of the preference labels.

Figures & tables

Figure 1: Flowchart of the WIM method. An LLM produces a natural language output s1s_{1} . s1s_{1} is then evaluated by a human or an LLM judge. The judge’s goal is to produce s2s_{2} , a response containing what is missing in s1s_{1} . Both s1s_{1} and s2s_{2} are passed through a sentence embedding model to produce high dimensional vectors S1S_{1} and S2S_{2} . The similarity of S1S_{1} and S2S_{2} is calculated using cosine similarity and the resulting similarity score is the WIM rating. A higher similarity between S1S_{1} and S2S_{2} implies that there is less missing from the LLM’s output. nn model outputs are rated by the WIM method and then sorted to produce a ranking. The ranking of the outputs is then passed to a preference learning algorithm.
Figure 2: 2D visualization of missingness
Figure 3: Histogram of ratings from the Numerical rating system and the WIM rating system
Figure 4: Histogram of rating deltas from the Numerical Rating System and the WIM Rating System
MethodAverage Delta
Numerical0.928
WIM1.396
Table 1: Average rating delta per judging pair
MethodLoss Difference
Random-0.0011
Numerical-0.0020
WIM Changing Judge-0.0033
WIM Fixed Judge-0.0059
Table 2: Loss difference through training
MethodEntropy Difference
Random-61.27
Numerical-45.3
WIM Changing Judge-53.08
WIM Fixed Judge-106.94
Table 3: Model mean entropy change after training
Figure 5: Reward advantage trajectories
MethodWin Rate
Random49.9%
Numerical50.1%
WIM Moving Judge51.3%
WIM Fixed Judge52.0%
Table 4: Comparison of win rates
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

MethodBBHGPQA Diamond ZeroshotIfEvalMMLU
Base Model67.84%30.81%40.48%63.82%
Random67.72%27.78%39.74%63.89%
Numerical67.98%28.28%39.93%63.90%
WIM Changing Judge68.01%28.28%39.19%63.88%
WIM Fixed Judge68.15 %29.80%39.56%63.89%
Appendix
Table 5: Benchmark performance
Figure 6: Comparison of the chosen and rejected reward trajectories for the Numerical Rating System
Figure 7: Comparison of the chosen and rejected reward trajectories for the WIM Changing Judge Rating System
Figure 8: Comparison of the chosen and rejected reward trajectories for the WIM Fixed Judge Rating System
ParameterValue
rr16
α\alpha16
Target Modulesq_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Dropout0.0
Biasnone
Appendix
Table 6: LoRA hyperparameters
ParameterValue
α\alpha0.6
kk4
Appendix
Table 7: Contrastive search hyperparameters
ParameterValue
β\beta0.1
Temperature0.9
LossSigmoid
Log Probability Cutoff256 Tokens
Appendix
Table 8: ODPO hyperparameters
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,401 papers

Explore similar work

CardsList

  1. Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users

    Jun 18, 2026Haw-Shiuan Chang, Jeffrey Gomez, Mehul Patwari +2Reward ModelingLLM Alignment

    University of Massachusetts, Amherst, USA · York University, Canada

  2. Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

    Jul 6, 2026Robert Morabito, Tyler McDonald, Charitra Viswanath +4Human Preference EvaluationLLM Evaluation

    Brock University · Emory University · University of Southern California +2

  3. Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

    Aug 16, 2025Jenny Y. Huang, Yunyi Shen, Dennis Wei +1Pairwise Preference LearningLLM Evaluation

    Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research