Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict

Sourabrata Mukherjee; Sunayana Sitaram

cs.AISep 26, 2026

Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict

Authors: Sourabrata Mukherjee, Sunayana Sitaram

Organizations: Microsoft Research

Abstract

LLM judges are widely used to evaluate model outputs, but their verdicts can be unreliable: a judge may favor the worse answer for its position, length, or other surface features. When a judge is wrong, is the information needed to judge correctly absent from the model, or present in its internal representations but not reflected in the output? We study this across 64 open-weight evaluators and 14 datasets, including causal interventions on 41 judges (editing activations mid-run to see whether the verdict changes). On LLMBar, built so the superficially better answer is the worse one, the verdicts of 50 judges agree with human labels only 0.456 of the time, even after averaging both answer orders. Yet a small probe on the same judges' activations, with no weight updates, reaches 0.846, and 0.686 once surface features such as length and position are residualized out (0.507 with shuffled labels). The gap holds across eight benchmarks and model families, but is not universal: a score of how well surface features alone predict the human label, computed before any probe is trained, predicts the size of the gain (Spearman rho = 0.90). On rubric tasks that score one answer at a time, leaving no surface cue to exploit, reading the internals gives no advantage. The interventions also show that editing activations mid-network already changes the verdict, before it can be read off directly, and locate the pathways carrying position and length bias. At the same label budget, the recovered signal lets a judge flag cases where it is likely wrong and yields better labels for preference learning. A wrong verdict, then, does not mean the judge lacks the information, and a simple diagnostic shows when it is worth recovering.

Figures & tables

Panel of Figure 1 — Figure 1: A judge’s rich internal computation reaches the user through an output stage that exposes only a verdict. On adversarial LLMBar the verdict agrees with humans below chance ( $.456$ , mean over $50$ judges), while a small probe reading the same frozen activations reaches $.846$ . The four questions below organize the paper.

Work	Internal	Causal	Surface	Predicts	Downstream	Judge	Scale
	read-out	localization	controls	when useful	use	format	(models)
Maiya et al. (2025)	✓	–	pos. only	–	–	both	$4$ families
Lai et al. (2025)	✓	–	–	–	data sel.	point-wise	$6$
Girrbach et al. (2025)	✓	–	–	–	BoN, routing	point-wise	$5$
Li et al. (2026)	✓	–	–	–	data sel.	point-wise	$4$
Feldhus et al. (2026)	(✓)	✓	–	–	–	point-wise	$3$ families

Table 1: Positioning of our work against the four closest lines of prior work and one concurrent one. “Causal localization”: forward-pass interventions that change the emitted output; “surface controls”: the advantage is tested after removing length and position cues; “predicts when useful”: a diagnostic computable before the read-out is fitted, validated on an independent run, and including a negative case; “downstream use”: empirically demonstrated applications (BoN: best-of-

N

); only our work uses the recovered preferences as training labels (DPO) and to abstain on likely errors. Judge format: pairwise compares two answers in one prompt, point-wise scores one at a time. A dash marks an axis outside that paper’s scope; (✓) marks an appendix-level result. Extended discussion: App. A.11 .

Figure 2: The internal–output gap and the statistics that have to hold for it. (a) Reader hierarchy on LLMBar-adversarial (

50

judges, bootstrap

95\%

CIs): the raw verdict is below chance, the surface-only baseline is high by construction here (§ 6 ), and the internal read-outs survive surface residualization (hatched) well above the shuffled-label control. (b) Per-benchmark advantage bootstrapped over judges and over model families; clustering widens the intervals but leaves

7/8

excluding zero, the exception being hh-rlhf, the lowest-cf benchmark, where § 6 predicts the effect should vanish. (c) The diagnostic orders the gain; triangles are an independent replication seed. (d) Per judge, a residual patch alters the emitted verdict before a lens can decode it; causal decision depth is the onset of intervention sensitivity defined in § 3 .

Bench.	cf	raw	o - avg	probe	resid	$\Delta_{\text{r}}$	$\delta_{\text{o}}$
rb2	.96	.620	.684	.840	.907	$+.22$	$+.16$
llmbar	.80	.483	.456	.758	.846	$+.28$	$+.30$
rb-easy	.77	.690	.815	.868	.879	$+.18$	$+.05$
mtbench	.62	.626	.697	.739	.756	$+.11$	$+.04$
arena	.59	.566	.616	.623	.631	$+.06$	$+.01$
judgebench	.52	.516	.545	.540	.548	$+.02$	$-.01$

Table 2: (a) The evaluator ladder: mean accuracy against human labels over

50

judges, ordered by the surface-confound diagnostic cf , with

\Delta_{\text{r}}\!=\!

probe

-

raw and

\delta_{\text{o}}\!=\!

probe

-

order-average, the probe’s non-redundant value; the footer gives the eight-benchmark mean on seed

0

and on the independent replication seed

1

, and the number of benchmarks whose family-clustered CI for the gain over raw excludes zero. (b) Headline quantities of the forward-pass causal suite of § 5 , seed-stable across seeds

0/1/2

and measured inside a live forward pass. Amber marks the high-confound regime and the three central mechanistic numbers; green marks the best reader per row, the per-judge depth ordering, and the surrogate validation. cf is a diagnostic of where to read internals (§ 6 ).

Figure 3: Four consequences of reading internals, at a matched human-label budget. (a) The probe’s edge over output-level fixes is confined to confounded cells and vanishes, or reverses, on clean ones. (b) Its DPO labeling advantage grows as labels get scarce. (c) Ablating the position-bias heads removes much of the bias, also on unseen data. (d) Internal confidence supports better abstention than the judge’s own verdict margin, which on LLMBar is below chance. Full ten-panel version: Fig. 6 .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Section	What it is for	Tables and figures
§ A.1 Additional Definitions	Definitions of dimensional collapse and of the concept directions used in steering and ablation.	–
§ A.2 Claim–Evidence Matrix	Every load-bearing claim against the evidence for it.	Tab. 5 : claim, evidence, causal or not, scale, and the claims we do not make.
§ A.3 Main-Paper Figure and Table Guide	Panel and column definitions for Figs. 2 – 6 and Tabs. 2 a, 3 a and 3 b.	Tab. 6 : what every panel and column means.
§ A.4 Additional Figures	Companion figures to the main results.	Fig. 4 : where the decision forms. Fig. 5 : the read-out end to end. Fig. 6 : the ten-panel version of Fig. 3 .
§ A.5 Model Registry and Dataset Sheet	The 64 models and 14 datasets, with roles, sizes, and provenance.	Tab. 7 : all 64 models by role. Tab. 8 : instruments, controls, compute. Tab. 9 : the 8 pairwise benchmarks. Tab. 10 : the 6 auxiliary sets.
§ A.6 Deployable Read-Out Recipe	The read-out written out as runnable steps, with no weight updates.	Alg. 1: cache the residual stream, order-average, fit the probe.

Appendix

Table 4: Guide to the appendix: section purposes, and what each table and figure shows.

Claim	Evidence	Causal?	Scale	Where
Preference is decodable from frozen activations when the verdict is wrong	CV probe, $.456\!\to\!.758/.846$	no (L1)	50 judges, LLMBar-adv	§ 4
That signal is not only surface	9-dim residualization, length balancing, shuffle control	no (L1)	50 judges	§ 4
The gap generalizes	ladder, $8/8$ benchmarks, $89\%$ of cells	no	$400$ cells	§ 4
… and is not an artifact of treating judges as independent	family-clustered bootstrap; $7/8$ CIs exclude $0$ ; positive in $21/21$ families	no	$21$ families	§ 4
Intervention changes the verdict from mid-stack	residual patching, steering	yes (L2)	$41$ judges	§ 5
… before it is lens-decodable	paired per-judge test, $34/39$ , $p\!=\!2{\times}10^{-7}$	yes (L2)	$39$ judges	§ 5

Appendix

Table 5: Claim–evidence matrix for the main paper.

Panel	What it shows, and how it was computed
Fig. 4 : where the decision forms
–	A forward-pass residual patch places the causal decision at $0.58$ of depth, before the lens first decodes the verdict; per judge this ordering holds on $34/39$ judges (Fig. 2 d). On easy items the verdict is linearly readable without extra fitting; on adversarial items the untrained read-out is wrong at every depth, which is why the deployable reader is a trained probe.
Fig. 5 : the read-out, end to end (50 judges $\times$ 8 benchmarks)
(a)	A small-label lens probe (blue) and a residual-stream probe (green) exceed the stated verdict (rose) on all 8 benchmarks, mean $+0.12$ , every one significant under FDR-BH; the margin is largest where the verdict collapses to chance. Training-free order-averaging (sand) beats raw on $7/8$ .
(b)	Selective prediction: abstaining on the lowest-confidence items raises accuracy monotonically as coverage falls. At $25\%$ coverage accuracy reaches $0.91$ – $0.98$ .
(c)	Over all $400$ judge $\times$ benchmark cells, absolute probe accuracy rises with judge quality (slope $0.56$ ) and $89\%$ of cells sit above $y\!=\!x$ , so the weak-judge gain is range compression rather than headroom.

Appendix

Table 6: Panel and column guide for the main-paper figures and tables.

Figure 4: Where the decision forms. A forward-pass residual patch places the causal decision at

0.58

of depth, before the lens decodes the verdict. On easy items the verdict is linearly readable without extra fitting; on adversarial items the untrained read-out is wrong at every depth, which is why the deployable reader is a trained probe. Per-judge pairing of the two depths: Fig. 2 d.

Figure 5: The read-out end to end, over 50 judges and 8 benchmarks. (a) Both probes beat the stated verdict everywhere. (b) Abstaining raises accuracy monotonically. (c) Not a ceiling artifact. (d) The gain is predicted by cf (

\rho\!=\!0.90

). (e) At

256

labels the frozen probe recovers about half of the mean LoRA gain with zero gradient steps. Panel guide: Table 6 .

Figure 6: Downstream consequences of the internal read-out, the full ten-panel version of Fig. 3 . (a) All arms given the same number of labels; probe best. (b) Its edge over output-level fixes is confined to confounded cells. (c) Probe labels train a better DPO policy on four of five cells. (d) That gain grows as labels get scarce. (e) Against a full -parameter fine-tune of a 24B judge, holding safety where that fine-tune regresses. (f) Human-label-predictive signal survives surface residualization (the surface-only bar is high by construction on this adversarial set). (g) Ablating the bias heads removes the bias, also on unseen data. (h) Stacked mid-layers beat either component. (i) Internal confidence supports better abstention than the judge’s own verdict margin. (j) The measured gains decline with

\mathrm{cf}

. Panel guide: Table 6 .

Family	Models (sizes as loaded)
Role A, the flagship adversarial panel: 50 generative judges.
The forward-pass causal suite intervenes on the 41 of these that expose
hookable decoder blocks within our memory budget; all 50 are read out.
Qwen2.5	0.5b, 1.5b, 3b, 7b, 14b, 32b
Qwen3	8b, 14b, 30b-a3b, 32b
Llama-3	3.2-1b, 3.1-8b, tulu-3-8b

Appendix

Table 7: The full model registry, all 64 models by role. 50 panel judges

+

5 read-out-only large judges

+

9 scalar reward models

=

64 , spanning

0.36

B–

405

B and

20{+}

families. The forward-pass causal suite intervenes on 41 panel judges and 5 reward models; every other registry entry is read out but not intervened on.

63

of the

64

are loadable locally.

Component	What we use
Cached read-out surrogates
Lenses	logit lens; tuned lens (learned affine per layer)
Attribution	direct logit attribution over heads and MLPs
Patching	activation patching, path patching, causal scrubbing
Features	Gemma-Scope sparse autoencoders
Probes	$L_{2}$ -logistic, 5-fold CV, shuffled-label control

Appendix

Table 8: Instrument, control, and compute inventory for § 3 . Cached surrogates are validated against the forward-pass suite, which changes the actual generation.

Dataset	Label / form	Role in this paper
LLMBar (natural, adversarial )	human pairwise, surface decoupled from quality	flagship confounded regime; the verdict falls below chance here
RewardBench 2	human pairwise	highest confound score ( $\text{cf}\!=\!.96$ ); largest probe gain
RewardBench (easy)	human pairwise	mid-confound control
JudgeBench	human pairwise, length-neutral	the null : no surface cue, no probe gain
MT-Bench (human)	human pairwise	human–human agreement $0.646$ (inter-annotator, not a label ceiling)
Chatbot Arena	$55$ k human preferences	scale / in-the-wild preference

Appendix

Table 9: Dataset sheet for all

14

sets. The first eight are the pairwise-preference benchmarks of the evaluator ladder; the last six are the auxiliary sets (Table 10 ). Amber marks the confounded regime where the internal read-out wins most; green marks the two Indic rubric benchmarks. All fourteen datasets are existing, previously published resources; none is introduced by this work. Twelve are public, as is Pariksha ( Watts et al., 2024 ) . Samiksha ( Hamna et al., 2026 ) is not openly redistributable; researchers who need it should contact its authors, who can share it under the terms it was collected under. We release the schema, rubric definitions, language coverage, evaluation prompts, and every result computed from it.

Auxiliary set	exps.	Result reported in this paper
Samiksha	$31$	Out-of-domain confirmation over $50$ judges ( $n\!=\!80$ ): emitted score $\rho\!=\!-0.006$ , held-out best layer $-0.005$ , probe $+0.017$ ; best layer beats final on $52\%$ of judges, matching the $50\%$ the confound account predicts
Pariksha	$31$	Out-of-domain confirmation over $44$ judges ( $n\!=\!80$ ): $+0.013$ / $+0.003$ / $-0.053$ ; best layer beats final on $50\%$ of judges
BiGGen-Bench	$27$	Rubric-format control: standalone rubric scoring commits at $0.9$ – $1.0$ of depth, latest of any format
HelpSteer2	$27$	Multi-attribute rank-collapse measurements (§ 5 )
m-RewardBench	$23$	Cross-lingual direction geometry and transfer
UltraFeedback	$20$	Preference pool for the DPO labeling experiment (§ 7 )

Appendix

Table 10: The six auxiliary datasets. These do not enter the evaluator ladder, which needs pairwise labels and a length-only confound score, but each is used in

20

–

31

mechanism experiments. Green marks the two Indic rubric benchmarks, which supply the out-of-domain confirmation of § 6 .

Reader (256-label)	Mean	llmbar	judge	pku
Raw verdict	$0.550$	$0.437$	$0.547$	$0.667$
Probe (0 updates)	$0.634$	$0.774$	$0.537$	$0.592$
LoRA / PEFT fine-tune	$\mathbf{0.724}$	$0.881$	$0.537$	$0.753$

Appendix

Table 11: Reading the same frozen judge vs. training, given the same 256 human labels (4 judges

\times

3 benchmarks). The no-update probe (green) recovers about half of the mean LoRA/PEFT gain and most of it on confounded llmbar (amber); against a full -parameter fine-tune the probe wins outright (main Table 3 b).

Bench	AUC int.	AUC verd.	$\Delta$ / % win
llmbar	0.822	0.431	$+.391$ / 100%
rb2	0.921	0.744	$+.176$ / 98%
rb-easy	0.939	0.876	$+.063$ / 74%
mtbench	0.823	0.780	$+.042$ / 48%
pku-safe	0.697	0.677	$+.020$ / 56%
arena	0.671	0.667	$+.004$ / 32%

Appendix

Table 12: Risk–coverage AUC: internal confidence vs. verdict margin. On llmbar the verdict-margin AUC is

0.431

(below chance): the judge’s own confidence is anti -correlated with correctness.

Bench.	$\rho$	by verdict	by internals
rewardbench2	.58	skywork-crit-8b	unchanged
llmbar-adv	.20	skywork-crit-8b	mistral-sm-24b
rb-easy	.71	qwen2.5-14b	qwen2.5-32b
mtbench	.54	phi-4	ds-r1-qwen-32b
arena	.71	mistral-sm-24b	gemma-3-27b
judgebench	.32	qwen2.5-32b	ds-r1-qwen-32b

Appendix

Table 13: Deployment re-ranking: Spearman

\rho

between ranking the 50 judges by emitted verdict and by internal read-out, with the top-ranked judge under each. Rows follow the confound ordering of Table 2 . The best judge to deploy changes on 7 of 8 benchmarks and the two rankings agree only moderately (mean

\rho\!=\!0.55

). Agreement is lowest on LLMBar-adversarial (

0.20

), where the emitted verdict is least trustworthy, and also low on JudgeBench (

0.32

), where near-ties among judges make the ranking unstable under either view (amber). RewardBench-2 is the one benchmark whose top judge is stable. Names abbreviated (ds-r1

=

deepseek-r1, sm

=

small, crit

=

critic).

Arm	fwd/item	grad. steps	params upd.	stored
raw verdict	$1$	$0$	$0$	–
order-average	$2$	$0$	$0$	–
self-consistency ( $k{=}5$ )	$5$	$0$	$0$	–
calibrated prompt	$1$	$0$	$0$	–
lens probe	$2$	$0$	$41$	$164$ B
residual-stream probe	$2$	$0$	$49$	$\approx\!1$ MB

Appendix

Table 14: Cost of each arm when all arms are given the same

3{,}349

human labels in § 7 . Parameter counts are for Mistral-Small-24B (

40

layers, width

5120

); the lens probe learns one weight per layer plus a bias. Green marks the arms that touch no model weights; amber the arm they are compared against. This is an analytic accounting, not a wall-clock benchmark.

Property (behavioral probe)	Measure	Value
Bias & safety pathologies
Position-bias flip rate	swap-flip frac.	$0.95\text{--}1.0$
Swap non-invariance	$1-$ agreement	up to $1.0$
Eval-awareness shift	lens $\sigma$	$2.0\text{--}5.6$
Self-preference (cross-gen)	win rate	$0.553$
scaling with size	Spearman	$+0.63$

Appendix

Table 15: Behavioral pathologies the mechanism explains, with operational units. The last block are internal repairs.

Bench.	cf	Measure	Abstain	Layers	Budget	Label
	diagnostic	acc.	AUC	content	budget	DPO
rewardbench2	.96	$+.22$	$+.18$	$+.12$	$+.09$	$+.01$
llmbar-adv	.80	$+.28$	$+.39$	$+.17$	$+.31$	$+.33$
rb-easy	.77	$+.18$	$+.06$	$+.04$	–	–
mtbench	.62	$+.11$	$+.04$	$+.02$	–	–
arena	.59	$+.06$	$+.00$	$+.01$	$-.03$	–

Appendix

Table 16: One score tracks the measured gains. The measurement gain ( Measure , probe

-

verdict accuracy), selective-prediction advantage ( Abstain , risk–coverage AUC

\Delta

), mid-layer content ( Layers ), same-label advantage over cheap fixes ( Budget ), and downstream DPO gain ( Label ) all track the surface-confound diagnostic cf and shrink to

\approx 0

on length-neutral sets. Amber rows are the confounded regime where every column peaks (green); the three full columns span all

8

benchmarks, while Budget / Label were run on 4- and 2-benchmark grids (

\text{--}=

not in that grid).

bench $\cdot$ judge	pol.	verd.	probe	gold	$\Delta$
llmbar $\cdot$ q7b	1.5b	0.463	0.787	0.825	$+.325$
llmbar $\cdot$ q7b	3b	0.625	0.775	0.863	$+.150$
rb2 $\cdot$ q7b	1.5b	0.906	0.931	0.960	$+.026$
rb2 $\cdot$ g9b	1.5b	0.940	0.951	0.963	$+.011$
rb2 $\cdot$ q7b	3b	0.923	0.934	0.963	$+.011$
rb2 $\cdot$ 3b label-efficiency (pool $\to$ $\Delta$ probe $-$ verd.)

Appendix

Table 17: Probe- vs. verdict-labeled DPO, gold-test accuracy. The gain is largest on confounded llmbar (verdict-DPO below chance) and, on rb2, grows as the labeled pool shrinks.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 529 papers

Sep 29, 2026

JudgeProfile: Understanding and Steering Subjectivity in LLM Judges

LLM judges are inherently subjective, often favoring different responses in pairwise comparison when neither option is objectively wrong. To study this subjectivity, we introduce JudgeProfile, a framework that dissects LLM evaluation into perception (how a judge compares two responses across specific attributes like clarity, correctness, and detail) and prioritization (how much each attribute influences the final choice). We curate SubjectiveSet, a dataset of 50,013 response pairs from 17 public data sources, evaluated by 21 LLM judges across 87 attributes. We find a hidden consensus in perception: judges frequently agree on attribute judgments even when their overall choices diverge. Building on this separation, we first characterize each judge's prioritization using attribute weights estimated from its own overall choices. These weights differ across judges even when estimated from the same attribute judgments. We then learn new weights from reference labels to adapt their decisions to a target evaluation standard. Reweighting perceived attributes improves average held-out agreement with reference labels from 66.48% to 71.97%, outperforming fine-tuning and rubric prompting. Our findings show that understanding and steering the subjectivity of LLM judges requires attention not only to what they perceive, but also to how they prioritize it.

Qi Cao, Kangning Liu, Xuan Kan +10

Meta · University of California San Diego · The University of Hong Kong +1

Human Preference Evaluation LLM Evaluation LLM-as-a-Judge Preference Learning

Ask PDF

Oct 5, 2026

Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations

The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judges and Llama-3.1-8B. The linear probes achieve AUROCs of .621-.850 and outperform a combined baseline that uses verbalized confidence, verdict-label logits, response lengths, and the judge's initial choice by .062-.113 AUROC. Linear probes trained on JudgeBench and then frozen achieve AUROCs of .685-.853 on 1,802 MT-Bench comparisons without MT-Bench fitting or recalibration. These results show that pre-verdict activations support prediction of susceptibility to candidate order and outperform the non-activation predictors evaluated here.

Hashmath Shaik, Gnaneswar Villuri, Alex Doboli

Department of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11794, USA

LLM Evaluation LLM-as-a-Judge Bias in LLM-as-a-Judge Language Model Probing

Ask PDF

Oct 4, 2026

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.

Laurène Vaugrante, Thilo Hagendorff

AI Safety Research Group, University of Stuttgart

LLM Evaluation LLM-as-a-Judge LLM Reliability

Ask PDF

Abstract

LLM-as-a-Judge

Explore similar work

JudgeProfile: Understanding and Steering Subjectivity in LLM Judges

Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models