Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring

Wonjeong Jo; Gyuwan Kim; Ming Shan Hee

cs.CLOct 15, 2026

Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring

Authors: Wonjeong Jo, Gyuwan Kim, Ming Shan Hee

Organizations: UC Davis · UC Santa Barbara · MBZUAI

Abstract

Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied. To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. Sensitivity is greatest for toxicity, the only unipolar construct examined. Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.

Figures & tables

Figure 1: Experimental design for evaluating scale sensitivity.

Construct	Data Source	Sampled $N$
Political leaning	MBIC (Media Bias Annotation Dataset) Spinde et al. (2021)	1,700
Sentiment	VADER movie reviews Hutto and Gilbert (2014)	5,000
Toxicity	Jigsaw Unintended Bias in Toxicity Classification cjadams et al. (2019)	5,000

Table 1: Data sources and sample sizes used in the study. Sample size refers to the number of items extracted from each original dataset and analyzed in our study.

Format	Sign	Scale bounds	Range
Continuous	Signed	$[-1,+1]$	3
		$[-2,+2]$	5
		$[-5,+5]$	11
		$[-10,+10]$	21
	Unsigned	$[1,3]$	3
		$[1,5]$	5

Table 2: The 16 rating scale conditions tested in the study.

Figure 2: Scale sensitivity by rating scale design element. Dots show the estimate for each combination of task, prompting strategy, and model, colored by task. Black lines are 95% item bootstrap intervals.

Figure 3: (a) Format sensitivity on scales with different rating ranges. (b) Range sensitivity for each pair of ranges. Bars are 95% item bootstrap intervals.

Figure 4: Scale sensitivity across tasks, prompting strategies, and models (RQ2–RQ4), averaged across the remaining tasks, prompting strategies, models, and minimal pairs. The top row shows numerical sensitivity,

SS_{\mathrm{MAE}}

, and the bottom row shows semantic sensitivity,

SS_{\mathrm{switch}}

. Error bars indicate 95% item bootstrap confidence intervals.

Figure 5: Share of the variation in scale sensitivity explained by each factor. Black lines are 95% item bootstrap intervals. The full decomposition is reported in Table 7 in Appendix F .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Dimension and scope	Low-end evidence	High-end evidence
Political leaning	PC Party, candidate, movement, or political leaning. Political actors and groups.	Supports Democrats, liberals, or progressives, or criticizes conservatives from the left.	Supports Republicans, conservatives, MAGA, libertarians, or nationalists, or criticizes liberals from the right.
	EP Economic policy and redistribution. Economy, labor, taxation, markets, government intervention.	Supports redistribution, welfare, labor rights, progressive taxes, universal healthcare, or corporate regulation.	Supports lower taxes, deregulation, free markets, privatization, welfare cuts, limited government, or individual responsibility.
	SC Social and cultural values. Religion, immigration, reproductive and LGBTQ+ rights, diversity, secular governance.	Supports reproductive or LGBTQ+ rights, immigrant inclusion, multiculturalism, diversity, secularism, or progressive change.	Supports traditional or religious values, abortion restrictions, stricter immigration, assimilation, or opposition to progressive policies.
	GL Government authority, law, order, and security. State power, policing, justice, civil liberties.	Supports police accountability, justice reform, protest rights, civil liberties, rehabilitation, or limits on coercion.	Supports strong policing, punishment, enforcement, border control, military strength, surveillance, or public order.
	IG Identity, inclusion, and group-based equality. Discrimination, minority rights, systemic inequality, DEI.	Supports anti-discrimination, affirmative action, minority rights, systemic inequality claims, DEI, or marginalized groups.	Opposes affirmative action or DEI, or emphasizes meritocracy, colorblindness, or skepticism about systemic inequality.
Sentiment	Content Evaluation . Overall evaluation of the film.	Film portrayed as poor, frustrating, disappointing, weak, or not worth watching.	Film portrayed as enjoyable, successful, valuable, exceptional, or highly satisfying.

Appendix

Table 3: Condensed marking rubrics. Each rubric in the original prompt also has a middle column for neutral, mixed, or unclear evidence (ambiguous or moderate evidence for toxicity), which we omit here. The full prompts are presented at https://anonymous.4open.science/r/Scale-Sensitivity-C418/

Points	Political leaning	Sentiment	Toxicity
3	left, neutral, right	negative, neutral, positive	not, moderately, extremely toxic
5	far left, left, neutral, right, far right	very negative, negative, neutral, positive, very positive	not, slightly, moderately, strongly, extremely toxic
11	far / moderate / slight left, neutral, slight / moderate / far right	extremely / moderately / slightly negative, neutral, slightly / moderately / extremely positive	not, slightly, somewhat, moderately, considerably, strongly, extremely toxic
21	far / strong / moderate / slight left, neutral, slight / moderate / strong / far right	extremely / strongly / moderately / slightly negative, neutral, slightly / moderately / strongly / extremely positive	not, minimally, slightly, somewhat, moderately, considerably, strongly, severely, extremely toxic

Appendix

Table 4: Labels in ordinal scale instructions, ordered from lowest to highest score.

Model	Reasoning	Temp.	Top- $p$	Top- $k$
GPT 5.6 Luna	low	default	default	default
Gemini 3.5 Flash Lite	minimal	default	default	default
Qwen3.8 27B	low	1.0	0.95	20
DeepSeek V4 Flash	low	1.0	1.0	default
GLM 5.3 Flash	low	1.0	0.95	default

Appendix

Table 5: Generation settings for LLM scoring inference.

Figure 8: Cumulative distribution of normalized scores for each scale. Solid lines show the number of scale points, from light (3 points) to dark (21 points). The gray dashed line repeats the distribution of the human labels for that task in every panel. Dotted vertical lines mark the category boundaries at 1/3 and 2/3.

Figure 9: Share of texts in the low, middle, and high categories for each scale. Each row represents a task. Percentages are displayed for segments exceeding 8%.

Figure 10: Scale sensitivity at three reasoning levels for the three open models. The black line is the average over the three models. Bars are 95% item bootstrap intervals from 2,000 resamples.

Figure 11: Agreement between LLM scores and human labels for every model, prompting strategy, and rating scale condition.

		$SS_{\mathrm{MAE}}$		$SS_{\mathrm{switch}}$
Moderator	Design element	$b$	95% CI	$b$	95% CI
Task
Sentiment	Sign	$-0.006$	[ $-0.007$ , $-0.005$ ]	$+0.006$	[0.003, 0.008]
	Range	$+0.005$	[0.005, 0.006]	$-0.001$	[ $-0.003$ , 0.001]
Toxicity	Sign	$+0.029$	[0.028, 0.030]	$+0.080$	[0.078, 0.084]
	Range	$-0.031$	[ $-0.032$ , $-0.030$ ]	$-0.051$	[ $-0.053$ , $-0.049$ ]

Appendix

Table 6: Moderation of scale sensitivity by task, prompting strategy, and model. Entries are interaction coefficients

b

from weighted least squares regressions of the 1,200 minimal pair estimates, fitted separately for each metric. Brackets are 95% item bootstrap intervals. Intervals that include zero indicate no detectable moderation.

		$\eta^{2}$ (%)
Component	df	$SS_{\mathrm{MAE}}$	$SS_{\mathrm{switch}}$
Scale design	39	40.3	8.9
Design element changed	2	1.2	0.4
Format of compared scales	2	13.6	4.6
Sign of compared scales	2	3.2	1.6
Points or range pair	11	17.0	1.6

Appendix

Table 7: Full variance decomposition of the 1,200 minimal pair estimates (Figure 5 ). Shares come from the weighted least squares regression with all interactions between two factors and sum to 100% for each metric. Subcomponents may not add exactly to their total because of rounding. All 95% item bootstrap intervals lie within 1.2 percentage points of the estimates.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 789 papers

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias
Mar 31, 2026Filip J. Kucia, Anirban Chakraborty, Anna WróblewskaAutomated Essay Scoring Language Model Bias Evaluation
Faculty of Mathematics and Information Science, Warsaw University of Technology, 00-662 Warsaw, Poland · University of Wolverhampton, UK
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation
Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLLM Evaluation LLM-as-a-Judge
Japan Advanced Institute of Science and Technology

Oct 4, 2026J/K move · Enter open · S save

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Laurène Vaugrante, Thilo Hagendorff

AI Safety Research Group, University of Stuttgart

LLM Evaluation LLM-as-a-Judge LLM Reliability

Open paper AskPDF arXiv

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.

Abstract

LLM Reliability

Explore similar work

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias

Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation