Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement

Harry Lyu; Neil Thompson

cs.AIOct 1, 2026

Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement

Authors: Harry Lyu, Neil Thompson

Organizations: Massachusetts Institute of Technology

Abstract

LLM judges are increasingly used to assess whether AI outputs meet workplace requirements, but agreement on response rankings does not establish agreement on acceptance rates or occupational aggregates. We introduce O*NET-BENCH, an audit suite derived from an existing survey of 45,796 worker ratings, and evaluate 33 pre-existing judge configurations across six model families on 4,501 test ratings. Twenty-five configurations achieve tie-aware pair accuracy of at least 0.60, although a train-fitted response-only TF-IDF baseline nearly matches the strongest judge. Despite this ordering agreement, judges estimate that 3.0%-97.9% of responses are acceptable, compared with 61.1% for occupation-matched workers. In one fine-tuned lineage, changing from pointwise scoring to a bundled few-shot/listwise protocol improves response ordering while reducing agreement with worker means at the task and occupation levels; this reversal replicates on a task- and worker-disjoint validation split under prespecified criteria. Cross-validated calibration largely removes mean bias, but calibrated scores explain at most 8.5% of individual worker-rating variance. Prediction-assisted estimation yields at most small precision gains at the studied label budgets. These results show that ranking agreement alone is insufficient for occupational measurement. Judges should be validated against the acceptance rates and aggregates their scores will be used to estimate.

Figures & tables

Figure 1: The scale problem. (a) A real pair of test-split task instances, responses sorted by worker score: the judge reproduces every within-task ordering exactly, yet the task means (horizontal markers, joined by dashed lines) move oppositely and Task A’s majority verdict flips. (b) Test-split score distributions: judges reproduce neither the location nor the shape of the worker distribution; parenthesized values are mean bias. (c) The quantity these evaluations report, the share judged acceptable at threshold

\tau

, depends on the grader; judge curves are steep near the acceptance threshold, so small location errors become large level errors.

Component	Main measures	Evaluation level
Ordering agreement	Tie-aware pair accuracy, coverage, $\rho$	response; within-instance pairs
Score-level agreement	Mean bias, SD ratio, acceptance share	response
Aggregate agreement	Cell-mean corr., classification disagreement	task; occupation; family
Calibration and label efficiency	CV bias and $R^{2}$ ; PPI precision, coverage	response; pooled

Table 1: Audit overview. All test-split analyses are exploratory; preregistered validation covers the primary fine-tuned protocol contrast, in its response-ordering and aggregate-agreement directions (§ 5.7 ), and the separate eight-arm severity check was not preregistered.

Figure 2: Cardinal validity across the audit. (a) Mean bias (judge

-

worker) for all 33 configurations, with 95% task-clustered intervals, grouped by model family. Squares are pointwise arms, and the faded row is the partial-coverage reasoning variant. Every interval excludes zero. (b) Tie-aware pair accuracy against mean bias (point size: pair coverage).

Configuration group	$n$	Pair acc.	Bias	$\|$ Bias $\|$	Acc. (score ${\geq}7$ )	Cov.
proprietary pointwise	3	0.595	$-2.50$	2.50	0.117	0.66
few-shot/listwise prompting	8	0.629	$-1.61$	1.61	0.348	0.90
self-consistency	2	0.637	$-1.53$	1.53	0.336	0.93
ensembling	4	0.643	$-1.31$	1.31	0.353	0.92
scaffold prompting	3	0.620	$-0.82$	0.82	0.497	0.89
reasoning/decoding	3	0.643	$-0.74$	0.74	0.495	0.89

Table 2: Medians by configuration group for the 32 near-complete arms (

\geq

99.8% test coverage); acceptance uses scores

\geq 7

. Groups differ in models and protocols, so these are descriptive summaries, not controlled intervention comparisons.

Figure 3: Downstream consequences by threshold

\tau

. (a) Share of responses judged acceptable. (b) Share of occupations with a strict majority acceptable. (c) Share of occupations whose classification disagrees with the sampled worker majority.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Family	Present.	Cover.	Bias [95% CI]	Acc. (score ${\geq}7$ )	PairAcc	PairCov
Claude Opus 4.7 pointwise	Claude	pointwise	1.000	$-3.23$ $[-3.36,-3.11]$	0.030	0.595	0.618
Haiku pointwise	Claude	pointwise	1.000	$-1.41$ $[-1.56,-1.28]$	0.439	0.600	0.709
Haiku fs+lw	Claude	listwise	1.000	$-1.33$ $[-1.43,-1.22]$	0.377	0.632	0.929
GPT pointwise	GPT	pointwise	1.000	$-2.50$ $[-2.63,-2.37]$	0.117	0.568	0.657
GPT fs+lw	GPT	listwise	1.000	$-2.47$ $[-2.60,-2.35]$	0.214	0.601	0.869
GPT-OSS fs+lw	GPT-OSS	listwise	1.000	$-1.49$ $[-1.61,-1.37]$	0.337	0.632	0.855

Appendix

Table 3: Full registry. “Present.” is the presentation protocol; naming tokens are decoded in Table 4 , and column quantities are defined in § 4 . Bias intervals use 1,000 task-clustered bootstrap draws.

Token	Meaning
pointwise / lw	one response per prompt / all responses of a task instance (typically five) scored together in one prompt (§ 3 )
fs	few-shot exemplars in the prompt; 5ex / 10ex give the count where it is varied
rubric, ranked, analogical	alternative prompt scaffolds
think / noThink	hybrid reasoning enabled / disabled
t0.7, t1	decoding temperature (all other arms decode at 0)
5x	five sampled judgments per response (self-consistency), combined by average or majority

Appendix

Table 4: Configuration naming key for Table 3 , Figure 2 a, and Figure 5 .

Split	Ratings	Task IDs	Workers	Occup.	Families	Generators
Train	36,649	4,119	7,339	680	22	47
Validation	4,646	515	931	311	22	47
Test	4,501	515	901	307	22	47

Appendix

Table 5: Dataset splits. Task IDs and workers are disjoint across splits; occupations and response-generator variants overlap.

Figure 4: Split representativeness. (a) Worker manager_accept shares by split against the full corpus: split means are 6.92/7.00/6.97, and the largest per-score share difference between the test split and the full corpus is 0.008. (b) The two configurations run on the full corpus reproduce their test-split distributions (GPT-5.4-mini mean 4.54 versus 4.47; Haiku 5.64 versus 5.56), and both assign a score of 9 to fewer than 0.05% of the 45,796 rows.

Figure 5: Score distributions for the workers (top left) and all 33 configurations on the test split, grouped and colored by model family (Figure 2 b palette; legend below). Bars are shares of ratings on a common axis capped at 0.6; the one clipped bar is break-marked and annotated with its value, and the cased dark line in each judge panel is the workers’ distribution. Panel headers give valid rows and the mean (workers: 6.97); the dashed line marks each panel’s mean. No configuration reproduces the workers’ right-shifted distribution: most judges shift mass left of the worker outline, the pointwise fine-tunes overshoot onto 8–9, and Llama collapses to a near-constant 8.

Figure 6: Conditional distributions of judge scores (columns) given worker scores (rows), normalized within each worker-score row, for eight pointwise configurations on the test split. Panel titles report exact-match rates. The distributions show differences in score placement and concentration; Llama assigns nearly constant scores. Agreement in within-instance response ordering is evaluated separately using the metrics in § 5.1 .

Figure 7: Mean judge score conditional on each worker score for eight pointwise configurations on the test split. The diagonal marks equality between the conditional mean judge score and the worker score. Conditional means vary less than the full worker-score range, and their levels differ substantially across configurations. This descriptive view complements the marginal distributions in Figure 5 and the ordering metrics in § 5.1 .

Pointwise $\to$ few-shot/listwise	$\Delta$ response $\rho$	$\Delta$ task $\rho$	$\Delta$ occ. $\rho$
Qwen base	$+0.110$ [0.077,0.144]	$-0.006$ [ $-0.079$ ,0.072]	$-0.025$ [ $-0.079$ ,0.062]
Haiku	$+0.070$ [0.042,0.096]	$-0.011$ [ $-0.065$ ,0.044]	$-0.007$ [ $-0.058$ ,0.047]
GPT-OSS	$+0.109$ [0.078,0.142]	$-0.003$ [ $-0.078$ ,0.072]	$+0.029$ [ $-0.055$ ,0.090]
GPT	$+0.060$ [0.036,0.085]	$+0.031$ [ $-0.017$ ,0.077]	$+0.013$ [ $-0.026$ ,0.072]
Qwen FT1	$+0.043$ [0.014,0.072]	$-0.116$ [ $-0.207$ , $-0.023$ ]	$-0.145$ [ $-0.224$ , $-0.041$ ]
Qwen FT2	$+0.049$ [0.021,0.078]	$-0.119$ [ $-0.218$ , $-0.022$ ]	$-0.167$ [ $-0.296$ , $-0.043$ ]

Appendix

Table 6: Matched pairs with 95% task-clustered intervals; boldface marks estimates whose intervals exclude zero. Local ordering improves in all six pairs. Statistically clear aggregate declines are confined to the two checkpoints of one LoRA lineage, which differ by a single training epoch and are not independent. The FT2 row reports the canonical preregistered intervals of Table 8 ; the remaining rows use an earlier matched-pair bootstrap draw set.

Presentation	Exemplars	Resp. $\rho$	Task $\rho$	Occ. $\rho$	Resp. $\rho$	Task $\rho$	Occ. $\rho$
		Test			Validation
Pointwise	zero-shot	0.109	0.067	0.075	0.088	0.051	0.007
Pointwise	few-shot	0.138	0.087	0.058	0.124	0.018	$-0.066$
Listwise	zero-shot	0.191	0.087	0.036	0.196	0.092	0.015
Listwise	few-shot	0.219	0.060	0.049	0.200	0.056	$-0.049$

Appendix

Table 7:

2\times 2

on base Qwen3.5-9B, test and validation. Mean response-level effects, computed from unrounded cell means: presentation/schema

+0.082

(test) and

+0.092

[0.068,0.115]

(validation); exemplars

+0.028

and

+0.019

[0.001,0.038]

. Occupation-level factor effects are small and change sign across splits.

Level	Test $\Delta$ [95% CI]	Validation $\Delta$ [95% CI]	Pre-registered threshold
Response $\rho$	$+0.049$ $[+0.021,+0.078]$	$+0.033$ $[+0.003,+0.062]$	$\geq+0.025$
Task $\rho$	$-0.119$ $[-0.218,-0.022]$	$-0.112$ $[-0.185,-0.032]$	$\leq-0.05$
Occupation $\rho$	$-0.167$ $[-0.296,-0.043]$	$-0.108$ $[-0.196,-0.006]$	$\leq-0.10$

Appendix

Table 8: Same-checkpoint protocol reversal (few-shot/listwise minus pointwise, Qwen-FT-2ep, temperature 0), task-clustered intervals. Thresholds were fixed before the validation runs; all three replicate.

Grouping	Worker $V_{G}$	Pointwise $V_{G}$	Listwise $V_{G}$	$\Delta V_{G}$ [95% CI]
Task	.227	.295	.118	$-$ .177 [ $-$ .212, $-$ .143]
Occupation	.145	.207	.078	$-$ .128 [ $-$ .181, $-$ .111]
Job family	.027	.072	.016	$-$ .056 [ $-$ .085, $-$ .041]
Generator	.058	.132	.346	$+$ .214 [.186,.239]

Appendix

Table 9: One-way variance shares for the primary matched comparison (shares overlap; do not sum). Task-clustered bootstrap intervals.

Figure 8: Variance patterns and scope of the protocol reversal (§ 5.7 ). (a) In the primary fine-tuned comparison, the few-shot/listwise scores have smaller task- and occupation-associated variance shares and a larger generator-associated share than the pointwise scores; the one-way shares are descriptive and non-additive. (b) All six matched protocol changes improve response correlation; task-clustered intervals for the occupation-level difference exclude zero only for the two LoRA checkpoints (orange). (c) The primary task- and occupation-level declines persist across the 15 aggregation specifications; horizontal bars mark medians.

Selection criterion	$\|$ Acc. err $\|$ (pct. pts)	$\|$ Bias $\|$	Pair acc.
Pair accuracy	10.5 [7.7, 24.1]	0.66	0.657
Acceptance-share error	5.8 [3.6, 8.3]	0.53	0.639
Mean-bias magnitude	17.3 [14.4, 19.9]	0.12	0.546
Pair accuracy within 5 pts of acceptance	5.8 [3.6, 8.3]	0.53	0.639

Appendix

Table 10: Held-out performance by selection criterion over 1,000 task half-splits (means; brackets are the 2.5–97.5 percentile range of the acceptance error). Selecting for the reported estimand roughly halves the acceptance error at a cost of 0.018 pair accuracy; selecting for mean bias alone is worst on the acceptance share.

Condition	Raw bias	Post-cal. bias	CV $R^{2}$	Residual SD
Qwen-FT-2ep 10ex	$-0.67$	$+0.002$	0.085	1.85
Qwen-FT-2ep fs+lw	$-0.65$	$-0.001$	0.082	1.86
Qwen-FT-1ep fs+lw	$-0.59$	$-0.001$	0.078	1.86
EnsHQF average (H+Q+FT)	$-1.19$	$+0.002$	0.077	1.86
EnsHQF majority (H+Q+FT)	$-1.18$	$+0.001$	0.064	1.87
Qwen-FT-2ep 10ex noThink/t0.7	$-0.74$	$+0.001$	0.061	1.88

Appendix

Table 11: Task-grouped 5-fold cross-validated isotonic calibration, ten highest-

R^{2}

conditions (worker SD: 1.94). Calibration removes location error. The remaining signal explains at most 8.5% of individual-rating variance.

Figure 9: Joint distributions of judge and worker scores (grey, share of rows) for eight pointwise configurations on the test split. Red curves are isotonic fits predicting judge score from worker score; the dotted line marks equal scores. The curves describe how judge scores vary across worker ratings. They are distinct from the cross-validated calibrators in Table 11 , which predict worker ratings from judge scores, and should not be interpreted as those calibrators or as a direct explanation of their $R^{2}$ .

Mean rating
Predictor	5%	10%	25%	50%	Eff. $n$ gain at 25%	$\|$ Bias $\|_{\max}$	Coverage
FT2 pointwise	1.064	1.028	1.018	1.034	-3.4%	0.006	0.95–0.96
FT2 listwise	1.078	1.052	1.042	1.052	-7.9%	0.004	0.95–0.97
Haiku pointwise	1.120	1.079	1.065	1.072	-11.8%	0.013	0.94–0.98
TF–IDF (response only)	1.089	1.055	1.035	1.045	-6.6%	0.009	0.94–0.96
Acceptance (score $\geq 7$ )

Appendix

Table 12: Simulation of task-level cross-fitted prediction-assisted estimation with isotonic calibration, using 500 repetitions per condition. Entries are interval-width ratios relative to human-only estimation; values above 1 indicate wider intervals. The width-based effective-sample-size summary at the 25% label budget is

(w_{\mathrm{human}}/w)^{2}-1

. The final columns report maximum absolute bias and the empirical coverage range of nominal 95% intervals across budgets; human-only coverage is 0.90–0.97. The interval construction includes the heuristic calibrator-fitting term described in Appendix I , so the reported coverage assessment applies to this simulation. Across conditions, empirical estimator SD ratios have median 1.00 and range 0.96–1.08. Judge-only estimates retain biases of

+1.21

-0.65

-1.41

rating points and

+0.29

-0.10

-0.17

in acceptance share.

	$\|$ Acceptance err. $\|$ (pct. pts)		$\|$ Mean err. $\|$ (rating pts)
Calibrator	median	max	median	max
Fold-mean constant (no judge input)	0.0	0.0	0.001	0.001
Raw score, threshold at 7	24.0	58.1	1.223	3.231
Intercept-corrected, threshold at 7	11.2	35.2	0.002	0.003
Affine-calibrated, threshold at 7	11.6	39.6	0.001	0.002
Isotonic mean-calibrated, threshold at 7	10.7	39.6	0.001	0.003

Appendix

Table 13: Calibrator–estimand match under task-grouped 5-fold cross-validation, summarized over the 32 near-complete configurations (median and maximum absolute error). Mean-targeted calibrators recover the mean rating almost exactly, but thresholding their output does not recover the acceptance share; calibrating the acceptance probability directly does. The fold-mean constant shows that near-zero pooled error is attainable with no judge input, so aggregate unbiasedness alone is not evidence of judge information.

	Mean rating		Acceptance (score $\geq 7$ )
Predictor	cross-fit	tuned	cross-fit	tuned
FT2 pointwise	1.018–1.064	1.014–1.054	1.029–1.067	1.019–1.038
FT2 listwise	1.042–1.078	1.038–1.070	1.046–1.084	1.035–1.056
Haiku pointwise	1.065–1.120	1.060–1.104	1.062–1.109	1.047–1.058
TF–IDF (response only)	1.035–1.089	1.029–1.073	1.035–1.084	1.019–1.033

Appendix

Table 14: Tuned-

\lambda

heuristic (cluster-sum plug-in weight) under the same cross-fitted design: CI-width ratio ranges versus the human-only estimator over label fractions 0.05–0.50. Tuning does not produce consistent gains here either (empirical SD ratio median 1.00), and the LLM judges are no better predictors than the response-only TF–IDF control.

Figure 10: CI-width ratio of the cross-fitted prediction-assisted estimator relative to the human-only estimator, by labeled fraction, for the four predictors of Table 12 ; values above one are wider.

Judge	Majority-acceptable share	Disagreement with workers
Claude Opus 4.7 (pointwise)	0.003 $[0.000,0.010]$	0.684 $[0.635,0.736]$
Claude Haiku 4.5 (pointwise)	0.378 $[0.326,0.430]$	0.505 $[0.450,0.564]$
Qwen-FT-2ep (pointwise)	0.974 $[0.954,0.990]$	0.313 $[0.264,0.362]$
Qwen-FT-2ep (few-shot/listwise)	0.479 $[0.423,0.534]$	0.489 $[0.436,0.541]$
GPT-OSS-120B (pointwise)	0.873 $[0.834,0.909]$	0.309 $[0.261,0.365]$
TF–IDF full context (control)	0.410 $[0.358,0.469]$	0.466 $[0.407,0.521]$

Appendix

Table 15: Occupation classification at scores

\geq 7

(strict majority; 307 occupations with

\geq 5

ratings; workers: 0.687 majority-acceptable). Intervals from 1,000 occupation-cell bootstrap draws. Each reported configuration disagrees with the sampled worker-majority classification for at least 30.9% of occupations; the final row is a constant classifier that marks every occupation acceptable.

Pair	Pointwise	Listwise	Hybrid	$\Delta$ vs. pointwise [95% CI]	Recovered
Qwen base	.109	.219	.150	$+$ .041 [ $+$ .027, $+$ .055]	37%
Haiku	.149	.219	.162	$+$ .013 [ $+$ .004, $+$ .021]	18%
GPT-OSS	.093	.203	.153	$+$ .059 [ $+$ .038, $+$ .081]	54%
GPT	.091	.151	.100	$+$ .010 [ $-$ .000, $+$ .020]	16%
Qwen FT1	.215	.258	.219	$+$ .004 [ $-$ .009, $+$ .015]	9%
Qwen FT2	.214	.263	.228	$+$ .013 [ $+$ .001, $+$ .027]	27%

Appendix

Table 16: Response Spearman for the hybrid across all six matched pairs; boldface marks estimates whose intervals exclude zero. “Recovered” is the share of each pair’s listwise local gain.

Worker target	$n$	Response $\rho$	$\mathrm{PairAcc}_{1/2}$	Pair coverage
Understandability	4,501	.122	.537	.249
Relevance	4,501	.127	.541	.187
Factual accuracy	4,241	.097	.531	.182
Tone	4,501	.114	.541	.181
Employee quality	4,501	.236	.621	.529
Manager acceptance	4,501	.214	.596	.510

Appendix

Table 17: Per-field agreement for the pointwise FT2 judge (“accuracy not applicable” treated as missing).

Figure 11: Per-field Spearman correlation with the worker over the full corpus (45,796 ratings) for the two full-corpus pointwise runs. Time-to-complete estimates, task frequency, the primary required-information type, and failure cost reach

\rho=0.46

–

0.73

; every graded response-quality field, including the target (highlighted), stays below

0.17

for both judges.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge
Sep 11, 2026Shreyas Krishnan, Serina Chang, Abhishek NagarajLLM Evaluation Factual Knowledge in Language Models
University of California, Berkeley · National Bureau of Economic Research (NBER)
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
May 25, 2026Delip Rao, Chris Callison-BurchInter-Rater Reliability LLM Evaluation
University of Pennsylvania
AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
Aug 27, 2026Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian +1LLM-as-a-Judge LLM Agent Evaluation
ServiceNow AI Hyderabad, India

Sep 11, 2026J/K move · Enter open · S save

ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge

Shreyas Krishnan, Serina Chang, Abhishek Nagaraj

University of California, Berkeley · National Bureau of Economic Research (NBER)

LLM Evaluation Factual Knowledge in Language Models Knowledge-Intensive QA

Open paper AskPDF arXiv

We present ORQA, a method for testing occupation-level knowledge in large language models. Prior methods either map abstract LLM skills to occupations via task definitions or utilize expert knowledge which is difficult to obtain at scale and expensive. ORQA complements both of these methods by connecting O*NET occupations to trusted occupation-specific websites (such as regulatory agencies, licensing bodies, professional organizations, and government publications) and converting these into source-traceable question-answer pairs. A combination of an automated pipeline and human review produces a set of high quality questions about occupations. The question set created via our method covers 116 occupations from all 21 major groups in the SOC, with 480 questions sourced from 187 different websites. Each question is designed to probe a real-world skill question that is relevant to the occupation in question. We test 15 state-of-the-art frontier and open-weight models via this method. Claude Opus 4.6, GPT-5.4 and Claude Sonnet 4.6 all perform the best at approximately 58-62% while smaller open-weight models achieve approximately 33-41% performance. Performance varies significantly across occupations. Healthcare-related occupations achieve the highest performance (78%) while Office and Administrative Support achieve approximately 40%. Performance on individual occupations (e.g. Sheet Metal Workers and Fish and Game Wardens) is essentially zero. We also find that open-ended questions and weighting by wage bill do not significantly affect the ranking of models on this benchmark. We believe that leveraging existing trusted occupation-specific information to test LLM knowledge in professional domains may be a scalable and useful method for evaluating occupation-level AI performance in the future. Results and data are available at orqabench.org.

Abstract

LLM-as-a-Judge

Explore similar work

ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling