Predicting Human-Gain Curves under Local Proxy Optimization from Repeated Ratings

Moonwon Choi; Kisung Nam; Seunggeun Lee

cs.AIOct 30, 2026

Predicting Human-Gain Curves under Local Proxy Optimization from Repeated Ratings

Authors: Moonwon Choi, Kisung Nam, Seunggeun Lee

Organizations: Graduate School of Data Science, Seoul National University · Massachusetts General Hospital, Harvard Medical School · Broad Institute of MIT and Harvard

Abstract

Optimizing a proxy objective can improve human-perceived quality, but the resulting gain can vary with optimization strength. Average proxy-human correlation alone does not reveal how human-perceived quality changes as optimization strength increases. We propose PACE (Proxy Alignment and Curvature Expansion), a pre-optimization diagnostic that uses repeated human ratings to predict the human-gain curve induced by a fixed proxy. The method models proxy optimization as a KL-regularized exponential tilt. At KL radius $d$ , the local gain is approximated by $A\sqrt{d}+Bd$ , where $A$ measures first-order proxy-human alignment and $B$ captures the leading second-order correction along the proxy-optimization path. We estimate $A$ and $B$ from repeated human ratings on calibration items and test the resulting gain-curve prediction on held-out evaluation items. Across SummEval, analysis-eligible WMT MQM clusters, an OpenMEVA-MAGS ROC/WP repeated-rating slice, and a GPT-4o judge proxy experiment, PACE consistently reduces held-out gain-prediction error relative to $A$ -only, zero-gain, and reliability-only baselines. With the same pre-pool information, it is competitive with flexible proxy-conditional calibration, and a LoRA-DPO experiment provides a modest bridge to actual optimization. Transfer experiments across datasets and evaluation slices show that source coefficients are useful priors, but target calibration further improves prediction. The resulting diagnostic is local and one-sided: it characterizes gentle optimization along the positive tilt branch of a fixed item pool.

Figures & tables

Figure 1: Using PACE. (a) Proxy scores

r

and

k\geq 2

human ratings

Y_{1},\dots,Y_{k}

on calibration items give plug-in estimates of the alignment coefficient

A

and the curvature coefficient

B

, which forecast the local human-gain curve before optimization. A planned rule that is not a raw-score tilt, such as Best-of-

N

, is first translated into its induced KL radius and relative effective sample size (RelESS) on an unlabeled, proxy-scored pilot (dashed). No human labels of the target pool are used. (b) Schematic with

\hat{B}<0

\hat{A}\sqrt{d}

alone overstates the gain, and

\hat{B}d

corrects it inside the checked window; beyond it the realized gain may saturate or decline, and no forecast is made.

$\hat{h}$	Predictor	SummEval	GPT-4o	WMT	OpenMEVA
	Paths / cells	32 / 7,989	4 / 1,000	141 / 35,250	10 / 2,500
	Pre-pool $A{+}B$ (reference)	0.01185	0.00806	0.05921	0.00991
Binned	Projection	0.02782	0.00860	0.06557	0.01330
	Direct integration	0.02914	0.00865	0.06843	0.01429
Spline	Projection	0.01040	0.00780	0.05667	0.00726
	Direct integration	0.01027	0.00777	0.05699	0.00728

Table 2: Target-pool-aware CurveMAE: the proxy scores of the target pool are available, but its human labels are not. For each calibrator, the same fitted

\hat{h}

is used by projection (the two-term moment formula applied to

(\hat{h}(r),r)

on the target pool) and by direct integration (the exact tilt integral of

\hat{h}

). The pre-pool

A{+}B

row, from Table 1 b, sees less information and is a reference, not a matched competitor. Metric, support, and units as in Table 1 b; paired intervals are in Appendix B.7 .

Figure 2: What adding

B

contributes in the pre-pool setting (30% calibration, 50 splits per path, held-out exact tilt gain; panel (a) uses the common cells of Table 1 b). (a) Relative endpoint-MAE reduction

1-\mathrm{MAE}_{A+B}(d)/\mathrm{MAE}_{A\text{-only}}(d)

on identical cells, a within-dataset ratio. Error bars are 95% path-cluster bootstrap intervals; GPT-4o has 4 paths and OpenMEVA 10, and OpenMEVA’s intervals include zero for

d\leq 0.05

. (b) Median share

R_{B}(d)=|\hat{B}d|/(|\hat{A}\sqrt{d}|+|\hat{B}d|)

of the predicted gain over all splits, a magnitude share rather than net

B

or an error reduction. The improvement is aggregate:

A{+}B

has lower CurveMAE than

A

-only on 55.6–67.0% of splits.

MAE of calibration-only prediction
	Realized	Idealized
Mean $\pm$ SD over 3 seeds	likelihood-ratio gain	tilt gain
Mean KL radius $d$	0.03968 $\pm$ 0.00136
Mean silver gain	0.02081 $\pm$ 0.00062	0.04197 $\pm$ 0.00017
Idealized minus realized gain	0.02116 $\pm$ 0.00074
$A{+}B$	0.01721 $\pm$ 0.00116	0.00395 $\pm$ 0.00047

Table 3: LoRA-DPO bridge: mean

\pm

sample SD over three training seeds, after averaging the 4/3/3 checkpoints per seed whose mean prompt KL is at most

0.10

. Gains are in silver-target units. Realized : self-normalized DPO/reference likelihood-ratio reweighting of fixed pools of 32 candidates per prompt, relative to uniform; it is not a fresh-generation, human-rated, or exact full-policy gain. Idealized : raw-score exponential tilt of the same pool at the same prompt radius. Predictors use common calibration-only coefficients. This is a restricted predictor set; all 46 comparator variants are in Appendix B.16 . Lower MAE is better.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: When the skewness correction matters. (a) Correction share as a function of proxy skewness

\mathbb{E}[Z^{3}]

for the synthetic skew-normal family (line) and the two SummEval coherence proxies (points). (b) The two components of

B

and their sum for the synthetic family: with positive alignment, right skew contributes a negative correction and left skew a positive one. Other moments also vary across this family, so total

B

need not decrease with skewness.

Figure B.1: SummEval reliability by dimension and rater pool. The plot visualizes Table B.3 : reliability is a property of the observation protocol, not a single scalar attached to the benchmark. Error bars, where shown, are bootstrap intervals for the ICC estimate under the split construction.

Figure B.2: Expert–crowd reliability lift. Consistency and fluency show large measurement gains from expert raters. Relevance does not, which motivates separating measurement improvement from preference heterogeneity. Error bars show bootstrap intervals for the expert-minus-crowd ICC difference.

Figure B.3: ICC reliability profiles across the repeated-rating analysis slices. Panel A shows SummEval dimension-level ICC(A,1), with mean ICC

\approx 0.22

. Panel B shows one-way ICC(1,1) for eligible raw WMT MQM splits, with mean ICC

\approx 0.51

; these are the direct-validation splits, not the prepared prospective clusters. Panel C shows OpenMEVA-MAGS reliability, with mean ICC

\approx 0.75

. The dashed line in each panel marks the slice mean.

Figure B.4: SummEval exact-tilt validation example. One human rater defines the proxy and LOO ratings define

J

. The checked window is shaded through

d=0.10

; the dash-dot line marks the nominal fitting cutoff

d=0.20

. Curves stop at the computed exact path near

d=0.15

, and the unattainable region is shaded gray.

Figure B.5: Overview of endpoint prediction and OOD transfer. Panel A compares endpoint gain-prediction error at

d=0.10

for SummEval human-rater proxy paths and the GPT-4o judge proxy experiment. Panel B shows the SummEval

\rightarrow

WMT OOD transfer comparison under the stricter z_cal protocol. Source-prior denotes the fixed target-heavy shrinkage estimator. Error bars are 95% path-cluster bootstrap intervals for pooled-split MAE; GPT-4o has only four dimension paths. Lower MAE is better.

Figure B.6: Prospective prediction diagnostics across eligible human-proxy analysis slices. Panel A compares

A{+}B

MAE against the reliability-only baseline (

A_{\nu}

) at

d=0.10

on SummEval, WMT MQM clusters, and OpenMEVA-MAGS ROC/WP; the broken

y

-axis separates the high-error

A_{\nu}

/zero baselines from the lower coefficient errors. Panel B reports agreement between the calibration moment coefficient sign and the evaluation curve-fit coefficient sign,

\operatorname{sign}(\hat{B}_{\rm cal})=\operatorname{sign}(B_{\rm eval}^{\rm fit})

, together with mean realized gain

\Delta(0.10)

for the same human-proxy slices. The evaluation coefficient is fitted over

0<d\leq 0.10

and is used only for retrospective scoring. The 50% and 90% lines are visual references, not calibrated acceptance thresholds. GPT-4o results are in Tables B.13 and B.15 .

Figure B.7: Sensitivity of

\hat{B}

to the calibration gold on SummEval. (a) Mean

|\hat{B}-B_{\rm eval}|

against the number of clean gold raters. (b) Sign agreement of

\hat{B}

with the clean estimate at the same rater count under injected noise (nominal scale, all gold raters). (c) Paired CurveMAE difference

A{+}B

minus

A

-only with 95% path-cluster intervals: the sign of

\hat{B}

stays stable, but its predictive benefit disappears at 1.0 SD for experts and 0.5 SD for crowd workers.

Figure B.8: SummEval coherence forecasts across 50 matched item splits (480 calibration / 1,120 held-out items). Top: mean exact held-out tilt gain (black), calibration-only PACE (

A{+}B

, blue), and

A

-only forecasts (orange dashed). Bottom: absolute error computed within each split, then averaged. Percentages give PACE’s relative MAE change against

A

-only at

d=0.10

. Shading marks the between-method gap, not uncertainty: the mean

\hat{B}d

correction above and the MAE difference below. Axes are zoomed to show

d\in[0.025,0.10]

; neither vertical axis starts at zero. The repeated splits reuse the same 1,600 items.

Figure B.9: KL-radius audit for the shared local window. The shaded region and vertical dashed line mark

d\leq 0.10

, the window used for the main prospective prediction tables. Panel A plots MAE versus KL radius on the human-proxy setting: the

A{+}B

predictor remains below the

A

-only and zero baselines throughout the shared local grid. Panel B gives the same comparison for the GPT-4o proxy, where

A{+}B

remains below

A

-only,

A_{\nu}

, and zero from

d=0.005

d=0.20

. Ribbons or intervals, where shown, are 95% path-cluster bootstrap intervals for pooled-split MAE. Panel C separates split coverage (the line) from path counts (annotations). For SummEval, coverage is relative to 1,589 retained splits: 100% through

d=0.10

1{,}472/1{,}589=92.6\%

0.15

, and

1{,}350/1{,}589=85.0\%

0.20

; the corresponding path counts are 32, 31, and 27. All four GPT-4o paths remain reachable.

Figure B.10: Tail and outlier robustness audit. Panel A reports stability of the curvature coefficient

B

, measured as correlation with the original-preprocessing coefficient. Panel B reports the median absolute change in exact endpoint gain at

d=0.10

relative to the original preprocessing; this is a gain-scale shift, not the prediction error of the local law. Mild winsorization and controlled gold-side outliers are comparatively benign in this audit. Rank-normalization can change endpoint gain scale, especially for WMT, and proxy-side outliers can substantially perturb curvature, especially in OpenMEVA. Sign-agreement and

A

-stability metrics are reported in Table B.36 .

Figure B.11: Radius and calibration-size sensitivity: panels A/C are SummEval human-proxy paths; B/D are GPT-4o proxy paths. Panels A and B report MAE as the KL radius

d

increases; the shaded region and dashed line mark the shared

d\leq 0.10

local window and main endpoint. Panels C and D report MAE at

d=0.10

as the calibration split fraction varies on the same proxy–gold paths; larger calibration fractions leave smaller evaluation splits. Ribbons or intervals, where shown, are 95% path-cluster bootstrap intervals for pooled-split MAE. The shared legend compares coefficient

A{+}B

A

-only, the empirical fit, reliability-only

A_{\nu}

, and zero gain.

Figure B.12: Finite-range residual and selective-use diagnostics. Panels A/B are same-pool approximation audits for SummEval/GPT-4o: moments and empirical fits use the same pool and exact curve whose median maximum residual is reported over

d\leq D

. They do not measure held-out prediction error. Panel C is the GPT-4o external-proxy selective-use audit:

\tau

thresholds the predicted endpoint-gain margin before applying the optimize-if-positive rule, and lower-margin cases are abstained rather than treated as confident positive selections.

Figure B.13: Prompt-conditional Best-of-

N

scope audit. Panel A reports the mean prompt-level KL of exact with-replacement prompt-conditional BoN, with the

d=0.10

window shown by the dashed line. Panel B reports the fraction of paths whose mean prompt KL is attainable by a pooled raw-score tilt within

d\leq 0.25

, accounting for paths whose tied top scores cap the pooled tilt. Mean BoN gains are in Table B.46 .

Figure B.14: Legacy pooled-item BoN resampling audit, not prompt-conditional selection or a theorem test. Candidates are drawn with replacement from the pooled item set, and each candidate’s proxy is an independently drawn rating or RewardBench model column; gold is the itemwise mean over all columns, not LOO. Panels A–D use the bridge coordinate $u=\log N$ , not the induced KL. Panels B–C compare fitted and theory coefficients, and Panel D shows fitted $B$ as the fit range grows. Panels E/F show the proxy–gold gap/gold level, not gold gain. The displayed $\nu$ is a square-root variance ratio, not ICC.

Figure B.15: Finite-range diagnostics across eligible analysis slices. Panel A reports median minimum tilted variance on the left scale and median minimum effective sample size on the accompanying labels/right scale for each slice on the shared local window

d\leq 0.1

; tail labels summarize the share of paths whose tilted tail gold falls below the reference baseline. Panel B relates median minimum tilted variance to curvature-sign agreement with the evaluation curve-fit coefficient. This is a numerical stability diagnostic, not a guarantee of accurate prediction.

Figure B.16: Protocol split-half stability. Panel A reports coherence split-half correlations under mean, median, and trimmed-mean proxy construction. Panel B gives the same comparison for consistency. Panel C gives the same comparison for fluency. Panel D gives the same comparison for relevance. Error bars summarize random rater-half splits on the same items. The panels describe sensitivity to rater composition, while Table B.48 shows that improving measurement changes both signal and curvature.

Figure B.17: Proxy perturbation stress test. The bars show realized endpoint gains

\Delta(0.10)

under noisy, oracle, shuffled, random-normal, and sign-flipped proxy variants. Coefficient values and baseline prediction errors are reported in Table B.49 ; the multi-dataset extension is Figure B.18 . The shuffle condition shows that the result is not a proxy-scale artifact: preserving the marginal score distribution while destroying itemwise alignment collapses realized gain.

Figure B.18: Proxy perturbation stress test across eligible analysis slices. Panel A shows SummEval, Panel B WMT MQM clusters, and Panel C OpenMEVA-MAGS ROC/WP. In each panel, the exact first-order coefficient

A

tracks alignment direction across perturbations, while the reliability-only

A_{\nu}

baseline fails to distinguish aligned from unaligned proxies.

Language Model Performance Prediction

See all 41 papers

Statistical Learning Theory for Direct Preference Optimization
Oct 19, 2026Joel Wills, Yuanfan Li, Yiming Ying +1Direct Preference Optimization Pairwise Preference Learning
School of Mathematics and Statistics, The University of Sydney
Forecasting Downstream Performance of LLMs With Proxy Metrics
May 18, 2026Arkil Patel, Siva Reddy, Marius Mosbach +1Language Model Pretraining LLM Evaluation
Mila – Quebec AI Institute & McGill University Canada · ServiceNow Research · Periodic Labs
Distributional Process Reward Models: Calibrated Prediction of Future Rewards via Conditional Optimal Transport
May 7, 2026Rachel Ma, Dylan Hadfield-Menell, Kristjan GreenewaldProcess Reward Models Inference-Time Scaling
MIT CSAIL · MIT-IBM Watson AI Lab · IBM Research

Oct 19, 2026J/K move · Enter open · S save

Statistical Learning Theory for Direct Preference Optimization

Joel Wills, Yuanfan Li, Yiming Ying, Ding-Xuan Zhou

School of Mathematics and Statistics, The University of Sydney

Direct Preference Optimization Pairwise Preference Learning Statistical Learning Theory Bradley–Terry Model

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) learns from pairwise preferences through a logistic objective whose score-difference parameterization follows the Bradley--Terry model. We study how failure of this modeling assumption limits DPO, distinguishing structural misspecification from limited model capacity and finite-sample estimation. Our main contribution is a characterization of structural approximation error through the triangle-curl of the preference log-odds, which vanishes exactly when preferences admit a Bradley--Terry representation. We upper bound the achievable excess logistic risk by a score-class approximation term and a triangle-curl term, and establish a lower bound proportional to triangle-curl under additional boundedness assumptions. For ReLU score models, including clipped models, we derive explicit approximation rates showing that increasing capacity can eliminate the score-approximation term while leaving the structural obstruction unchanged. We further establish finite-sample bounds for clipped empirical risk minimizers that separate network approximation, structural misspecification, and estimation error. We place these results within a surrogate-risk framework by identifying contextual preference accuracy (CPA) as a natural measure of agreement between preference judgments and changes in pairwise probability ratios relative to the reference policy. Standard logistic consistency and calibration results connect DPO's objective to this policy property and transfer our excess-risk upper bounds to CPA, without treating CPA as a complete measure of generated-response quality.

Abstract

Language Model Performance Prediction

Explore similar work

Statistical Learning Theory for Direct Preference Optimization

Forecasting Downstream Performance of LLMs With Proxy Metrics

Distributional Process Reward Models: Calibrated Prediction of Future Rewards via Conditional Optimal Transport