Can You Trust Your Preference Models? Stackelberg Games for Alignment

Riccardo Formenti; Aditya Agrawal; Vikas Garg

cs.AIOct 10, 2026

Can You Trust Your Preference Models? Stackelberg Games for Alignment

Authors: Riccardo Formenti, Aditya Agrawal, Vikas Garg

Organizations: Department of Mathematics ETH Zürich · Department of Computer Science Aalto University · YaiYai Ltd and Aalto University

Abstract

Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are unavailable during policy training, we use disagreement among an ensemble of preference models as a proxy for prediction error. We formulate this search as a zero-sum Stackelberg game between two opposing policies, a Leader which tries to minimize disagreement, and a Follower which maximizes it. For a general class of finite contextual Stackelberg games, we establish equilibrium existence and, under regularization, uniqueness and a closed-form characterization. We introduce Stackelberg Mirror Descent-Ascent (SMIDA), a two-timescale algorithm with linear last-iterate convergence in the tabular setting, and develop a neural policy-gradient analogue. Experiments with a synthetic oracle support disagreement as an informative proxy for prediction error. On held out data the trained policies outperform reference policies and increase the fraction of comparisons that exceed a disagreement threshold and reverse the oracle's preference.

Figures & tables

\pi_{t+1}(\cdot\mid x)\;\propto\;\pi_{t}(\cdot\mid x)^{1-\eta^{L}\tau^{L}}\,\pi^{\mathrm{ref}}(\cdot\mid x)^{\eta^{L}\tau^{L}}\exp\!\bigl(-\eta^{L}\,\widehat{r}_{t}(x,\cdot)\bigr)

Algorithm 1 Stackelberg Mirror Descent–Ascent ( smida ), tabular form

Figure 1: Ensemble variance and prediction difficulty on held out comparisons. (a) Oracle agreement across variance deciles. (b) Ensemble error rate among comparisons selected in decreasing order of variance. Error bars and shading show

95\%

prompt-bootstrap intervals.

Figure 2: Online candidate yield and oracle-confirmed adversarial yield on holdout prompts. (a) Fraction of

5{,}000

triplets per policy pair meeting the common reference-quantile threshold at level

\alpha

. (b) Fraction of all

1{,}600

oracle-labelled triplets per policy pair that are both

\alpha

-online candidates and

\varepsilon

-adversarial. Blue solid curves denote trained policies; red dashed curves denote reference policies. Lighter curves indicate larger oracle margins. Shading shows

95\%

prompt-bootstrap intervals

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

\phi_{t+1}=\phi_{t}+\eta_{t}^{F}\widehat{\nabla_{\phi}J^{F}}_{t},\qquad\theta_{t+1}=\theta_{t}-\eta_{t}^{L}\widehat{\nabla_{\theta}J^{L}}_{t}.

Appendix

Algorithm 2 Deep Stackelberg Mirror Descent–Ascent ( deep-smida )

Summary statistics		Selected percentiles
Statistic	Value	Percentile	Value
Mean	$0.006739$	$25$	$0.000588$
Median	$0.002931$	$75$	$0.008959$
Variance	$9.0507\times 10^{-5}$	$90$	$0.018687$
Standard deviation	$0.009514$	$95$	$0.026660$
Maximum	$0.100228$	$99$	$0.044667$

Appendix

Table 1: Distribution of ensemble variance on the held out UltraFeedback subset.

Optimization
Parameter	Value
Optimizer	AdamW
Leader learning rate $\eta^{L}$	$8\times 10^{-6}$
Follower learning rate $\eta^{F}$	$2\times 10^{-5}$
Initial learning rate (both policies)	$10^{-6}$
Warm-up steps	$50$

Appendix

Table 2: Hyperparameters for training the Leader and Follower. Unless specified otherwise, settings apply to both policies.

Figure 3: Preference-ensemble error detection on

37{,}614

held-out comparisons, including

9{,}155

errors relative to the binary oracle labels. (a) ROC curves for decision uncertainty (blue) and Jensen–Shannon divergence (JSD; red), with AUROCs of

0.707

and

0.654

, respectively. The dashed diagonal indicates chance performance. (b) Oracle-relative error rates among comparisons selected in descending order of each score. Selecting the top

10\%

gives error rates of

47.5\%

for decision uncertainty (

95\%

CI:

46.0

–

49.1\%

) and

39.7\%

for JSD (

95\%

CI:

38.2

–

41.3\%

), compared with

24.3\%

overall. Shading shows pointwise

95\%

prompt-bootstrap intervals.

Figure 4: Ensemble disagreement after and during training. (a) Empirical CDFs of ensemble variance on

5{,}000

triplets from

2{,}500

held-out prompts per policy pair, generated after training. Vertical lines mark reference-quantile thresholds; shading shows

95\%

prompt-bootstrap intervals. (b) Shares of

\alpha

-online candidates in training minibatches of

64

triplets, using the same fixed thresholds. Thin curves show per-step shares, thick curves their moving averages, and horizontal dotted lines the nominal reference rates

\alpha

Figure 5: Adversarial precision of online candidates. Solid curves show the fraction of

\alpha

-online candidates that are

\varepsilon

-adversarial. Candidate thresholds are fixed by the reference variance distribution, as in Section 5 Lighter curves correspond to larger oracle margins. Horizontal dotted lines show the corresponding adversarial fractions among all labelled triplets, giving the expected precision under random selection. Shading shows a 95% prompt-bootstrap interval for

\varepsilon=0

Training and rollout sampling
Parameter	NLHF	SLHF
Training prompts	$2{,}000$	$2{,}000$
Training epochs	$1$	$1$
Optimizer steps	$63$	$63$
Prompts per optimizer step	$32$	$32$
Prompts in final step	$16$	$16$

Appendix

Table 3: Training and generation settings for the NLHF (Nash-MD-PG) and SLHF (StackelbergGDA) baselines. Both methods fully fine-tune Qwen2.5-1.5B-Instruct on the same

2{,}000

training prompts using the frozen preference model

\mathcal{P}_{0}

. Each uses a separate frozen copy of the initial policy as its KL reference.

	Column policy
Row policy	NLHF	SLHF Leader	Our Leader
NLHF	—	$0.4508$	$0.4511$
SLHF Leader	$\mathbf{0.5492}$	—	$0.4930$
Our Leader	$\mathbf{0.5489}$	$\mathbf{0.5070}$	—

Appendix

Table 4: Pairwise oracle preference rates on

500

held-out prompts. Each entry estimates the probability that the row policy’s response is preferred to the column policy’s response, using

11

oracle queries per response pair and averaging across prompts. Values above

0.5

favor the row policy.

Loose accuracy by instruction category
Metric	Stackelberg Follower	Follower-only
Loose: prompt	35.12	28.28
Loose: instruction	46.04	38.49
Strict: prompt	30.31	23.66
Strict: instruction	41.25	33.93
Case	22.47	33.71

Appendix

Table 5: Leader ablation on IFEval. Accuracy (%) is reported for the Follower trained with an adaptive Leader (Stackelberg Follower) and with a fixed Leader (follower-only), evaluated on the same

541

prompts and

834

verifiable instructions. The higher value in each row is bolded. Category rows report loose instruction-level accuracy.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

Alignment Makes Language Models Normative, Not Descriptive
Mar 17, 2026Eilam Shapira, Moshe Tennenholtz, Roi ReichartLLM Alignment Preference Alignment
Technion – Israel Institute of Technology
Robust Nash Alignment under Preference Uncertainty
Sep 30, 2026Shihab Ahmed, Debamita Ghosh, David Tang +3LLM Alignment Preference Alignment
University of Central Florida · Arizona State University · University of Colorado Boulder
Inference-Time Nash Alignment
Sep 8, 2026Hadi Hosseini, Debmalya Mandal, Duohan ZhangInference-Time Optimization Preference Alignment
Penn State University, USA · University of Warwick, UK

Mar 17, 2026J/K move · Enter open · S save

Alignment Makes Language Models Normative, Not Descriptive

Eilam Shapira, Moshe Tennenholtz, Roi Reichart

Technion – Israel Institute of Technology

LLM Alignment Preference Alignment Normative Conformity in Language Models Human Behavior Modeling Human Behavior Prediction

Open paper AskPDF arXiv

Post-training alignment optimizes language models to match human preference signals, but this objective is not equivalent to modeling observed human behavior. We compare 120 base-aligned model pairs on more than 10,000 real human decisions in multi-round strategic games - bargaining, persuasion, negotiation, and repeated matrix games. In these settings, base models outperform their aligned counterparts in predicting human choices by nearly 10:1, robustly across model families, prompt formulations, and game configurations. This pattern reverses, however, in settings where human behavior is more likely to follow normative predictions: aligned models dominate on one-shot textbook games across all 12 types tested and on non-strategic lottery choices - and even within the multi-round games themselves, at round one, before interaction history develops. This boundary-condition pattern suggests that alignment induces a normative bias: it improves prediction when human behavior is relatively well captured by normative solutions, but hurts prediction in multi-round strategic settings, where behavior is shaped by descriptive dynamics such as reciprocity, retaliation, and history-dependent adaptation. These results reveal a fundamental trade-off between optimizing models for human use and using them as proxies for human behavior.

Abstract

LLM Alignment

Explore similar work

Alignment Makes Language Models Normative, Not Descriptive

Robust Nash Alignment under Preference Uncertainty

Inference-Time Nash Alignment