Dual Optimal: Make Your LLM Peer-like with Dignity

Xiangqi Wang; Yue Huang; Haomin Zhuang; Kehan Guo; Xiangliang Zhang

cs.CLApr 1, 2026

Dual Optimal: Make Your LLM Peer-like with Dignity

Authors: Xiangqi Wang, Yue Huang, Haomin Zhuang, Kehan Guo, Xiangliang Zhang

Organizations: Department of Computer Science and Engineering, University of Notre Dame

Abstract

Current aligned language models exhibit a dual failure mode we term the Evasive Servant: they sycophantically validate flawed user beliefs while deflecting responsibility with boilerplate disclaimers. We propose the Dignified Peer framework, which counters servility with anti-sycophancy and trustworthiness, and mitigates evasiveness through empathy and creativity. Realizing this agent requires overcoming significant challenges in data supervision, objective collapse, and evaluation bias. We address these issues by introducing the PersonaKnob dataset which features a compositional partial order structure of multiple persona preference. This data is utilized alongside a tolerant constrained Lagrangian DPO algorithm that dynamically balances all persona dimensions to prevent behavioral collapse. Additionally, we employ a psychometrically calibrated Item Response Theory evaluation protocol to disentangle latent model persona capability from confounders like judge biases. Extensive empirical studies demonstrate that our approach successfully build a LLM agent with both dignity and peer.

Figures & tables

Figure 1: Failure vs. desired behavior in one example scenario (more in Figure 12 ).

Dataset	Dimensions	Paradigm	Multi-Persona
Model-Written Evals ( Perez et al., 2022 )	A only	Selection	No
TruthfulQA ( Lin et al., 2022 )	T only	Selection	No
Empathetic Dialogues ( Rashkin et al., 2019 )	E only	Generation	No
WritingPrompts ( Team, 2024 )	C only	Generation	No
PersonaKnob (ours)	A+T+E+C	Both	Yes

Table 1: Comparison of PersonaKnob with existing datasets.

Figure 2: Construction pipeline of PersonaKnob: from masked sampling to final human-in-the-loop review.

Figure 3: An illustrative instance of PersonaKnob, consisting of one reference (golden) response and three negative (rejected) responses generated under the active traits {A,E,C}. Note that the displayed responses are abstracted from the full original outputs, as they are too long to include.

Figure 4: Distribution statistics of PersonaKnob.

Figure 5: The three-stage evaluation based on MFRM. The process transitions from multiple-judge rubric annotation to multi-facet calibration, ultimately synthesizing metrics equipped with Fisher Uncertainty, resulting in Peer $\hat{P}$ and Dignity $\hat{D}$ scores.

LLM	Raw Prob.		Z-Score		MFRM
	$P$ $\uparrow$	$D$ $\uparrow$	$P_{z}$ $\uparrow$	$D_{z}$ $\uparrow$	$\hat{P}$ $\uparrow$	$\hat{D}$ $\uparrow$
Llama-3.1-70B-Ins.	$0.718\pm 0.271$	$0.693\pm 0.259$	$0.002\pm 0.661$	$-0.046\pm 0.723$	$7.115\pm 0.018$	$7.937\pm 0.241$
Qwen-2.5-72B	$0.709\pm 0.279$	$0.672\pm 0.360$	$0.008\pm 0.664$	$-0.062\pm 0.729$	$6.332\pm 0.010$	$8.036\pm 0.097$
Mistral-Large-3-2512	$0.778\pm 0.207$	$0.711\pm 0.340$	$0.085\pm 0.642$	$0.057\pm 0.681$	$11.886\pm 0.253$	$6.101\pm 0.190$
Claude-3.7-Sonnet	$0.789\pm 0.173$	$0.661\pm 0.326$	$-0.019\pm 0.664$	$0.038\pm 0.686$	$12.117\pm 0.009$	$4.542\pm 0.024$
GPT-4.1	$0.691\pm 0.288$	$0.776\pm 0.201$	$-0.047\pm 0.674$	$-0.027\pm 0.715$	$6.200\pm 0.010$	$10.480\pm 0.027$

Table 2: Comparison of scoring methods for Peer (

P

) and Dignity (

D

). Raw Prob. computes the average proportion of satisfied rubric items; Z-Score normalizes per-judge scores to zero mean and unit variance; MFRM calibrates for judge severity, rubric difficulty, and question complexity jointly. Significant differences across models are observed under MFRM, but not under Raw Prob. or Z-score .

Panel of Figure 6 — Figure 6: Left: Radar chart of utility and ATEC trait scores for six persona configurations. Right: Correlation among A, T, E and C.

Method	Data	Dual Grad.	Constrained	Mechanism
SFT-Anchor	A	✗	✗	SFT only.
SFT-Combined	A+PK	✗	✗	SFT only.
Multi-Neg DPO ( Rafailov et al., 2023 )	PK	✗	✗	Uniform loss averaging.
PCGrad-DPO ( Yu et al., 2020 )	PK	✗	✓	Non-conflicted gradient
SACPO ( Wachi et al., 2024 )	PK	✗	✓	Stepwise dimension constraint.
SafeRLHF ( Dai et al., 2023 )	PK	✓	✓	Single-objective and constraint Lagrangian.

Table 3: Overview of compared methods. PK = PersonaKnob; A = Anchor.

Model Family	Method	A $\uparrow$	T $\uparrow$	E $\uparrow$	C $\uparrow$
Llama-3-8B-Ins.	Vanilla	$-2.782\pm 0.030$	$-2.524\pm 0.030$	$2.939\pm 0.044$	$0.476\pm 0.016$
	SFT-Anchor	$2.024\pm 0.024$	$1.949\pm 0.026$	$-2.301\pm 0.027$	$2.029\pm 0.022$
	SFT-Combined	$2.146\pm 0.021$	$\mathbf{3.093\pm 0.028}$	$1.875\pm 0.018$	$-0.615\pm 0.012$
	Multi-Neg DPO	$-4.806\pm 0.018$	$-6.575\pm 0.034$	$3.193\pm 0.031$	$\mathbf{4.999\pm 0.046}$
	PCGrad-DPO	$4.306\pm 0.976$	$1.594\pm 0.013$	$3.802\pm 0.049$	$-1.817\pm 0.012$
	SACPO	$0.812\pm 0.015$	$0.890\pm 0.022$	$-3.648\pm 0.030$	$3.356\pm 0.052$

Table 4: Performance Comparison of Preference Optimization Algorithms. Our Lag-DPO effectively balances all persona dimensions (

A,T,E,C

), avoiding negative interference between objectives. (Values reported in IRT Logit Space

\theta\pm SE

)

Setting	A $\uparrow$	T $\uparrow$	E $\uparrow$	C $\uparrow$
Lag-DPO (Default)	$\mathbf{14.542\pm 0.998}$	$\mathbf{13.050\pm 0.995}$	$\mathbf{4.889\pm 0.031}$	$\mathbf{3.977\pm 0.039}$
$\epsilon=0.0$	$1.412\pm 0.074$	$1.687\pm 0.058$	$3.089\pm 0.216$	$2.311\pm 0.079$
$\epsilon=0.8$	$13.470\pm 0.040$	$-4.104\pm 0.063$	$3.235\pm 0.031$	$1.758\pm 0.039$
$\eta_{\lambda}=0.001$	$4.505\pm 0.998$	$4.795\pm 0.049$	$3.239\pm 0.031$	$2.729\pm 0.022$
$\eta_{\lambda}=0.10$	$-2.046\pm 1.011$	$11.617\pm 1.986$	$3.042\pm 0.828$	$3.507\pm 0.974$

Table 5: Ablation Study of Lag-DPO. We analyze the impact of constraint margin

\epsilon

and dual learning rate

\eta_{\lambda}

on persona performance (A, T, E, C).

Figure 7: General capability evaluation on MMLU and BBH. Lag-DPO matches or exceeds baselines across both model families while achieving superior persona alignment.

Figure 8: Representation drift on BBH hard negatives. Cosine similarity to the Base model across layers. SFT-Anchor diverges strongly in mid-to-late layers, consistent with capability corruption.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Auditor	Criterion 1	Criterion 2	Criterion 3	Aggregate
1: Spurious Scenario	LLM	2.1% (S1)	2.8% (S2)	1.9% (S3)	5.6%
1: Spurious Scenario	Human	0.8% (S1)	1.2% (S2)	0.7% (S3)	2.4%
2: Unfaithful Reference	LLM	3.2% (R1)	2.9% (R2)	2.4% (R3)	7.4%
2: Unfaithful Reference	Human	1.4% (R1)	1.3% (R2)	1.1% (R3)	3.5%
3: Weak Negative	LLM	2.5% (N1)	4.1% (N2)	3.6% (N3)	8.8%
3: Weak Negative	Human	1.0% (N1)	2.3% (N2)	2.1% (N3)	4.9%

Appendix

Table 6: Residual hallucination rates among accepted instances, broken down by rubric criterion. An instance fails a type if any criterion scores 0.

Model	A $\uparrow$	T $\uparrow$	E $\uparrow$	C $\uparrow$
Tulu-SFT-2-7b	$-1.927\pm 0.019$	$-0.365\pm 0.019$	$0.969\pm 0.040$	$0.258\pm 0.028$
Tulu-DPO-2-7b	$-2.770\pm 0.013$	$2.150\pm 0.078$	$1.765\pm 0.047$	$-1.060\pm 0.026$
$\Delta$ (DPO - SFT)	-0.843	+2.515	+0.796	-1.318

Appendix

Table 7: Comparison of calibrated scores between the baseline SFT model and its H&H-tuned DPO counterpart. Dimensions denote Anti-Sycophancy (A), Trustworthiness (T), Empathy (E), and Creativity (C). Following multi-facet calibration, we observe that DPO significantly enhances Trustworthiness and Empathy capabilities. However, this comes at the cost of Anti-Sycophancy and Creativity, empirically demonstrating the “evasive servant” alignment tax.

Figure 9: Persona vector steering success rate. Each point reports the mean success rate steering from $P$ to $D$ or from $D$ to $P$ .

Figure 10: Multi-Neg DPO loss score comparison between two methods across training steps. Lower loss indicates stronger preference alignment.

Figure 11: Anchoring Dataset Cards. Four single-dimension datasets grounding PersonaKnob, spanning both selection -based ( $A$ , $T$ ) and open-ended generation ( $E$ , $C$ ) paradigms. Filled badges indicate evaluation format; outlined badges indicate task sub-type.

Panel of Figure 12 — Figure 12: Model Behavior Cards. Six models evaluated on the Evasive Servant vs. Dignified Peer axis across three behavioral dimensions. Only $M_{5}$ (Lag-DPO) achieves Dignified Peer . $M_{3}$ (SFT-Combined) exhibits a distinct Deflecting Promoter failure: it avoids the task but outputs manipulative social-media astroturfing scripts, demonstrating that SFT alone without preference optimization cannot suppress harmful compliance.

Figure 13: Calibration Visualization at Dimension Trustworthiness

Figure 14: The interface of our human annotation platform.

Dim	Kendall $\tau$	Spearman $\rho$
$A$	$0.898$	$0.943$
$T$	$0.895$	$0.956$
$E$	$0.964$	$0.978$
$C$	$0.936$	$0.949$

Appendix

Table 8: Question-level ranking stability after adding Gemini as a third judge.

Dataset Splits
Statistic	Value
Train samples	1331
Test samples	220
Task Paradigm
Selection-based instances (%)	50.3
Generation-based instances (%)	49.7

Appendix

Table 9: Summary statistics of the PersonaKnob dataset.

Dim.	Name	Failure Mode	Desired Behavior
A	Anti-Sycophancy	Validates flawed premise under social pressure	Evidence-backed pushback
T	Trustworthiness	Confident confabulation of uncertain claims	Signals uncertainty; suggests verification
E	Empathy	Cold or dismissive to affective context	Acknowledges feelings; non-judgmental
C	Creativity	Hollow, buzzword-recycling outputs	Actionable, original, well-structured help

Appendix

Table 10: Behavioral signatures of the four Dignified Peer dimensions.

Method	SycophancyEval		XSTest
Method	Syco. Rate $\downarrow$	Consist. $\uparrow$	Safe Compl. $\uparrow$	Unsafe Ref. $\uparrow$
Vanilla	0.872	0.129	95.0%	38.0%
SFT-Anchor	0.976	0.024	97.0%	33.0%
SFT-Combined	0.876	0.125	98.1%	44.8%
Multi-Neg DPO	0.894	0.106	97.1%	36.0%
PCGrad-DPO	0.896	0.104	95.2%	37.0%

Appendix

Table 11: OOD evaluation on Qwen3-4B-Instruct-2507. Best per column in bold .

Panel of Figure 15 — Figure 15: Prompt used for PersonaKnob question synthesis. Given a subset of active persona dimensions, the prompt elicits a single JSON-structured first-person question that requires all active dimensions simultaneously.

Panel of Figure 16 — Figure 16: Binary scoring rubrics for the four persona dimensions used in MFRM-IRT calibration. Each dimension has three rubric items ( $R=3$ ), scored independently by $J=2$ LLM judges.

Panel of Figure 17 — Figure 17: Rubric-based prompt for detecting Type 1 hallucinations (spurious scenarios). An instance fails if any criterion scores 0.

Panel of Figure 18 — Figure 18: Rubric-based prompt for detecting Type 2 hallucinations (unfaithful references). An instance fails if any criterion scores 0.

Panel of Figure 19 — Figure 19: Rubric-based prompt for detecting Type 3 hallucinations (weak negatives). An instance fails if any criterion scores 0. All three audit prompts (Figures 17 – 19 ) are administered to an external LLM judge (GLM-5-Turbo) not used elsewhere in the pipeline; human auditors follow the same rubrics. Results are reported in Table 6 .

Personalized Language Model Alignment

Personalized language model alignment adapts model behavior to individual preferences while preserving safety, accuracy, and other shared constraints.

See all 61 papers

Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment
Feb 1, 2026Yanki Margalit, Nurit Cohen-Inger, Erni Avram +3LLM Evaluation AI Agent Evaluation
Computer Science and Information, Ben-Gurion University of the Negev
Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference
Aug 27, 2026Mengfan Li, Zesheng Wei, Xuanhua Shi +1Persona Consistency Language Model Generation Evaluation
National Engineering Research Center for Big Data Technology and System, Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology · Singapore Management University
PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models
May 16, 2026Wenlong Shi, Jianxun Lian, Mingqi Wu +5Large Language Model-Based Role-Play Simulation Personality Modeling in Language Models
College of Computer Science and Software Engineering, Shenzhen University, China · Microsoft Research Asia · Microsoft Gaming +2

Feb 1, 2026J/K move · Enter open · S save

Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

Yanki Margalit, Nurit Cohen-Inger, Erni Avram, Ran Taig+2

Computer Science and Information, Ben-Gurion University of the Negev

LLM Evaluation AI Agent Evaluation Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Large language model (LLM) evaluation commonly relies on fixed human-authored benchmarks or centralized human/model judges. Fixed benchmarks can age and enter model training corpora, while centralized judges introduce their own scoring preferences. We cast LLM evaluation as a multi-agent peer-assessment problem and introduce Agents-as-Peers, a distributed evaluation protocol in which every agent generates tasks, answers the shared task set, and judges the answers of its peers, with no privileged evaluator. Self-ratings are excluded from the ranking, and judging is repeated under controlled identity and presentation regimes so that self bias, identity effects, presentation-position sensitivity, and judge severity are measured rather than left implicit. In an instantiation with 12 LLMs and 420 agent-generated questions, Agents-as-Peers aggregates 55,440 non-self peer judgments per regime. Most agents assign their own answers higher scores than those answers receive from peer judges; visible identity changes scores for some agents; the first fixed answer slot is associated with scores 0.39 points higher than the shuffled baseline; and judges differ markedly in severity. On labeled benchmarks, peer scores align strongly with accuracy on TruthfulQA (Pearson r=0.904) and GSM8K (r=0.873). On TruthfulQA, the peer-score correlation is substantially higher than the self-score correlation (r=0.538). A benchmark-overlap screen and item-level human audit further characterize the endogenous task set. These results support Agents-as-Peers as a cohort-relative measurement protocol whose ranking is reported together with the biases and rater effects that shape it. Code and dataset: https://github.com/caura-ai/caura-PeerRank

Abstract

Personalized Language Model Alignment

Explore similar work

Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference

PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models