Neither Over- nor Under-Constrained: Evaluating LLM Personalization under a Constraint Qualification Audit

Xuran Li; Guanqin Zhang; Imran Razzak; Hakim Hacid; Eleanna Kafeza; Hao Xue; Flora D. Salim

cs.AIOct 19, 2026

Neither Over- nor Under-Constrained: Evaluating LLM Personalization under a Constraint Qualification Audit

Authors: Xuran Li, Guanqin Zhang, Imran Razzak, Hakim Hacid, Eleanna Kafeza, Hao Xue, Flora D. Salim

Organizations: University of New South Wales · Mohamed bin Zayed University of Artificial Intelligence · The Technology Innovation Institute · The Hong Kong University of Science and Technology

Abstract

Accurate evaluation of LLM personalization is hindered by current evaluation paradigms that over- or under-constrain the response space, yielding frequent false rejections and false acceptances. To reduce these false rates, we introduce ConQual-Audit, an audit-based framework that evaluates LLM personalization through a three-stage process: (1) extracting semantic constraints from user profiles and task-grounded reference; (2) qualifying constraints by pruning unnecessary constraints and recovering missing constraints; and (3) verifying candidate responses against the audited constraints via natural language inference. Across PrefEval and LaMP, ConQual-Audit outperforms nine baselines, attaining top AUCs (0.88 and 0.90) and the lowest false rate (0.21) on the leanest constraint sets at an amortizable audit cost. Crucially, this constraint audit is portable: integrating it into ExPerT and PREF helps revise smaller, higher-quality constraint sets to reduce their evaluation false rates.

Figures & tables

Figure 1: Personalized Response Space.

Figure 2: Evalauation pipeline of ConQual-Audit framework

Method	Pos. $\uparrow$	Neg. $\downarrow$	AUC $\uparrow$	FRR $\downarrow$	FAR $\downarrow$	False Rate $\downarrow$	Tokens $\downarrow$	#Cons.
BLEU-4	0.35	0.16	0.79	0.65	0.14	0.39	0.0	–
ROUGE-L	0.65	0.46	0.70	0.22	0.46	0.34	0.0	–
BERTScore	0.95	0.92	0.74	0.00	1.00	0.50	28.3	–
MoverScore	0.65	0.57	0.76	0.08	0.79	0.44	29.0	–
GEMBA	0.52	0.30	0.67	0.47	0.26	0.36	198.3	–
PerSE	0.41	0.26	0.66	0.62	0.20	0.41	1,212.6	–

Table 1: Results averaged over LaMP-1, LaMP-2 and LaMP-4. Best in bold , second underlined .

Method	Pos. $\uparrow$	Neg. $\downarrow$	AUC $\uparrow$	FRR $\downarrow$	FAR $\downarrow$	False Rate $\downarrow$	Tokens $\downarrow$	#Cons.
GEMBA	0.90	0.68	0.82	0.03	0.84	0.44	670.1	–
PerSE	0.87	0.55	0.81	0.13	0.45	0.29	1,889.2	–
FActScore	0.60	0.58	0.52	0.35	0.62	0.49	37,430.2	–
ExPerT	0.08	0.06	0.53	0.97	0.04	0.50	9,663.9	8.23
PREF	0.92	0.81	0.79	0.02	1.00	0.51	1,641.9	5.11
ExPerT-Audit	0.06	0.09	0.45	0.94	0.04	0.49	24,790.6	6.67

Table 2: Results on PrefEval. Best in bold , second underlined .

	ExPerT		PREF
	Orig.	Audit	Orig.	Audit
#Cons.	8.23	6.67	5.11	3.64
Removed / added	1.78 / 0.22		1.51 / 0.05
AUC $\uparrow$	0.53	0.45	0.82	0.78
FRR $\downarrow$	0.96	0.94	0.02	0.03
FAR $\downarrow$	0.04	0.04	1.00	0.89

Table 3: Auditing the constraints of ExPerT and PREF.

Figure 3: ConQual-Audit. (1) The generator $G$ derives one preference constraint and three answer constraints from the user preference and the truth-grounded reference (if verify directly, they accept the negative response ( $R=2/4=\tau$ )). (2) Qualification prunes ac_3 , which the reference makes optional, and finds no duplicate and no gap. (3) The verifier $V$ checks responses against the qualified constraint set: the positive response is accepted and the negative one rejected.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Evaluating LLM Personalization via Semantic Constraint Verification
Jun 15, 2026Xuran Li, Guanqin Zhang, Imran Razzak +4LLM Evaluation LLM Personalization
University of New South Wales · 2Mohamed bin Zayed University of Artificial Intelligence · 3The Technology Innovation Institute +1
Re-Centering Humans in LLM Personalization
Jun 4, 2026Lechen Zhang, Jiarui Liu, Tal AugustHuman Preference Evaluation LLM Evaluation
†University of Illinois Urbana-Champaign · ♯Carnegie Mellon University
PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?
Oct 13, 2026Xinyi Liu, Hooshang Nayyeri, Emine Yilmaz +3LLM Evaluation LLM-as-a-Judge
Amazon · University of Illinois Urbana-Champaign · University College London

Jun 15, 2026J/K move · Enter open · S save

Evaluating LLM Personalization via Semantic Constraint Verification

Xuran Li, Guanqin Zhang, Imran Razzak, Hakim Hacid+3

University of New South Wales · 2Mohamed bin Zayed University of Artificial Intelligence · 3The Technology Innovation Institute+1

LLM Evaluation LLM Personalization

Open paper AskPDF arXiv

Current evaluation paradigms for Large Language Model (LLM) personalization rely heavily on brittle surface-matching metrics or computationally expensive LLM-as-a-judge protocols, both of which lack interpretability. To address these limitations, we introduce Natural Language Inference Constraint Verification (NLICV), a scalable, semantically invariant framework that maps sentence meanings to truth-condition sets to verify personalization constraints via a Natural Language Inference (NLI) model. Moving beyond binary scoring, NLICV categorizes LLM behaviors into four distinct modes: personalization, generalization, sycophancy, and failure. Extensive experiments demonstrate that NLICV aligns closely with human annotations while drastically reducing the latency and token costs associated with LLM judges (up to 2100 inference speedup). Finally, through an ablation-based procedure, NLICV pinpoints the exact sentences driving the constraint verification, yielding faithful, understandable evidence for its evaluations.

Abstract

LLM Evaluation

Explore similar work

Evaluating LLM Personalization via Semantic Constraint Verification

Re-Centering Humans in LLM Personalization

PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?