Wrong Together: Shared Bias in User Simulation

Saber Zerhoudi

cs.CLOct 31, 2026

Wrong Together: Shared Bias in User Simulation

Authors: Saber Zerhoudi

Organizations: University of Passau Passau, Germany

Abstract

User simulators generate responses in place of human participants, allowing researchers to estimate the result of a study. To use these estimates before collecting human responses, researchers need a way to assess their reliability. We test whether disagreement among simulators can identify unreliable estimates of a group's response proportions and of differences between groups. We evaluate this signal using 6 language-model simulators and existing GlobalOpinionQA responses for 60 questions in 12 countries. Disagreement predicts invalidity more accurately for differences between two groups than for a single group's response proportions (AUC 0.87 versus 0.71). Errors shared by the simulators remain invisible to this signal. When such an error affects both groups equally, it cancels in the comparison, helping explain the stronger result for group differences. We test this explanation by estimating the shared bias using one set of groups and subtracting it from predictions for a separate set. This correction reduces prediction error without changing disagreement among the simulators. In a retrospective analysis, we simulate referring the 20% of group comparisons with the greatest disagreement for human assessment. Among the remaining comparisons with a conclusive validity judgment, 54.8% are judged invalid, compared with 63.7% before referral.

Figures & tables

Analysis	Quantity	Count
Study	Questions	60
	Populations	12
	Simulators	6
	Model families	4
Primary: proportions	Option-level estimates	2616
	Pairwise contrasts	14388

Table 1: Study scale and numbers of evaluated estimates. Primary estimates use response proportions. Secondary analyses use questions with ordered response options.

Figure 1: Error prediction, bias correction, and selective referral. (A) Disagreement predicts error better for contrasts than absolute levels (

\Delta\mathrm{AUC}=0.16

, question-clustered

95\%

CI). (B) Each point is test level error (MAE) on the test countries: before correction (

0.194

), after subtracting a per-question bias estimated on the

7

calibration countries (

0.129

), and after subtracting an offset from unrelated questions as a placebo (

0.246

). Only the real correction lowers the error. (C) Verdicts of the equivalence test on the contrasts that simulation still answers, as the highest-disagreement share is referred to humans. Referral shrinks the contradicted share, but about half of the answered contrasts stay unresolved at every threshold, so referral orders the human checks without certifying the rest. Among resolved contrasts, invalidity falls from

63.7\%

when simulation answers all of them to

54.8\%

when it answers the lowest-disagreement

80\%

Evaluation	Absolute level	Contrast	Difference d
All pairwise contrasts	0.71 [0.67, 0.75]	0.87 [0.85, 0.90]	$+$ 0.16 [0.13, 0.19]
Test-question AUC	0.73	0.86	$+$ 0.14
Test-population pairs only	0.72	0.88	$+$ 0.16 a
Reference rotated (non-US)	—	0.87	b
Invalidity, full coverage	—	63.7%	c
Invalidity, $80\%$ coverage	—	54.8%	c

Table 2: Level and contrast AUC on response-option proportions. Prediction is evaluated on all pairs, test questions, and test-population pairs. Referral retains the lowest-disagreement

80\%

of contrasts. AUC is question-weighted, with question-clustered

95\%

confidence intervals. Validity uses a fixed equivalence margin.

Figure 2: Shared and simulator-specific error.

58\%

of error variance is shared across simulators, and

65\%

of that component is constant across countries. A contrast cancels the country-constant component. Country-varying shared error and simulator-specific error remain.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Selective simulation on test populations. (a) Share of invalid estimates in each fifth of disagreement, ranked within family so the families share one axis. The trend differs by family: contrast invalidity rises from the lowest to the highest fifth, distribution invalidity does not. (b) Referring the highest-disagreement estimates lowers invalidity among the answered contrast estimates; the same rule gives a smaller reduction across all families combined. The dotted line marks $80\%$ coverage.

Table 6

Panel of Figure 4 — Figure 4: Predictability and mechanism by family. (a) ROC curves, one per estimand family, using disagreement to catch invalid estimates: contrasts and direction rise above the diagonal, means and distributions stay near it (AUC is shown for each family, with a dotted line at $0.5$ ). (b) Fraction of estimand-error variance shared across simulators (invisible to disagreement) versus simulator-specific. Contrast error contains a larger simulator-specific component than mean or distribution error. Family-level invalidity rates are in Table 6 .

Figure 5: The contrast signal is stable and grows with the ensemble. (A) Contrast AUC against the equivalence margin. (B) Contrast AUC against the assumed human sample size. (C) Verdicts on all estimates against the same sample size: the invalid share among resolved estimates stays between

56\%

and

58\%

, while the unresolved share falls from

22.3\%

5.4\%

as the assumed human study grows. An uncertain respondent count therefore changes how many estimates are resolved more than how the resolved ones split. (D) Contrast AUC against the number of simulators

K

(bars are standard deviations over random

K

-subsets). Filled markers are the defaults. The dashed line is

0.5

Table 9

Panel of Figure 6 — Figure 6: From risk score to decision (contrast, test populations). (a) Probability calibration on test populations (ECE $0.080$ ). (b) Expected cost under the assigned error and referral costs.

User Simulation

User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.

See all 135 papers

Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
Investigating Assistant Bias in LLM User Simulators Using a Role Vector
Sep 1, 2026Daeheon Jeong, Yoonjoo Lee, Eugene Choi +2Large Language Model-Based Role-Play Simulation User Simulation
1KAIST · University of Michigan · 3Seoul National University +2
Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
May 8, 2026Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar +4Human Behavior Simulation User Simulation
University of Illinois Urbana-Champaign · Microsoft Research · University of California, Berkeley

Mar 11, 2026J/K move · Enter open · S save

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie+7

Carnegie Mellon University, Language Technologies Institute

User Simulation Sim-to-Real Transfer LLM Agent Evaluation Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full

τ

-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Abstract

User Simulation

Explore similar work

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Investigating Assistant Bias in LLM User Simulators Using a Role Vector

Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors