Multi-dimensional Bias in Modeling Multi-dimensional Preferences: Evaluating the Ability of Synthetic Agents to Replace Human Participants in Conjoint Experiments

Ho Ting Hung; Nachiket Midha; Victor Y. Wu; Yiwen Zhang

cs.MAAug 13, 2026

Multi-dimensional Bias in Modeling Multi-dimensional Preferences: Evaluating the Ability of Synthetic Agents to Replace Human Participants in Conjoint Experiments

Authors: Ho Ting Hung, Nachiket Midha, Victor Y. Wu, Yiwen Zhang

Organizations: Department of Politics and International Relations and the Nuffield Centre for Experimental Social Sciences at the University of Oxford · Department of Politics and International Relations, University of Oxford · Oxford Computational Political Science Group · Department of Political Science, Stanford University ·

Abstract

Despite growing interest in using LLMs to add robustness or reduce data-collection costs in survey experiments, their efficacy in conjoint design---an increasingly popular method in political science---remains underexplored. This paper addresses that gap by investigating whether synthetic agents can reproduce the multi-dimensional human preference patterns that conjoint is designed to capture. It replicates published conjoint studies and compares the results generated by synthetic agents with original human data along three dimensions: representational correspondence, inferential correspondence, and procedural stability. Our analysis evaluates the alignment of choice distributions as well as the statistical and substantive similarity of estimates, and the results are uneven across these dimensions and studies replicated. This implies that the validity of synthetic participants should be considered claim-dependent and hierarchical. Reproducing a figure or obtaining strong sign agreement is evidence of similar aggregate outputs, but not enough to support replacing human respondents. Our results suggest that the discipline as a whole must first map this innovation's boundaries across various levels before considering synthetic agents a robust substitute for human samples.

Figures & tables

Authors	Topic	Tasks $\times$ Sample
7	Climate Migration	19,440
9	Global Climate Agreements	34,000
29	Immigration	7,035
30	Housing	15,095
47	Candidate Gender Effects	15,830
61	Women in Politics	11,955

Table 1: Selected Studies for Replication

Hypothesis	Test Metric	Methodology	Threshold	Substantive Justification
Hypothesis 1 : Distribution	Wasserstein Distance ( $W$ )	Wasserstein Distance using Manhattan distance on one-hot profiles; $B=2000$ permutation test	$p\geq 0.05$	Observed distance falls within the 95% CI of human-to-human resampling; indistinguishable from natural sampling noise
	Hellinger Distance ( $H$ )	Distance between marginal choice probability vectors	95% CI upper bound $<0.3$	Ensures aggregate distributional alignment
	Pearson’s Correlation ( $r$ )	Correlation of empirical choice shares across all $M$ attribute levels	95% CI lower bound $>0.7$	Indicates aggregate preference weight alignment
	Weighted F1-Score	Weighted average of profile-level alignment across category classes	Score $>0.7$	Individual-level choices are captured with significantly higher accuracy than random-guess baselines
Hypothesis 2 : Similarity	Pearson’s Correlation ( $r$ )	Correlation between the vectors of human and synthetic AMCEs	95% CI lower bound $>0.7$	Assesses strong pattern alignment across the entire set of estimated treatment effects
	Root Mean Squared Error (RMSE)	Root mean squared difference between $\hat{\beta}_{m}^{A}$ and $\hat{\beta}_{m}^{H}$ across $M$ levels	RMSE $<0.05$	Replicated effects do not deviate by more than a common median human AMCE (0.05)

Table 2: Summary of Statistical Tests

Figure 1: Permutation-Calibrated Excess Wasserstein Distance

Figure 4: Conditional AMCE Heterogeneity across Studies

Figure 5: Study-Level Median Stability Ratios

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Original Main Estimand	Original Heterogeneity Estimand	Included Heterogeneity Variables or Subsets	Excluded Heterogeneity Variables or Subsets
7	AMCE and/or MM for conjoint profiles	Subgroup marginal means	Empathy mean/quartile; age mean/quartile; education college; employed; unemployed; US partisanship	US border-state indicators; US urban indicators; Germany East/West state indicators; Germany urban indicators; state-region indicators
9	Survey-weighted AMCE and MM	Interaction-style heterogeneity	Left/right ideology	Support for international environmental cooperation; reciprocity support / related attitudinal moderators
29	Post-stratification–weighted AMCE, with restricted-design adjustment for constrained attribute combinations	Interaction-based conditional AMCE / AICE	Education; party ID; income; White/non-white; Hispanic/not Hispanic; ideology; gender; age	Ethnocentrism; industry/fiscal exposure; ZIP diversity; immigration attitude
30	AMCE	AMCE estimated inside filtered subsets	Homeowner/renter $\times$ market-rate/affordable; city-rent quintiles $\times$ affordability; ZIP-rent quintiles $\times$ affordability; employed renters $\times$ rent-burden quintiles $\times$ affordability; homeowner income split; homeowner ideology split	Price anxiety / city-interest subset; Proposition I ban-support subset; citywide supply-support subset
47	AMCE	Interaction-based conditional AMCE / AICE; subset conditional AMCE	Respondent sex; education; age group; class; region; race; partisanship; same-party vs different-party pairings $\times$ respondent partisanship	N/A
61	AMCE	Respondent-level heterogeneity; candidate-gender subset AMCE; social-role profile marginal means	Respondent gender; respondent party; candidate gender subsets; candidate gender $\times$ marriage/spouse $\times$ children social-role profile category	Respondent age; respondent education; respondent income

Appendix

Table A.1: Overview of Original Estimands and Heterogeneity Variables across Included Studies

Study	Model	Temperature	Proportion
7 (Germany)	Llama 3.2	0.5	0.9942
7 (Germany)	GPT-4o	0.5	0.9999
7 (Germany)	GPT-4o mini	0.5	0.9996
7 (US)	GPT-4o	0.5	0.9999
7 (US)	GPT-4o mini	0.5	0.9994
29	Llama 3.2	0.5	0.9989

Appendix

Table C.1: Valid Choice Extraction Proportions across Studies and Models (<100%)

Comparison Pattern	n	Percent
Both flag model-human divergence	121	73.3%
Neither flags divergence	39	23.6%
Raw suggests divergence; permutation does not	5	3.03%

Appendix

Table D.1: Comparison of Divergence Flags Between Raw and Permutation-Calibrated Wasserstein Distance

Study	Mean $N$	Mean $\|\hat{\mathcal{S}}\|$	$N/\|\hat{\mathcal{S}}\|$	Singleton %	$\leq 2$ %	$\leq 5$ %
7 (Germany)	10,417.8	5,556.4	1.88	47.68	77.07	99.32
7 (US)	10,762.6	5,746.2	1.87	48.07	77.54	99.07
9	34,000.0	2,812.2	12.09	2.94	6.29	18.71
29	7,954.4	7,003.9	1.14	99.55	99.99	99.99
30	15,094.2	5,155.7	2.93	24.82	48.81	90.86
47 (Congress)	7,868.3	7,866.7	1.00	99.98	100.00	100.00

Appendix

Table D.2: Empirical Support Diagnostics across Replicated Studies

Figure G.1: Conditional MM Heterogeneity across Studies

Figure G.2: Profile MM Heterogeneity across Studies

Figure G.3: Subset AMCE Heterogeneity across Studies

Figure G.4: Subset cAMCE Heterogeneity across Studies

Figure H.1: Bootstrap CI Classification for Stability Ratios

Figure H.2: Distribution of Stability Ratios Across Analytic Choices

Figure I.1: Random-Effects Model Results: Between-Setting Variance

Figure I.2: Range of Estimates Across Analytic Choices

Synthetic Data Evaluation

Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.

See all 235 papers

Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data
Feb 10, 2026Jason Miklian, Kristian Hoelscher, John E. KatsosHuman Behavior Simulation Synthetic Data Generation
University of Oslo · Peace Research Institute Oslo · American University of Sharjah
Synthetic Cultural Agents from Aggregate Anchors
Oct 5, 2026Augusto Gonzalez-Bonorino, Kseniia Biriukova, Monica CapraLarge Language Model-Based Social Simulation Synthetic Data Generation for Language Models
Department of Economics, Arizona State University · EconLLM Lab · Department of Information Systems, Arizona State University +1
Your Reviews Replicate You: LLM-Based Agents as Customer Digital Twins for Conjoint Analysis
Mar 6, 2026Bin Xuan, Jungmin Hwang, Hakyeon LeePreference Elicitation Digital Twins
Department of Data Science Seoul National University of Science and Technology Seoul, Republic of Korea · Department of Industrial Engineering Seoul National University of Science and Technology Seoul, Republic of Korea

Feb 10, 2026J/K move · Enter open · S save

Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data

Jason Miklian, Kristian Hoelscher, John E. Katsos

University of Oslo · Peace Research Institute Oslo · American University of Sharjah

Human Behavior Simulation Synthetic Data Generation Synthetic Data Evaluation

Open paper AskPDF arXiv

How well can AI-derived synthetic research data replicate the responses of human participants? An emerging literature has begun to engage with this question, which carries deep implications for organizational research practice. This article presents a comparison between a human-respondent survey of 420 Silicon Valley coders and developers and synthetic survey data designed to simulate real survey takers generated by five leading Generative AI Large Language Models: ChatGPT Thinking 5 Pro, Claude Sonnet 4.5 Pro plus Claude CoWork 1.123, Gemini Advanced 2.5 Pro, Incredible 1.0, and DeepSeek 3.2. Our findings reveal that while AI agents produced technically plausible results that lean more towards replicability and harmonization than assumed, none were able to capture the counterintuitive insights that made the human survey valuable. Moreover, deviations grouped together for all models, leaving the real data as the outlier. Our key finding is that while leading LLMs are increasingly being used to scale, replicate and replace human survey responses in research, these advances only show an increased capacity to parrot conventional wisdom in harmony with each other rather than revealing novel findings. If synthetic respondents are used in future research, we need more replicable validation protocols and reporting standards for when and where synthetic survey data can be used responsibly, a gap that this paper fills. Our results suggest that synthetic survey responses cannot meaningfully model real human social beliefs within organizations, particularly in contexts lacking previously documented evidence. We conclude that synthetic survey-based research should be cast not as a substitute for rigorous survey methods, but as an increasingly reliable pre- or post-fieldwork instrument for identifying societal assumptions, conventional wisdoms, and other expectations about research populations.

Abstract

Synthetic Data Evaluation

Explore similar work

Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data

Synthetic Cultural Agents from Aggregate Anchors

Your Reviews Replicate You: LLM-Based Agents as Customer Digital Twins for Conjoint Analysis