Shared Autoregressive Context Can Distort Relationships in Synthetic Data

Thomas S. Robinson

cs.LGSep 26, 2026

Shared Autoregressive Context Can Distort Relationships in Synthetic Data

Authors: Thomas S. Robinson

Organizations: London School of Economics and Political Science

Abstract

Large language models can generate several records within one autoregressive completion, making earlier answers available as context for later records. This paper shows that such shared-completion batching can distort relationships among variables in the resulting synthetic data, using controlled tests on synthetic survey respondents. In a matched experiment on 2,000 European Social Survey profiles, generating ten rather than one respondent per request increases mean absolute error in within-country correlations by 48-58% for Qwen3.8-27B and 114-127% for Llama-3.3-70B-Instruct across three seeds, holding profiles, examples, questions and decoding parameters fixed. The distortion primarily reflects exaggerated relationship strength, while retaining substantial agreement with the human ordering of correlations. Controlled interventions establish answer history as a causal channel: re-pairing the same preceding values, with profiles and marginal distributions fixed, changes correlations among subsequently generated responses. Hiding preceding answers reduces correlation error in the tested settings but worsens marginal accuracy. Exploratory corrections across social-attitude, health and economic data likewise show that lower correlation error can coexist with worse marginal distributions and regression estimates. Request construction is therefore part of the data-generating process, and synthetic-data validity must be evaluated against the analyses the generated data are intended to support.

Figures & tables

Figure 1: Sharing a completion increases correlation error in the matched ESS experiment. (A,B) Mean absolute error against human data across

171

within-country correlations, with equal country weights and the same accepted respondents in all four conditions. Each line shows one seed. Error increases by

48\text{--}58\%

for Qwen and

114\text{--}127\%

for Llama. (C) The increase from one to ten respondents, with

95\%

paired-country percentile bootstrap intervals (

500

resamples). These intervals condition on the generated answers and twenty selected countries; they omit uncertainty over new seeds and other countries. Conditions below

99\%

validity are marked as quality failures. Variable pairs are not independent replications.

Figure 2: Manipulating earlier answers changes later relationships. The schematic shows the interventions; ranks illustrate pairing, with actual values fixed. (A) Mean correlation among newly generated tenth respondents, averaged across prompt templates, after positive or negative pairing of the same preceding values. This panel measures generated relationships, without comparing them with human data. Llama uses a separately validated NF4 runtime. (B) Qwen/NHANES correlation MAE for profiles accepted in all six conditions. (C) Tenth-respondent extensions, matching profiles across the two conditions. Filled markers average three repetitions; hollow markers show each repetition. Hiding history skips the earlier answers’ state updates while retaining the target’s logical token positions. Mean marginal error rises in every comparison (Tables 6 – 7 ).

Figure 3: Lower correlation error can coexist with worse marginal accuracy. Open (unfilled) points show errors for the uncorrected ten-person batches; filled points show errors after applying the full dependence map. Each arrow connects the results for one domain and model, averaged across three seeds on matched evaluation profiles. Left and down are better; every arrow moves left and up.

M

uses range-normalized Wasserstein distance; SD-normalized ACS results show the same trade-off. ACS quality failures remain included. This result concerns the full map; exact marginal preservation gives a separate diagnostic.

Appendix figures & tables54 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Seed	Policy	Size	Valid respondents	Invalid req.	Quality
Qwen3.8-27B	0	Independent	1	1,987/2,000 (99.4%)	13/2000	Pass
Qwen3.8-27B	0	Independent	10	1,980/2,000 (99.0%)	2/200	Pass
Qwen3.8-27B	0	Shared	1	1,986/2,000 (99.3%)	14/2000	Pass
Qwen3.8-27B	0	Shared	10	1,990/2,000 (99.5%)	1/200	Pass
Qwen3.8-27B	1	Independent	1	1,987/2,000 (99.4%)	13/2000	Pass
Qwen3.8-27B	1	Independent	10	1,980/2,000 (99.0%)	2/200	Pass

Appendix

Table 1: Production validity under whole-request acceptance: an invalid ten-person request loses all ten respondents. Failed-quality arms remain reported.

Model	Seed	$N$ common	MAE, size 1	MAE, size 10	$\Delta_{10-1}$	Conditional interval
Qwen3.8-27B	0	1,945	0.179	0.282	+0.103	[+0.067, +0.138]
Qwen3.8-27B	1	1,967	0.185	0.276	+0.091	[+0.073, +0.112]
Qwen3.8-27B	2	1,936	0.186	0.276 †	+0.089 †	[+0.057, +0.122]
Llama-3.3-70B	0	1,938	0.187	0.399 †	+0.213 †	[+0.190, +0.235]
Llama-3.3-70B	1	1,954	0.187	0.425 †	+0.238 †	[+0.210, +0.259]
Llama-3.3-70B	2	1,975	0.187	0.420	+0.233	[+0.207, +0.263]

Appendix

Table 2: Independent-seed request-size results on four-arm common IDs. MAE averages

171

within-country correlation errors; intervals use the conditional paired-country bootstrap.

\dagger

denotes an arm below

99\%

validity.

Model	Seed	Policy	$\Delta$ , 171 pairs	$\Delta$ , 136 pairs	136-pair interval
Qwen3.8-27B	0	Independent	+0.103	+0.118	[+0.080, +0.155]
Qwen3.8-27B	0	Shared	+0.144	+0.154	[+0.113, +0.174]
Qwen3.8-27B	1	Independent	+0.091	+0.098	[+0.075, +0.120]
Qwen3.8-27B	1	Shared	+0.053	+0.062	[+0.012, +0.107]
Qwen3.8-27B	2	Independent	+0.089 †	+0.096 †	[+0.063, +0.130]
Qwen3.8-27B	2	Shared	+0.109 †	+0.130 †	[+0.083, +0.166]

Appendix

Table 3: Matched-ID sensitivities:

\Delta

is size-ten minus size-one MAE. Excluding age/year leaves

136

pairs. Intervals use the same conditional country bootstrap. A dagger flags any failed-quality arm in four-arm matching, including the other seed policy.

Model	Domain	Positive	Zero	Negative	Hidden	$\Delta_{+,-}$	95% summary
Qwen	NHANES	0.963	0.182	-0.670	0.238	1.633	$[1.525,1.728]$
Qwen	ESS	0.985	0.145	-0.647	0.611	1.632	$[1.568,1.681]$
Llama-70B (NF4)	NHANES	0.837	–	-0.329	-0.036	1.166	withheld
Llama-70B (NF4)	ESS	0.725	-0.173	0.041	0.090	0.684	$[0.546,0.737]$

Appendix

Table 4: Equal-template correlations in newly generated tenth responses. Hidden history skips earlier-answer updates;

\Delta_{+,-}

is positive minus negative pairing. Intervals describe conditional order-mixture sensitivity. A dash is undefined; Llama/NHANES’s interval is withheld.

Model	Domain	Valid +	Valid 0	Valid $-$	Valid hidden	Matched/template	Undefined bootstraps
Qwen	NHANES	432/432	430/432	431/432	429/432	52–54	0/5000
Qwen	ESS	1080/1080	1080/1080	1080/1080	1079/1080	53–54	0/5000
Llama-70B (NF4)	NHANES	432/432	432/432	431/432	432/432	53–54	1393/5000
Llama-70B (NF4)	ESS	1080/1080	1080/1080	1080/1080	1080/1080	54–54	0/5000

Appendix

Table 5: Controlled-history validity and matched support under one strict rule across arms. Invalid answers remain archived; undefined bootstrap draws are counted, not discarded.

Position	$A_{\rm visible}$	$A_{\rm hidden}$	Change in $A$	95% interval	$M_{\rm visible}$	$M_{\rm hidden}$	Valid targets
2	0.324	0.213	-0.111	$[-0.144,-0.078]$	0.072	0.078	2383/2400
5	0.374	0.210	-0.164	$[-0.196,-0.129]$	0.067	0.074	2370/2400
10	0.367	0.197	-0.169	$[-0.223,-0.125]$	0.067	0.075	2375/2400

Appendix

Table 6: Qwen/NHANES, six-arm common support: three-repetition means of correlation MAE

A

and normalized marginal Wasserstein error

M

. Change is hidden minus visible; intervals resample paired profile blocks. Valid-target counts precede six-arm matching; each position plans

1{,}200

histories.

Model	Domain	$A_{\rm visible}$	$A_{\rm hidden}$	Reduction	95% interval for $\Delta A$	$M_{\rm visible}$	$M_{\rm hidden}$
Qwen	ACS	0.166	0.103	38%	$[-0.117,-0.015]$	0.0390	0.0399
Llama-70B (NF4)	NHANES	0.577	0.300	48%	$[-0.306,-0.243]$	0.0767	0.1060

Appendix

Table 7: Tenth-respondent extensions: three-repetition mean correlation MAE

A

and normalized marginal Wasserstein error

M

. Reduction is relative to visible history;

\Delta A

is hidden minus visible. All

400

profiles per repetition are retained; intervals use paired profile blocks.

Continuation arm	$A$	$P$	$A_{\rm or}$	$M_{\rm SD}$	$E_{\beta}$
Singleton	0.198	0.895	0.071	0.482	0.133
Ten profiles, target first	0.247	0.905	0.067	0.476	0.115
Visible history	0.363	0.851	0.080	0.402	0.157
Hidden, position gap	0.192	0.896	0.065	0.441	0.100
Hidden, contiguous	0.193	0.903	0.063	0.446	0.098

Appendix

Table 8: Same-runtime Qwen/NHANES continuations, three-repetition means on five-arm common support.

P

is correlation-vector Pearson agreement,

A_{\rm or}

human-oracle scalar MAE, and

E_{\beta}

the fixed PHQ-9 coefficient error. All five arms use forced target syntax.

Contrast	$\Delta A$	Fixed reference 95%	Resampled reference 95%
First minus singleton	0.048	[+0.031, +0.069]	[+0.031, +0.069]
Visible minus first	0.116	[+0.080, +0.167]	[+0.079, +0.168]
Hidden minus visible	-0.171	[-0.221, -0.127]	[-0.222, -0.127]
Contiguous minus gapped	0.001	[-0.015, +0.016]	[-0.015, +0.017]

Appendix

Table 9: Prespecified contrasts within the exploratory context follow-up. Paired bootstrap draws resample target IDs within cells, carrying all arms and repetitions. The second interval additionally resamples human reference rows within cells; neither is survey-design uncertainty. All

2{,}000

draws are defined in each sensitivity.

Position	Visible	Hidden	Hidden minus visible	Repetitions improved
2	0.105	0.079	-0.026	3/3
5	0.134	0.093	-0.040	3/3
10	0.156	0.101	-0.055	3/3

Appendix

Table 10: Exploratory PHQ-9 regression: three-repetition mean standardized-coefficient MAE

E_{\beta}

on the Qwen/NHANES six-arm common cohort. Hiding histories lowers error by

25\%

30\%

and

35\%

at positions two, five and ten, respectively. “Improved” refers to the mean over four coefficients, not each coefficient.

	ESS		NHANES		ACS †
Method	Qwen	Llama	Qwen	Llama	Qwen	Llama
Untouched batching	0.105	0.101	0.157	0.151	0.032	0.249
Marginal-only mapping	0.072	0.085	0.112	0.162	0.185	0.305
Learned dependence map	0.058	0.070	0.115	0.151	0.124	0.354
One-person row resampling	0.066	0.043	0.126	0.094	0.219	0.331
Identity-dependence target	0.135	0.129	0.135	0.078	0.183	0.263

Appendix

Table 11: All correction variants: three-repetition mean standardized-coefficient MAE

E_{\beta}

for the fixed domain-specific regressions. Lower is better; all variants share the original matched evaluation IDs within each model and repetition.

\dagger

: the six original ACS size-ten generation-quality failures remain included.

Domain	Model	$A_{0}$	One-person	Batch	Full map	Scalar	Gain retained
ESS	Qwen	0.193	0.182	0.286	0.147	0.156	93.2%
ESS	Llama	0.193	0.188	0.417	0.125	0.129	98.5%
NHANES	Qwen	0.160	0.249	0.396	0.212	0.210	100.9%
NHANES	Llama	0.160	0.332	0.517	0.204	0.217	95.9%
ACS †	Qwen	0.356	0.094	0.226	0.077	0.086	94.0%
ACS †	Llama	0.356	0.213	0.252	0.130	0.135	96.7%

Appendix

Table 12: A single overall adjustment retains most of the full correction’s improvement. Three-seed mean correlation MAE on matched evaluation respondents.

A_{0}

is the error from predicting zero correlation; one-person outputs provide another comparator. Gain retained is

(A_{B}-A_{\rm scalar})/(A_{B}-A_{\rm full})

. Both corrections use one-person fitting marginals and exclude human evaluation answers from fitting.

\dagger

: all six ACS ten-person quality failures are included.

Model	Method	$A$	$\overline{\|r\|}$	$S_{0}$	$P$	$\rho$	Sign	$A_{\rm or}$	$M_{\rm SD}$
Qwen	Batch	0.286	0.467	-0.483	0.785	0.859	0.904	0.099	0.517
Qwen	One-person	0.182	0.359	0.054	0.801	0.835	0.906	0.098	0.569
Qwen	Marginals only	0.225	0.404	-0.169	0.781	0.842	0.903	0.100	0.598
Qwen	Row resampling	0.187	0.361	0.030	0.801	0.820	0.908	0.098	0.576
Qwen	Identity target	0.174	0.060	0.096	0.417	0.320	0.571	0.174	0.606
Qwen	Full map	0.147	0.304	0.240	0.789	0.819	0.891	0.100	0.599

Appendix

Table 13: ESS calibration cohort: three-seed means,

A_{0}=0.193

P/\rho

: Pearson/Spearman correlation-vector agreement; Sign: sign agreement;

A_{\rm or}

: human-oracle scalar MAE. Exact variants preserve evaluated batch marginals; answer-fit scalar selects its parameter using generated fitting answers.

M_{\rm SD}

is defined in Equation 4 .

Model	Method	$A$	$\overline{\|r\|}$	$S_{0}$	$P$	$\rho$	Sign	$A_{\rm or}$	$M_{\rm SD}$
Qwen	Batch	0.396	0.527	-1.468	0.845	0.782	0.900	0.082	0.396
Qwen	One-person	0.249	0.380	-0.553	0.881	0.794	0.900	0.074	0.496
Qwen	Marginals only	0.339	0.470	-1.111	0.825	0.790	0.900	0.090	0.524
Qwen	Row resampling	0.303	0.434	-0.893	0.872	0.790	0.900	0.075	0.469
Qwen	Identity target	0.149	0.082	0.072	0.454	0.487	0.767	0.147	0.534
Qwen	Full map	0.212	0.330	-0.320	0.872	0.749	0.867	0.085	0.525

Appendix

Table 14: NHANES calibration cohort: same diagnostics as Table 13 , with

A_{0}=0.160

. The full and scalar maps improve raw MAE while remaining worse than the constant-zero correlation predictor.

Model	Method	$A$	$\overline{\|r\|}$	$S_{0}$	$P$	$\rho$	Sign	$A_{\rm or}$	$M_{\rm SD}$
Qwen	Batch	0.226	0.582	0.365	0.915	0.924	1.000	0.100	0.376
Qwen	One-person	0.094	0.429	0.736	0.919	0.752	1.000	0.071	0.397
Qwen	Marginals only	0.117	0.472	0.672	0.964	0.886	1.000	0.066	0.425
Qwen	Row resampling	0.101	0.457	0.717	0.959	0.943	1.000	0.059	0.437
Qwen	Identity target	0.177	0.211	0.502	0.922	0.276	0.778	0.177	0.424
Qwen	Full map	0.077	0.408	0.785	0.967	0.695	1.000	0.057	0.423

Appendix

Table 15: ACS calibration cohort: same diagnostics,

A_{0}=0.356

. All six original ten-person quality failures remain included. SD-normalized marginal error is foregrounded because the range scale downweights errors on the broad monetary ranges. There are only six pairs.

Cohort	Model	One-person	Batch	Full	Scalar
ESS primary	Qwen	0.208	0.337	—	—
ESS primary	Llama	0.212	0.591	—	—
ESS	Qwen	0.207	0.350	0.163	0.172
ESS	Llama	0.212	0.601	0.136	0.141
NHANES	Qwen	0.284	0.493	0.240	0.231
NHANES	Llama	0.386	0.666	0.224	0.235

Appendix

Table 16: Fisher-

z

correlation MAE, three-seed means. The first two rows use the original ESS request experiment; subsequent rows use the disjoint fitting/evaluation calibration study. No parameter is chosen using this sensitivity. ACS retains its quality failures.

Figure 4: Nested singleton-reference sizes, keeping all batched fitting rows. Shading is the observed seed/subset-order range, not a confidence interval. Full matrix and scalar use the same references and singleton marginal assignment. All ACS quality failures remain included; five/ten rows per group are exploratory low-support variants.

Domain	Model	Latent MAE	Latent mag. bias	Answer MAE	Answer mag. bias
ESS	Qwen	0.018	0.017	0.039	-0.034
ESS	Llama	0.047	0.045	0.075	-0.067
NHANES	Qwen	0.025	0.025	0.063	-0.024
NHANES	Llama	0.042	0.042	0.113	-0.113
ACS †	Qwen	0.017	0.017	0.060	-0.003
ACS †	Llama	0.046	0.046	0.082	-0.064

Appendix

Table 17: Fitting-target reproduction, three-seed means. Magnitude bias averages actual minus target

|r|

at each stage. Final answer correlations undershoot singleton fitting magnitudes in all six settings, despite latent overshoot. ACS quality failures remain retained.

Variant	ESS Q	ESS L	NHANES Q	NHANES L	ACS Q	ACS L
Full, $\gamma=0.1$	0.147	0.125	0.212	0.204	0.077	0.130
Full, $\gamma=0$	0.139	0.114	0.199	0.184	0.080	0.111
Full, $\gamma=0.01$	0.140	0.115	0.202	0.187	0.078	0.114
Full, $\gamma=0.05$	0.143	0.120	0.205	0.194	0.076	0.121
Full, $\gamma=0.25$	0.155	0.143	0.225	0.230	0.079	0.148
Full, $\gamma=0.5$	0.167	0.170	0.253	0.268	0.087	0.169

Appendix

Table 18: Transformation sensitivity: mean correlation MAE on unchanged matched evaluation cohorts. Q/L denote Qwen/Llama. Identity-linear variants retain the nonlinear rank/CDF steps; they differ from the identity target . All ACS quality failures remain included.

Model	Arm	$A$	$A_{\rm req}$	$\overline{\|r\|}$	$\rho$	$A_{\rm or}$	$L_{1}/p$	$L_{1,\rm req}/p$
Qwen	One-person	0.183	0.191	0.360	0.840	0.097	0.411	0.417
Qwen	Batch	0.278	0.370	0.459	0.864	0.098	0.505	0.593
Llama	One-person	0.187	0.216	0.355	0.798	0.099	0.415	0.447
Llama	Batch	0.415	0.480	0.596	0.866	0.099	0.669	0.736

Appendix

Table 19: Original ESS matched cohort, three-seed means.

A_{\rm req}

removes matched request means, retaining the group-centered human comparator.

L_{1}/p

is the largest correlation eigenvalue’s share, before/after request centering. Human mean

|r|

0.193

Target outcome	Visible slope	Cluster SE	Hidden slope	Cluster SE
Poverty–income ratio	0.043	0.023	-0.047	0.034
BMI	-0.042	0.031	0.031	0.032
Self-rated health	-0.060	0.027	0.055	0.030
PHQ-9	-0.081	0.029	0.004	0.034
Sedentary minutes	-0.071	0.040	0.015	0.041

Appendix

Table 20: Exploratory position-two predecessor regressions, conditional on profile and repetition fixed effects. Visible slopes oppose the predecessor loadings for all five outcomes, but are small. These are descriptive associations, not identified causal mediation effects.

Model/domain/position	History	$A$	$\overline{\|r\|}$	$S_{0}$	$P$	Sign	$A_{\rm or}$	$M_{\rm SD}$
Qwen/NHANES/2	Visible	0.324	0.455	-1.019	0.879	0.900	0.074	0.432
Qwen/NHANES/2	Hidden	0.213	0.336	-0.326	0.898	0.867	0.068	0.466
Qwen/NHANES/5	Visible	0.374	0.505	-1.334	0.865	0.900	0.075	0.402
Qwen/NHANES/5	Hidden	0.210	0.341	-0.310	0.868	0.900	0.073	0.439
Qwen/NHANES/10	Visible	0.367	0.498	-1.286	0.848	0.900	0.081	0.402
Qwen/NHANES/10	Hidden	0.197	0.328	-0.231	0.894	0.900	0.065	0.442

Appendix

Table 21: History diagnostics on each study’s original common cohort. Three-seed means use six-arm support for Qwen/NHANES and two-arm support for the extensions.

A_{0}=0.160

for NHANES and

0.356

for ACS. The strong raw Llama/NHANES gain disappears after oracle attenuation; Qwen/NHANES at position ten retains an improvement.

Domain	Model	$A_{\rm full}-A_{\rm scalar}$	Reference-only 95% interval
ESS	Qwen	-0.0095	[-0.0102, -0.0086]
ESS	Llama	-0.0043	[-0.0051, -0.0035]
NHANES	Qwen	0.0016	[-0.0077, +0.0106]
NHANES	Llama	-0.0130	[-0.0224, -0.0071]
ACS †	Qwen	-0.0090	[-0.0115, -0.0067]
ACS †	Llama	-0.0040	[-0.0048, -0.0040]

Appendix

Table 22: Full-map minus scalar MAE, with human-reference-only percentile intervals. Negative differences favour the full map. Generated outputs and fitted parameters are held fixed; these intervals do not include generation or survey-design uncertainty.

Domain	Model	Batch	Full	Scalar	Full/exact	Scalar/exact
ESS	Qwen	0.105	0.058	0.043	0.058	0.041
ESS	Llama	0.101	0.070	0.087	0.093	0.093
NHANES	Qwen	0.157	0.115	0.081	0.145	0.126
NHANES	Llama	0.151	0.151	0.151	0.177	0.167
ACS †	Qwen	0.032	0.124	0.110	0.074	0.027
ACS †	Llama	0.249	0.354	0.355	0.117	0.152

Appendix

Table 23: Fixed downstream coefficient MAE on matched evaluation respondents. Full/scalar use singleton fitting marginals; exact variants preserve evaluated batch marginals. Lower is better. ACS retains its six original failed-quality ten-person arms.

Domain	Model	Condition	$A_{1}$	$A_{10}$	$\Delta A$	Valid 1	Valid 10
ESS	Qwen	$T=0.5$	0.249	0.342	0.093	100.0%	100.0%
ESS	Qwen	$T=1.0$	0.185	0.272	0.087	99.9%	99.5%
ESS	Qwen	$T=1.5$ †	—	—	—	91.3%	51.5%
ESS	Qwen	Omit sentence, $T=1$ †	0.192	0.281	0.090	99.9%	99.0%
ESS	Llama	$T=0.5$	0.187	0.420	0.233	100.0%	100.0%
ESS	Llama	$T=1.0$	0.187	0.405	0.219	100.0%	100.0%

Appendix

Table 24: Correlation accuracy under each temperature and prompt condition. MAEs average three seeds, matching accepted profiles across request sizes within each condition as planned;

\Delta A=A_{10}-A_{1}

. Validity uses all planned requests before matching. A dagger marks a condition in which any size or seed has less than

99\%

validity or any truncation. Dashes indicate a three-seed mean that cannot be calculated under the support rule. These secondary estimates have no confidence intervals. Llama’s means also equal its primary estimates.

Domain	Model	Condition	$\Delta A$	95% interval	Change vs. $T=1$	95% interval
ESS	Qwen	$T=0.5$	—	Unsupported	—	Unsupported
ESS	Qwen	$T=1.0$	—	Unsupported	—	Unsupported
ESS	Qwen	$T=1.5$ †	—	Unsupported	—	Unsupported
ESS	Qwen	Omit sentence, $T=1$ †	—	Unsupported	—	Unsupported
ESS	Llama	$T=0.5$	0.233	[0.216, 0.255]	+0.014	[-0.003, +0.031]
ESS	Llama	$T=1.0$	0.219	[0.204, 0.241]	—	—

Appendix

Table 25: Primary contrasts using profiles accepted in all eight condition-by-size arms. Intervals use

2{,}000

paired request-block bootstrap draws within groups, carrying all conditions and seeds together and holding the human reference fixed. The last two columns compare the batching penalty with the new original-prompt

T=1

control. The fixed analysis rule determines which estimates and intervals are withheld. Intervals are conditional and pointwise, without multiplicity adjustment. Daggers follow Table 24 ’s rule.

Domain	Model	Seed	Shared $N$	Min./group	Valid high- $T$	Truncated	Supported
ESS	Qwen	0	885	9	98/200	38	No
ESS	Qwen	1	996	0	109/200	35	No
ESS	Qwen	2	940	26	102/200	37	Yes
ESS	Llama	0	2000	100	200/200	0	Yes
ESS	Llama	1	2000	100	200/200	0	Yes
ESS	Llama	2	2000	100	200/200	0	Yes

Appendix

Table 26: Profile counts for the primary comparison and validity at high temperature. Shared

N

and the minimum group count apply to all eight arms within a seed. Validity and truncation refer to original-prompt

T=1.5

ten-person requests. “Supported” indicates that primary point estimates can be calculated; it does not guarantee an interval.

Domain	Model	Condition	$b$	$\overline{\|r\|}$	$P$	$A_{\rm or}$	$M_{\rm SD}$	$E_{\beta}$
ESS	Qwen	$T=0.5$	1	0.429	0.769	0.104	0.616	0.071
ESS	Qwen	$T=0.5$	10	0.524	0.771	0.102	0.530	0.116
ESS	Qwen	$T=1.0$	1	0.362	0.804	0.097	0.563	0.062
ESS	Qwen	$T=1.0$	10	0.454	0.792	0.098	0.500	0.081
ESS	Qwen	Omit sentence, $T=1$ †	1	0.370	0.806	0.097	0.559	0.059
ESS	Qwen	Omit sentence, $T=1$ †	10	0.463	0.791	0.098	0.502	0.095

Appendix

Table 27: Relationship strength, pattern and analytic accuracy for the same matched profiles as Table 24 . Every mean requires all three seeds; Qwen/ESS

T=1.5

fails this rule.

b

is request size,

P

is Pearson agreement between the correlation vectors,

A_{\rm or}

is MAE after the human-oracle magnitude adjustment,

M_{\rm SD}

is marginal Wasserstein error and

E_{\beta}

is regression coefficient MAE. Daggers use the rule above.

Figure 5: Predicting generated correlations does not establish their accuracy. The reference panel supplies

738

model-by-pair comparisons, excluding household size. Absolute synthetic correlations exceed the real reference in 98.2% of comparisons; these are not independent tests. Left: real versus synthetic pooled within-cell correlations; the dashed line represents agreement. Right: synthetic within-cell versus real between-cell correlations. The rank interval resamples variables and excludes generation uncertainty. Opinion surveys supply

684

comparisons; the other domains supply eighteen each.

Model	Mean absolute error	Median $\|r_{real}\|$	Median $\|r_{synth}\|$
Claude Opus 5	0.278	0.127	0.476
Llama-3.3-70B	0.410	0.127	0.696
Qwen3-235B	0.413	0.127	0.644
GPT-oss-120B	0.410	0.127	0.641

Appendix

Table 28: Correlation accuracy in the reference opinion panel,

171

pairs per model. Mean absolute error is the mean of

|r_{W}^{synth}-r_{W}^{real}|

across pairs.

Domain	Rows	Spearman $\rho$	Slope on aggregates	$B$
Opinion survey	684	$0.823\ [0.57,0.87]$	$0.917\ [0.44,1.17]$	1000
Health	18	$0.868\ [0.34,0.87]$	$1.220\ [1.14,1.74]$	640
Labor market	18	$0.868\ [0.68,0.87]$	$1.545\ [1.28,1.65]$	627
Expenditure	18	$0.310\ [-0.62,0.80]$	$0.596\ [-2.27,2.63]$	672
All domains	738	$0.796\ [0.58,0.86]$	$0.903\ [0.57,1.12]$	1000

Appendix

Table 29: Aggregate predictability on the primary battery. Intervals resample variables within domains;

B

counts usable replicates out of

1{,}000

. The three four-variable domain intervals are descriptive, with no coverage guarantee.

Model	Respondents/request	Max. output tokens	Access
Claude Opus 5	50	24,576	Anthropic Batch API
Llama-3.3-70B-Instruct	20	16,384	Together Turbo
Qwen3-235B-A22B-2507	20	16,384	OpenRouter, fp8
GPT-oss-120B	20	16,384	Together

Appendix

Table 30: Opinion-panel generation settings. Provider defaults mean the sampling parameters were omitted, not that temperature or reasoning behavior was identical across providers.

Frame	Countries	Income	Happiness	Democracy
Real	35	0.062	0.205	0.380
Claude Opus 5	35	0.371	0.495	0.664
Llama-3.3-70B	35	0.711	0.703	0.752
Qwen3-235B	35	0.602	0.673	0.768
GPT-oss-120B	35	0.652	0.689	0.767
Real	32	0.063	0.206	0.379

Appendix

Table 31: Strict common-country, common-weight checks. Entries correlate trust in police with the column item. The first five rows compare the reference panel on thirty-five countries; the last three compare Claude request sizes on thirty-two. Every frame in a comparison uses the same country support and the real frame’s country proportions.

Round	Edition	Release year	DOI
1	6.7	2023	10.21338/ess1e06_7
2	3.6	2012	10.21338/ess2e03_6
3	3.7	2018	10.21338/ess3e03_7
4	4.6	2023	10.21338/ess4e04_6
5	3.6	2025	10.21338/ess5e03_6
6	2.7	2025	10.21338/ess6e02_7

Appendix

Table 32: ESS ERIC source editions. Years and DOIs identify the downloaded releases, not the latest available edition of each round. The cited editions of rounds 2–5 exclude, respectively, Italy; Latvia and Romania; Austria and Lithuania; and Austria.

Figure 6: Request-size sensitivity under the archival shared-seed protocol. Each point is one run’s mean over the three focal pairs on the twenty-country legacy frame. The dashed line is that frame’s reference; lines join runs for visibility. Examples also change between runs. Only request sizes one, ten and fifty were evaluated.

Respondents per request	Seed 0	Seed 1	Seed 2	Median
50	0.614703	0.634653	0.650875	0.634653
10	0.621342	0.647921	0.593515	0.621342
1	0.080197	0.061348	0.045335	0.061348
Real reference	0.192457

Appendix

Table 33: Mean focal correlations by request size and seed at temperature

1.0

, computed directly from the retained draws. Seed ranges summarize three runs varying decoding and exemplars.

Respondents per request	$T=0.3$	$T=0.7$	$T=1.0$	$T=1.3$
50	0.736	0.714	0.635	0.702
1	0.078	0.075	0.061	0.116

Appendix

Table 34: Temperature sensitivity: median focal correlation over three seeds. The legacy real reference on the twenty requested countries is

0.192

; the

T=1.3

single-respondent arm retains only eighteen countries.

Study	Groups	Variables / pairs	Human source	Human reference	Frontier profiles	Transfer fit / eval.
ESS	$20$	$19/171$	$79{,}990$	$19{,}896$	$2{,}000$	$1{,}000/1{,}000$
NHANES	$8$	$5/10$	$2{,}489$	$1{,}666$	$800$	$400/400$
ACS	$8$	$4/6$	$22{,}218$	$15{,}085$	$800$	$400/400$

Appendix

Table 35: Frozen populations and planned synthetic cohorts. Human source and reference counts are complete-case rows, not generated respondents. Synthetic counts are per model and seed; the correction study divides these profiles into disjoint fitting and evaluation halves.

Domain	Model	$1$	$2$	$5$	$10$	$\Delta_{10-1}$	Size-10 validity	Failed
ESS	Qwen	0.184	0.210	0.251	0.277	+0.094	99.0–100.0%	0
ESS	Llama	0.184	0.270	0.365	0.410	+0.226	100.0–100.0%	0
NHANES	Qwen	0.265	–	–	0.413	+0.148	100.0–100.0%	0
NHANES	Llama	0.337	–	–	0.518	+0.181	100.0–100.0%	0
ACS	Qwen	0.090	–	–	0.214	+0.124	95.0–97.5%	3
ACS	Llama	0.200	–	–	0.223	+0.023	93.8–98.8%	3

Appendix

Table 36: Request-size comparisons under constrained output formats. Entries are matched-ID pair MAE means over three generation seeds.

\Delta_{10-1}

is size ten minus size one. A size without a registered arm is shown as unavailable. The validity range is the fraction of requested size-ten respondents retained; failed arms remain included.

Figure 7: Request-size comparisons under constrained output formats. Lines join the matched-ID pair MAE means at the registered sizes within each domain and model. ESS has sizes

1,2,5,10

; NHANES and ACS have sizes

1,10

. The ACS size-ten arms have six quality failures in total, retained in the plotted means.

Domain	Model	Failed arms	Size-10 validity
ESS	Qwen	0	99.00%–100.00%
ESS	Llama	0	100.00%–100.00%
NHANES	Qwen	0	100.00%–100.00%
NHANES	Llama	0	100.00%–100.00%
ACS	Qwen	3	95.00%–97.50%
ACS	Llama	3	93.75%–98.75%

Appendix

Table 37: Quality flags for the structured request-size comparisons. The gate is at least

99\%

retained respondents and zero truncations for every model–seed–size arm.

Domain	Model	$A$ : batch	$A$ : learned	$\Delta A$	$M$ : batch	$M$ : learned	$\Delta M$
ESS	Qwen	0.286	0.147	-0.140	0.115	0.134	+0.019
ESS	Llama	0.417	0.125	-0.292	0.108	0.148	+0.039
NHANES	Qwen	0.396	0.212	-0.184	0.066	0.090	+0.024
NHANES	Llama	0.517	0.204	-0.313	0.069	0.108	+0.039
ACS †	Qwen	0.226	0.077	-0.149	0.043	0.051	+0.008
ACS †	Llama	0.252	0.130	-0.121	0.049	0.064	+0.015

Appendix

Table 38: Correlation and individual-variable accuracy after correction.

A

is correlation MAE on matched held-out profiles and

M

is normalized marginal Wasserstein-1 error; lower is better. Entries average three generation seeds. Deltas are learned dependence minus untouched batch. The ACS dagger marks six size-ten quality-failed model–seed arms, which remain included.

Domain	Model	Batch 10	Marginal map	Learned dependence	Row resampling	Identity target	Singleton anchor
ESS	Qwen	0.286	0.225	0.147	0.187	0.174	0.182
ESS	Llama	0.417	0.277	0.125	0.195	0.167	0.188
NHANES	Qwen	0.396	0.339	0.212	0.303	0.149	0.249
NHANES	Llama	0.517	0.357	0.204	0.378	0.135	0.332
ACS	Qwen	0.226	0.117	0.077	0.101	0.177	0.094
ACS	Llama	0.252	0.196	0.130	0.212	0.216	0.213

Appendix

Table 39: Association-error variants. Entries are matched-evaluation means over three generation seeds. The full correction does not always have the lowest error. The table includes every comparison specified in the original protocol.

Domain	Model	Batch 10	Marginal map	Learned dependence	Row resampling	Identity target	Singleton anchor
ESS	Qwen	0.115	0.133	0.134	0.129	0.135	0.127
ESS	Llama	0.108	0.146	0.148	0.144	0.148	0.144
NHANES	Qwen	0.066	0.089	0.090	0.081	0.092	0.084
NHANES	Llama	0.069	0.106	0.108	0.100	0.107	0.107
ACS	Qwen	0.043	0.051	0.051	0.051	0.051	0.048
ACS	Llama	0.049	0.064	0.064	0.063	0.065	0.061

Appendix

Table 40: Marginal-error variants. Entries are matched-evaluation means of normalized Wasserstein-1 error over three generation seeds. The same output cohorts and locked variants as Table 39 are used.

Model	Original-study variant	Raw (6 pairs)	Omit income–wages (5)	Asinh money (6)
Qwen	Uncorrected batch	0.226	0.253	0.210
Qwen	Marginals only	0.117	0.124	0.104
Qwen	Original learned map	0.077	0.077	0.076
Qwen	Whole-row resampling	0.101	0.104	0.100
Qwen	Identity target	0.177	0.200	0.273
Qwen	Held-out one-person	0.094	0.094	0.111

Appendix

Table 41: Prespecified ACS correlation-error sensitivities for all six original variants. Lower is better. All six size-ten quality failures are retained.

	ESS		NHANES		ACS †
Method ( $A/M$ )	Qwen	Llama	Qwen	Llama	Qwen	Llama
Uncorrected batch	0.286/ 0.115	0.417/ 0.108	0.396/ 0.066	0.517/ 0.069	0.226/ 0.043	0.252/ 0.049
Marginals only	0.225/0.133	0.277/0.146	0.339/0.089	0.357/0.106	0.117/0.051	0.196/0.064
Original learned map	0.147 /0.134	0.125 /0.148	0.212/0.090	0.204/0.108	0.077 /0.051	0.130 /0.064
Whole-row resampling	0.187/0.129	0.195/0.144	0.303/0.081	0.378/0.100	0.101/0.051	0.212/0.063
Identity target	0.174/0.135	0.167/0.148	0.149 /0.092	0.135 /0.107	0.177/0.051	0.216/0.065

Appendix

Table 42: Original and first exploratory controls: correlation MAE

A

and range-normalized marginal Wasserstein error

M

. Means use the same matched evaluation cohorts and three seeds. Bold denotes minima within these eight original methods, including ties, not all later scalar controls. Exact-batch preserves

M

\dagger

: six ACS size-ten quality failures retained. SD-normalized ACS results appear in Table 15 .

Figure 8: All eight methods on the same matched evaluation cohorts, averaged across three generation seeds. Lower is better on both axes. The exact-preservation control has the raw batch’s marginal error by construction; its correlation improvement is measured. ACS panels retain all six failed size-ten quality arms. Means do not imply improvement in every seed.

		Original map: $\Delta A$			Exact batch: $\Delta A$
Domain	Model	Seed 0	Seed 1	Seed 2	Seed 0	Seed 1	Seed 2
ESS	Qwen	-0.145	-0.145	-0.128	-0.103	-0.113	-0.106
ESS	Llama	-0.278	-0.302	-0.297	-0.217	-0.236	-0.238
NHANES	Qwen	-0.196	-0.120	-0.236	-0.167	-0.103	-0.176
NHANES	Llama	-0.311	-0.311	-0.317	-0.193	-0.205	-0.244
ACS †	Qwen	-0.120	-0.170	-0.158	-0.055	-0.058	-0.083

Appendix

Table 43: Seed-level correlation-error changes relative to the same uncorrected matched batch. Negative values improve accuracy. Exact preservation improves

17/18

contrasts; ACS/Llama seed 2 worsens by approximately

9\%

\dagger

: original ACS quality failures, retained without regeneration.

Panel	Model	Pairs	Common cells	Corr. MAE	W1 / ref. SD
ESS/WVS	Claude Opus 5	171	33–83	0.257	0.622
ESS/WVS	GPT-oss-120B	171	33–83	0.415	0.630
ESS/WVS	Llama-3.3-70B	171	33–83	0.428	0.621
ESS/WVS	Qwen3-235B	171	33–83	0.434	0.584
NHANES	GPT-oss-120B	6	69	0.345	0.632
NHANES	Llama-3.3-70B	6	69	0.659	0.720

Appendix

Table 44: Direct marginal audit on common pair-specific cells and equal cell weights. All

738

model–pair endpoints are supported. Entries average over pairs; cell ranges reflect missingness. Across thirteen domain–model arms, mean marginal errors span

0.275

–

0.720

human SDs. No equivalence threshold or test is specified; sampling error contributes. This is not a fully crossed four-model by four-domain experiment.

	ESS		NHANES		ACS †
Method	Qwen	Llama	Qwen	Llama	Qwen	Llama
Uncorrected batch	0.151	0.212	0.240	0.263	0.113	0.159
Marginals only	0.128	0.136	0.185	0.183	0.108	0.122
Original learned map	0.126	0.129	0.183	0.182	0.108	0.121
Whole-row resampling	0.138	0.134	0.233	0.231	0.153	0.158
Identity target	0.126	0.127	0.188	0.188	0.114	0.128

Appendix

Table 45: Profile-contrast error normalized by human pooled within-group SD, averaged over three seeds. Relative to untouched batching, the original and exact-preservation maps reduce domain–model means by approximately

4

–

39\%

and

1

–

10\%

, respectively; neither is uniformly best. All methods use the same metadata-selected support. Lower is better.

\dagger

: existing ACS quality failures retained.

	ESS		NHANES		ACS †
Method	Qwen	Llama	Qwen	Llama	Qwen	Llama
Uncorrected batch	0.576	0.568	0.519	0.559	0.395	0.454
Marginals only	0.638	0.686	0.596	0.676	0.435	0.507
Original learned map	0.638	0.689	0.594	0.679	0.435	0.504
Whole-row resampling	0.616	0.673	0.553	0.650	0.441	0.508
Identity target	0.640	0.689	0.594	0.670	0.434	0.507

Appendix

Table 46: Conditional marginal Wasserstein distance within supported profile categories, normalized by the same human pooled within-group SD. The original map raises every domain–model mean; exact preservation lowers every mean. The latter is measured, not guaranteed by preserving group marginals. Lower is better.

	ESS		NHANES		ACS †
Method	Qwen	Llama	Qwen	Llama	Qwen	Llama
Uncorrected batch	0.355	0.446	0.352	0.413	0.203	0.246
Marginals only	0.358	0.416	0.346	0.377	0.169	0.237
Original learned map	0.361	0.418	0.351	0.380	0.171	0.238
Whole-row resampling	0.339	0.417	0.355	0.398	0.215	0.282
Identity target	0.372	0.424	0.365	0.383	0.182	0.257

Appendix

Table 47: Absolute conditional mean error on the same profile support and human SD scale.

Domain	Attribute	Groups	Cells	Min. human $n$	Min. generated $n$
ESS	gender	20	40	266	15
ESS	marital status	20	61	34	5
ESS	education	19	60	50	5
ESS	citizen	3	6	49	6
ESS	belong religion	20	45	40	6
ESS	voted	18	39	57	5

Appendix

Table 48: Metadata-selected profile support, common across all models, seeds and variants. Minimum generated counts refer to one model–seed evaluation cohort, not pooled draws.

Group	Natural	Permuted	Aligned	$\Delta$
Men, 20-34	0.724	0.338	0.766	+0.428
Men, 35-49	0.481	-0.184	0.570	+0.754
Men, 50-64	0.386	0.098	0.566	+0.468
Men, 65+	0.340	0.203	0.658	+0.455
Women, 20-34	0.543	-0.072	0.649	+0.721
Women, 35-49	0.257	0.314	0.832	+0.518

Appendix

Table 49: BMI–sedentary correlations among newly generated tenth respondents only;

\Delta

is aligned minus permuted history. Injected records are excluded.

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 293 papers

Sep 12, 2026

Synthetic Data in Marketing Research: How to Evaluate and When to Trust

Debate over synthetic data in marketing research has polarized between claims that large language models (LLMs) make human respondents obsolete and calls to avoid them entirely. We argue that both positions obscure the more useful question: not whether synthetic respondents work, but when. Building on Brand, Israeli, and Ngwe (2026), we make three contributions. First, we distinguish three types of synthetic data (ungrounded LLM responses, segment-level personas, and individual-level digital twins) and map each to the decisions it can support. Second, we develop a taxonomy of four families of accuracy measures and suggest that the wide range of reported twin accuracy, from near-perfect to near-chance, largely reflects differences in what is being measured rather than in method quality. Aggregate measures often perform well even when little information is supplied to the LLM, and can mask a complete absence of respondent-level differentiation. Third, we introduce the forgotten question problem, in which a question is omitted from a fielded study, as a setting for twin-based augmentation of existing data. We propose an ex-ante answerability diagnostic that requires no ground truth: the R^2 of a random forest predicting twin outputs from the data used to construct the twins. Across 108 attitude questions from a nationally representative survey (N = 3,063), screening at R^2 above 0.7 raises the mean twin-human individual-level correlation by 15% and reduces the share of poorly answered questions from 25.9% to 4.3%. Embedding similarity and experienced-researcher judgment provide correlated but weaker screens.

Oded Netzer, Rajan Sambandam

Arthur J. Samberg Professor of Business, Columbia Business School · President, TRC Insights

Synthetic Data Evaluation Synthetic Data Augmentation Synthetic Data Generation

Ask PDF

Sep 28, 2026

Simulating Respondents, Not Single Questions: Coherent Survey Generation with Large Language Models

Large language models are increasingly used to simulate response distributions in social surveys. Prior work has achieved accurate population-level simulation for individual questions. Real questionnaires, however, ask each respondent a sequence of related questions. A simulated respondent should show coherent preferences across the whole questionnaire, not merely accurate distributions for isolated items. Existing single-item methods cannot accurately reproduce how the same person answers a complete survey. We propose FullRespondent-LLM (FR-LLM), which fine-tunes two specialized LLMs: a marginal model for each item's response distribution and a respondent-level autoregressive model for dependencies across answers. Marginal-Constrained Joint Projection (MCJP) then projects the autoregressive joint distribution onto the set satisfying the item-level marginals learned by the first model. This yields complete questionnaires with realistic cross-item relationships while retaining strong item-level accuracy. On two real-world social survey datasets, FR-LLM more accurately reproduces multi-question response patterns, maintains competitive single-item accuracy, and generalizes better to unseen populations and questions. In a small commercial-survey dataset, we use simulated responses to make pricing and stocking decisions; FR-LLM achieves the highest realized profit.

Ji Huang, Mengfei Li, Shuai Shao

School of Computer Science, University of Science and Technology of China · School of Management, Fudan University

Large Language Model-Based Social Simulation Conditional Generative Modeling

Ask PDF

Oct 22, 2026

Silicon Samples Match the Answers but Not the Respondents

Silicon sampling generates survey respondents by having a large language model (LLM) answer a survey as a person with given demographic attributes, and the resulting synthetic populations are increasingly used in place of human ones. Existing evaluations focus on individual-answer accuracy, per-question answer distributions, and correlations across questions. None of these compares the respondents themselves, so a generator whose respondents have combinations of answers found in no human respondent can still score well if it reproduces every answer distribution. We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples. On two US social surveys, all six LLMs we evaluate leave human respondents uncovered, and under our main settings, they fall behind a baseline that samples each answer independently from the human marginals. Across the prompting and inference settings we test, every LLM has a higher RCG than a Gaussian copula fitted to the human data.

Yusuke Kumagae, Masahiro Kaneko

Hakuhodo DY Holdings Inc. · MBZUAI

Synthetic Population Generation Synthetic Data Evaluation Large Language Model-Based Social Simulation

Ask PDF

Abstract

Synthetic Data Generation for Language Models

Explore similar work

Synthetic Data in Marketing Research: How to Evaluate and When to Trust

Simulating Respondents, Not Single Questions: Coherent Survey Generation with Large Language Models

Silicon Samples Match the Answers but Not the Respondents