Do LLMs Reproduce the Age Gap in Loneliness? Evaluating Survey Simulation against European Survey Data

Yuliya Li; Achim Rettinger; Dominik Beyer

cs.CLOct 18, 2026

Do LLMs Reproduce the Age Gap in Loneliness? Evaluating Survey Simulation against European Survey Data

Authors: Yuliya Li, Achim Rettinger, Dominik Beyer

Organizations: Trier University, Germany · FZI Research Center for Information Technology, Karlsruhe, Germany

Abstract

Large language models (LLMs) are often used to simulate survey respondents, but matching survey averages does not establish accurate demographic differences. We test whether LLM outputs reproduce within-country age gaps in loneliness using the 2022 EU Loneliness Survey, comparing adults aged 18-34 and 65 or older in France, Italy, Poland, and Sweden. We evaluate simulation with and without age information and direct distribution estimation against a probability-based sample, with a non-probability sensitivity reference. Both human samples show higher mean loneliness among younger adults. Under age-specified simulation with standard response-option order, Mistral Small 4 and GPT-5.6 Terra reverse this ordering under all three prompt framings in three and two countries, respectively, mainly by overestimating older-adult means. DeepSeek V4 Flash preserves the ordering everywhere but underestimates both group means, with errors largely cancelling in France and Italy. Against the probability-based reference, direct estimation has lower absolute age-gap error than simulation for Mistral and Terra in every country. For DeepSeek, simulation has lower gap error but direct estimation has lower group-mean error; these comparisons depend on low-validity older-adult simulation cells. Evaluating LLM outputs as proxies for age groups within countries therefore requires checking both group means and their contrast against human survey data.

Figures & tables

	$n$ ( $n_{\mathrm{eff}}$ )		Weighted mean		Older-minus-younger gap
Country	18–34	65+	18–34	65+	Estimate	95% CI
EU4: probability-based sample
France	196 (138)	271 (168)	5.05	4.06	$-1.00$	[ $-1.36$ , $-0.62$ ]
Italy	176 (102)	263 (163)	5.09	4.19	$-0.89$	[ $-1.31$ , $-0.49$ ]
Poland	219 (93)	98 (70)	5.38	4.27	$-1.10$	[ $-1.63$ , $-0.57$ ]
Sweden	203 (196)	294 (290)	5.25	3.95	$-1.30$	[ $-1.59$ , $-1.02$ ]

Table 1: Human reference estimates.

n

: complete respondents with valid weights;

n_{\mathrm{eff}}

: Kish effective sample size. Means use the supplied country weights and the UCLA-3 scale (3–9). Gaps compare ages 65–100 with 18–34; intervals are pointwise 95% percentile-bootstrap intervals. Displayed estimates are rounded.

Country	Model	Cond.	Mean 18–34	Mean 65+	Gap	Signed gap error [95% CI]
France	Mistral Small 4	S	5.09	6.02	0.93	1.92 [1.45, 2.38]
	Mistral Small 4	D	4.77	4.70	$-0.08$	0.92 [0.52, 1.31]
	DeepSeek V4 Flash	S	4.34	3.40 †	$-0.93^{\dagger}$	0.06 † [IW]
	DeepSeek V4 Flash	D	4.48	4.11	$-0.37$	0.62 [0.25, 1.00]
	GPT-5.6 Terra	S	5.43	5.19	$-0.23$	0.76 [0.37, 1.15]
	GPT-5.6 Terra	D	4.95	3.99	$-0.95$	0.04 [ $-0.33$ , 0.41]

Table 2: EU4 estimates under S (age-specified simulation) and D (direct estimation), averaged equally across the three framings. Gap: older minus younger; gap error: model gap minus EU4 human gap. Intervals are pointwise 95% bootstrap intervals combining model and fixed-weight human resampling, conditional on the three selected framings. Bold marks the smaller absolute gap error in each model–country S/D pair. † Depends on a low-validity cell; IW: interval withheld.

Figure 1: Older-minus-younger UCLA-3 gaps; negative values indicate higher mean loneliness among younger adults. S and D estimates average the three framings equally. Bars show model-only pointwise 95% intervals conditional on the three selected framings. Hollow circles depend on a low-validity cell (

<45

valid outputs); dependent intervals are withheld.

Figure 2: Point estimates of errors in the younger and older group means against EU4, one point per model, country, and method. A point’s height above the dashed diagonal equals its signed gap error (Equation ( 3 )); only points near the origin match both group means. Hollow markers depend on a low-validity cell. FR, IT, PL, SE: France, Italy, Poland, Sweden.

Model	$L_{N}$	$L_{S}$	$L_{D}$	$L_{N}$ $-L_{S}$	$\|E_{S}\|$ $-\|E_{D}\|$
France
Mistral	0.50	1.00	0.46	$-0.50$	1.00
DeepSeek	0.89	0.69 †	0.31	0.20 †	$-0.56^{\dagger}$
Terra	0.54	0.75	0.09	$-0.22$	0.72
Italy
Mistral	0.45	1.06	0.47	$-0.62$	1.18

Table 3: EU4 error comparison.

L

: mean absolute error across the two age-group means;

E

: signed gap error. Positive

L_{N}-L_{S}

favors S over N; positive

|E_{S}|-|E_{D}|

favors D over S. Bold marks the lowest displayed

L

per row, including rounding ties. † Depends on a low-validity cell.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	EU4	EU27
Released sample	4,029	25,646
Four countries	4,029	4,008
Valid age, 16–100	4,029	4,006
Target age groups	1,748	1,727
Complete UCLA-3	1,720	1,682
Valid weight	1,720	1,682

Appendix

Table 4: Respondents remaining after each filtering step. EU27 is restricted to the four EU4 countries before age and response exclusions.

Figure 3: Weighted human UCLA-3 means for ages 18–34 and 65–100. Circles: EU4; squares: matched EU27. Bars show pointwise 95% percentile-bootstrap intervals for the means, not the spread of individual scores.

Final outcome	Draws
Valid	8,911
Technical failure	12
Malformed	70
Item nonresponse	5
Invalid values	2
Refusal	0

Appendix

Table 5: Final main-run outcomes. All rates use scheduled draws as the denominator.

Country	Model	f1 gap	f2 gap	f3 gap
France	Mistral	1.245	0.260	1.280
	DeepSeek	$-0.451$	$-1.191^{\dagger}$	$-1.161$
	Terra	0.500	$-0.820$	$-0.380$
Italy	Mistral	1.347	0.820	1.525
	DeepSeek	$-0.611$	$-1.177^{\dagger}$	$-0.794$
	Terra	0.960	0.900	0.080

Appendix

Table 6: Framing-specific S older-minus-younger gaps, rounded to three decimals. † The older-adult cell is a low-validity cell. Point estimates only.

		EU4		EU27
Country	Model	S	D	S	D
France	Mistral	1.924	0.919	1.447	0.442
	DeepSeek	$0.062^{\dagger}$	0.623	$0.416^{\dagger}$	0.146
	Terra	0.762	0.043	0.285	0.434
Italy	Mistral	2.124	0.939	2.583	1.398
	DeepSeek	$0.032^{\dagger}$	0.579	$0.492^{\dagger}$	1.038

Appendix

Table 7: Absolute gap errors under both human references, using unchanged model outputs. † Depends on a low-validity cell. The S/D ranking changes in France for DeepSeek and Terra.

Large Language Model-Based Social Simulation

Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.

See all 257 papers

Can Open-Weight Large Language Models (LLMs) Simulate Human Survey Populations? A Cross-Instrument Calibration Study
Sep 26, 2026Grandee Lee, Wang YueLarge Language Model-Based Social Simulation LLM Evaluation
Singapore University of Social Sciences Singapore, Singapore
Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data
Jun 27, 2026Eun Cheol Choi, Youngrae Kim, Prabhu Pugalenthi +2LLM Alignment LLM Fine-Tuning
University of Southern California, CA, USA.
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
Jul 6, 2026Mantas Lukauskas, Viktorija ŠarkauskaitėPsychometric Validation Synthetic Data Evaluation
Hostinger · Kaunas University of Technology · AI Insight Lab +1

Sep 26, 2026J/K move · Enter open · S save

Can Open-Weight Large Language Models (LLMs) Simulate Human Survey Populations? A Cross-Instrument Calibration Study

Grandee Lee, Wang Yue

Singapore University of Social Sciences Singapore, Singapore

Large Language Model-Based Social Simulation LLM Evaluation Language Model Calibration

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to generate synthetic survey respondents and digital twins of real people, but whether their output preserves real human statistical structure, rather than surface plausibility, remains unresolved, and most existing evidence comes from proprietary models rather than open-weight ones. We evaluate three open-weight LLM families on a cross-instrument calibration task: conditioning personas on real respondents' verbatim answers to one psychometric instrument and measuring them on a second, construct-distance-controlled instrument, checked against a 2,058-person human panel. Across a 139-pair grid, the simulated cross-instrument correlation tracks the real human correlation at r = 0.70 - 0.73 in every model, driven mainly by correct sign rather than precise magnitude and concentrated in pairs of moderate construct distance. A correlation of this magnitude, obtained from untuned open-weight models conditioned only on individual-level survey data, is a substantively encouraging result for LLM-based behavioral simulation and digital-twin applications: specific model families and releases already reproduce a meaningful share of real human cross-instrument structure without any fine-tuning. This capability does not, however, improve monotonically across model releases: on a matched panel, the newest of three tested Llama releases performs worst on two of three headline metrics, so realizing its promise in practice requires release-specific, distance-aware verification rather than a one-time benchmark.

Abstract

Large Language Model-Based Social Simulation

Explore similar work

Can Open-Weight Large Language Models (LLMs) Simulate Human Survey Populations? A Cross-Instrument Calibration Study

Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents