Survey simulations require realistic combinations of answers, but predictive accuracy alone does not establish that a model can generate them. We evaluate the option probabilities returned by Jev, a TypeSafe decision model, using linked responses to 20 binary items from five archived survey waves. We compare population-only prediction, prediction given recorded answers, and sequential completion that uses generated answers to predict later responses. Joint comparisons hold the starting population information fixed, and two three-question cases are completed in both question orders. We evaluate predictive scores, answer frequencies, and associations against the survey records. Recorded answers can improve predictive scores while associations remain distorted. Large conditional errors also occur. For example, among men who described themselves as having traditional values, the model predicted that 16% would see men and women as similar in workplace abilities, compared with 56% in the survey. Sequential completion has mixed effects on joint validity, and reversing question order produces different joint distributions in both tested cases. Order checks and classical marginal bounds clarify what can be assessed before linked validation answers are available. The findings support validating the joint events a simulation uses and separating empirical constraints from model assumptions. The evidence concerns selected binary tasks and one model version.
Figures & tables
Block
Given answer or attribute
Target answer(s)
Evaluation n
W92 pairs
Economic system: unfair or fair
Corporate profit, government size, federal health coverage, labor unions
4891–10052
W54 pairs
Savings account
Retirement account, medical-bill debt
3365, 3377
W41 pairs
Race relations in 2050
Public education, political division in 2050
1088, 1084
W29 predictions
Recorded sex and traditional values
Similarity of men/women in parenting and workplace abilities
1242
Finance1 (W54)
Savings account
Retirement account (B), medical-bill debt (C)
3360
Finance2 (W54)
Investment ownership
Credit-card debt (B), car loan (C)
3360
Table 1: Selected case material. Counts include evaluation respondents whose required answers fall within the two listed options. Full wording and options appear in the appendix.
Comparison
Categories
Brier gain
Respondent interval
Output range
Direction across outputs
W92 pairs (mean)
2
0.0416
[0.0358,0.0473] (95%)
[0.0388,0.0433]
Positive
W54 pairs (mean)
4
0.0153
[0.0112,0.0193] (95%)
[0.0123,0.0187]
Positive
W41 pairs (mean)
4
0.0468
[0.0356,0.0583] (95%)
[0.0409,0.0510]
Positive
W29 targets (mean)
2
−0.1259
[ −0.1772 , −0.0737 ] (95%)
[ −0.1456 , −0.1094 ]
Negative
Finance1
8
−0.0013
[ −0.0018 , −0.0008 ] (95%)
[ −0.0035 ,0.0002]
Crosses zero
Finance2
8
−0.0003
[ −0.0026 ,0.0020] (95%)
[ −0.0027 ,0.0026]
Crosses zero
Table 2: Brier gains relative to the comparison-specific control. W92 uses the same mean forecast for everyone as its control. W54/W41 use population-only prediction, W29 uses prediction given recorded sex, and the triples use separate completion. Positive gains favor recorded-answer conditioning or sequential completion. The Categories column counts the scored outcomes: two answers, four answer pairs, or eight answer triples. Respondent intervals hold mean model probabilities fixed. Output ranges vary the collected model responses at fixed human records and reference shares. Gains use different controls and outcome spaces across rows.
Wave
Target
Human
Model
Error (pp)
W54
Retirement account
56.4
69.7
13.3
W54
Medical-bill debt
30.0
32.0
2.0
W41
Public education in 2050
44.7
28.7
16.0
W41
Political division in 2050
70.0
76.7
6.7
W29
Parenting similarity
33.5
57.0
23.5
W29
Workplace similarity
63.3
61.3
1.9
Table 3: Population-only requests. Human and model columns give first-option shares (%). Error is their absolute difference in percentage points. The table covers the six selected population-only requests.
Figure 1: Survey answer shares (horizontal axis) versus mean model probabilities (vertical axis) for 26 question/group/information conditions in W54, W41, and W29. Both axes refer to each question’s first displayed option. Circles use population information only, while triangles add recorded respondent information. Points on the dashed diagonal match human frequencies. Horizontal bars are approximate simultaneous human-share bands under person resampling, wider than pointwise intervals. Vertical bars show the range of the three collected model outputs. The annotated women’s workplace condition has the largest observed error, 42.7 pp. Both sex groups are included. Answer wording and coding appear in the item dictionary.
Figure 2: Human association strength (horizontal axis) versus model association strength (vertical axis) for 13 answer-pair comparisons, measured by log odds ratios. Each symbol marks one comparison. The dashed diagonal indicates equal associations. Zero indicates independence. A point farther from zero vertically than horizontally has a stronger model association. Horizontal bars are pointwise nominal 95% human respondent-bootstrap intervals. Vertical bars show the range across collected model outputs at fixed human composition/reference shares. Circles show AB pairs, triangles AB pooled over sex using empirical (X,A) composition, and squares sequential BC pooled over A. Colors identify waves. Black arrows identify 9 Parenting and 10 Workplace.
Wave
Target B
Separate
Conditional
W54
Retirement account
13.3
4.7
W54
Medical-bill debt
8.1
4.0
W41
Public education in 2050
23.4
12.0
W41
Political division in 2050
17.5
4.2
Table 4: Joint TV (%) for the four pair generators sharing an empirical A marginal. Point errors use each pair cohort.
Figure 3: Joint shares of answers B and C, pooled over A. Cell values are percentages. Each panel shows four combinations that total 100% before rounding. For example, the upper-left cell of the Finance2 human panel means that 28.0% have both credit-card debt and a car loan. Rows identify cases and columns identify human records or completion rules. Question labels apply across each row. Titles give pooled odds ratios. Both model rules use the same reference A shares. Race options are abbreviated as Unique and Common.
Case family
Contrast
Estimate
Respondent interval
Output range
Race/W29
Race reverse minus forward gain
0.0279
[0.0206,0.0352]
[0.0204,0.0364]
Race/W29
Workplace masked minus named gain
−0.0474
[ −0.0581 , −0.0359 ]
[ −0.0728 , −0.0097 ]
Race/W29
Race masked minus named forward gain
0.0068
[0.0049,0.0086]
[ −0.0091 ,0.0242]
W54/Finance2
W54 masked minus named mean pair gain
0.0002
[ −0.0023 ,0.0028]
[ −0.0052 ,0.0077]
W54/Finance2
Finance2 reverse minus forward gain
0.0009
[ −0.0017 ,0.0036]
[ −0.0055 ,0.0069]
Table 5: Five Brier control contrasts. Interval levels follow the separate families: 98.333% for Race/W29 and 97.5% for W54/Finance2. Collected-output ranges separately vary observed responses at fixed human data and reference shares. Positive values favor the first-named alternative.
Available information
Identified quantity or assumption
Implication in these cases
No target-population human labels
Only the model’s proposed distribution is observed. Frequency error cannot be measured.
Evaluation labels reveal raw errors up to 23.5 pp, which the probability vector alone does not disclose.
One eligible-cohort A marginal
A shares are supplied. Missing conditional frequencies remain model-dependent.
Four anchored pair generators improve but retain 4–12% joint TV. Applicability depends on the cohort represented by the supplied A shares.
A person’s recorded answer
Fixes this person’s A answer, specifying the query q(B∣A=a) . Estimating the corresponding human share P(B∣A=a) requires linked answers or additional assumptions.
W92 gains and W29 losses require separate linked-human evaluation.
B and C target margins
Conjunction shares lie within Fréchet bounds. A particular association is not identified.
Finance2 permits both-answer prevalence anywhere from 0 to 40.6%.
Separate linked (A,B) and (A,C) tables
Conditional bounds are available. B⊥C∣A selects one completion through an added assumption.
Finance2’s human-margin conditional-independence completion gives 21.2%, versus the observed 28.0%.
Linked validation records for A,B,C
Permit estimating joint cells, associations, event errors, and paired score differences in the evaluated cohort.
Expose Finance2’s conjunction deficit and Race’s distorted association.
Table 6: Human information and the conclusions it supports. Each row describes a different information setting. Empirical shares and bounds also have sampling uncertainty.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Case block
Selection basis
Prior exposure and interpretation
W92 pairs
Four binary targets chosen by wording and availability of linked answers
Four targets evaluated under the same fixed-mean comparison.
W54 pairs
Savings, retirement, and medical debt chosen from related self-report items
Two targets evaluated as a pair block. No random domain sampling.
W29 predictions
Traditional values and two gender-opinion items
Recorded sex was selected after checking conditional-group eligibility and before forecasting or evaluation decoding.
Finance1
Existing finance block used to test a new completion operation
Exploratory on previously analyzed item content.
Finance2
Unqueried investment item and first two unqueried debt items by codebook order
Same W54 wave and split. Comparisons specified before model requests for these items.
Appendix
Table 7: Case selection and data reuse. The cases were not sampled from a defined task population. Exploratory comparisons and controls that reuse material are identified separately.
Task
n
Categories
Gain
Respondent interval
Output range
W92 Corporate profit
9960
2
0.0740
[0.0600,0.0874]
[0.0714,0.0755]
W92 Government size
9958
2
0.0275
[0.0175,0.0374]
[0.0245,0.0292]
W92 Federal health coverage
10052
2
0.0402
[0.0356,0.0451]
[0.0358,0.0437]
W92 Labor unions
4891
2
0.0246
[0.0158,0.0332]
[0.0235,0.0250]
W54 Retirement account
3365
4
0.0258
[0.0179,0.0338]
[0.0232,0.0298]
W54 Medical-bill debt
3377
4
0.0048
[0.0024,0.0072]
[0.0013,0.0076]
Appendix
Table 8: All 13 reported task comparisons. Intervals are 95% respondent-bootstrap intervals conditional on the mean probabilities. Race and the historical-context check also have the two-endpoint 97.5% intervals. W29 rows use the original cohort description, while the historical-context check adds the 2017 field period. Some rows share questions or respondents.
Association
Human OR
Human interval
Model OR
Model output range
W92 Corporate profit
6.677
[5.753,7.817]
9.898
[8.609,11.870]
W92 Government size
0.324
[0.276,0.380]
0.216
[0.183,0.251]
W92 Federal health coverage
4.807
[4.114,5.574]
2.283
[2.000,2.567]
W92 Labor unions
2.780
[2.266,3.435]
2.698
[2.357,3.217]
W54 Retirement account
4.795
[3.844,6.057]
5.182
[4.644,5.881]
W54 Medical-bill debt
0.477
[0.379,0.594]
0.528
[0.452,0.596]
Appendix
Table 9: All 13 descriptive association estimates at original mean probabilities. The first eight rows concern AB. W29 pools over sex using empirical (X,A) composition, rather than an A-only generator. The last three concern sequential BC pooled over A. Human intervals are pointwise nominal 95% weighted person-bootstrap intervals, computed separately for each association. Model ranges vary collected outputs at fixed human composition/reference shares.
Case
Stratum
Human [interval]
Separate
Sequential
Separate range
Sequential range
Finance1
All A
0.48 [0.39,0.59]
0.88
0.97
[0.85,0.92]
[0.86,1.08]
Finance1
A=0
0.50 [0.39,0.66]
1.00
1.05
[1.00,1.00]
[0.95,1.15]
Finance1
A=1
0.73 [0.48,1.08]
1.00
1.01
[1.00,1.00]
[0.88,1.14]
Finance2
All A
3.28 [2.70,3.97]
1.04
1.69
[1.02,1.06]
[1.44,1.91]
Finance2
A=0
2.74 [2.08,3.61]
1.00
1.58
[1.00,1.00]
[1.41,1.75]
Finance2
A=1
3.70 [2.87,4.78]
1.00
1.68
[1.00,1.00]
[1.42,1.91]
Appendix
Table 10: BC odds ratios and collected-output ranges. Human values include pointwise nominal 95% person-bootstrap intervals. All-A rows collapse over A. The Separate rule has conditional OR=1 within A by construction. Output ranges are stress checks at fixed human data, not confidence intervals.
Target
Sex
Human shares
Model shares
Human OR
Model OR
Model OR range
Parenting similarity
Male
31.4 / 46.9
12.3 / 74.7
0.52 [0.28,0.96]
0.048
[0.039,0.053]
Parenting similarity
Female
29.7 / 38.1
15.0 / 77.0
0.69 [0.38,1.28]
0.053
[0.042,0.065]
Workplace similarity
Male
56.3 / 78.0
16.0 / 80.7
0.36 [0.16,0.71]
0.046
[0.036,0.054]
Workplace similarity
Female
62.4 / 72.7
19.7 / 84.7
0.62 [0.29,1.19]
0.044
[0.038,0.051]
Appendix
Table 11: W29 conditioned first-option shares (%) and descriptive ORs. Each share pair gives the values for “describes me well and “does not describe me well, in that order. The target first option is “men and women are basically similar”. Both targets are shown for men and women. Human ORs include pointwise nominal 95% person-bootstrap intervals. Model OR ranges combine the collected conditional outputs.
Item
Wording and options
ECONFAIR_W92
Please choose the statement that comes closer to your own views. First: The economic system in this country unfairly favors powerful interests Second: The economic system in this country is generally fair to most Americans
BUSPROFIT_W92
Please choose the statement that comes closer to your own views. First: Business corporations make too much profit Second: Most corporations make a fair and reasonable amount of profit
GOVSIZE1_W92
If you had to choose, would you rather have First: A smaller government providing fewer services Second: A bigger government providing more services
GOVTHC_W92
Do you think it is the responsibility of the federal government to make sure all Americans have health care coverage? First: Yes, it is Second: No, it is not
INSTN_LBRUN_W92
Are labor unions having a positive or negative effect on the way things are going in the country these days? First: Positive effect Second: Negative effect
FINANCEa_W54
Do you have a savings account? First: Yes, have this Second: No, do not have this
Appendix
Table 12: Item wording and displayed answer coding (1/2). Identifiers are retained here for reproducibility and were present inside model-visible request content.
Item
Wording and options
HAPPENj_W41
Still thinking ahead 30 years, which do you think is more likely to happen in the U.S.? The country will be more/less politically divided First: The country will be more politically divided Second: The country will be less politically divided
MESUM2_FB_W29
How well, if at all, do the following words or phrases describe you? Have traditional values First: Describes me well Second: Does not describe me well
DIFF1C_W29
Do you think men and women are basically similar or basically different when it comes to their approach to parenting? First: Men and women are basically similar Second: Men and women are basically different
DIFF1E_W29
Do you think men and women are basically similar or basically different when it comes to the things they are good at in the workplace? First: Men and women are basically similar Second: Men and women are basically different
FINANCEc_W54
Do you have personal investments in stocks, bonds or mutual funds other than those held in an IRA or 401K? First: Yes, have this Second: No, do not have this
DEBTa_W54
Do you have the following type of loans or debt: Credit card debt First: Yes, have this Second: No, do not have this
Appendix
Table 13: Item wording and displayed answer coding (2/2).
Analysis family
Size
Comparisons covered
Reported level
W29 historical context and Race
2
W29 historical-context gain and Race sequential-completion gain
97.5% per contrast
Race/W29 controls
3
Race reverse minus forward gain, W29 masked minus named gain, Race masked minus named forward gain
98.333% per contrast
W54/Finance2 controls
2
W54 masked minus named mean pair gain and Finance2 reverse minus forward gain
97.5% per contrast
Other score and OR estimates
n/a
Other original Brier comparisons, matched-mean joint diagnostics, secondary arm gains, and human odds ratios
95% pointwise
Human-share bands
26 coordinates
Centered within-wave maxima with 0.05/3 allocation across three waves
Approximate 95% simultaneous bands
Appendix
Table 14: Interval families and correction scope. Brier families retain their specified contrasts. Pointwise intervals and human-share bands have separate descriptive roles. Model-output ranges summarize the collected responses at fixed human data.
Large language models are increasingly used to simulate response distributions in social surveys. Prior work has achieved accurate population-level simulation for individual questions. Real questionnaires, however, ask each respondent a sequence of related questions. A simulated respondent should show coherent preferences across the whole questionnaire, not merely accurate distributions for isolated items. Existing single-item methods cannot accurately reproduce how the same person answers a complete survey. We propose FullRespondent-LLM (FR-LLM), which fine-tunes two specialized LLMs: a marginal model for each item's response distribution and a respondent-level autoregressive model for dependencies across answers. Marginal-Constrained Joint Projection (MCJP) then projects the autoregressive joint distribution onto the set satisfying the item-level marginals learned by the first model. This yields complete questionnaires with realistic cross-item relationships while retaining strong item-level accuracy. On two real-world social survey datasets, FR-LLM more accurately reproduces multi-question response patterns, maintains competitive single-item accuracy, and generalizes better to unseen populations and questions. In a small commercial-survey dataset, we use simulated responses to make pricing and stocking decisions; FR-LLM achieves the highest realized profit.
Ji Huang, Mengfei Li, Shuai Shao
School of Computer Science, University of Science and Technology of China · School of Management, Fudan University
Demographic synthetic survey panels are often validated by matching aggregate answers to published surveys. We test what that certificate establishes across six multiselect batteries from four survey organisations in three countries. The headline analysis is restricted to three instruments whose synthetic cohort and human target share the stated population frame; three other batteries remain sensitivity analyses. The response contract dominates measured fidelity. In the aligned instruments, committed sets leave 66 of 128 model-battery option slots empty in panels of up to 500 respondents, versus 0 of 128 under per-option probability elicitation. Across eight uncapped model-instrument comparisons, probabilities reduce option-marginal MAE by 4.53 to 7.30 points. The capped instrument reverses on two models until the vectors are projected onto its stated maximum. These are measurement effects: human targets are realised check-all responses, whereas the vectors are latent inclusion propensities. Published marginal agreement also fails to discriminate respondent simulation from direct population estimation. On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons. Constraint-aware probability vectors average 5.34 and beat the query on four of nine, so the baseline challenges the validation criterion rather than proving direct estimation uniformly best. On three unpublished demographic cells, neither approach beats reciting the national distribution. Population-marginal agreement is therefore evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.
Language models are increasingly used to create synthetic survey samples, also called silicon samples. Decision models are a new class of language models that return a probability for each point of a rating scale instead of text. We ask whether these models, or the newest generative LLMs, can stand in for human respondents on a psychometric survey. We test fourteen decision models from eleven providers and 47 generative LLMs on an open Lithuanian organisational-psychology survey (n=263, 68 items), with 100 real respondent profiles, the Psychometric Similarity Score (PSS) and statistical baselines. The best generative model, MiniMax M3, reaches PSS 0.813. This is level with a Gaussian copula fit only on other respondents (0.824) and below a copula fit in sample (0.927). A second sample of 100 real people scores below the best models, so at this size PSS cannot rank them against real respondents. In 5 of 10 product lines the newest generative release scores below an earlier release of the same line beyond resampling noise, and none gains beyond it. With the default rounded readout, 13 of 14 decision models score below the generative reference gpt-5.4-mini. Of the 47 generative models, 33 reproduce the pattern of item correlations better than any decision model. Averaged over respondents, the probability vectors are much closer to the human item marginals than rounded answers. They also rank respondents about as well as a profile regression fit on 163 real people. As forecasts of each person's own answers, however, every decision model is worse than the answer frequencies of other respondents, mainly because its average vector differs from them. The profile regression improves on these frequencies by only 1.0%, so the profile fields say little about how a person answers. Synthetic respondents built on such profiles should therefore aim at sample-level fidelity.
Mantas Lukauskas, Viktorija Šarkauskaitė
Hostinger; Kaunas University of Technology; AI Insight Lab · AI Insight Lab; Independent Researcher