Simulating Respondents, Not Single Questions: Coherent Survey Generation with Large Language Models
Organizations: School of Computer Science, University of Science and Technology of China · School of Management, Fudan University
Abstract
Large language models are increasingly used to simulate response distributions in social surveys. Prior work has achieved accurate population-level simulation for individual questions. Real questionnaires, however, ask each respondent a sequence of related questions. A simulated respondent should show coherent preferences across the whole questionnaire, not merely accurate distributions for isolated items. Existing single-item methods cannot accurately reproduce how the same person answers a complete survey. We propose FullRespondent-LLM (FR-LLM), which fine-tunes two specialized LLMs: a marginal model for each item's response distribution and a respondent-level autoregressive model for dependencies across answers. Marginal-Constrained Joint Projection (MCJP) then projects the autoregressive joint distribution onto the set satisfying the item-level marginals learned by the first model. This yields complete questionnaires with realistic cross-item relationships while retaining strong item-level accuracy. On two real-world social survey datasets, FR-LLM more accurately reproduces multi-question response patterns, maintains competitive single-item accuracy, and generalizes better to unseen populations and questions. In a small commercial-survey dataset, we use simulated responses to make pricing and stocking decisions; FR-LLM achieves the highest realized profit.
Figures & tables
| Dataset | Train | M1 test | M2 test | M3 test |
|---|---|---|---|---|
| ESS11 | 28,268 | 28,268 | 7,312 | 7,312 |
| TALIS 2018 | 12,000 | 8,000 | 8,000 | 8,000 |
| Mode | Method | Qwen3.5-9B | Ministral-3-8B | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Cronbach’s MAE | AVE MAE | Item JSD | Construct JSD | Cronbach’s MAE | AVE MAE | Item JSD | Construct JSD | ||
| M1 | Zero-shot | .507 | .348 | .075 | .124 | .442 | .316 | .106 | .168 |
| Single FT | .363 | .272 | .024 | .037 | .367 | .274 | .031 | .048 | |
| Sequential FT | .084 | .075 | .044 | .047 | .114 | .103 | .089 | .091 | |
| FR-LLM | .078 | .071 | .024 | .021 | .046 | .040 | .031 | .032 | |
| M2 | Zero-shot | .625 | .405 | .084 | .175 | .524 | .370 | .107 | .201 |
| Mode | Method | Qwen3.5-9B | Ministral-3-8B | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Cronbach’s MAE | AVE MAE | Item JSD | Construct JSD | Cronbach’s MAE | AVE MAE | Item JSD | Construct JSD | ||
| M1 | Zero-shot | .809 | .395 | .061 | .296 | .808 | .392 | .141 | .474 |
| Single FT | .642 | .362 | .017 | .144 | .831 | .396 | .020 | .168 | |
| Sequential FT | .094 | .142 | .015 | .058 | .114 | .181 | .029 | .075 | |
| FR-LLM | .008 | .007 | .017 | .043 | .049 | .058 | .020 | .052 | |
| M2 | Zero-shot | .786 | .387 | .069 | .309 | .811 | .388 | .150 | .486 |
| Method | Train | Test | MAE | AVE MAE | Item JSD | Construct JSD |
|---|---|---|---|---|---|---|
| Zero-shot | 0 | 1,000 | .494 | .329 | .073 | .131 |
| Single FT | 1,000 | 1,000 | .406 | .290 | .054 | .042 |
| FR-LLM | 1,000 | 1,000 | .066 | .061 | .054 | .022 |
| Single FT | 28,268 | 28,268 | .363 | .272 | .024 | .037 |
| FR-LLM | 28,268 | 28,268 | .078 | .071 | .024 | .021 |
| Background change | Zero-shot | Single FT | FR-LLM |
|---|---|---|---|
| Low to high education | .261 | .153 | .149 |
| Age 18–29 to 60+ | .399 | .231 | .216 |
| Urban to rural | .123 | .112 | .111 |
| Low/young to high/old | .298 | .198 | .177 |
| Urban/male to rural/female | .229 | .150 | .168 |
| Macro average | .262 | .169 | .164 |
| Human | Excess error above human resampling | FR gap | ||||
|---|---|---|---|---|---|---|
| Metric | error | Zero-shot | Single FT | Sequential FT | FR-LLM | closed |
| MAE | .007 | .500 | .356 | .077 | .071 | 85.9% |
| AVE MAE | .007 | .342 | .265 | .068 | .065 | 81.1% |
| Item JSD | .007 | .067 | .017 | .037 | .017 | 74.4% |
| Construct JSD | .001 | .124 | .037 | .047 | .021 | 82.9% |
| Configuration | MAE | AVE MAE | Item JSD | Construct JSD |
|---|---|---|---|---|
| Zero-shot | .507 | .348 | .075 | .124 |
| Single FT | .363 | .272 | .024 | .037 |
| Joint proposal (16 samples) | .076 | .070 | .027 | .028 |
| MCJP (1 sample) | .087 | .080 | .024 | .022 |
| MCJP (16 samples) | .078 | .071 | .024 | .022 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Data / backbone | Method | M1 | M2 | M3 |
|---|---|---|---|---|
| ESS11 / Qwen | Zero-shot | .247 | .366 | .296 |
| Single FT | .099 | .142 | .160 | |
| Sequential FT | .110 | .165 | .184 | |
| FR-LLM | .059 | .081 | .123 | |
| ESS11 / Ministral | Zero-shot | .291 | .376 | .340 |
| Single FT | .119 | .150 | .182 |
| Candidate questionnaires | Item JSD | Two-construct JSD | Maximum marginal error | Adjusted cells | Effective sample fraction |
| 1 | .0244 | .0693 | .1952 | 469 | .330 |
| 2 | .0244 | .0621 | .0748 | 201 | .422 |
| 4 | .0244 | .0596 | .0190 | 76 | .506 |
| 8 | .0244 | .0596 | .0112 | 18 | .555 |
| 16 | .0244 | .0592 | .0013 | 2 | .578 |
| Item JSD | Construct JSD | Two-construct JSD | Support adjustment | Effective sample fraction | |
|---|---|---|---|---|---|
| 0.00 | .0267 | .0280 | .0708 | .0000 | 1.000 |
| 0.25 | .0251 | .0254 | .0661 | .0013 | .962 |
| 0.50 | .0242 | .0240 | .0636 | .0027 | .860 |
| 0.75 | .0240 | .0224 | .0606 | .0040 | .722 |
| 1.00 | .0244 | .0217 | .0596 | .0053 | .578 |