cs.CLSep 24, 2026
SaveArtificial Societies Benchmark: A Validation Framework for Synthetic Research
Organizations: Artificial Societies University of Oxford, UK
Abstract
A synthetic survey can reproduce the average answer while misrepresenting how people differ, how their answers relate to one another, or how they respond to changes in conditions. We introduce the Artificial Societies Benchmark to help researchers assess whether synthetic populations support their intended analyses. The framework combines eleven tests across internal, construct, and external validity, drawing on twenty human sources and comparing nine language models. It connects each research use to the evidence it requires and tests how results change with the information we supply about respondents. Importantly, strong performance in one domain does not establish fidelity in the others. Models often answer too consistently, compress response scales, and alter relationships between traits whilst richer profiles improve prediction for some models and worsen it for others. The resulting scorecard helps researchers identify which aspects of a synthetic population can support their analysis and where researchers need further human evidence.
Figures & tables
| Test | Measure | Comparison | What it captures |
|---|---|---|---|
| Internal validity | |||
| IV-1 | Repeated questions | Repeat the same item | Excess consistency or variability |
| IV-2 | Linked questions | Check logical constraints | Departures from human violation patterns |
| IV-3 | Paraphrased questions | Ask equivalent wording | Changes in answers across wording |
| IV-4 | Perturbed surveys | Change survey administration | Human sensitivity to the instrument |
| Construct validity | |||
Table 1: What each test asks about a synthetic population. Use the intended analysis to select relevant tests across the three domains. Appendix A gives formal definitions.
Figure 1: Choosing tests for a proposed study. Following relevant branches shows which tests a study requires. The top branch applies across study purposes. Colours identify the three validity domains.
Figure 2: Which datasets yield estimates for each test? Filled squares indicate at least one model yields an estimate, coloured outlines indicate a test we attempted without an estimate, and pale outlines indicate a combination we did not run. Counts appear in Table 6 .
| Model | Weight availability | Temperature | Thinking |
|---|---|---|---|
| GPT-5.6 Sol | Proprietary | 1 | Off |
| Claude Opus 5 | Proprietary | 1 | Off |
| Gemini 3.8 Flash | Proprietary | Ignored | Low |
| Grok 4.3 | Proprietary | 1 | Off |
| Mistral Small 2603 | Open-weight | 1 | Off |
| DeepSeek Flash | Open-weight | 1 | Off |
Table 2: Models and generation settings. Open-weight denotes public availability of model weights. Gemini’s backend ignores temperature; Scout has no thinking mode.
Figure 3: Model departures from human responses across eleven validity tests. Each row represents a model, and each column shows one diagnostic from the dataset named in its heading, using the first generation. Cell values report differences or prediction errors in the units of that diagnostic. For signed differences, positive values mean that the model statistic exceeds the human statistic, and negative values mean that it falls below it. For example, in IV-1 means that the model repeats its answers 22 percentage points more often than humans. Absolute errors and prediction losses are nonnegative. Across all columns, values closer to zero indicate closer agreement or lower error. White represents zero; darker shading indicates larger absolute values relative to the largest among the nine models within that column. Each column has its own units and colour scale, so comparisons apply within columns. CV-1 and CV-3 use the UK Global Flourishing Study (GFS). EV-4 measures error in how strongly SAPA facet items relate to their traits. A dash marks an unavailable estimate. Appendix B.2 defines each diagnostic and explains unavailable estimates.
Figure 4: IV-1: repeated-answer agreement in Bainbridge sample 1. Each point compares a model with humans on one of three identically worded item pairs. Zero means equal repeat agreement, positive values mean that the model repeats itself more often, and negative values mean that it repeats itself less often. Lines show paired 95 percent participant-bootstrap intervals. Matching human agreement, rather than maximising consistency, is the comparison of interest.
Figure 5: CV-3: Big Five trait relationships in SAPA. Each model or control panel places synthetic correlations below the diagonal and correlations for its matched humans above it. The black-bordered bottom-right panel shows the full 512-person human reference. E, N, C, A, and O denote extraversion, neuroticism, conscientiousness, agreeableness, and openness. Compare corresponding cells: agreement with the human pattern is desirable rather than high correlation across the matrix. Panel subtitles give mean absolute correlation error against the matched humans; incomplete summaries say “partial”. NA indicates a constant synthetic trait with no defined correlation. The two statistical controls isolate the contribution of item dependence.
Figure 6: What overall response agreement leaves unresolved. Each point represents one model and task in the first generation. Left (EV-1, EV-2), smaller horizontal values mean closer overall answer frequencies, and smaller vertical values mean closer agreement for the worst eligible item–subgroup combination; the lower-left corner indicates smaller errors on both. Right (EV-1, EV-3), FSD3587 points compare response-distribution error with mean error in the effect of changing the messenger, in 0–10 rating points. Again, smaller values on both axes indicate closer human agreement. The panels use different distance measures and do not establish statistical independence between tests.
Figure 7: EV-1: response-distribution error by dataset and model. Points show average item-level error for the first generation, with smaller values indicating closer human agreement. The left panel compares categorical answer frequencies using total variation distance; the right compares numerical responses using Wasserstein distance divided by the response range. Both have zero at exact agreement, but they measure different discrepancies. Only tasks with complete estimable item support enter these means.
Figure 8: EV-3: messenger-effect recovery in FSD3587. Each panel shows 24 contrasts on the 0–10 argument-rating scale. Points on the diagonal match the human effect; departures show its attenuation, amplification, or reversal. Values with opposite signs indicate reversal. The scorecard retains model-minus-human error intervals for each contrast.
Figure 9: EV-2: does personal information improve prediction over demographics alone? Each point gives the change in average prediction error for one model and repetition among the same 430 people. Points left of zero indicate improvement, and points right of zero indicate worsening; the two points per row represent separate generations. Lines show paired 95 percent participant-bootstrap intervals, and an interval crossing zero leaves the direction uncertain. The three panels add original self-descriptions, distilled descriptions, or survey history plus distilled descriptions. These primary comparisons use jointly valid answers; Table 8 also accounts for unanswered questions.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Test | Measure | Human-referenced diagnostics |
|---|---|---|
| IV-1 | Repeated questions | Repeated-answer agreement, kappa, and numerical change |
| IV-2 | Linked questions | Constraint violations and their joint patterns |
| IV-3 | Paraphrased questions | Agreement and response-transition distance |
| IV-4 | Perturbed surveys | Response shifts under instrument perturbations |
| CV-1 | Item covariance | Reliability and within-construct item associations |
| CV-2 | Independent criteria | Associations with independently measured criteria |
Table 3: Eleven validity tests and their principal diagnostics.
| Condition | Information we supply to the model | Output | Sources |
|---|---|---|---|
| L1 aggregate | Survey context, question, and options for the national adult population | Percentages over options | GSS |
| L2 composition | L1 plus the unweighted demographic composition of the task roster in five declared age bands | Percentages over options | GSS |
| L3 demographic | One respondent’s recorded age, sex, ethnicity, education, urbanicity, and political affiliation, or the recorded fields a source provides | One answer | GSS and every profiled task |
| L3 plus history | L3 plus an extractive summary of the same person’s earlier closed-ended answers | One answer | Twin |
| L4 verbatim | L3 plus original free text, being three self-descriptions in Twin and pre-election likes, dislikes, and problem statements in ANES 2020 | One answer | Twin, ANES 2020 |
| L4 distilled | L3 plus a seven-field summary of the free text, which one shared preprocessing model extracts as quotations and we audit against the source, with and without history | One answer | Twin, 430 people |
Table 4: Persona information and elicitation conditions. Levels L1–L4 follow Gordon et al. Gordon et al. [2026] ; we declare the remaining rows as extensions.
| Model | Provider, route | Regime | Decoding controls |
|---|---|---|---|
| gpt-5.6-sol | OpenAI, batch | Thinking disabled | Temperature 1, reasoning effort none, caps 100/512 |
| claude-opus-5 | Anthropic, batch | Thinking disabled | Temperature 1, thinking off, effort high (provider default), caps 100/512 |
| gemini-3.8-flash | Google, online | Low thinking, separate | Backend ignores temperature, low thinking, cap 2,048 |
| grok-4.3 | xAI, batch | Thinking disabled | Temperature 1, reasoning off, caps 100/512 |
| mistral-small-2603 | Mistral, batch | Thinking disabled | Temperature 1, caps 100/512; both synchronous and Batch image limits refused one ESS continuation |
| deepseek-flash | DeepSeek, online | Thinking disabled | Temperature 1, thinking off, caps 100/512 |
Table 5: The nine evaluated systems. Output caps are tokens for individual answers and for GSS distribution requests. Local systems answer the hosted request bodies through vLLM 0.29 on one H100.
| Test | IV-1 | IV-2 | IV-3 | IV-4 | CV-1 | CV-2 | CV-3 | EV-1 | EV-2 | EV-3 | EV-4 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| yielding an estimate | 5 | 5 | 5 | 12 | 8 | 5 | 4–5 | 17 | 14 | 9 | 5 |
| run | 5 | 5 | 5 | 12 | 9 | 5 | 5 | 17 | 15 | 9 | 6 |
Table 6: Study families yielding an estimate and study families run. CV-3 yields four families for OpenAI, Claude, Gemini, and Gemma and five for the other models. Counts otherwise agree across models. A run can yield a diagnostic failure or insufficient support for an estimate.
| Model | Original text | Distilled text | History and distilled text |
|---|---|---|---|
| GPT-5.6 Sol | , | , | , |
| Claude Opus 5 | , | , | , |
| Gemini 3.8 Flash | , | , | , |
| Grok 4.3 | , | , | , |
| Mistral Small 2603 | , | , | , |
| DeepSeek Flash | , | , | , |
Table 7: Change in Twin prediction loss from demographic profiles. Each cell gives repetitions 1 and 2, in that order. Negative values favour the added information. All paired comparisons are estimable; intervals appear in Figure 9 and the scorecard.
| Model | Original text | Distilled text | History and distilled text |
|---|---|---|---|
| GPT-5.6 Sol | |||
| Claude Opus 5 | |||
| Gemini 3.8 Flash | |||
| Grok 4.3 | |||
| Mistral Small 2603 | |||
| DeepSeek Flash |
Table 8: Twin missing-answer sensitivity: bootstrap envelopes for changes in full-target prediction loss relative to demographics. Each cell gives repetitions 1 and 2, respectively. An entirely negative range supports improvement throughout the envelope; a range crossing zero leaves direction uncertain. These envelopes include both missing-answer bounds and participant resampling, unlike the primary intervals in Figure 9 .