Model-Free Assessment of Simulator Fidelity via Quantile Curves
Organizations: Department of IEOR and Data Science Institute, Columbia University.
Abstract
As generative AI models are increasingly used to simulate real-world systems, quantifying the ``sim-to-real'' gap is critical. For each input setting of interest -- which we call a scenario, such as a survey question or operating condition -- the real and simulated systems are associated with unobserved latent population parameters, and their discrepancy varies across scenarios. A fundamental challenge is that, for any given scenario, this discrepancy cannot be observed directly, since both systems are accessible only through finite samples, often of heterogeneous sizes across scenarios. Standard predictive inference methods are therefore ill-suited, as they quantify uncertainty in observable outputs rather than latent population parameters. To address this, we construct confidence sets for these latent parameters and use them to derive a robust proxy for the sim-to-real discrepancy. We then estimate the quantile function of this proxy to obtain a distribution-level risk profile of the simulator, which supports a broad range of statistical summaries, including statistical inference for the real output distribution in a new scenario, the calculation of risk measures like Conditional Value-at-Risk (CVaR), and principled comparisons across simulators. Our method is model-agnostic and handles general output spaces, such as categorical survey responses and continuous multi-dimensional data. We demonstrate the practical utility of this method by evaluating the alignment of four major LLMs with human populations on the WorldValueBench dataset.
Figures & tables
| Scenario | True Distribution ( ) | Real Output ( ) | |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
| Scenario drawn from the scenario population. | |
| Real-world respondent drawn from the population distribution. | |
| (or ) | Real-world population parameter for scenario . |
| (or ) | Simulator-side population parameter for scenario under fixed simulator settings. |
| Finite-sample estimators of and . | |
| Human sample size for scenario . |
| Discrepancy | Primary sensitivity |
| Mean loss | Differences in the average outcome. |
| Wasserstein distance | Distributional shifts measured by the distance over which probability mass is transported. |
| Total variation | Redistribution of probability mass across categories. |
| KL divergence | Outcomes to which the simulator assigns too little probability relative to the real distribution. |
| MMD or energy distance | Distributional differences emphasized by the chosen kernel or outcome-space distance. |
| Demographic | Categories |
| Sex | Male; Female |
| Country | 64 countries/societies |
| Age | Numeric (years) |
| Language | Text |
| Migration status | Native-born; First-generation immigrant; Second-generation; Other/missing |
| Marital status | Married; Living together as married; Divorced; Separated; Widowed; Never married/Single |
| GPT-4o | GPT-5-mini | Llama-3.3 | Qwen-3 | |||||
| Target | Coverage | Width | Coverage | Width | Coverage | Width | Coverage | Width |
| 32 (91.4) | 1.009 | 32 (91.4) | 1.080 | 29 (82.9) | 1.220 | 31 (88.6) | 1.130 | |
| 32 (91.4) | 1.163 | 34 (97.1) | 1.231 | 32 (91.4) | 1.498 | 32 (91.4) | 1.327 | |
| 33 (94.3) | 1.324 | 34 (97.1) | 1.452 | 34 (97.1) | 1.875 | 32 (91.4) | 1.473 | |
| 35 (100.0) | 1.571 | 35 (100.0) | 1.817 | 35 (100.0) | 1.967 | 35 (100.0) | 1.940 | |
| 35 (100.0) | 2.000 | 35 (100.0) | 2.000 | 35 (100.0) | 2.000 | 35 (100.0) | 2.000 | |
Sim-to-Real Transfer
Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.
See all 347 papers