Can Model Answer Probabilities Represent Survey Populations? Conditional Prediction and Joint Validity in Jev Survey Simulation
Organizations: University of Osaka, Japan · Nagoya University, Japan
Abstract
Survey simulations require realistic combinations of answers, but predictive accuracy alone does not establish that a model can generate them. We evaluate the option probabilities returned by Jev, a TypeSafe decision model, using linked responses to 20 binary items from five archived survey waves. We compare population-only prediction, prediction given recorded answers, and sequential completion that uses generated answers to predict later responses. Joint comparisons hold the starting population information fixed, and two three-question cases are completed in both question orders. We evaluate predictive scores, answer frequencies, and associations against the survey records. Recorded answers can improve predictive scores while associations remain distorted. Large conditional errors also occur. For example, among men who described themselves as having traditional values, the model predicted that 16% would see men and women as similar in workplace abilities, compared with 56% in the survey. Sequential completion has mixed effects on joint validity, and reversing question order produces different joint distributions in both tested cases. Order checks and classical marginal bounds clarify what can be assessed before linked validation answers are available. The findings support validating the joint events a simulation uses and separating empirical constraints from model assumptions. The evidence concerns selected binary tasks and one model version.
Figures & tables
| Block | Given answer or attribute | Target answer(s) | Evaluation n |
|---|---|---|---|
| W92 pairs | Economic system: unfair or fair | Corporate profit, government size, federal health coverage, labor unions | 4891–10052 |
| W54 pairs | Savings account | Retirement account, medical-bill debt | 3365, 3377 |
| W41 pairs | Race relations in 2050 | Public education, political division in 2050 | 1088, 1084 |
| W29 predictions | Recorded sex and traditional values | Similarity of men/women in parenting and workplace abilities | 1242 |
| Finance1 (W54) | Savings account | Retirement account (B), medical-bill debt (C) | 3360 |
| Finance2 (W54) | Investment ownership | Credit-card debt (B), car loan (C) | 3360 |
| Comparison | Categories | Brier gain | Respondent interval | Output range | Direction across outputs |
|---|---|---|---|---|---|
| W92 pairs (mean) | 2 | 0.0416 | [0.0358,0.0473] (95%) | [0.0388,0.0433] | Positive |
| W54 pairs (mean) | 4 | 0.0153 | [0.0112,0.0193] (95%) | [0.0123,0.0187] | Positive |
| W41 pairs (mean) | 4 | 0.0468 | [0.0356,0.0583] (95%) | [0.0409,0.0510] | Positive |
| W29 targets (mean) | 2 | [ , ] (95%) | [ , ] | Negative | |
| Finance1 | 8 | [ , ] (95%) | [ ,0.0002] | Crosses zero | |
| Finance2 | 8 | [ ,0.0020] (95%) | [ ,0.0026] | Crosses zero |
| Wave | Target | Human | Model | Error (pp) |
|---|---|---|---|---|
| W54 | Retirement account | 56.4 | 69.7 | 13.3 |
| W54 | Medical-bill debt | 30.0 | 32.0 | 2.0 |
| W41 | Public education in 2050 | 44.7 | 28.7 | 16.0 |
| W41 | Political division in 2050 | 70.0 | 76.7 | 6.7 |
| W29 | Parenting similarity | 33.5 | 57.0 | 23.5 |
| W29 | Workplace similarity | 63.3 | 61.3 | 1.9 |
| Wave | Target B | Separate | Conditional |
|---|---|---|---|
| W54 | Retirement account | 13.3 | 4.7 |
| W54 | Medical-bill debt | 8.1 | 4.0 |
| W41 | Public education in 2050 | 23.4 | 12.0 |
| W41 | Political division in 2050 | 17.5 | 4.2 |
| Case family | Contrast | Estimate | Respondent interval | Output range |
|---|---|---|---|---|
| Race/W29 | Race reverse minus forward gain | 0.0279 | [0.0206,0.0352] | [0.0204,0.0364] |
| Race/W29 | Workplace masked minus named gain | [ , ] | [ , ] | |
| Race/W29 | Race masked minus named forward gain | 0.0068 | [0.0049,0.0086] | [ ,0.0242] |
| W54/Finance2 | W54 masked minus named mean pair gain | 0.0002 | [ ,0.0028] | [ ,0.0077] |
| W54/Finance2 | Finance2 reverse minus forward gain | 0.0009 | [ ,0.0036] | [ ,0.0069] |
| Available information | Identified quantity or assumption | Implication in these cases |
|---|---|---|
| No target-population human labels | Only the model’s proposed distribution is observed. Frequency error cannot be measured. | Evaluation labels reveal raw errors up to 23.5 pp, which the probability vector alone does not disclose. |
| One eligible-cohort A marginal | A shares are supplied. Missing conditional frequencies remain model-dependent. | Four anchored pair generators improve but retain 4–12% joint TV. Applicability depends on the cohort represented by the supplied A shares. |
| A person’s recorded answer | Fixes this person’s A answer, specifying the query . Estimating the corresponding human share requires linked answers or additional assumptions. | W92 gains and W29 losses require separate linked-human evaluation. |
| B and C target margins | Conjunction shares lie within Fréchet bounds. A particular association is not identified. | Finance2 permits both-answer prevalence anywhere from 0 to 40.6%. |
| Separate linked (A,B) and (A,C) tables | Conditional bounds are available. selects one completion through an added assumption. | Finance2’s human-margin conditional-independence completion gives 21.2%, versus the observed 28.0%. |
| Linked validation records for A,B,C | Permit estimating joint cells, associations, event errors, and paired score differences in the evaluated cohort. | Expose Finance2’s conjunction deficit and Race’s distorted association. |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Case block | Selection basis | Prior exposure and interpretation |
|---|---|---|
| W92 pairs | Four binary targets chosen by wording and availability of linked answers | Four targets evaluated under the same fixed-mean comparison. |
| W54 pairs | Savings, retirement, and medical debt chosen from related self-report items | Pair evaluation uses disjoint reference/evaluation rows. Finance1 reuses the item content. |
| W41 pairs | Related opinions in the archived 2050 scenario | Two targets evaluated as a pair block. No random domain sampling. |
| W29 predictions | Traditional values and two gender-opinion items | Recorded sex was selected after checking conditional-group eligibility and before forecasting or evaluation decoding. |
| Finance1 | Existing finance block used to test a new completion operation | Exploratory on previously analyzed item content. |
| Finance2 | Unqueried investment item and first two unqueried debt items by codebook order | Same W54 wave and split. Comparisons specified before model requests for these items. |
| Task | n | Categories | Gain | Respondent interval | Output range |
|---|---|---|---|---|---|
| W92 Corporate profit | 9960 | 2 | 0.0740 | [0.0600,0.0874] | [0.0714,0.0755] |
| W92 Government size | 9958 | 2 | 0.0275 | [0.0175,0.0374] | [0.0245,0.0292] |
| W92 Federal health coverage | 10052 | 2 | 0.0402 | [0.0356,0.0451] | [0.0358,0.0437] |
| W92 Labor unions | 4891 | 2 | 0.0246 | [0.0158,0.0332] | [0.0235,0.0250] |
| W54 Retirement account | 3365 | 4 | 0.0258 | [0.0179,0.0338] | [0.0232,0.0298] |
| W54 Medical-bill debt | 3377 | 4 | 0.0048 | [0.0024,0.0072] | [0.0013,0.0076] |
| Association | Human OR | Human interval | Model OR | Model output range |
|---|---|---|---|---|
| W92 Corporate profit | 6.677 | [5.753,7.817] | 9.898 | [8.609,11.870] |
| W92 Government size | 0.324 | [0.276,0.380] | 0.216 | [0.183,0.251] |
| W92 Federal health coverage | 4.807 | [4.114,5.574] | 2.283 | [2.000,2.567] |
| W92 Labor unions | 2.780 | [2.266,3.435] | 2.698 | [2.357,3.217] |
| W54 Retirement account | 4.795 | [3.844,6.057] | 5.182 | [4.644,5.881] |
| W54 Medical-bill debt | 0.477 | [0.379,0.594] | 0.528 | [0.452,0.596] |
| Case | Stratum | Human [interval] | Separate | Sequential | Separate range | Sequential range |
|---|---|---|---|---|---|---|
| Finance1 | All A | 0.48 [0.39,0.59] | 0.88 | 0.97 | [0.85,0.92] | [0.86,1.08] |
| Finance1 | A=0 | 0.50 [0.39,0.66] | 1.00 | 1.05 | [1.00,1.00] | [0.95,1.15] |
| Finance1 | A=1 | 0.73 [0.48,1.08] | 1.00 | 1.01 | [1.00,1.00] | [0.88,1.14] |
| Finance2 | All A | 3.28 [2.70,3.97] | 1.04 | 1.69 | [1.02,1.06] | [1.44,1.91] |
| Finance2 | A=0 | 2.74 [2.08,3.61] | 1.00 | 1.58 | [1.00,1.00] | [1.41,1.75] |
| Finance2 | A=1 | 3.70 [2.87,4.78] | 1.00 | 1.68 | [1.00,1.00] | [1.42,1.91] |
| Target | Sex | Human shares | Model shares | Human OR | Model OR | Model OR range |
|---|---|---|---|---|---|---|
| Parenting similarity | Male | 31.4 / 46.9 | 12.3 / 74.7 | 0.52 [0.28,0.96] | 0.048 | [0.039,0.053] |
| Parenting similarity | Female | 29.7 / 38.1 | 15.0 / 77.0 | 0.69 [0.38,1.28] | 0.053 | [0.042,0.065] |
| Workplace similarity | Male | 56.3 / 78.0 | 16.0 / 80.7 | 0.36 [0.16,0.71] | 0.046 | [0.036,0.054] |
| Workplace similarity | Female | 62.4 / 72.7 | 19.7 / 84.7 | 0.62 [0.29,1.19] | 0.044 | [0.038,0.051] |
| Item | Wording and options |
|---|---|
| ECONFAIR_W92 | Please choose the statement that comes closer to your own views. First: The economic system in this country unfairly favors powerful interests Second: The economic system in this country is generally fair to most Americans |
| BUSPROFIT_W92 | Please choose the statement that comes closer to your own views. First: Business corporations make too much profit Second: Most corporations make a fair and reasonable amount of profit |
| GOVSIZE1_W92 | If you had to choose, would you rather have First: A smaller government providing fewer services Second: A bigger government providing more services |
| GOVTHC_W92 | Do you think it is the responsibility of the federal government to make sure all Americans have health care coverage? First: Yes, it is Second: No, it is not |
| INSTN_LBRUN_W92 | Are labor unions having a positive or negative effect on the way things are going in the country these days? First: Positive effect Second: Negative effect |
| FINANCEa_W54 | Do you have a savings account? First: Yes, have this Second: No, do not have this |
| Item | Wording and options |
|---|---|
| HAPPENj_W41 | Still thinking ahead 30 years, which do you think is more likely to happen in the U.S.? The country will be more/less politically divided First: The country will be more politically divided Second: The country will be less politically divided |
| MESUM2_FB_W29 | How well, if at all, do the following words or phrases describe you? Have traditional values First: Describes me well Second: Does not describe me well |
| DIFF1C_W29 | Do you think men and women are basically similar or basically different when it comes to their approach to parenting? First: Men and women are basically similar Second: Men and women are basically different |
| DIFF1E_W29 | Do you think men and women are basically similar or basically different when it comes to the things they are good at in the workplace? First: Men and women are basically similar Second: Men and women are basically different |
| FINANCEc_W54 | Do you have personal investments in stocks, bonds or mutual funds other than those held in an IRA or 401K? First: Yes, have this Second: No, do not have this |
| DEBTa_W54 | Do you have the following type of loans or debt: Credit card debt First: Yes, have this Second: No, do not have this |
| Analysis family | Size | Comparisons covered | Reported level |
|---|---|---|---|
| W29 historical context and Race | 2 | W29 historical-context gain and Race sequential-completion gain | 97.5% per contrast |
| Race/W29 controls | 3 | Race reverse minus forward gain, W29 masked minus named gain, Race masked minus named forward gain | 98.333% per contrast |
| W54/Finance2 controls | 2 | W54 masked minus named mean pair gain and Finance2 reverse minus forward gain | 97.5% per contrast |
| Other score and OR estimates | n/a | Other original Brier comparisons, matched-mean joint diagnostics, secondary arm gains, and human odds ratios | 95% pointwise |
| Human-share bands | 26 coordinates | Centered within-wave maxima with allocation across three waves | Approximate 95% simultaneous bands |