The As-If Machine: Personalized counterfactuals make distant policy risks feel close

Joshua Ashkinaze; Yubo Zhou; Pengyu Chen; Avi Bhattacharya; Hyebin Park; Yanman Zhang; Stephanie Preston; Eric Gilbert

cs.HCOct 23, 2026

The As-If Machine: Personalized counterfactuals make distant policy risks feel close

Authors: Joshua Ashkinaze, Yubo Zhou, Pengyu Chen, Avi Bhattacharya, Hyebin Park, Yanman Zhang, Stephanie Preston, Eric Gilbert,

Organizations: University of Michigan

Abstract

Life experiences can shape policy attitudes, but we only live one life. Citizens may therefore fail to support policies addressing risks they have never personally experienced. Narratives can make such risks immediate and personal, but personalized stories remain difficult to produce at scale. Here we ask whether AI can make distant scenarios feel concrete by simulating a participant's life as if that scenario occurred. We present the As-If Machine (AIM), a multi-agent system through which people co-write personalized counterfactual narratives---stories about their own lives under different policy futures. In two pre-registered and quota-matched experiments on science funding (N=1,007) and satellite cybersecurity (N=894), we compared AIM with two controls: reading an expert-written vignette on the same topic and co-writing with AIM about an unrelated topic. AIM increased psychological closeness---how close a risk feels to the self---by nearly identical margins in the two domains (0.36 and 0.38 standard deviations (SDs) against the vignette, 0.80 and 0.90 against unrelated writing). AIM increased psychological closeness even among participants with unfavorable views of AI. For policy attitudes, AIM's effects were 0.16 SDs against the vignette and 0.10 SDs (n.s.) against unrelated writing for science funding; the corresponding effects for satellite cybersecurity were 0.54 and 0.67 SDs. Across 23,496 human-AI autocomplete interactions, participants and the system played distinct roles: AIM supplied concrete detail that grounded the scenario, and participants supplied first-person perspective and emotion. Co-writing with the As-If Machine makes distant policy risks feel close.

Figures & tables

Figure 1: Panel a , Participant view: Participants first complete an intake form. They then continue a scenario prompt that the system built from their answers, shown by orange outlines. Whenever they pause, the system suggests a continuation, which they can accept by pressing Tab. Panel b , System view: Four steps run once before writing begins. The system turns the intake answers into a text profile (1). The Initializer writes a personalized event onset, which becomes the visible prompt for the participant (2). The Predictor forecasts how the event would affect the participant (3). The Storyteller combines the onset and effects into a plot (4). During writing, the Autocompleter reads the profile, event, plot, and current text and offers continuations (5).

Figure 4: Suggestive mechanisms. A , Serial multiple-mediator model (closeness predicts transportation), AIM versus vignette, both studies pooled (

n=1{,}265

). Line widths correspond to effect size. B , D , Pearson correlations of transportation items (B) and system ratings (D) with each outcome among AIM participants, with

95\%

intervals; opacity indicates

p<.05

. C , Change in the AIM effect per standard deviation of AI attitudes, from each causal forest’s best linear projection. E , Mean system ratings by study and condition, rescaled to 1–10; stars test each mean against the scale midpoint. For all plots,

{}^{*}p<.05

{}^{**}p<.01

{}^{***}p<.001

; n.s., not significant.

Measure	AIM	Vignette	Placebo
Study 1: Science funding	n =333	n =331	n =343
Closeness index	0.40 (0.79)	0.04 (0.95)	-0.40 (1.09)
Effects on me (1–9)	7.55 (1.63)	6.66 (2.03)	5.73 (2.37)
Relevance to my life (1–9)	7.32 (1.83)	6.74 (2.02)	5.84 (2.29)
Attitude index	0.07 (1.00)	-0.09 (1.06)	-0.02 (0.95)
Personal benefit of U.S. science (1–5)	3.84 (0.97)	3.73 (1.01)	3.83 (0.97)

Table 1: Outcome means (SD) by study and condition. Bold rows are the two outcome indices, z-scored within study (mean 0, SD 1). Indented rows are the items that form each index, in their original units; the response scale follows each item. Study 2’s attitude index averages three subscales (importance, spending, action).

Study	Outcome	Control	$n$	$b$	$SE$	$z$	95% CI	$p$
Science	Closeness	vignette	1007	0.356	0.068	5.271	[0.224, 0.489]	$1.36\times 10^{-7}$
Science	Closeness	placebo	1007	0.801	0.073	10.956	[0.657, 0.944]	$6.25\times 10^{-28}$
Science	Attitudes	vignette	1007	0.160	0.080	2.000	[0.003, 0.317]	$.0455$
Science	Attitudes	placebo	1007	0.098	0.075	1.308	[-0.049, 0.244]	$.191$
Satellite	Closeness	vignette	894	0.379	0.069	5.468	[0.243, 0.515]	$4.55\times 10^{-8}$
Satellite	Closeness	placebo	894	0.896	0.078	11.545	[0.744, 1.048]	$7.8\times 10^{-31}$

Table 2: Primary treatment effects. AIM-minus-control estimates from condition-only OLS models. Outcomes are the closeness and attitude indices, z-scored within study, so

b

is in standard deviations; Extended Data Table 1 lists the items in each index.

study	outcome	sample	control	n	Effect [95% CI]	$p$
Science	Closeness	Low	vignette	488	0.55 [0.35, 0.76]	$2.00\times 10^{-7}$
Science	Closeness	Low	placebo	488	1.09 [0.87, 1.31]	$1.65\times 10^{-22}$
Science	Closeness	High	vignette	519	0.16 [0.00, 0.31]	$.049$
Science	Closeness	High	placebo	519	0.52 [0.35, 0.70]	$4.44\times 10^{-9}$
Science	Attitudes	Low	vignette	488	0.32 [0.11, 0.54]	$.003$
Science	Attitudes	Low	placebo	488	0.21 [0.00, 0.42]	$.047$

Table 3: Familiarity split at 2. AIM-minus-control estimates with

95\%

confidence intervals.

study	outcome	sample	control	n	Effect [95% CI]	$p$
Science	Closeness	Low	vignette	662	0.47 [0.30, 0.65]	$1.30\times 10^{-7}$
Science	Closeness	Low	placebo	662	1.00 [0.82, 1.19]	$5.50\times 10^{-27}$
Science	Closeness	High	vignette	345	0.15 [-0.03, 0.34]	$.100$
Science	Closeness	High	placebo	345	0.36 [0.17, 0.56]	$2.81\times 10^{-4}$
Science	Attitudes	Low	vignette	662	0.23 [0.05, 0.41]	$.012$
Science	Attitudes	Low	placebo	662	0.15 [-0.02, 0.31]	$.087$

Table 4: Familiarity split at 3. AIM-minus-control estimates with

95\%

confidence intervals.

study	control	n	path	Effect [95% CI]
Science	vignette	664	Total	0.160 [0.003, 0.317]
Science	vignette	664	Direct	-0.009 [-0.156, 0.138]
Science	vignette	664	Indirect	0.169 [0.100, 0.247]
Science	placebo	676	Total	0.098 [-0.049, 0.245]
Science	placebo	676	Direct	-0.229 [-0.375, -0.083]
Science	placebo	676	Indirect	0.327 [0.247, 0.421]

Table 5: Single-mediator models by study and control. Indirect effects use bias-corrected bootstrap intervals.

Subscale	Control	$n$	$b$	95% CI	$z$	$p$
Importance	Vignette	894	0.495	[0.349, 0.642]	6.636	$3.22\times 10^{-11}$
Importance	Placebo	894	0.603	[0.450, 0.756]	7.732	$1.06\times 10^{-14}$
Spending	Vignette	894	0.508	[0.354, 0.663]	6.453	$1.09\times 10^{-10}$
Spending	Placebo	894	0.562	[0.412, 0.713]	7.306	$2.75\times 10^{-13}$
Action	Vignette	894	0.407	[0.249, 0.566]	5.030	$4.89\times 10^{-7}$
Action	Placebo	894	0.563	[0.414, 0.712]	7.399	$1.37\times 10^{-13}$

Table 6: Secondary satellite attitude subscales. AIM-minus-control estimates from three-arm OLS models with HC1-robust intervals and two-sided tests.

Figure 1: AIM interface for an illustrative scenario in which gravity disappears. The next panel shows the configuration file that generates it; neither experiment used this scenario.

Figure 2: AIM interface and scenario configuration, continued. Example YAML configuration for the gravity-disappearance scenario shown in the preceding panel.

Figure 3: Satellite interface and autocomplete prompt. Top, the co-writing interface; bottom, the autocomplete request from step 5 of Figure 1 , with colors marking the source of each element.

Best Practices for Narrative Transportation	Our System
Connect the character to the user’s life	AIM uses participants’ intake responses to personalize the character and generate a realistic scenario onset grounded in their circumstances.
Use material familiar to the reader	Intake questions capture participants’ routines and daily lives, giving the agents familiar material to incorporate into each story.
Cue “remindings” of the reader’s own experiences	Intake questions ask participants to describe their own lives, and the scenario is constructed from these responses.
Use rich mental imagery	Agent prompts instruct AIM to describe concrete surroundings, actions, and consequences.
Maintain psychological realism	The Initializer and Predictor Agents are instructed to be “very specific and very realistic” and not to exaggerate, while the scenario onset is constrained to follow from the participant’s stated circumstances.
Keep stories cohesive	The agent pipeline builds each plot from a scenario onset and its predicted effects, preserving continuity across the narrative.

Table 7: Embedding narrative transportation best practices [ 18 ] into the system.

Outcome	Control	All participants $b$ [95% CI]	Excluding 08-21 $b$ [95% CI]	Effect $\times$ period $b$ [95% CI]	$p$
Closeness	Placebo	0.90 [0.74, 1.05]	0.86 [0.70, 1.03]	-0.23 [-0.63, 0.18]	.280
Closeness	Vignette	0.38 [0.24, 0.51]	0.42 [0.27, 0.57]	0.20 [-0.16, 0.55]	.276
Attitudes	Placebo	0.67 [0.52, 0.82]	0.66 [0.49, 0.82]	-0.09 [-0.45, 0.28]	.645
Attitudes	Vignette	0.54 [0.39, 0.69]	0.55 [0.38, 0.72]	0.00 [-0.37, 0.37]	.989

Table S1: Sensitivity of the Study 2 effects to the timer change of 2026-08-22, shown on all participants and again excluding the 146 who began before it. Estimates are AIM-minus-control differences in SDs with HC1 confidence intervals; the last column is the effect

\times

period interaction, the formal test of whether the effect differs across the change.

Variable	Treat	Placebo	Vignette	Test	q
Age	45.62 (15.43)	45.39 (15.36)	45.70 (15.69)	F=0.04	1.000
Male (%)	45.9%	47.2%	49.8%	$\chi^{2}$ =1.05	1.000
Education (Recoded 1–8)	3.15 (0.98)	3.16 (1.14)	3.25 (1.29)	F=0.81	1.000
Income (Recoded 1–7)	4.06 (1.90)	4.22 (1.93)	4.38 (1.82)	F=2.38	1.000
Political Ideology (+ = conservative)	3.63 (1.79)	3.71 (1.75)	3.83 (1.83)	F=1.06	1.000
Familiarity w/ sci. funding	2.87 (1.54)	2.76 (1.49)	2.82 (1.54)	F=0.42	1.000

Table S2: Study 1 (science funding) participant descriptives by arm: pre-treatment covariates (M (SD) for continuous; % for binary). One-way ANOVA

F

-tests for continuous variables;

\chi^{2}

for binary. Party ID (Democrat / Independent / Republican) is a

\chi^{2}

test on participants with Prolific party data, missing for 5 participants.

q

is the sharpened false-discovery-rate

q

value, correcting for multiple comparisons across covariates.

Variable	Treat	Placebo	Vignette	Test	q
Age	45.81 (15.23)	45.61 (14.65)	47.29 (14.36)	F=1.17	0.329
Male (%)	40.9%	46.8%	49.2%	$\chi^{2}$ =4.33	0.221
Education (Recoded 1–8)	3.27 (1.13)	3.22 (1.10)	3.18 (1.04)	F=0.48	0.567
Income (Recoded 1–7)	4.29 (1.89)	4.10 (1.81)	3.87 (1.86)	F=3.78	0.133
Political Ideology (+ = conservative)	3.56 (1.72)	3.83 (1.76)	3.82 (1.85)	F=2.21	0.221
Familiarity w/ satellite cybersecurity	2.72 (1.60)	2.80 (1.51)	3.05 (1.55)	F=3.72	0.133

Table S3: Study 2 (satellite cybersecurity) participant descriptives by arm: pre-treatment covariates (M (SD) for continuous; % for binary). One-way ANOVA

F

-tests for continuous variables;

\chi^{2}

for binary. Party ID (Democrat / Independent / Republican) is a

\chi^{2}

test on participants with Prolific party data, missing for 6 participants.

q

is the sharpened false-discovery-rate

q

value, correcting for multiple comparisons across covariates.

Sample	Study	Mediator	Slope [95% CI]	p	n
All participants	Study 1	Closeness	0.41 [0.36, 0.47]	<.001	1007
All participants	Study 2	Closeness	0.62 [0.57, 0.68]	<.001	894
All participants	Study 2 minus Study 1	Closeness	0.21 [0.13, 0.29]	<.001	1901
AIM Treatment	Study 1	Closeness	0.38 [0.22, 0.54]	<.001	333
AIM Treatment	Study 2	Closeness	0.65 [0.51, 0.78]	<.001	298
AIM Treatment	Study 2 minus Study 1	Closeness	0.27 [0.06, 0.47]	.011	631

Table S4: Conversion of mediators into attitudes. Each slope is the change in attitudes (SD) per SD of the mediator, by study. Study 2 minus Study 1 rows report the study x mediator interaction from models pooling both studies within each sample. HC1 intervals.

Sample	Study 1 R 2 (%)	Study 2 R 2 (%)	Difference (points)	n (Study 1, Study 2)
Both controls	3.9 [2.0, 7.7]	0.7 [0.1, 3.2]	3.3 [0.2, 6.6]	674, 596
All arms	3.3 [1.8, 5.9]	0.4 [0.1, 2.2]	2.9 [0.5, 5.2]	1007, 894

Table S5: Variance in attitudes explained by ideology: the R 2 of attitudes on ideological direction, strength, and their interaction, among control participants and among all participants, by study. Difference is Study 1 minus Study 2 in percentage points. Brackets give 95% percentile intervals from 5,000 bootstrap draws, resampling participants within arm.

Sample	Label	Study 1 (%)	Study 2 (%)	n (Study 1, Study 2)	z	p
AIM and vignette	Crystallized (either type)	13.7	6.5	664, 600	4.21	<.001
AIM and vignette	Type 1: prior familiarity	13.1	6.2	664, 600	4.14	<.001
AIM and vignette	Type 2: reinforced	2.0	0.7	664, 600	1.99	.047
Vignette only	Crystallized (either type)	14.5	6.3	331, 302	3.35	<.001
Vignette only	Type 1: prior familiarity	13.9	6.0	331, 302	3.31	<.001
Vignette only	Type 2: reinforced	1.8	1.0	331, 302	0.87	.384

Table S6: Share of participants whose open-ended responses showed a crystallized attitude, by type, with two-proportion z-tests comparing the studies. Labels from gpt-4.1-mini, using the prompt above.

Sample	Raters	$\kappa$	95% CI	Agreement (%)	Disagreements	Positives
All ( $N=200$ )	Rater A vs Rater B	0.83	[0.74, 0.90]	92.5	15	63 / 62
All ( $N=200$ )	LLM vs Rater A	0.91	[0.83, 0.96]	96.0	8	61 / 63
All ( $N=200$ )	LLM vs Rater B	0.80	[0.71, 0.89]	91.5	17	61 / 62
Random ( $N=100$ )	Rater A vs Rater B	0.80	[0.56, 0.96]	96.0	4	9 / 13
Random ( $N=100$ )	LLM vs Rater A	0.71	[0.39, 0.93]	95.0	5	10 / 9
Random ( $N=100$ )	LLM vs Rater B	0.66	[0.39, 0.86]	93.0	7	10 / 13

Table S7: Inter-annotator agreement. Two human raters independently coded 200 open-ended responses as crystallized or not, and the LLM coded the same responses with a codebook fixed before human coding began. A response counts as crystallized if it shows prior familiarity or reinforcement, judged separately as subtypes. The first 100 responses were drawn at random; the remaining 100 combine random responses with responses the LLM coded as crystallized, shuffled together, to ensure enough positive cases. The table reports agreement between the two human raters and between the LLM and each human rater, on all responses and on the random subset alone.

\kappa

is Cohen’s kappa with a 95% percentile bootstrap interval (2,000 resamples of responses). Positives gives the number of responses each rater coded 1, in the order the raters are listed.

Rating	Study 2 $-$ Study 1 [95% CI]	$p$	$q$
Fun	-0.20 [-0.45, 0.05]	.111	.384
Tailored	-0.15 [-0.40, 0.11]	.257	.522
Realistic	-0.30 [-0.57, -0.04]	.024	.166
Helped	-0.08 [-0.33, 0.18]	.556	.589
Easy	0.06 [-0.07, 0.20]	.369	.536
Agency	0.09 [-0.14, 0.32]	.436	.536

Table S8: UX changes. Each row is one system rating, on its administered scale, among placebo participants only. The placebo task was identical in the two studies, so these differences isolate the interface change from the change of scenario. Estimates are Study 2 minus Study 1 with

95\%

HC1 confidence intervals,

p

values, and sharpened FDR

q

-values across the six ratings. n = 636 placebo participants.

Model	Gap vs vignette	Gap vs placebo
Simple	0.40 [0.18, 0.62]	0.57 [0.36, 0.79]
Composition	0.44 [0.24, 0.65]	0.55 [0.35, 0.75]
Restrictions	0.40 [0.18, 0.62]	0.58 [0.36, 0.79]
Composition + restrictions	0.44 [0.24, 0.65]	0.55 [0.35, 0.75]

Table S9: Composition effects and study restrictions. Each row models attitudes on arm, study, and their interactions, with the AIM arm omitted. Composition adds 13 pretreatment covariates; Restrictions adds Safari use (Study 2 excluded Safari users) and the number of words written, measured in hundreds. Gap vs vignette is Study 2’s AIM effect against the vignette minus Study 1’s corresponding effect; gap vs placebo is defined analogously. Brackets give 95% confidence intervals (HC1). If participant composition or the study restrictions explained the difference between studies, these gaps would shrink toward zero after adjustment.

Study	Mean similarity	Slope [95% CI]	$p$	$n$
Study 1	0.150	-0.08 [-0.20, 0.04]	.176	330
Study 2	0.127	0.09 [-0.01, 0.19]	.079	298

Table S10: Tighter connection to the DVs. Similarity is the mean cosine similarity between an AIM participant’s story and the mean embedding of that study’s attitude items (SBERT all-mpnet-base-v2). Slope is the change in attitudes (SD) per SD of similarity among AIM participants, with a

95\%

confidence interval (HC1) and

p

value. Study 1 stories were the more similar of the two to their own attitude items (Welch

t(591)=3.04

p=.002

), the opposite of what an explanation running through item overlap would require.

Study	Item	At scale max placebo $\to$ AIM	OLS [95% CI]	Tobit [95% CI]
Study 1	Preferred NIH budget share	0.3% $\to$ 0.6%	0.12 [-0.03, 0.27]	0.13 [-0.03, 0.29]
Study 1	Science benefits people like me	27.4% $\to$ 27.6%	0.02 [-0.13, 0.17]	0.02 [-0.19, 0.23]
Study 2	Personal importance	13.7% $\to$ 24.8%	0.62 [0.46, 0.77]	0.72 [0.54, 0.90]
Study 2	Importance vs other issues	6.5% $\to$ 14.8%	0.53 [0.38, 0.69]	0.59 [0.42, 0.76]
Study 2	Spending: reduce risk	3.8% $\to$ 10.7%	0.49 [0.34, 0.65]	0.53 [0.36, 0.70]
Study 2	Spending: priority	2.7% $\to$ 12.8%	0.59 [0.44, 0.74]	0.65 [0.47, 0.82]

Table S11: Ceiling effects on the attitude items. Each item is z-scored within its study, and effects are AIM minus placebo in item standard deviations. The two-sided Tobit treats values as censored and estimates the effect on the latent scale, rescaled by the observed standard deviation so OLS and Tobit are comparable. comparable. Two columns show percentage of each condition’s participants at the maximum item.

Quantity	Study 1	Study 2
Vignette emotion	4.57	3.65
AIM emotion	4.02	4.13
a: AIM effect on emotion	-0.55	0.48
b: Emotion -> attitudes	0.073	0.136
c: AIM effect on attitudes	0.16	0.54
Extra AIM effect on emotion in Study 2	1.024

Table S12: Whether greater emotional engagement in Study 2 can account for its larger AIM attitude effect. Path a is AIM’s effect on emotional engagement relative to the vignette. Path b is the association between emotional engagement and attitudes, adjusting for treatment and closeness. Path c is AIM’s total effect on attitudes relative to the vignette. The predicted additional attitude effect is (a2-a1)b1, and the share accounted for is 100(a2-a1)b1/(c2-c1).

Study	Outcome	vs.	Unfavourable $b$ [95% CI]	Favourable $b$ [95% CI]	Interaction $b$ [95% CI]	$p$
Study 1	Closeness	Vignette	+0.53 [+0.24, +0.81]	+0.30 [+0.15, +0.45]	-0.08 [-0.22, +0.05]	.229
Study 1	Closeness	Placebo	+0.76 [+0.44, +1.07]	+0.81 [+0.65, +0.97]	+0.05 [-0.10, +0.21]	.494
Study 1	Attitudes	Vignette	+0.25 [-0.02, +0.51]	+0.13 [-0.06, +0.32]	-0.10 [-0.26, +0.06]	.209
Study 1	Attitudes	Placebo	+0.03 [-0.24, +0.31]	+0.12 [-0.05, +0.29]	-0.02 [-0.17, +0.13]	.785
Study 2	Closeness	Vignette	+0.41 [+0.13, +0.70]	+0.36 [+0.21, +0.52]	-0.05 [-0.19, +0.09]	.475
Study 2	Closeness	Placebo	+0.99 [+0.67, +1.31]	+0.86 [+0.69, +1.03]	-0.14 [-0.30, +0.02]	.093

Table S13: AIM’s effects by prior attitude toward AI, split at the midpoint of the 1–10 AI-attitude scale. Estimates are AIM-minus-control differences in outcome SDs with HC1 confidence intervals. Interaction is the arm by AI-attitude term, in SDs per SD of AI attitude.

Study	Control	Proposed chain	$n$	Unadjusted treatment coefficient	Adjusted treatment coefficient	Tolerated
Pooled	Vignette	AIM $\rightarrow$ transportation $\rightarrow$ closeness	1265	+0.37 [+0.27, +0.46]	+0.26 [+0.18, +0.35]	No
Pooled	Vignette	AIM $\rightarrow$ closeness $\rightarrow$ transportation	1265	+0.23 [+0.12, +0.35]	+0.01 [-0.09, +0.11]	Yes
Study 1	Vignette	AIM $\rightarrow$ transportation $\rightarrow$ closeness	664	+0.36 [+0.22, +0.49]	+0.31 [+0.20, +0.43]	No
Study 1	Vignette	AIM $\rightarrow$ closeness $\rightarrow$ transportation	664	+0.10 [-0.05, +0.26]	-0.10 [-0.24, +0.04]	Yes
Study 2	Vignette	AIM $\rightarrow$ transportation $\rightarrow$ closeness	601	+0.38 [+0.24, +0.51]	+0.21 [+0.09, +0.32]	No
Study 2	Vignette	AIM $\rightarrow$ closeness $\rightarrow$ transportation	601	+0.38 [+0.22, +0.54]	+0.13 [-0.01, +0.28]	Yes

Table S14: Treatment coefficients before and after adjusting for the proposed upstream mediator. Coefficients are in SD units with 95% HC1 confidence intervals.

Figure S2: Serial multiple-mediator model (primary specification), every study and control comparison. Closeness predicts transportation; see § 14.9 for why.

Figure S3: Parallel multiple-mediator model (alternative), every study and control comparison.

Type	Theme	%	Representative quote
Success Mode	Made issue more personal	30.1%	Having to contemplate a real-life scenario for myself made it personal and, thus, more effective.
Success Mode	Scenario specifics were effective	29.0%	Simply the idea of having to wait 1 year + for an appointment
Success Mode	The experience caused reflection	16.6%	Thinking about what could happen and how it could affect my life was effective.
Success Mode	The activity made the issue feel real	9.3%	Writing the story showed me how funding cuts can directly affect people’s health, which made the issue feel more real.
Success Mode	Writing was emotionally charged	8.3%	I felt that it was totally effecting [sic] my own life in that I was suffering and unable to find a solution, so that everyday went by while I seemed to be on hold.
Success Mode	AI predictive ability	6.7%	The responses after I typed a couple of words were effective and captured what I was trying to say.

Table S15: Qualitative themes in Study 1 (science funding) open-text responses from the AIM arm. Participants described what was effective and what was ineffective about the co-writing experience. We clustered responses with BERTopic on INSTRUCTOR embeddings and excluded clusters that restated the question or gave no substantive answer (e.g., “nothing was ineffective”). Percentages use the 193 substantive responses about what was effective and the 133 about what was ineffective.

Type	Theme	%	Representative quote
Success Mode	Realizing the scale of disruption	31.4%	It made me realize how big a threat this is to my everyday life. My job, communication, and navigating abilities would be severely disrupted.
Success Mode	Rehearsing what I would do	23.1%	It made me feel like I was in the situation and forced me to think about what I would actually do.
Success Mode	Everyday dependencies made it concrete	13.2%	Writing out a first person scenario grounded the abstract idea of satellite systems into immediate, everyday consequences like failed card payment, loss of phone service and bank outages.
Success Mode	Topic had never been considered	10.7%	I haven’t really pondered cybersecurity satellites in the past so it was a bit eye opening.
Success Mode	Recognizing personal reliance	10.7%	Making me more aware of what I use in my life that would be effected by this. I didn’t think I did, but as I started navigating a night in my life, I realized that a lot of what I use works on these systems.
Success Mode	Surfaced an unnoticed vulnerability—not having cash on hand	10.7%	Making it very obvious that my habit of never having cash on hand probably isn’t a good idea.

Table S16: Qualitative themes in Study 2 (satellite cybersecurity) open-text responses from the AIM arm. Participants described what was effective and what was ineffective about the co-writing experience. We clustered responses with BERTopic on INSTRUCTOR embeddings and excluded clusters that restated the question or gave no substantive answer (e.g., “nothing was ineffective”). Percentages use the 121 substantive responses about what was effective and the 78 about what was ineffective.

Study	Pairs	Mean AUC (SD)
Science funding	285	0.914 (0.026)
Satellite	259	0.940 (0.028)

Table S17: Human–AI text classification with all 109 linguistic features (Biber and Empath). Mean AUC, with its SD across five participant-grouped folds.

Model	$n$	$\chi^{2}$	df	CFI	TLI	RMSEA	$p$
Three factors	894	122.584	11	0.973	0.948	0.107	$5.47\times 10^{-21}$
One factor	894	932.935	14	0.775	0.662	0.271	$3.77\times 10^{-190}$

Table S18: Satellite attitude measurement models. Confirmatory factor analysis of the seven attitude items (

n=894

Moderator	Closeness $B$ [95% CI]	Closeness $p$	Closeness $q$	Attitudes $B$ [95% CI]	Attitudes $p$	Attitudes $q$
Intercept	0.90 [0.71, 1.09]	$7.2\times 10^{-20}$		0.17 [-0.05, 0.39]	$.123$
Familiarity	-0.36 [-0.53, -0.20]	$1.4\times 10^{-5}$	$\leq.001$	-0.17 [-0.35, 0.01]	$.0611$	$.913$
Age	-0.21 [-0.37, -0.05]	$.00967$	$.084$	-0.18 [-0.35, -0.01]	$.039$	$.913$
Mental imagery	0.11 [-0.07, 0.28]	$.229$	$.684$	0.13 [-0.02, 0.28]	$.0842$	$.913$
IRI fantasy	0.08 [-0.09, 0.26]	$.346$	$.687$	-0.09 [-0.26, 0.09]	$.322$	$.913$
IRI perspective	-0.15 [-0.33, 0.03]	$.109$	$.444$	-0.09 [-0.25, 0.07]	$.253$	$.913$

Table S19: Study 1: Science funding, AIM versus placebo: causal-forest best linear projections (

n=644

). Gender uses female as the reference.

q

is the sharpened false-discovery-rate q-value, with the moderator coefficients of each outcome as one family.

Figure S4: Best linear projections: Study 1: Science funding, psychological closeness, AIM versus placebo.

Figure S5: Best linear projections: Study 1: Science funding, policy attitudes, AIM versus placebo.

Figure S6: Conditional effects: Study 1: Science funding, psychological closeness, AIM versus placebo.

Figure S7: Conditional effects: Study 1: Science funding, policy attitudes, AIM versus placebo.

Moderator	Closeness $B$ [95% CI]	Closeness $p$	Closeness $q$	Attitudes $B$ [95% CI]	Attitudes $p$	Attitudes $q$
Intercept	0.45 [0.26, 0.64]	$2.7\times 10^{-6}$		0.04 [-0.21, 0.28]	$.777$
Familiarity	-0.09 [-0.23, 0.05]	$.195$	$.451$	-0.10 [-0.29, 0.08]	$.279$	$.674$
Age	-0.11 [-0.25, 0.02]	$.0956$	$.451$	-0.14 [-0.31, 0.03]	$.109$	$.674$
Mental imagery	0.08 [-0.09, 0.24]	$.369$	$.528$	0.17 [0.01, 0.33]	$.0425$	$.674$
IRI fantasy	0.03 [-0.14, 0.21]	$.713$	$.528$	-0.13 [-0.32, 0.06]	$.19$	$.674$
IRI perspective	-0.03 [-0.19, 0.13]	$.746$	$.528$	-0.00 [-0.19, 0.18]	$.961$	$.674$

Table S20: Study 1: Science funding, AIM versus vignette: causal-forest best linear projections (

n=638

). Gender uses female as the reference.

q

is the sharpened false-discovery-rate q-value, with the moderator coefficients of each outcome as one family.

Figure S8: Best linear projections: Study 1: Science funding, psychological closeness, AIM versus vignette.

Figure S9: Best linear projections: Study 1: Science funding, policy attitudes, AIM versus vignette.

Figure S10: Conditional effects: Study 1: Science funding, psychological closeness, AIM versus vignette.

Figure S11: Conditional effects: Study 1: Science funding, policy attitudes, AIM versus vignette.

Moderator	Closeness $B$ [95% CI]	Closeness $p$	Closeness $q$	Attitudes $B$ [95% CI]	Attitudes $p$	Attitudes $q$
Intercept	0.96 [0.76, 1.16]	$8.64\times 10^{-19}$		0.66 [0.46, 0.85]	$1.25\times 10^{-10}$
Familiarity	-0.27 [-0.44, -0.11]	$.00116$	$.021$	-0.11 [-0.27, 0.05]	$.172$	$1.000$
Age	-0.21 [-0.38, -0.04]	$.0147$	$.134$	-0.14 [-0.30, 0.03]	$.111$	$1.000$
Mental imagery	-0.00 [-0.17, 0.16]	$.973$	$1.000$	0.02 [-0.14, 0.18]	$.837$	$1.000$
IRI fantasy	0.09 [-0.12, 0.29]	$.4$	$1.000$	0.12 [-0.06, 0.30]	$.183$	$1.000$
IRI perspective	0.03 [-0.16, 0.23]	$.731$	$1.000$	0.00 [-0.17, 0.18]	$.974$	$1.000$

Table S21: Study 2: Satellite cybersecurity, AIM versus placebo: causal-forest best linear projections (

n=564

). Gender uses female as the reference.

q

is the sharpened false-discovery-rate q-value, with the moderator coefficients of each outcome as one family.

Figure S12: Best linear projections: Study 2: Satellite cybersecurity, psychological closeness, AIM versus placebo.

Figure S13: Best linear projections: Study 2: Satellite cybersecurity, policy attitudes, AIM versus placebo.

Figure S14: Conditional effects: Study 2: Satellite cybersecurity, psychological closeness, AIM versus placebo.

Figure S15: Conditional effects: Study 2: Satellite cybersecurity, policy attitudes, AIM versus placebo.

Moderator	Closeness $B$ [95% CI]	Closeness $p$	Closeness $q$	Attitudes $B$ [95% CI]	Attitudes $p$	Attitudes $q$
Intercept	0.53 [0.35, 0.72]	$3.93\times 10^{-8}$		0.58 [0.37, 0.79]	$1.1\times 10^{-7}$
Familiarity	-0.07 [-0.21, 0.07]	$.33$	$.881$	-0.11 [-0.27, 0.05]	$.186$	$1.000$
Age	-0.10 [-0.25, 0.05]	$.179$	$.881$	-0.00 [-0.15, 0.15]	$.996$	$1.000$
Mental imagery	-0.12 [-0.28, 0.05]	$.175$	$.881$	-0.12 [-0.27, 0.03]	$.122$	$1.000$
IRI fantasy	0.05 [-0.15, 0.24]	$.635$	$1.000$	0.08 [-0.11, 0.27]	$.401$	$1.000$
IRI perspective	0.08 [-0.09, 0.25]	$.38$	$.881$	-0.06 [-0.25, 0.12]	$.523$	$1.000$

Table S22: Study 2: Satellite cybersecurity, AIM versus vignette: causal-forest best linear projections (

n=579

). Gender uses female as the reference.

q

is the sharpened false-discovery-rate q-value, with the moderator coefficients of each outcome as one family.

Figure S16: Best linear projections: Study 2: Satellite cybersecurity, psychological closeness, AIM versus vignette.

Figure S17: Best linear projections: Study 2: Satellite cybersecurity, policy attitudes, AIM versus vignette.

Figure S18: Conditional effects: Study 2: Satellite cybersecurity, psychological closeness, AIM versus vignette.

Figure S19: Conditional effects: Study 2: Satellite cybersecurity, policy attitudes, AIM versus vignette.

Figure S20: Best linear projection coefficients in both studies. Points show each moderator’s coefficient with its

95\%

confidence interval for Study 1 (science funding) and Study 2 (satellite cybersecurity). Moderators are standardized within study. Each panel is one outcome and AIM–control comparison, and

r

is the correlation between the two studies’ coefficients across that panel’s moderators. The vignette comparison omits the four app-usage moderators, which only AIM and placebo participants have.

Large Language Model-Based Role-Play Simulation

Large language model-based role-play simulation creates interactive agents that portray people, characters, or users for training, testing, and research.

See all 109 papers

Apr 5, 2026

From Plausible to Causal: Counterfactual Semantics for Policy Evaluation in Simulated Online Communities

LLM-based social simulations can generate believable community interactions, enabling

policy wind tunnels'' where governance interventions are tested before deployment. But believability is not causality. Claims like

intervention

A

reduces escalation'' require causal semantics that current simulation work typically does not specify. We propose adopting the causal counterfactual framework, distinguishing necessary causation (would the outcome have occurred without the intervention?) from sufficient causation (does the intervention reliably produce the outcome?). This distinction maps onto different stakeholder needs: moderators diagnosing incidents require evidence about necessity, while platform designers choosing policies require evidence about sufficiency. We formalize this mapping, show how simulation design can support estimation under explicit assumptions, and argue that the resulting quantities should be interpreted as simulator-conditional causal estimates whose policy relevance depends on simulator fidelity. Establishing this framework now is essential: it helps define what adequate fidelity means and moves the field from simulations that look realistic toward simulations that can support policy changes.

Agam Goyal, Yian Wang, Eshwar Chandrasekharan +1

University of Illinois Urbana-Champaign, Urbana, Illinois, USA

Counterfactual Evaluation Large Language Model-Based Social Simulation Causal Inference Actual Causality

Ask PDF

Apr 12, 2026

Thinking Fast, Thinking Wrong: Intuitiveness Modulates LLM Counterfactual Reasoning in Policy Evaluation

Large language models (LLMs) are increasingly used for causal and counterfactual reasoning, yet their reliability in real-world policy evaluation remains underexplored. We construct a benchmark of 40 empirical policy evaluation cases drawn from economics and social science, each grounded in peer-reviewed evidence and classified by intuitiveness -- whether the empirical finding aligns with (obvious), is unclear relative to (ambiguous), or contradicts (counter-intuitive) common prior expectations. We evaluate four frontier LLMs across five prompting strategies with 8,000 experimental trials and analyze the results using mixed-effects logistic regression. Our findings reveal three key results: (1) a chain-of-thought (CoT) paradox, where chain-of-thought prompting dramatically improves performance on obvious cases but this benefit is substantially attenuated on counter-intuitive ones (interaction OR = 0.278,

p < 0.001

); (2) intuitiveness as the dominant factor, with case-level variance exceeding that of model choice or prompting strategy (ICC = 0.671); and (3) a knowledge-reasoning dissociation, where citation-based familiarity is unrelated to accuracy (

p = 0.84

), suggesting models possess relevant knowledge but fail to reason with it when findings contradict intuition. We frame these results through the lens of dual-process theory (System 1 vs. System 2) and argue that current LLMs' "slow thinking" achieves only partial inhibition of intuitive priors -- producing the form of deliberative reasoning without fully delivering its substance.

Yanjie He

Independent Researcher

Counterfactual Reasoning in Language Models Causal Reasoning in Language Models CoT Reasoning

Ask PDF

Aug 28, 2026

The Illusion of $\textit{What If}$ : Evaluating the Breakdown of Counterfactual Reasoning in LLMs

Counterfactual reasoning requires models to reason beyond the observed world and explain how altered conditions propagate through downstream consequences. Existing benchmarks largely target bounded settings with fixed variables or single gold outcomes, overlooking open-domain scenarios requiring causal-process evaluation. To this end, we present

\textbf{WhatIfBench}

, a diagnostic benchmark for open-domain, open-form, long-horizon counterfactual causal reasoning, containing 220 what-if questions across STEM, HSS, and Hybrid scenarios. To evaluate free-form responses, we further propose

\textbf{PRISM}

, which first converts each natural-language explanation into a Response-Derived Semantic Causal Graph of events, states, and mechanisms. On top of this graph, PRISM then jointly applies a Process Metric assessing graph-level causal validity and a Rubric Metric assessing answer-level explanatory adequacy. Evaluating six frontier LLMs with this framework, we find that WhatIfBench remains far from saturated: even the strongest model reaches only a 64.62% final score. Further analysis reveals persistent causal gaps, premise drift, and topology fragmentation, suggesting that fluent counterfactual narratives often mask fragile causal processes. The benchmark, code, and evaluation scripts are available at WhatIfBench.

Yucheng Wang, Yuetian Du, Zhengyi Liu +8

Zhejiang University · Alibaba Group · City University of Hong Kong

Counterfactual Reasoning in Language Models Counterfactual Reasoning Counterfactual Evaluation Causal Reasoning in Language Models Faithfulness of Language Model Explanations

Ask PDF

Abstract

Large Language Model-Based Role-Play Simulation

Explore similar work

From Plausible to Causal: Counterfactual Semantics for Policy Evaluation in Simulated Online Communities

Thinking Fast, Thinking Wrong: Intuitiveness Modulates LLM Counterfactual Reasoning in Policy Evaluation

The Illusion of What If\textit{What If}What If: Evaluating the Breakdown of Counterfactual Reasoning in LLMs

The Illusion of $\textit{What If}$ : Evaluating the Breakdown of Counterfactual Reasoning in LLMs