Too Late to Prune: Generative AI, the Screen, and Team Creative Decisions

Difang Huang; Shumin Yu; Fadong Chen; Weihua Zhou

econ.GNOct 10, 2026

Too Late to Prune: Generative AI, the Screen, and Team Creative Decisions

Authors: Difang Huang, Shumin Yu, Fadong Chen, Weihua Zhou

Organizations: Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China · School of Management, Zhejiang University, Hangzhou, China

Abstract

Generative AI makes options cheap and shifts the bottleneck of creative work from generating options to screening them. We argue that AI's value depends on whether its options arrive before or after the team's screen. In our model, early options compete for a place in the portfolio; late options arrive after that competition and are kept unscreened. In a preregistered experiment, we randomly assigned 148 three-person teams to five schedules of access to the same language model during a product-design task with timed search and selection stages. Moving the same 15 minutes of access from search to selection lowered expert-rated creativity by 0.62 standard deviations, although teams used the model at similar rates. Platform logs locate the difference at the screen. Early access enlarged the pool of ideas fivefold, and teams pruned it. Late-access teams reached the screen with the same small pool as teams without AI and then searched again, devoting 29% of their selection-stage queries to generating new options, compared with 11% for teams with access throughout. Late access added breadth, not quality: these teams submitted 73% more proposals than did teams without AI. Language-model judges detected the benefit of access but not the cost of late timing.

Figures & tables

Figure 1: Experimental design. Teams worked for 40 minutes in three platform-enforced stages. Colored cells mark when the assistant was available. All conditions used the same model (DeepSeek V3.2, temperature 0.1) with the same stage-specific system prompts.

Figure 2: Rated creativity by AI-access schedule. Points are teams (

n=148

); diamonds are condition means with 95% confidence intervals. Brackets report Hedges

g

and two-sided Welch

p

values.

Figure 3: The idea funnel. (a) Distinct ideas in the members’ last personal drafts of the search stage (the pool at the gate); gray segments are the team’s own ideas and colored segments ideas derived from the assistant’s search-stage output; bars are means and lines 95% confidence intervals of the total. (b) Share of pool ideas that reappear in the final submission. (c) Share of submission sentences that match the team’s search-stage drafts, chat, or assistant output. Early: early-search access (C1); Warning: early-search access with warning (C5); Full: full-process access (C3); Late: later-selection access (C2). Points are condition means with 95% confidence intervals.

Figure 4: How teams used access and what they submitted. (a) Composition of the 2,124 assistant queries within each access window, coded blind to condition, stage, and team; numbers at right are queries per team in the window. (b) Rater-coded proposals per submission; points are teams, diamonds are means with 95% confidence intervals. (c) Proposal count and rated creativity, both residualized on condition and task fixed effects; the line is the OLS fit.

Condition	window	Teams	Participants	(S1 / S2 / S3)	team (s.d.)	$\geq 1$ query
	AI access			Assistant queries	Queries per	Teams with
Early-search access (C1)	Search (S1)	30	90	448 / 0 / 0	14.9 (5.3)	30
Later-selection access (C2)	Selection (S2)	30	90	0 / 413 / 0	13.8 (4.4)	30
Full-process access (C3)	S1–S3	30	90	438 / 319 / 144	30.0 (9.8)	30
Early-search + warning (C5)	Search (S1)	29	87	362 / 0 / 0	12.5 (5.5)	28
No-AI control (C4)	None	29	87	0 / 0 / 0	–	0

Table 1: Experimental conditions and observed assistant use. The 40-minute task comprised a 15-minute divergent-search stage (S1), a 15-minute convergent-selection stage (S2), and a 10-minute final-review stage (S3). Assistant queries are logged requests to the integrated assistant (DeepSeek V3.2, temperature 0.1), after correction of one group-name typo in the log. No query was logged outside an assigned window. In C5 every assistant response was followed by a persistent statement that AI output is only an initial reference and should be discussed with teammates. One C5 team issued no query.

	(1)	(2)	(3)	(4)	(5)
Early-search access (C1)	$0.486^{***}$	$0.486^{***}$	$0.441^{***}$	$0.486^{***}$	$0.488^{***}$
	$(0.154)$	$(0.147)$	$(0.168)$	$(0.141)$	$(0.163)$
Later-selection access (C2)	$0.130$	$0.124$	$0.073$	$0.124$	$0.007$
	$(0.165)$	$(0.165)$	$(0.195)$	$(0.159)$	$(0.182)$
Full-process access (C3)	$0.479^{***}$	$0.473^{***}$	$0.523^{***}$	$0.473^{***}$	$0.449^{***}$
	$(0.152)$	$(0.148)$	$(0.162)$	$(0.142)$	$(0.155)$

Table 2: Effects of the AI-access schedule on rated creativity. The dependent variable is the calibrated creativity composite from four blind expert raters, averaged over the proposals in a team’s submission (columns 1–3), the rater’s mean over a team’s proposals (column 4), or the rating of each proposal (column 5). Coefficients are differences from the no-AI control (C4). Columns 1–3 report HC3 heteroskedasticity-robust standard errors; columns 4–5 report standard errors clustered by team (148 clusters). Column 5 weights teams by their number of proposals and therefore estimates a proposal-weighted rather than a team-weighted contrast. Team covariates (column 3) are the ten pretreatment measures in Table C.2 , mean-centered. Randomization inference re-randomizes condition labels between the two compared arms within task strata (10,000 draws); the Westfall–Young step-down procedure re-randomizes all five labels within task strata (10,000 draws) and controls the family-wise error rate across all eight condition contrasts. All eight contrasts with Welch, RI, BH, Holm and WY

p

values are reported in Online Appendix Table E.1 .

{}^{*}p<0.10

{}^{**}p<0.05

{}^{***}p<0.01

(two-sided).

	(1)	(2)	(3)	(4)	(5)
	Originality	Appropriateness	Elaboration	Originality quality	Text quality
Early-search access (C1)	$0.431^{**}$	$0.530^{***}$	$0.695^{***}$	$0.483^{***}$	$0.502^{***}$
	$(0.172)$	$(0.121)$	$(0.157)$	$(0.147)$	$(0.130)$
Later-selection access (C2)	$0.068$	$0.188$	$0.326^{*}$	$0.123$	$0.183$
	$(0.192)$	$(0.121)$	$(0.178)$	$(0.153)$	$(0.135)$
Full-process access (C3)	$0.461^{***}$	$0.428^{***}$	$0.579^{***}$	$0.442^{***}$	$0.408^{***}$

Table 3: Effects by rated dimension. OLS with task fixed effects and HC3 standard errors; each dependent variable is the team mean of a rated dimension (1–7 scale; originality also includes the capped feature-count score and naming bonus). Originality, appropriateness and elaboration enter the composite with weights 0.70, 0.15 and 0.15; originality quality is the main component of originality; text quality is rated separately and does not enter the composite. WY

p

values adjust the C1–C2 contrast for the five dimensions by step-down max-

|t|

over 10,000 stratified re-randomizations, using the same task-fixed-effects HC3

t

statistic as the table.

{}^{*}p<0.10

{}^{**}p<0.05

{}^{***}p<0.01

(two-sided).

Panel A. Submission and process measures
	(1)	(2)	(3)	(4)	(5)	(6)
	Proposals per submission	Rated elaboration	Submission length (chars)	Characters per proposal	Stage-1 chat (1,000 chars)	Semantic dispersion
Early-search access (C1)	$0.74^{*}$	$0.70^{***}$	$220.5^{***}$	$28.6$	$1.02^{***}$	$0.033^{***}$
	$(0.43)$	$(0.16)$	$(72.7)$	$(27.9)$	$(0.23)$	$(0.008)$
Later-selection access (C2)	$1.53^{***}$	$0.33^{*}$	$310.9^{***}$	$15.2$	$-0.02$	$0.032^{***}$
	$(0.54)$	$(0.18)$	$(77.4)$	$(31.7)$	$(0.08)$	$(0.009)$

Table 5: Submission composition, team process, and the use of assistant access. Panel A: OLS with task fixed effects and HC3 standard errors (

n=148

teams); all outcomes are post-treatment. Proposals per submission is the rater-coded number of distinct proposals; characters per proposal is computed within team and then averaged; semantic dispersion is the mean pairwise TF–IDF cosine distance among paragraph-level idea units of the submission. Panel B: each of the 2,124 assistant queries was coded by a large language model applying a written codebook, blind to condition, stage and team, into one of five request types (Generate new options; Develop a specified option; Evaluate or select; Write or polish for submission; Other). Shares pool queries within the cell. A second, independent blind coding of 200 random queries agreed with the codes in 87.0% of cases (Cohen’s

\kappa=0.82

;

\kappa=0.89

for generate versus other). In the selection window, later-selection teams devoted a team-average 29.1% of queries to generating new options against 11.0% for full-process teams (difference 18.1 percentage points,

p<0.001

), and issued 4.07 versus 1.20 generation queries per team (

p<0.001

; Welch

t

-tests, 30 teams each). Because the selection window was later-selection teams’ first contact with the assistant, we also drop each team’s first three queries in the window: the shares become 21.1% and 8.9% (

p=0.011

{}^{*}p<0.10

{}^{**}p<0.05

{}^{***}p<0.01

(two-sided).

Panel A. Share of the assistant’s idea units, by access window (team means)
		Entered the	Entered near-	Shared in	Entered without
	Idea units	submission	verbatim	team chat	being shared
Early-search (C1), search window	1,850	14.6%	8.7%	27.7%	3.9%
Warning (C5), search window	1,492	13.6%	8.1%	27.6%	4.1%
Full-process (C3), search window	1,798	6.1%	3.2%	22.2%	2.5%
Later-selection (C2), selection window	1,617	23.7%	15.8%	26.3%	9.6%

Table 6: The fate of assistant output: Adoption, verbatim use, and sharing with teammates. Each of the 2,124 assistant responses is split into idea units at list items and headings (8,603 units). A unit entered the submission if one of its sentences has a TF–IDF (character 2–3-gram) cosine similarity of at least 0.35 with a sentence of the team’s final submission, near-verbatim if the similarity is at least 0.70, and shared if it reaches the same threshold with a team-chat message posted after the response arrived. Assistant responses were visible only to the member who asked, so a unit that entered the submission without being shared never reached the team’s common channel. Applied to other teams’ submissions in the same task, the threshold matches 0.4% of unit–submission pairs (Online Appendix Table G.1 ). Panel B regresses unit indicators on the window contrast with task fixed effects (and, in the second row of each pair, fixed effects for the five request types); standard errors are clustered by team. Online Appendix G reports alternative thresholds, sentence-embedding similarity, generation requests only, and timing within the window.

{}^{*}p<0.10

{}^{**}p<0.05

{}^{***}p<0.01

(two-sided).

Baseline
	Early-search vs.	Early-search vs.	Full-process vs.
Specification	later-selection (C1–C2)	no AI (C1–C4)	no AI (C3–C4)
Welch $t$ -test, calibrated score	0.356 [0.064, 0.648]	0.486 [0.182, 0.790]	0.479 [0.179, 0.779]
Specification
OLS, task FE	0.361 [0.061, 0.662]	0.486 [0.198, 0.773]	0.473 [0.184, 0.763]
OLS, task FE + team covariates	0.369 [0.040, 0.698]	0.441 [0.112, 0.771]	0.523 [0.206, 0.839]

Table 7: Robustness of the three primary contrasts. Entries are differences in creativity score points with 95% confidence intervals (team-level unless stated). † The interval includes zero. Raters are labeled A–D in order of increasing raw mean (severity), as in Online Appendix Table D.2 ; A and D are the two industry experts and B and C the two doctoral raters. Leave-one-out and subset scores average the remaining raters. Severity calibration does not change team scores, because every rater scored every submission and additive severity differences cancel in team means. Randomization inference re-randomizes condition labels between the two compared arms within tasks (10,000 draws). The crossed random-effects row is a REML linear mixed model of proposal-level ratings with task fixed effects and random intercepts for teams, proposals and raters (Online Appendix E.2 ). The proposal-level row weights each team by its number of proposals. The off-schedule row drops teams whose logged search or selection stage deviated from 15 minutes by more than 3 minutes (most often because a member joined late).

Panel A. Condition effects by evaluator (standardized team scores)
	(1)	(2)	(3)	(4)
	Human panel (4 raters)	AI panel (10 models)	AI panel excl. DeepSeek V3	AI minus human
Early-search access (C1)	$0.788^{***}$	$0.703^{***}$	$0.715^{***}$	$-0.085$
	$(0.238)$	$(0.230)$	$(0.228)$	$(0.230)$
Later-selection access (C2)	$0.202$	$0.779^{***}$	$0.814^{***}$	$0.577^{**}$
	$(0.267)$	$(0.255)$	$(0.248)$	$(0.225)$

Table 8: Human versus language-model evaluators. Panel A: OLS with task fixed effects and HC3 standard errors (

n=148

). Each evaluator’s team score is standardized across teams (mean 0, s.d. 1); AI scores are standardized within model, averaged over models, and then re-standardized. Column 4 is the difference between the AI and human

z

-scores, so it measures relative rather than absolute leniency. The proposal-count row is from a separate model that adds the rater-coded proposal count to the condition and task indicators. Panel B stacks the raw human and pooled AI team scores (296 observations) and regresses them on proposal count, an AI indicator and their interaction, with condition and task fixed effects that are allowed to differ between the two scales, and team-clustered standard errors; brackets are 95% confidence intervals. The ten models are Claude Sonnet 4.6, Claude Opus 4.6 and 4.7, GPT-5.5 and GPT-5.5 Pro, Gemini 3.1 Flash and Pro, DeepSeek V3 and V4 Pro, and Kimi K2.6. The experimental assistant ran on DeepSeek V3.2; column 3 drops DeepSeek V3, and the last row of Panel B drops both DeepSeek models.

{}^{*}p<0.10

{}^{**}p<0.05

{}^{***}p<0.01

(two-sided).

$\beta$	$\rho$	None	Early	Late	Mean	Best	Total	$K_{L}$	Early	Late
		Mean quality			Early $-$ late gap				Effort per proposal
0.0	1.0	5.439	5.691	4.726	0.965	0.284	$-$ 26.43	8.00	0.500	0.125
0.0	0.6	5.437	5.690	4.860	0.831	0.291	$-$ 15.71	5.60	0.500	0.189
0.3	1.0	5.438	5.689	4.767	0.922	0.242	$-$ 26.76	8.00	0.500	0.163
0.3	0.6	5.437	5.688	4.911	0.778	0.240	$-$ 15.98	5.60	0.500	0.245
0.6	1.0	5.438	5.689	4.801	0.888	0.208	$-$ 27.03	8.00	0.500	0.200

Table A.1: Numerical analysis of the search–selection model. Parameters:

\theta\sim\mathcal{N}(4.3,0.6^{2})

n=a=6

k=2

B=1

h(e)=0.8\sqrt{e}

; 100,000 replications per cell. In every cell early access dominates late access in mean quality, the gap in the best proposal is positive and smaller, and the gap in total quality is negative, as Proposition 4 states for

\rho=1

Remaining time $R$	Cutoff $\hat{\theta}$	$\Delta(\mu,R)$	Benefit of comparing	Cost of comparing	Decision	Retention $\rho(R)$
0.15	3.94	0.119	0.0340	0.0926	Keep without comparing	1.00
0.20	3.96	0.113	0.0357	0.0741	Keep without comparing	1.00
0.30	3.99	0.103	0.0388	0.0569	Keep without comparing	1.00
0.40	4.02	0.094	0.0415	0.0479	Keep without comparing	1.00
0.50	4.04	0.086	0.0441	0.0422	Compare	0.67
0.60	4.06	0.079	0.0464	0.0382	Compare	0.65

Table A.2: Optimal retention of a late option (Lemma 1 ). Parameters as in Table A.1 :

\theta\sim\mathcal{N}(4.3,0.6^{2})

h(e)=0.8\sqrt{e}

, a portfolio of

K=2

proposals holding the two best of six own ideas, perceived quantity credit

\gamma_{T}=0.35

, and comparison time

\tau=0.10

(in units of the selection-stage development budget,

B=1

). The benefit of comparing is

\mathbb{E}[(\hat{\theta}-\theta)^{+}]/(K+1)

; the cost is

h(R/K)-h((R-\tau)/K)

\Delta(\mu,R)\geq 0

means that an average option is worth keeping if it is not compared. Retention is 1 when the option is kept without comparison and

1-F(\hat{\theta})

when it is compared.

Condition	Backpack + pet	Refrigerator + calendar	Shoes + map	Teams	Participants
C1	11	10	9	30	90
C2	10	10	10	30	90
C3	10	10	10	30	90
C4	10	9	10	29	87
C5	10	10	9	29	87
Total	51	49	48	148	444

Table C.1: Realized allocation of teams to conditions and tasks. The allocation sequence contained ten slots for each of the 15 condition-by-task cells. Two recruited teams did not submit study data because of technical failures and were excluded before rating. Realized cell sizes range from 9 to 11; in C1 one team more than planned completed the backpack-and-pet task and one fewer the shoes-and-map task.

Covariate	C1	C2	C3	C5	C4	$p$
Age (years)	21.41	21.50	21.46	21.60	21.97	0.783
Share female	0.711	0.744	0.800	0.667	0.759	0.339
Generative-AI use frequency (1–5)	4.12	4.17	4.09	4.14	4.08	0.901
AI acceptance (7–35)	25.14	24.98	25.02	25.41	24.86	0.751
Need for cognition (18–90)	58.04	58.07	56.83	57.52	56.07	0.654
Openness (2–10)	6.88	6.94	6.61	6.91	6.57	0.349

Table C.2: Pretreatment balance across conditions (team means). Entries are means of team-level averages of the registration survey (

n=148

teams; 444 participants). Ranges in parentheses refer to the summed scale (a single item for AI-use frequency). The

p

column reports a one-way ANOVA across the five conditions. Joint tests regress the treatment indicator on all ten covariates and task fixed effects within each pair of conditions (linear probability model, HC3 covariance) and test that all covariate coefficients are zero. The omnibus test sums the ten ANOVA

F

statistics and compares the sum with 2,000 re-randomizations of condition labels within task strata.

Covariate	C1	C2	C3	C5	C4	$p$	Balance statistic
Age (years)	21.41	21.51	21.46	21.60	21.97	0.649	0.144
Generative-AI use frequency (1–5)	4.12	4.17	4.09	4.14	4.08	0.915	0.070
AI acceptance (7–35)	25.14	24.98	25.02	25.41	24.86	0.793	0.110
Need for cognition (18–90)	58.04	58.07	56.83	57.52	56.07	0.658	0.121
Openness (2–10)	6.88	6.94	6.61	6.91	6.57	0.306	0.139
Conscientiousness (2–10)	6.84	6.92	6.59	6.90	6.72	0.585	0.133

Table C.3: Participant-level pretreatment balance.

n=444

participants (age missing for two). Continuous variables: condition means, one-way ANOVA

p

, and the largest absolute standardized deviation of a condition mean from the sample mean. Categorical variables: shares,

\chi^{2}

test

p

, and Cramér’s

V

. The Big Five dimensions are two-item sums; need for cognition and AI acceptance are summed scales.

Measure	ICC(2,1)	ICC(2, $k$ )	$\alpha$	$n$	ICC(2,1)	ICC(2, $k$ )	$\alpha$	$n$
	Team level				Proposal level
Creativity composite	0.176	0.461	0.033	148	0.165	0.442	0.032	387
Originality	0.185	0.476	0.065	148	0.162	0.436	0.053	387
Appropriateness	0.070	0.232	$-$ 0.105	148	0.081	0.260	$-$ 0.077	387
Elaboration	0.183	0.473	0.044	148	0.216	0.525	0.092	387
Originality quality	0.095	0.296	$-$ 0.070	148	0.091	0.285	$-$ 0.055	387

Table D.1: Agreement among the four human raters. ICC(2,1) and ICC(2,

k

) are two-way random-effects, absolute-agreement intraclass correlations for one rater and for the mean of

k=4

raters, computed on units scored by all four raters.

\alpha

is Krippendorff’s interval alpha using all available ratings. Team-level units are the rater’s mean over a team’s proposals; proposal-level units are individual proposals. Fewer complete units are available for originality quality and text quality because some raters left these items blank for some proposals.

			Mean (s.d.)	Hedges $g$ from rater’s own scores			Correlation with rater
Rater	Type	Units	of composite	C1–C2	C1–C4	C3–C4	A	B	C	D
A	Industry expert	396	3.05 (1.24)	0.57	0.88	0.52	–	0.46	0.46	0.53
B	Doctoral researcher	387	4.19 (0.92)	0.24	0.12	0.30	0.46	–	0.55	0.35
C	Doctoral researcher	394	5.10 (0.74)	0.72	0.62	0.58	0.46	0.55	–	0.44
D	Industry expert	396	5.36 (0.97)	0.40	0.72	1.01	0.53	0.35	0.44	–

Table D.2: Rater characteristics, rater-specific effects, and inter-rater correlations. Raters are labeled in order of increasing mean (severity). Units are the proposals each rater scored (raters differed in how they segmented three submissions). Rater-specific effects are Welch contrasts on each rater’s team means. Correlations are Pearson correlations of team means across raters. All four raters rank early-search above later-selection submissions; rater B’s scores show little difference between AI and no-AI conditions.

Code	Category	Definition	Example queries (translated)
G	Generate new options	Requests new ideas, concepts, directions or alternative products, including the task prompt pasted with a request for designs, “more” or “other” ideas, and new angles.	“What innovative ideas come from combining a refrigerator and a calendar?” “Any other directions? This one is good. Give me some other directions.”
D	Develop a specified option	Asks to elaborate, detail, extend, add features to, name, or plan the implementation of an idea already specified.	“Can the service be extended? How would it work?” “Help us define the core user group and the technical challenges.”
E	Evaluate or select	Asks to compare options, judge feasibility, cost or market potential, choose the best, or identify flaws.	“Compare the pros and cons of a built-in versus an external calendar screen on the fridge.” “Compared with a real pet, what are its advantages and disadvantages?”
W	Write or polish	Asks to rewrite into the submission format, shorten or expand to a length, polish wording, or merge options into one text.	“Expand it into a plan of about 250 characters.” “Write it as one paragraph, in the plainest language everyone can understand.”
I	Information or other	Factual or background questions, clarifications, conversational or empty input.	“What are the uses and characteristics of pets?” “Why do you know we are having a team discussion?”

Table D.3: Codebook for assistant requests. Decision rules: a query that proposes its own idea and also asks for more ideas is coded G; a query that pastes an existing idea and asks to refine it is coded D; naming a given product is D; a request dominated by a length or format instruction is W. Queries were shuffled and stripped of team, condition, stage and time stamps before coding.

Condition	Stage	Generate	Develop	Evaluate	Write	Other	Total
Early-search (C1)	S1	247 (55.1%)	121 (27.0%)	20 (4.5%)	37 (8.3%)	23 (5.1%)	448
Later-selection (C2)	S2	122 (29.5%)	167 (40.4%)	35 (8.5%)	78 (18.9%)	11 (2.7%)	413
Full-process (C3)	S1	256 (58.4%)	114 (26.0%)	24 (5.5%)	17 (3.9%)	27 (6.2%)	438
Full-process (C3)	S2	36 (11.3%)	165 (51.7%)	24 (7.5%)	64 (20.1%)	30 (9.4%)	319
Full-process (C3)	S3	12 (8.3%)	43 (29.9%)	10 (6.9%)	60 (41.7%)	19 (13.2%)	144
Warning (C5)	S1	188 (51.9%)	113 (31.2%)	11 (3.0%)	27 (7.5%)	23 (6.4%)	362

Table D.4: Assistant requests by condition, stage, and type. Counts of coded queries with row percentages. No query occurred outside an assigned window; C4 had no assistant.

Contrast	(points)	$g$	Welch	RI	BH	Holm	WY
	Difference	Hedges	Unadjusted $p$		Adjusted $p$
Early-search vs. no AI (C1–C4)	0.486	0.83	$0.002$	$0.002$	$0.009$	$0.018$	$0.012$
Later-selection vs. no AI (C2–C4)	0.130	0.21	$0.425$	$0.433$	$0.486$	$1.000$	$0.669$
Full-process vs. no AI (C3–C4)	0.479	0.83	$0.002$	$0.002$	$0.009$	$0.018$	$0.012$
Warning vs. no AI (C5–C4)	0.319	0.48	$0.071$	$0.049$	$0.142$	$0.354$	$0.251$
Early-search vs. later-selection (C1–C2)	0.356	0.62	$0.018$	$0.021$	$0.048$	$0.107$	$0.086$

Table E.1: Inference for the family of eight condition contrasts. The outcome is the calibrated team creativity score (

n=148

). RI is a randomization-inference

p

value from 10,000 re-randomizations of condition labels between the two compared arms within task strata (studentized difference in means). BH controls the false-discovery rate; Holm and Westfall–Young (WY) step-down max-

|t|

(10,000 re-randomizations of all five labels within task strata) control the family-wise error rate across all eight contrasts. The family combines confirmatory and exploratory contrasts, so the FWER adjustments are conservative for the focal C1–C2 test.

Contrast	$n$ (a/b)	Mean a (s.d.)	Mean b (s.d.)	Difference [95% CI]	Hedges $g$ [95% CI]	$p$
Early-search vs. no AI (C1–C4)	30/29	4.88 (0.52)	4.39 (0.64)	0.486 [0.182, 0.790]	0.83 [0.33, 1.41]	0.002
Later-selection vs. no AI (C2–C4)	30/29	4.52 (0.61)	4.39 (0.64)	0.130 [ $-$ 0.194, 0.455]	0.21 [ $-$ 0.29, 0.73]	0.425
Full-process vs. no AI (C3–C4)	30/29	4.87 (0.50)	4.39 (0.64)	0.479 [0.179, 0.779]	0.83 [0.35, 1.37]	0.002
Warning vs. no AI (C5–C4)	29/29	4.71 (0.68)	4.39 (0.64)	0.319 [ $-$ 0.028, 0.667]	0.48 [ $-$ 0.02, 0.99]	0.071
Early-search vs. later-selection (C1–C2)	30/30	4.88 (0.52)	4.52 (0.61)	0.356 [0.064, 0.648]	0.62 [0.13, 1.20]	0.018
Warning vs. early-search (C5–C1)	29/30	4.71 (0.68)	4.88 (0.52)	$-$ 0.167 [ $-$ 0.485, 0.151]	$-$ 0.27 [ $-$ 0.84, 0.22]	0.297

Table E.2: Pairwise condition contrasts on rated creativity. Outcome: calibrated team creativity score. Differences and

p

values from two-sided Welch

t

-tests; intervals for Hedges

g

are percentile intervals from 5,000 team-level bootstrap resamples (seed 20261003).

Dimension	Contrast	Coefficient [95% CI]	Hedges $g$	$p$	BH $p$	Holm $p$
Originality	C1–C4	0.431 [0.093, 0.769]	0.62	0.012	0.023	0.124
	C2–C4	0.068 [ $-$ 0.308, 0.443]	0.10	0.724	0.724	0.839
	C3–C4	0.461 [0.128, 0.793]	0.70	0.007	0.013	0.073
	C5–C4	0.332 [ $-$ 0.036, 0.700]	0.44	0.077	0.103	0.469
Appropriateness	C1–C4	0.530 [0.292, 0.768]	1.16	$<0.001$	$<0.001$	$<0.001$
	C2–C4	0.188 [ $-$ 0.049, 0.425]	0.42	0.120	0.150	0.602

Table E.3: Dimension-level contrasts with the no-AI control and multiplicity adjustment. Coefficients from OLS of each team-level dimension score on condition indicators and task fixed effects (HC3). The family comprises the 20 tests shown (four contrasts with the control

\times

five dimensions); BH and Holm adjustments are applied across all 20.

	Mean of remaining raters (main text): $g$ ( $p$ )			Severity-adjusted mixed model: $g$ [95% CI]
Dropped rater	C1–C2	C1–C4	C3–C4	C1–C2	C1–C4	C3–C4
A	0.54 (0.039)	0.65 (0.015)	0.88 (0.001)	0.550 [0.045, 1.080]	0.633 [0.139, 1.183]	0.882 [0.410, 1.418]
B	0.68 (0.010)	0.93 ( $<0.001$ )	0.89 (0.001)	0.692 [0.218, 1.280]	0.931 [0.419, 1.546]	0.887 [0.391, 1.483]
C	0.54 (0.038)	0.82 (0.003)	0.79 (0.003)	0.544 [0.030, 1.115]	0.804 [0.304, 1.397]	0.790 [0.316, 1.369]
D	0.62 (0.017)	0.74 (0.006)	0.61 (0.022)	0.590 [0.117, 1.172]	0.706 [0.216, 1.283]	0.608 [0.134, 1.101]

Table E.4: Leave-one-rater-out estimates under two aggregation rules. Left panel: Hedges

g

and Welch

p

from contrasts on the mean of the three remaining raters’ team scores (score-point estimates and intervals are in Table 7 of the main text). Right panel: Hedges

g

with 95% interval after re-estimating rater severity on the remaining three raters’ proposal-level ratings (REML linear mixed model with crossed random intercepts for proposals, raters, teams and tasks; severities are the centered predicted rater effects), averaging the adjusted ratings to teams and bootstrapping teams 5,000 times. Under both rules, every interval for the C1–C2 contrast excludes zero.

Task $\times$ condition interaction: $F(8,133)=1.22$ , $p=0.289$
Task	C1–C2	C1–C4	C2–C4	C3–C4	C5–C4
Backpack + pet accessory	0.17 (0.24; 0.594)	0.58 (1.22; 0.009)	0.41 (0.57; 0.207)	0.65 (1.41; 0.004)	0.31 (0.51; 0.250)
Refrigerator + calendar	0.64 (1.20; 0.013)	0.20 (0.31; 0.489)	$-$ 0.44 ( $-$ 0.79; 0.101)	0.07 (0.10; 0.815)	0.27 (0.36; 0.417)
Shoes + map	0.28 (0.57; 0.216)	0.65 (1.09; 0.022)	0.37 (0.63; 0.161)	0.67 (1.21; 0.013)	0.33 (0.57; 0.200)

Table E.5: Condition contrasts within each task. Entries are score-point differences, with Hedges

g

and Welch

p

in parentheses (9–11 teams per cell). The interaction test compares OLS models with and without task-by-condition interactions. The timing contrast is positive in all three tasks; cell-level estimates are imprecise by construction.

	(1)	(2)	(3)	(4)
	Mean proposal score (primary)	Best proposal score	Total score (sum)	Number of proposals
Early-search access (C1)	$0.486^{***}$	$0.646^{***}$	$4.25^{***}$	$0.74^{*}$
	$(0.147)$	$(0.129)$	$(1.63)$	$(0.43)$
Later-selection access (C2)	$0.124$	$0.426^{***}$	$6.31^{***}$	$1.53^{***}$
	$(0.165)$	$(0.129)$	$(1.95)$	$(0.54)$
Full-process access (C3)	$0.473^{***}$	$0.541^{***}$	$2.85$	$0.39$

Table E.6: The value of the submitted portfolio under alternative aggregations. OLS with task fixed effects and HC3 standard errors. For each rater, the creativity composite of a team’s proposals is averaged (column 1), maximized (column 2) or summed (column 3); the rater-level values are then averaged over the four raters. Column 1 reproduces the primary outcome. Column 4 is the rater-coded number of proposals. Columns 2–4 are implications of the search–selection framework (Section 2) rather than separate confirmatory outcomes.

{}^{*}p<0.10

{}^{**}p<0.05

{}^{***}p<0.01

(two-sided).

Measure	C1	C2	C4	C1 $-$ C4	C1 $-$ C2	C2 $-$ C4
	Condition means			Differences ( $p$ )
Pool: ideas in any search-stage draft	23.50	6.30	6.69	$+16.81$ ( $<0.001$ )	$+17.20$ ( $<0.001$ )	$-0.39$ (0.659)
Pool: de-duplication at 0.40	19.57	3.77	3.76	$+15.81$ ( $<0.001$ )	$+15.80$ ( $<0.001$ )	$+0.01$ (0.989)
Pool: de-duplication at 0.60	19.90	3.87	4.00	$+15.90$ ( $<0.001$ )	$+16.03$ ( $<0.001$ )	$-0.13$ (0.820)
Pool: embedding de-duplication (0.85)	18.53	3.37	3.69	$+14.84$ ( $<0.001$ )	$+15.17$ ( $<0.001$ )	$-0.32$ (0.527)
Pre-gate text share: TF–IDF threshold 0.25	0.74	0.07	0.26	$+0.48$ ( $<0.001$ )	$+0.67$ ( $<0.001$ )	$-0.19$ ( $<0.001$ )

Table F.1: Robustness of the funnel measures. Welch tests. Pool measures count distinct ideas in the members’ search-stage drafts under alternative rules; pre-gate text shares are the share of submission sentences matching the team’s search-stage drafts, chat or assistant output. Placebo match rates of the TF–IDF rule against other teams’ pre-gate material in the same task: 2.3% (0.25), 0.8% (0.35, main) and 0.2% (0.50). Embedding thresholds (0.836 and 0.885) are the cosine similarities exceeded by 5% and 1% of placebo matches; the embedding model is multilingual MiniLM, which recognizes paraphrase.

Panel A. Pre-gate origin and proposal rating (creativity composite, rater mean)
Sample	Model	Estimate (s.e.)	$p$	Units (teams)
all units	condition + task FE	$+0.022$ (0.090)	0.805	353 (142)
all units	within team	$+0.027$ (0.128)	0.837	167 (42)
exact or single segmentation	condition + task FE	$+0.058$ (0.096)	0.544	292 (127)
exact or single segmentation	within team	$+0.089$ (0.136)	0.515	141 (37)
Panel B. Share of proposals of pre-gate origin, by condition (all units)

Table F.2: Proposal-level origin and within-condition associations. Each submission is segmented into the number of units the raters scored (353 units in 142 teams; 86% of teams segmented exactly by their own headers or as a single unit, the rest by length). Within multi-unit teams, unit length correlates 0.27 with the raters’ feature count and 0.37 with rated elaboration, which supports the alignment. A unit is of pre-gate origin if at least half of its sentences match the team’s pre-gate material. Panel A: OLS with team-clustered standard errors; the within-team model uses teams with units of both origins. All associations are descriptive.

Condition	Pool	Control pool	$k$	$\sigma_{\theta}$	Implied premium (perfect screen)	Observed vs. control	Ratio
C1	20	4	2	0.473	0.461	0.486	1.06
C3	13	4	2	0.473	0.356	0.479	1.35
C5	14	4	2	0.473	0.375	0.319	0.85

Table F.3: Screening premium implied by the measured pools. The implied premium is

\sigma_{\theta}[M_{k}(N)-M_{k}(N_{0})]

for standard normal order statistics, where

N

and

N_{0}

are the mean pool sizes at the gate (rounded) in the condition and the control,

k=2

ideas are retained, and

\sigma_{\theta}

is the standard deviation of the rater-mean proposal composite multiplied by the square root of its reliability (ICC(2,

k

)

=0.44

). It assumes perfect screening and equal quality of AI and own ideas; it is a check on magnitudes, not an estimate.

Similarity measure	Threshold	Placebo match rate (other teams’ submissions)	Match rate (own submission)
TF–IDF (character 2–3 grams)	0.250	1.1%	21.1%
TF–IDF (character 2–3 grams)	0.300	0.6%	18.3%
TF–IDF (character 2–3 grams) (main)	0.350	0.4%	16.4%
TF–IDF (character 2–3 grams)	0.400	0.2%	15.0%
TF–IDF (character 2–3 grams)	0.500	0.1%	12.8%
TF–IDF (character 2–3 grams)	0.700	0.0%	9.6%

Table G.1: Calibration of the tracing rule. Each assistant idea unit is compared with its own team’s final submission and with the submission of every other team that worked on the same task. The placebo match rate is the share of unit–other-team pairs in which some sentence pair reaches the threshold; it is the false-positive rate of the rule. Embedding thresholds (multilingual MiniLM, cosine similarity of normalized sentence embeddings) are set so that their placebo match rates equal those of the TF–IDF thresholds named in parentheses.

TF–IDF, threshold 0.25
	Entered submission		Near-verbatim		Entered unshared
Contrast (window)	Raw	Type FE	Raw	Type FE	Raw	Type FE
C2 (S2) $-$ C1 (S1)	$+10.7$ ***	$+6.3$ *	$+6.1$ **	$+3.4$	$+4.5$ ***	$+3.8$ **
C2 (S2) $-$ C1 (S1), generation requests	$+8.0$ **	–	$+2.2$	–	$+3.7$ **	–
C2 (S2) $-$ C3 (S2)	$+0.8$	$+1.9$	$+6.6$ **	$+8.0$ ***	$-5.1$ **	$-4.9$ *
C3 (S2) $-$ C3 (S1)	$+17.3$ ***	$+10.0$ ***	$+4.9$ ***	$+1.6$	$+10.1$ ***	$+8.8$ ***

Table G.2: Robustness of the tracing results. Entries are differences in percentage points of idea units, from unit-level linear probability models with task fixed effects and team-clustered standard errors; “Type FE” adds fixed effects for the five request types. Near-verbatim uses a TF–IDF similarity of 0.70 (for embeddings, the threshold with the same placebo match rate). Generation rows restrict to units from requests for new options, so request-type fixed effects do not apply.

{}^{*}p<0.10

{}^{**}p<0.05

{}^{***}p<0.01

Outcome	Units	Slope per minute, search window (pp)	Difference in slope, selection window (pp)	Units
Entered submission	all units	$+1.04$ ( $<0.001$ )	$+0.39$ (0.259)	7,973
Entered submission	generation requests only	$+0.48$ (0.010)	$+0.55$ (0.510)	3,551
Near-verbatim	all units	$+0.70$ ( $<0.001$ )	$+0.24$ (0.413)	7,973
Near-verbatim	generation requests only	$+0.37$ (0.010)	$-0.12$ (0.794)	3,551
Entered unshared	all units	$+0.38$ (0.002)	$+0.38$ (0.088)	7,973
Entered unshared	generation requests only	$+0.07$ (0.361)	$+0.93$ (0.175)	3,551

Table G.3: Timing within the access window. Unit-level linear probability models of the outcome on minutes elapsed in the window, a selection-window indicator and its interaction with minutes, with team fixed effects and team-clustered standard errors (

p

in parentheses). Units from the search windows of C1, C3 and C5 and the selection windows of C2 and C3. Lemma 1 implies a steeper slope in the selection window for options kept without comparison; the differences are small, all but one positive, and none is significant at the 5% level.

Panel A. Content shares (team means; Welch tests)
	C2 $-$ C1	C2 $-$ C3	C1 $-$ C4	C5 $-$ C1
AI-derived share of submission sentences	$+0.167$ (0.008)	$-0.110$ (0.009)	$+0.650$ ( $<0.001$ )	$-0.182$ (0.020)
of which never shared in chat	$+0.062$ (0.236)	$-0.201$ (0.005)	$+0.133$ (0.001)	$-0.049$ (0.260)
of which shared in chat	$+0.104$ (0.088)	$+0.091$ (0.187)	$+0.517$ ( $<0.001$ )	$-0.133$ (0.058)
Panel B. Association with standardized team scores within conditions
	Human panel	Language-model panel	AI $-$ human

Table G.4: AI-derived content, its pricing by the two panels, and typicality. Panel A: each submission sentence is matched to its most similar assistant idea unit of the same team (TF–IDF, threshold 0.35); it is AI-derived if the match reaches the threshold and never shared if that unit did not reappear in the team chat;

p

values in parentheses. Matched against the assistant output of five other teams in the same task, the same rule classifies 1.4% of submission sentences as AI-derived (against 71.7% for the team’s own output). Panel B: OLS of standardized team scores on the three variables with condition and task fixed effects (

n=148

; HC3 standard errors in parentheses); because condition is held fixed, these are descriptive associations. Panel C: mean TF–IDF similarity of a submission to the other submissions in its task, and to the pooled assistant output received by other teams in its task; defined for every team, so the contrasts preserve randomization (

p

in parentheses).

Process variable	Indicator	$n$	$a$	$p_{a}$	$b$	$p_{b}$	Total	Direct	Indirect $a\times b$ [95% CI]
Semantic dispersion	Stage-1 AI	148	0.033	$<0.001$	4.191	0.020	0.486	0.346	0.140 [0.019, 0.268]
Stage-1 chat (1,000 characters)	Stage-1 AI	148	1.022	$<0.001$	0.115	0.031	0.486	0.368	0.117 [0.021, 0.247]
Rater-coded proposal count	Stage-2 AI	148	1.526	0.005	$-$ 0.179	$<0.001$	0.124	0.398	$-$ 0.273 [ $-$ 0.462, $-$ 0.097]
Chat messages, all stages	Stage-1 AI	148	$-$ 15.9	0.109	0.0009	0.468	0.486	0.501	$-$ 0.015 [ $-$ 0.072, 0.028]
Chat messages, all stages	Stage-2 AI	148	$-$ 13.3	0.242	0.0009	0.468	0.124	0.137	$-$ 0.012 [ $-$ 0.066, 0.030]
Chat characters, all stages (1,000)	Stage-1 AI	148	2.037	$<0.001$	0.030	0.360	0.486	0.425	0.060 [ $-$ 0.041, 0.186]

Table H.1: Exploratory decomposition of condition effects through process variables. The

a

path is the task-adjusted effect of the access indicator on the process variable (in the variable’s units); the

b

path is the adjusted association between the process variable and creativity; total and direct effects are the indicator’s coefficients without and with the process variable; indirect effects use percentile intervals from 5,000 team bootstraps. Stage-1 and Stage-2 indicators follow the saturated coding (C1, C3, C5 and C2, C3, respectively). All process variables are post-treatment, so these are descriptive decompositions rather than mediation estimates. Rows below the line show that total interaction volume carries no indirect association.

Panel A. Share of submission sentences whose nearest source is an assistant output
Condition	Embedding	TF–IDF
Early-search (C1)	0.130 [0.073, 0.187]	0.116 [0.053, 0.179]
Later-selection (C2)	0.258 [0.176, 0.340]	0.260 [0.171, 0.350]
Full-process (C3)	0.391 [0.296, 0.487]	0.388 [0.265, 0.511]
Warning (C5)	0.125 [0.069, 0.180]	0.060 [0.027, 0.093]
Panel B. Condition differences in semantic proximity among assistant users ( $n=118$ ; reference C1)

Table H.2: Engagement with assistant outputs. Semantic proximity is the similarity between a team’s final submission and the assistant outputs it received (sentence embeddings, cosine similarity; robustness with TF–IDF vectors). Embedding- and TF–IDF-based team measures correlate at

r=0.91

–

0.92

. All models use HC3 standard errors. Proximity is undefined for the no-AI control, so Panels B and C are observational comparisons among assistant users and do not preserve randomization; one assigned C5 team that never used the assistant is excluded.

Model	Family	Mean	s.d.	Share $\geq 6$	Pearson $r$	Spearman $\rho$	Mean diff.
Claude Sonnet 4.6	Claude	4.53	0.80	0.010	0.483	0.427	0.11
Claude Opus 4.6	Claude	4.72	0.76	0.028	0.395	0.325	0.30
Claude Opus 4.7	Claude	5.42	0.75	0.208	0.516	0.430	0.99
GPT-5.5	GPT	5.17	0.78	0.117	0.565	0.514	0.74
GPT-5.5 Pro	GPT	5.16	0.81	0.137	0.548	0.499	0.74
Gemini 3.1 Flash	Gemini	4.86	0.81	0.048	0.562	0.503	0.44

Table I.1: The language-model evaluator panel. Each model scored the 394 proposal units of the locked template once at temperature 0.1, using the human rubric, the full submission text, the expected number of proposals, and a JSON response schema. Mean and s.d. refer to the 1–7 composite; share

\geq 6

is the fraction of proposals scored 6 or higher. Agreement statistics compare each model with the mean human composite for the same proposal; mean difference is AI minus human. Because ICC(2,

k

) rises with the number of raters, the panel’s higher reliability is not evidence of validity.

Credit $\gamma$ per proposal	C1 $-$ C2 [95% CI]	$p$	Hedges $g$	C2 $-$ C4	$p$
0.0	0.356 [0.064, 0.648]	0.018	0.62	0.130	0.425
0.1	0.279 [0.047, 0.512]	0.019	0.61	0.283	0.043
0.2	0.203 [ $-$ 0.022, 0.428]	0.076	0.46	0.436	0.001
0.3	0.126 [ $-$ 0.147, 0.399]	0.359	0.24	0.589	$<0.001$
0.4	0.049 [ $-$ 0.306, 0.405]	0.782	0.07	0.742	$<0.001$
0.5	$-$ 0.027 [ $-$ 0.481, 0.427]	0.905	$-$ 0.03	0.895	$<0.001$

Table I.2: Evaluators that credit breadth. Upper panel: the human team score plus

\gamma

points per rater-coded proposal (Corollary 1 ). The timing gap vanishes at

\gamma^{*}=0.46

, the C1–C2 difference in the human score divided by the C2–C1 difference in proposals. Lower panel: originality

=0.6\times

quality

+\,0.4\times

a score for the number of proposals (1

=2

, 2

=4

, 3

=6

\geq 4=7

), and creativity is the unweighted mean of originality, appropriateness and elaboration, computed per rater and averaged. Welch tests.

Panel A. AI-minus-human gap (standardized) regressed on condition and submission features
Specification	C1 $-$ C2	C2	C3	Added variables	Adj. $R^{2}$
Conditions only	$-0.662$ ***	$+0.577$ **	$+0.642$ ***	–	0.107
+ proposal count	$-0.547$ **	$+0.354$	$+0.585$ ***	proposals $+0.146$ ***	0.207
+ AI-derived sentence share	$-0.591$ ***	$+0.434$ *	$+0.426$ *	AI share $+0.553$	0.115
+ both	$-0.527$ **	$+0.312$	$+0.514$ **	proposals $+0.142$ ***; AI share $+0.187$	0.203
+ both + rated text quality	$-0.455$ **	$+0.431$ *	$+0.629$ **	proposals $+0.086$ ; AI share $+0.283$ ; text quality $-0.322$ *	0.228

Table I.3: Sources of the divergence between the human and language-model panels. Panel A: OLS with task fixed effects and HC3 standard errors (

n=148

); the dependent variable is the difference between the standardized model-panel and human-panel team scores (Table 8 , column 4 of the main text). Proposal count is rater-coded; AI share is the share of submission sentences whose nearest source is an assistant output (sentence embeddings; zero without access); rated text quality is the human team mean, centered. Panel B: Welch contrasts of standardized team scores (

p

in parentheses); model-panel dimension scores are standardized within model, averaged over models, and re-standardized; the last column is the correlation of the two panels’ team scores across all 148 teams.

{}^{*}p<0.10

{}^{**}p<0.05

{}^{***}p<0.01

Team-level moderator	Stage-1 AI $\times$ moderator	Stage-2 AI $\times$ moderator	Warning $\times$ moderator
AI-use frequency, mean	$-$ 0.021 (0.844)	0.332 (0.002)	0.281 (0.105)
Conscientiousness, s.d.	0.031 (0.795)	$-$ 0.141 (0.218)	$-$ 0.306 (0.013)
Conscientiousness, range	0.032 (0.784)	$-$ 0.141 (0.208)	$-$ 0.311 (0.016)
AI optimism, mean	$-$ 0.052 (0.576)	0.205 (0.026)	0.197 (0.246)
Prefer AI interaction, mean	$-$ 0.062 (0.599)	$-$ 0.241 (0.029)	$-$ 0.220 (0.290)
Emotional stability, mean	$-$ 0.250 (0.034)	0.017 (0.883)	0.144 (0.291)

Table J.1: Exploratory moderator screen (eight smallest

p

values). Each row is a separate model that adds the moderator and its interactions with the Stage-1, Stage-2 and warning indicators to the saturated condition model with task fixed effects (HC3); the moderator is standardized, and cells give

b

(

p

). The screen covered all team means, standard deviations and ranges of the registration-survey scales. No adjustment for multiplicity is applied, and none of these interactions would survive one; we report them for transparency and do not interpret them.

Human-AI Decision Making

Human-AI decision making studies how people and AI systems share, support, or delegate decisions across tasks and institutions.

See all 173 papers

Offline-to-Online Creative Optimization with Generative Models and Adaptive Testing
Jul 26, 2026Kevin Lee, Benjamin Letham, Zhiyuan Jerry Lin +5Online Advertising Generative Modeling
University of Michigan · USA · Meta
Human diversity fuels collective creativity that large language models cannot simulate or sustain
Jul 29, 2026Mengchen Dong, Hiromu YakuraComputational Creativity Diverse Text Generation
Center for Humans and Machines, Max Planck Institute for Human Development, Berlin, Germany
Thinking Outside the Bots: Human Ideas in AI's Semantic Blind Spots Are Not Inferior
Oct 12, 2026Kibum Moon, Dan R. Johnson, Kostadin Kushlev +6Computational Creativity Generative AI Evaluation
Department of Psychology, Georgetown University · Department of Cognitive and Behavioral Science, Washington and Lee University · The Wharton School, University of Pennsylvania +4

Jul 26, 2026J/K move · Enter open · S save

Offline-to-Online Creative Optimization with Generative Models and Adaptive Testing

Kevin Lee, Benjamin Letham, Zhiyuan Jerry Lin, Elodie Samson+4

University of Michigan · USA · Meta

Online Advertising Generative Modeling A/b Testing

Open paper AskPDF arXiv

Ad creative optimization is increasingly constrained by evaluation rather than generation. Generative models can produce many plausible creatives, but reliable evaluation requires online experiments, in which only a limited slate can be tested. We study how to use data from historical A/B tests to generate and select the candidates in that slate. We developed and deployed a performance-driven offline-to-online workflow that guides creative generation with a predictive model as an inference-time critic. In the offline phase, we use a predictive model trained on historical experiments to rank and refine variants created by a generative model. A final test slate is then deployed in an online adaptive experiment. In a 50-arm field experiment, we found that the best creative generated with this method yielded 45.1% higher engagement than the best human-authored creative. Two additional experiments showed the same upper-tail pattern, with lifts of 46.7% and 36.2%. We found that despite the predictive model being too noisy to directly identify the best creative offline, it effectively guides the generative model toward creating strong candidates that can be efficiently evaluated in an adaptive experiment. The results suggest a design principle for creative optimization with generative models: use predictive models to guide generation of a slate to test, judge the slate by whether it contains high-performing candidates at a feasible test size, and use adaptive experiments to select among candidates while limiting traffic lost to weak arms.

Abstract

Human-AI Decision Making

Explore similar work

Offline-to-Online Creative Optimization with Generative Models and Adaptive Testing

Human diversity fuels collective creativity that large language models cannot simulate or sustain

Thinking Outside the Bots: Human Ideas in AI's Semantic Blind Spots Are Not Inferior