Organizations: Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China · School of Management, Zhejiang University, Hangzhou, China
Generative AI makes options cheap and shifts the bottleneck of creative work from generating options to screening them. We argue that AI's value depends on whether its options arrive before or after the team's screen. In our model, early options compete for a place in the portfolio; late options arrive after that competition and are kept unscreened. In a preregistered experiment, we randomly assigned 148 three-person teams to five schedules of access to the same language model during a product-design task with timed search and selection stages. Moving the same 15 minutes of access from search to selection lowered expert-rated creativity by 0.62 standard deviations, although teams used the model at similar rates. Platform logs locate the difference at the screen. Early access enlarged the pool of ideas fivefold, and teams pruned it. Late-access teams reached the screen with the same small pool as teams without AI and then searched again, devoting 29% of their selection-stage queries to generating new options, compared with 11% for teams with access throughout. Late access added breadth, not quality: these teams submitted 73% more proposals than did teams without AI. Language-model judges detected the benefit of access but not the cost of late timing.
Figures & tables
Figure 1: Experimental design. Teams worked for 40 minutes in three platform-enforced stages. Colored cells mark when the assistant was available. All conditions used the same model (DeepSeek V3.2, temperature 0.1) with the same stage-specific system prompts.
Figure 2: Rated creativity by AI-access schedule. Points are teams ( n=148 ); diamonds are condition means with 95% confidence intervals. Brackets report Hedges g and two-sided Welch p values.
Figure 3: The idea funnel. (a) Distinct ideas in the members’ last personal drafts of the search stage (the pool at the gate); gray segments are the team’s own ideas and colored segments ideas derived from the assistant’s search-stage output; bars are means and lines 95% confidence intervals of the total. (b) Share of pool ideas that reappear in the final submission. (c) Share of submission sentences that match the team’s search-stage drafts, chat, or assistant output. Early: early-search access (C1); Warning: early-search access with warning (C5); Full: full-process access (C3); Late: later-selection access (C2). Points are condition means with 95% confidence intervals.
Figure 4: How teams used access and what they submitted. (a) Composition of the 2,124 assistant queries within each access window, coded blind to condition, stage, and team; numbers at right are queries per team in the window. (b) Rater-coded proposals per submission; points are teams, diamonds are means with 95% confidence intervals. (c) Proposal count and rated creativity, both residualized on condition and task fixed effects; the line is the OLS fit.
AI access
Assistant queries
Queries per
Teams with
Condition
window
Teams
Participants
(S1 / S2 / S3)
team (s.d.)
≥1 query
Early-search access (C1)
Search (S1)
30
90
448 / 0 / 0
14.9 (5.3)
30
Later-selection access (C2)
Selection (S2)
30
90
0 / 413 / 0
13.8 (4.4)
30
Full-process access (C3)
S1–S3
30
90
438 / 319 / 144
30.0 (9.8)
30
Early-search + warning (C5)
Search (S1)
29
87
362 / 0 / 0
12.5 (5.5)
28
No-AI control (C4)
None
29
87
0 / 0 / 0
–
0
Table 1: Experimental conditions and observed assistant use. The 40-minute task comprised a 15-minute divergent-search stage (S1), a 15-minute convergent-selection stage (S2), and a 10-minute final-review stage (S3). Assistant queries are logged requests to the integrated assistant (DeepSeek V3.2, temperature 0.1), after correction of one group-name typo in the log. No query was logged outside an assigned window. In C5 every assistant response was followed by a persistent statement that AI output is only an initial reference and should be discussed with teammates. One C5 team issued no query.
(1)
(2)
(3)
(4)
(5)
Early-search access (C1)
0.486∗∗∗
0.486∗∗∗
0.441∗∗∗
0.486∗∗∗
0.488∗∗∗
(0.154)
(0.147)
(0.168)
(0.141)
(0.163)
Later-selection access (C2)
0.130
0.124
0.073
0.124
0.007
(0.165)
(0.165)
(0.195)
(0.159)
(0.182)
Full-process access (C3)
0.479∗∗∗
0.473∗∗∗
0.523∗∗∗
0.473∗∗∗
0.449∗∗∗
(0.152)
(0.148)
(0.162)
(0.142)
(0.155)
Table 2: Effects of the AI-access schedule on rated creativity. The dependent variable is the calibrated creativity composite from four blind expert raters, averaged over the proposals in a team’s submission (columns 1–3), the rater’s mean over a team’s proposals (column 4), or the rating of each proposal (column 5). Coefficients are differences from the no-AI control (C4). Columns 1–3 report HC3 heteroskedasticity-robust standard errors; columns 4–5 report standard errors clustered by team (148 clusters). Column 5 weights teams by their number of proposals and therefore estimates a proposal-weighted rather than a team-weighted contrast. Team covariates (column 3) are the ten pretreatment measures in Table C.2 , mean-centered. Randomization inference re-randomizes condition labels between the two compared arms within task strata (10,000 draws); the Westfall–Young step-down procedure re-randomizes all five labels within task strata (10,000 draws) and controls the family-wise error rate across all eight condition contrasts. All eight contrasts with Welch, RI, BH, Holm and WY p values are reported in Online Appendix Table E.1 . ∗p<0.10 , ∗∗p<0.05 , ∗∗∗p<0.01 (two-sided).
(1)
(2)
(3)
(4)
(5)
Originality
Appropriateness
Elaboration
Originality quality
Text quality
Early-search access (C1)
0.431∗∗
0.530∗∗∗
0.695∗∗∗
0.483∗∗∗
0.502∗∗∗
(0.172)
(0.121)
(0.157)
(0.147)
(0.130)
Later-selection access (C2)
0.068
0.188
0.326∗
0.123
0.183
(0.192)
(0.121)
(0.178)
(0.153)
(0.135)
Full-process access (C3)
0.461∗∗∗
0.428∗∗∗
0.579∗∗∗
0.442∗∗∗
0.408∗∗∗
Table 3: Effects by rated dimension. OLS with task fixed effects and HC3 standard errors; each dependent variable is the team mean of a rated dimension (1–7 scale; originality also includes the capped feature-count score and naming bonus). Originality, appropriateness and elaboration enter the composite with weights 0.70, 0.15 and 0.15; originality quality is the main component of originality; text quality is rated separately and does not enter the composite. WY p values adjust the C1–C2 contrast for the five dimensions by step-down max- ∣t∣ over 10,000 stratified re-randomizations, using the same task-fixed-effects HC3 t statistic as the table. ∗p<0.10 , ∗∗p<0.05 , ∗∗∗p<0.01 (two-sided).
(1)
(2)
(3)
(4)
(5)
(6)
Proposals per submission
Rated elaboration
Submission length (chars)
Characters per proposal
Stage-1 chat (1,000 chars)
Semantic dispersion
Panel A. Submission and process measures
Early-search access (C1)
0.74∗
0.70∗∗∗
220.5∗∗∗
28.6
1.02∗∗∗
0.033∗∗∗
(0.43)
(0.16)
(72.7)
(27.9)
(0.23)
(0.008)
Later-selection access (C2)
1.53∗∗∗
0.33∗
310.9∗∗∗
15.2
−0.02
0.032∗∗∗
(0.54)
(0.18)
(77.4)
(31.7)
(0.08)
(0.009)
Table 5: Submission composition, team process, and the use of assistant access. Panel A: OLS with task fixed effects and HC3 standard errors ( n=148 teams); all outcomes are post-treatment. Proposals per submission is the rater-coded number of distinct proposals; characters per proposal is computed within team and then averaged; semantic dispersion is the mean pairwise TF–IDF cosine distance among paragraph-level idea units of the submission. Panel B: each of the 2,124 assistant queries was coded by a large language model applying a written codebook, blind to condition, stage and team, into one of five request types (Generate new options; Develop a specified option; Evaluate or select; Write or polish for submission; Other). Shares pool queries within the cell. A second, independent blind coding of 200 random queries agreed with the codes in 87.0% of cases (Cohen’s κ=0.82 ; κ=0.89 for generate versus other). In the selection window, later-selection teams devoted a team-average 29.1% of queries to generating new options against 11.0% for full-process teams (difference 18.1 percentage points, p<0.001 ), and issued 4.07 versus 1.20 generation queries per team ( p<0.001 ; Welch t -tests, 30 teams each). Because the selection window was later-selection teams’ first contact with the assistant, we also drop each team’s first three queries in the window: the shares become 21.1% and 8.9% ( p=0.011 ). ∗p<0.10 , ∗∗p<0.05 , ∗∗∗p<0.01 (two-sided).
Entered the
Entered near-
Shared in
Entered without
Idea units
submission
verbatim
team chat
being shared
Panel A. Share of the assistant’s idea units, by access window (team means)
Early-search (C1), search window
1,850
14.6%
8.7%
27.7%
3.9%
Warning (C5), search window
1,492
13.6%
8.1%
27.6%
4.1%
Full-process (C3), search window
1,798
6.1%
3.2%
22.2%
2.5%
Later-selection (C2), selection window
1,617
23.7%
15.8%
26.3%
9.6%
Table 6: The fate of assistant output: Adoption, verbatim use, and sharing with teammates. Each of the 2,124 assistant responses is split into idea units at list items and headings (8,603 units). A unit entered the submission if one of its sentences has a TF–IDF (character 2–3-gram) cosine similarity of at least 0.35 with a sentence of the team’s final submission, near-verbatim if the similarity is at least 0.70, and shared if it reaches the same threshold with a team-chat message posted after the response arrived. Assistant responses were visible only to the member who asked, so a unit that entered the submission without being shared never reached the team’s common channel. Applied to other teams’ submissions in the same task, the threshold matches 0.4% of unit–submission pairs (Online Appendix Table G.1 ). Panel B regresses unit indicators on the window contrast with task fixed effects (and, in the second row of each pair, fixed effects for the five request types); standard errors are clustered by team. Online Appendix G reports alternative thresholds, sentence-embedding similarity, generation requests only, and timing within the window. ∗p<0.10 , ∗∗p<0.05 , ∗∗∗p<0.01 (two-sided).
Early-search vs.
Early-search vs.
Full-process vs.
Specification
later-selection (C1–C2)
no AI (C1–C4)
no AI (C3–C4)
Baseline
Welch t -test, calibrated score
0.356 [0.064, 0.648]
0.486 [0.182, 0.790]
0.479 [0.179, 0.779]
Specification
OLS, task FE
0.361 [0.061, 0.662]
0.486 [0.198, 0.773]
0.473 [0.184, 0.763]
OLS, task FE + team covariates
0.369 [0.040, 0.698]
0.441 [0.112, 0.771]
0.523 [0.206, 0.839]
Table 7: Robustness of the three primary contrasts. Entries are differences in creativity score points with 95% confidence intervals (team-level unless stated). † The interval includes zero. Raters are labeled A–D in order of increasing raw mean (severity), as in Online Appendix Table D.2 ; A and D are the two industry experts and B and C the two doctoral raters. Leave-one-out and subset scores average the remaining raters. Severity calibration does not change team scores, because every rater scored every submission and additive severity differences cancel in team means. Randomization inference re-randomizes condition labels between the two compared arms within tasks (10,000 draws). The crossed random-effects row is a REML linear mixed model of proposal-level ratings with task fixed effects and random intercepts for teams, proposals and raters (Online Appendix E.2 ). The proposal-level row weights each team by its number of proposals. The off-schedule row drops teams whose logged search or selection stage deviated from 15 minutes by more than 3 minutes (most often because a member joined late).
(1)
(2)
(3)
(4)
Human panel (4 raters)
AI panel (10 models)
AI panel excl. DeepSeek V3
AI minus human
Panel A. Condition effects by evaluator (standardized team scores)
Early-search access (C1)
0.788∗∗∗
0.703∗∗∗
0.715∗∗∗
−0.085
(0.238)
(0.230)
(0.228)
(0.230)
Later-selection access (C2)
0.202
0.779∗∗∗
0.814∗∗∗
0.577∗∗
(0.267)
(0.255)
(0.248)
(0.225)
Table 8: Human versus language-model evaluators. Panel A: OLS with task fixed effects and HC3 standard errors ( n=148 ). Each evaluator’s team score is standardized across teams (mean 0, s.d. 1); AI scores are standardized within model, averaged over models, and then re-standardized. Column 4 is the difference between the AI and human z -scores, so it measures relative rather than absolute leniency. The proposal-count row is from a separate model that adds the rater-coded proposal count to the condition and task indicators. Panel B stacks the raw human and pooled AI team scores (296 observations) and regresses them on proposal count, an AI indicator and their interaction, with condition and task fixed effects that are allowed to differ between the two scales, and team-clustered standard errors; brackets are 95% confidence intervals. The ten models are Claude Sonnet 4.6, Claude Opus 4.6 and 4.7, GPT-5.5 and GPT-5.5 Pro, Gemini 3.1 Flash and Pro, DeepSeek V3 and V4 Pro, and Kimi K2.6. The experimental assistant ran on DeepSeek V3.2; column 3 drops DeepSeek V3, and the last row of Panel B drops both DeepSeek models. ∗p<0.10 , ∗∗p<0.05 , ∗∗∗p<0.01 (two-sided).
Mean quality
Early − late gap
Effort per proposal
β
ρ
None
Early
Late
Mean
Best
Total
KL
Early
Late
0.0
1.0
5.439
5.691
4.726
0.965
0.284
− 26.43
8.00
0.500
0.125
0.0
0.6
5.437
5.690
4.860
0.831
0.291
− 15.71
5.60
0.500
0.189
0.3
1.0
5.438
5.689
4.767
0.922
0.242
− 26.76
8.00
0.500
0.163
0.3
0.6
5.437
5.688
4.911
0.778
0.240
− 15.98
5.60
0.500
0.245
0.6
1.0
5.438
5.689
4.801
0.888
0.208
− 27.03
8.00
0.500
0.200
Table A.1: Numerical analysis of the search–selection model. Parameters: θ∼N(4.3,0.62) , n=a=6 , k=2 , B=1 , h(e)=0.8e ; 100,000 replications per cell. In every cell early access dominates late access in mean quality, the gap in the best proposal is positive and smaller, and the gap in total quality is negative, as Proposition 4 states for ρ=1 .
Remaining time R
Cutoff θ^
Δ(μ,R)
Benefit of comparing
Cost of comparing
Decision
Retention ρ(R)
0.15
3.94
0.119
0.0340
0.0926
Keep without comparing
1.00
0.20
3.96
0.113
0.0357
0.0741
Keep without comparing
1.00
0.30
3.99
0.103
0.0388
0.0569
Keep without comparing
1.00
0.40
4.02
0.094
0.0415
0.0479
Keep without comparing
1.00
0.50
4.04
0.086
0.0441
0.0422
Compare
0.67
0.60
4.06
0.079
0.0464
0.0382
Compare
0.65
Table A.2: Optimal retention of a late option (Lemma 1 ). Parameters as in Table A.1 : θ∼N(4.3,0.62) , h(e)=0.8e , a portfolio of K=2 proposals holding the two best of six own ideas, perceived quantity credit γT=0.35 , and comparison time τ=0.10 (in units of the selection-stage development budget, B=1 ). The benefit of comparing is E[(θ^−θ)+]/(K+1) ; the cost is h(R/K)−h((R−τ)/K) . Δ(μ,R)≥0 means that an average option is worth keeping if it is not compared. Retention is 1 when the option is kept without comparison and 1−F(θ^) when it is compared.
Condition
Backpack + pet
Refrigerator + calendar
Shoes + map
Teams
Participants
C1
11
10
9
30
90
C2
10
10
10
30
90
C3
10
10
10
30
90
C4
10
9
10
29
87
C5
10
10
9
29
87
Total
51
49
48
148
444
Table C.1: Realized allocation of teams to conditions and tasks. The allocation sequence contained ten slots for each of the 15 condition-by-task cells. Two recruited teams did not submit study data because of technical failures and were excluded before rating. Realized cell sizes range from 9 to 11; in C1 one team more than planned completed the backpack-and-pet task and one fewer the shoes-and-map task.
Covariate
C1
C2
C3
C5
C4
p
Age (years)
21.41
21.50
21.46
21.60
21.97
0.783
Share female
0.711
0.744
0.800
0.667
0.759
0.339
Generative-AI use frequency (1–5)
4.12
4.17
4.09
4.14
4.08
0.901
AI acceptance (7–35)
25.14
24.98
25.02
25.41
24.86
0.751
Need for cognition (18–90)
58.04
58.07
56.83
57.52
56.07
0.654
Openness (2–10)
6.88
6.94
6.61
6.91
6.57
0.349
Table C.2: Pretreatment balance across conditions (team means). Entries are means of team-level averages of the registration survey ( n=148 teams; 444 participants). Ranges in parentheses refer to the summed scale (a single item for AI-use frequency). The p column reports a one-way ANOVA across the five conditions. Joint tests regress the treatment indicator on all ten covariates and task fixed effects within each pair of conditions (linear probability model, HC3 covariance) and test that all covariate coefficients are zero. The omnibus test sums the ten ANOVA F statistics and compares the sum with 2,000 re-randomizations of condition labels within task strata.
Covariate
C1
C2
C3
C5
C4
p
Balance statistic
Age (years)
21.41
21.51
21.46
21.60
21.97
0.649
0.144
Generative-AI use frequency (1–5)
4.12
4.17
4.09
4.14
4.08
0.915
0.070
AI acceptance (7–35)
25.14
24.98
25.02
25.41
24.86
0.793
0.110
Need for cognition (18–90)
58.04
58.07
56.83
57.52
56.07
0.658
0.121
Openness (2–10)
6.88
6.94
6.61
6.91
6.57
0.306
0.139
Conscientiousness (2–10)
6.84
6.92
6.59
6.90
6.72
0.585
0.133
Table C.3: Participant-level pretreatment balance. n=444 participants (age missing for two). Continuous variables: condition means, one-way ANOVA p , and the largest absolute standardized deviation of a condition mean from the sample mean. Categorical variables: shares, χ2 test p , and Cramér’s V . The Big Five dimensions are two-item sums; need for cognition and AI acceptance are summed scales.
Team level
Proposal level
Measure
ICC(2,1)
ICC(2, k )
α
n
ICC(2,1)
ICC(2, k )
α
n
Creativity composite
0.176
0.461
0.033
148
0.165
0.442
0.032
387
Originality
0.185
0.476
0.065
148
0.162
0.436
0.053
387
Appropriateness
0.070
0.232
− 0.105
148
0.081
0.260
− 0.077
387
Elaboration
0.183
0.473
0.044
148
0.216
0.525
0.092
387
Originality quality
0.095
0.296
− 0.070
148
0.091
0.285
− 0.055
387
Table D.1: Agreement among the four human raters. ICC(2,1) and ICC(2, k ) are two-way random-effects, absolute-agreement intraclass correlations for one rater and for the mean of k=4 raters, computed on units scored by all four raters. α is Krippendorff’s interval alpha using all available ratings. Team-level units are the rater’s mean over a team’s proposals; proposal-level units are individual proposals. Fewer complete units are available for originality quality and text quality because some raters left these items blank for some proposals.
Mean (s.d.)
Hedges g from rater’s own scores
Correlation with rater
Rater
Type
Units
of composite
C1–C2
C1–C4
C3–C4
A
B
C
D
A
Industry expert
396
3.05 (1.24)
0.57
0.88
0.52
–
0.46
0.46
0.53
B
Doctoral researcher
387
4.19 (0.92)
0.24
0.12
0.30
0.46
–
0.55
0.35
C
Doctoral researcher
394
5.10 (0.74)
0.72
0.62
0.58
0.46
0.55
–
0.44
D
Industry expert
396
5.36 (0.97)
0.40
0.72
1.01
0.53
0.35
0.44
–
Table D.2: Rater characteristics, rater-specific effects, and inter-rater correlations. Raters are labeled in order of increasing mean (severity). Units are the proposals each rater scored (raters differed in how they segmented three submissions). Rater-specific effects are Welch contrasts on each rater’s team means. Correlations are Pearson correlations of team means across raters. All four raters rank early-search above later-selection submissions; rater B’s scores show little difference between AI and no-AI conditions.
Code
Category
Definition
Example queries (translated)
G
Generate new options
Requests new ideas, concepts, directions or alternative products, including the task prompt pasted with a request for designs, “more” or “other” ideas, and new angles.
“What innovative ideas come from combining a refrigerator and a calendar?” “Any other directions? This one is good. Give me some other directions.”
D
Develop a specified option
Asks to elaborate, detail, extend, add features to, name, or plan the implementation of an idea already specified.
“Can the service be extended? How would it work?” “Help us define the core user group and the technical challenges.”
E
Evaluate or select
Asks to compare options, judge feasibility, cost or market potential, choose the best, or identify flaws.
“Compare the pros and cons of a built-in versus an external calendar screen on the fridge.” “Compared with a real pet, what are its advantages and disadvantages?”
W
Write or polish
Asks to rewrite into the submission format, shorten or expand to a length, polish wording, or merge options into one text.
“Expand it into a plan of about 250 characters.” “Write it as one paragraph, in the plainest language everyone can understand.”
I
Information or other
Factual or background questions, clarifications, conversational or empty input.
“What are the uses and characteristics of pets?” “Why do you know we are having a team discussion?”
Table D.3: Codebook for assistant requests. Decision rules: a query that proposes its own idea and also asks for more ideas is coded G; a query that pastes an existing idea and asks to refine it is coded D; naming a given product is D; a request dominated by a length or format instruction is W. Queries were shuffled and stripped of team, condition, stage and time stamps before coding.
Condition
Stage
Generate
Develop
Evaluate
Write
Other
Total
Early-search (C1)
S1
247 (55.1%)
121 (27.0%)
20 (4.5%)
37 (8.3%)
23 (5.1%)
448
Later-selection (C2)
S2
122 (29.5%)
167 (40.4%)
35 (8.5%)
78 (18.9%)
11 (2.7%)
413
Full-process (C3)
S1
256 (58.4%)
114 (26.0%)
24 (5.5%)
17 (3.9%)
27 (6.2%)
438
Full-process (C3)
S2
36 (11.3%)
165 (51.7%)
24 (7.5%)
64 (20.1%)
30 (9.4%)
319
Full-process (C3)
S3
12 (8.3%)
43 (29.9%)
10 (6.9%)
60 (41.7%)
19 (13.2%)
144
Warning (C5)
S1
188 (51.9%)
113 (31.2%)
11 (3.0%)
27 (7.5%)
23 (6.4%)
362
Table D.4: Assistant requests by condition, stage, and type. Counts of coded queries with row percentages. No query occurred outside an assigned window; C4 had no assistant.
Difference
Hedges
Unadjusted p
Adjusted p
Contrast
(points)
g
Welch
RI
BH
Holm
WY
Early-search vs. no AI (C1–C4)
0.486
0.83
0.002
0.002
0.009
0.018
0.012
Later-selection vs. no AI (C2–C4)
0.130
0.21
0.425
0.433
0.486
1.000
0.669
Full-process vs. no AI (C3–C4)
0.479
0.83
0.002
0.002
0.009
0.018
0.012
Warning vs. no AI (C5–C4)
0.319
0.48
0.071
0.049
0.142
0.354
0.251
Early-search vs. later-selection (C1–C2)
0.356
0.62
0.018
0.021
0.048
0.107
0.086
Table E.1: Inference for the family of eight condition contrasts. The outcome is the calibrated team creativity score ( n=148 ). RI is a randomization-inference p value from 10,000 re-randomizations of condition labels between the two compared arms within task strata (studentized difference in means). BH controls the false-discovery rate; Holm and Westfall–Young (WY) step-down max- ∣t∣ (10,000 re-randomizations of all five labels within task strata) control the family-wise error rate across all eight contrasts. The family combines confirmatory and exploratory contrasts, so the FWER adjustments are conservative for the focal C1–C2 test.
Contrast
n (a/b)
Mean a (s.d.)
Mean b (s.d.)
Difference [95% CI]
Hedges g [95% CI]
p
Early-search vs. no AI (C1–C4)
30/29
4.88 (0.52)
4.39 (0.64)
0.486 [0.182, 0.790]
0.83 [0.33, 1.41]
0.002
Later-selection vs. no AI (C2–C4)
30/29
4.52 (0.61)
4.39 (0.64)
0.130 [ − 0.194, 0.455]
0.21 [ − 0.29, 0.73]
0.425
Full-process vs. no AI (C3–C4)
30/29
4.87 (0.50)
4.39 (0.64)
0.479 [0.179, 0.779]
0.83 [0.35, 1.37]
0.002
Warning vs. no AI (C5–C4)
29/29
4.71 (0.68)
4.39 (0.64)
0.319 [ − 0.028, 0.667]
0.48 [ − 0.02, 0.99]
0.071
Early-search vs. later-selection (C1–C2)
30/30
4.88 (0.52)
4.52 (0.61)
0.356 [0.064, 0.648]
0.62 [0.13, 1.20]
0.018
Warning vs. early-search (C5–C1)
29/30
4.71 (0.68)
4.88 (0.52)
− 0.167 [ − 0.485, 0.151]
− 0.27 [ − 0.84, 0.22]
0.297
Table E.2: Pairwise condition contrasts on rated creativity. Outcome: calibrated team creativity score. Differences and p values from two-sided Welch t -tests; intervals for Hedges g are percentile intervals from 5,000 team-level bootstrap resamples (seed 20261003).
Dimension
Contrast
Coefficient [95% CI]
Hedges g
p
BH p
Holm p
Originality
C1–C4
0.431 [0.093, 0.769]
0.62
0.012
0.023
0.124
C2–C4
0.068 [ − 0.308, 0.443]
0.10
0.724
0.724
0.839
C3–C4
0.461 [0.128, 0.793]
0.70
0.007
0.013
0.073
C5–C4
0.332 [ − 0.036, 0.700]
0.44
0.077
0.103
0.469
Appropriateness
C1–C4
0.530 [0.292, 0.768]
1.16
<0.001
<0.001
<0.001
C2–C4
0.188 [ − 0.049, 0.425]
0.42
0.120
0.150
0.602
Table E.3: Dimension-level contrasts with the no-AI control and multiplicity adjustment. Coefficients from OLS of each team-level dimension score on condition indicators and task fixed effects (HC3). The family comprises the 20 tests shown (four contrasts with the control × five dimensions); BH and Holm adjustments are applied across all 20.
Mean of remaining raters (main text): g ( p )
Severity-adjusted mixed model: g [95% CI]
Dropped rater
C1–C2
C1–C4
C3–C4
C1–C2
C1–C4
C3–C4
A
0.54 (0.039)
0.65 (0.015)
0.88 (0.001)
0.550 [0.045, 1.080]
0.633 [0.139, 1.183]
0.882 [0.410, 1.418]
B
0.68 (0.010)
0.93 ( <0.001 )
0.89 (0.001)
0.692 [0.218, 1.280]
0.931 [0.419, 1.546]
0.887 [0.391, 1.483]
C
0.54 (0.038)
0.82 (0.003)
0.79 (0.003)
0.544 [0.030, 1.115]
0.804 [0.304, 1.397]
0.790 [0.316, 1.369]
D
0.62 (0.017)
0.74 (0.006)
0.61 (0.022)
0.590 [0.117, 1.172]
0.706 [0.216, 1.283]
0.608 [0.134, 1.101]
Table E.4: Leave-one-rater-out estimates under two aggregation rules. Left panel: Hedges g and Welch p from contrasts on the mean of the three remaining raters’ team scores (score-point estimates and intervals are in Table 7 of the main text). Right panel: Hedges g with 95% interval after re-estimating rater severity on the remaining three raters’ proposal-level ratings (REML linear mixed model with crossed random intercepts for proposals, raters, teams and tasks; severities are the centered predicted rater effects), averaging the adjusted ratings to teams and bootstrapping teams 5,000 times. Under both rules, every interval for the C1–C2 contrast excludes zero.
Table E.5: Condition contrasts within each task. Entries are score-point differences, with Hedges g and Welch p in parentheses (9–11 teams per cell). The interaction test compares OLS models with and without task-by-condition interactions. The timing contrast is positive in all three tasks; cell-level estimates are imprecise by construction.
(1)
(2)
(3)
(4)
Mean proposal score (primary)
Best proposal score
Total score (sum)
Number of proposals
Early-search access (C1)
0.486∗∗∗
0.646∗∗∗
4.25∗∗∗
0.74∗
(0.147)
(0.129)
(1.63)
(0.43)
Later-selection access (C2)
0.124
0.426∗∗∗
6.31∗∗∗
1.53∗∗∗
(0.165)
(0.129)
(1.95)
(0.54)
Full-process access (C3)
0.473∗∗∗
0.541∗∗∗
2.85
0.39
Table E.6: The value of the submitted portfolio under alternative aggregations. OLS with task fixed effects and HC3 standard errors. For each rater, the creativity composite of a team’s proposals is averaged (column 1), maximized (column 2) or summed (column 3); the rater-level values are then averaged over the four raters. Column 1 reproduces the primary outcome. Column 4 is the rater-coded number of proposals. Columns 2–4 are implications of the search–selection framework (Section 2) rather than separate confirmatory outcomes. ∗p<0.10 , ∗∗p<0.05 , ∗∗∗p<0.01 (two-sided).
Condition means
Differences ( p )
Measure
C1
C2
C4
C1 − C4
C1 − C2
C2 − C4
Pool: ideas in any search-stage draft
23.50
6.30
6.69
+16.81 ( <0.001 )
+17.20 ( <0.001 )
−0.39 (0.659)
Pool: de-duplication at 0.40
19.57
3.77
3.76
+15.81 ( <0.001 )
+15.80 ( <0.001 )
+0.01 (0.989)
Pool: de-duplication at 0.60
19.90
3.87
4.00
+15.90 ( <0.001 )
+16.03 ( <0.001 )
−0.13 (0.820)
Pool: embedding de-duplication (0.85)
18.53
3.37
3.69
+14.84 ( <0.001 )
+15.17 ( <0.001 )
−0.32 (0.527)
Pre-gate text share: TF–IDF threshold 0.25
0.74
0.07
0.26
+0.48 ( <0.001 )
+0.67 ( <0.001 )
−0.19 ( <0.001 )
Table F.1: Robustness of the funnel measures. Welch tests. Pool measures count distinct ideas in the members’ search-stage drafts under alternative rules; pre-gate text shares are the share of submission sentences matching the team’s search-stage drafts, chat or assistant output. Placebo match rates of the TF–IDF rule against other teams’ pre-gate material in the same task: 2.3% (0.25), 0.8% (0.35, main) and 0.2% (0.50). Embedding thresholds (0.836 and 0.885) are the cosine similarities exceeded by 5% and 1% of placebo matches; the embedding model is multilingual MiniLM, which recognizes paraphrase.
Panel A. Pre-gate origin and proposal rating (creativity composite, rater mean)
Sample
Model
Estimate (s.e.)
p
Units (teams)
all units
condition + task FE
+0.022 (0.090)
0.805
353 (142)
all units
within team
+0.027 (0.128)
0.837
167 (42)
exact or single segmentation
condition + task FE
+0.058 (0.096)
0.544
292 (127)
exact or single segmentation
within team
+0.089 (0.136)
0.515
141 (37)
Panel B. Share of proposals of pre-gate origin, by condition (all units)
Table F.2: Proposal-level origin and within-condition associations. Each submission is segmented into the number of units the raters scored (353 units in 142 teams; 86% of teams segmented exactly by their own headers or as a single unit, the rest by length). Within multi-unit teams, unit length correlates 0.27 with the raters’ feature count and 0.37 with rated elaboration, which supports the alignment. A unit is of pre-gate origin if at least half of its sentences match the team’s pre-gate material. Panel A: OLS with team-clustered standard errors; the within-team model uses teams with units of both origins. All associations are descriptive.
Condition
Pool
Control pool
k
σθ
Implied premium (perfect screen)
Observed vs. control
Ratio
C1
20
4
2
0.473
0.461
0.486
1.06
C3
13
4
2
0.473
0.356
0.479
1.35
C5
14
4
2
0.473
0.375
0.319
0.85
Table F.3: Screening premium implied by the measured pools. The implied premium is σθ[Mk(N)−Mk(N0)] for standard normal order statistics, where N and N0 are the mean pool sizes at the gate (rounded) in the condition and the control, k=2 ideas are retained, and σθ is the standard deviation of the rater-mean proposal composite multiplied by the square root of its reliability (ICC(2, k ) =0.44 ). It assumes perfect screening and equal quality of AI and own ideas; it is a check on magnitudes, not an estimate.
Similarity measure
Threshold
Placebo match rate (other teams’ submissions)
Match rate (own submission)
TF–IDF (character 2–3 grams)
0.250
1.1%
21.1%
TF–IDF (character 2–3 grams)
0.300
0.6%
18.3%
TF–IDF (character 2–3 grams) (main)
0.350
0.4%
16.4%
TF–IDF (character 2–3 grams)
0.400
0.2%
15.0%
TF–IDF (character 2–3 grams)
0.500
0.1%
12.8%
TF–IDF (character 2–3 grams)
0.700
0.0%
9.6%
Table G.1: Calibration of the tracing rule. Each assistant idea unit is compared with its own team’s final submission and with the submission of every other team that worked on the same task. The placebo match rate is the share of unit–other-team pairs in which some sentence pair reaches the threshold; it is the false-positive rate of the rule. Embedding thresholds (multilingual MiniLM, cosine similarity of normalized sentence embeddings) are set so that their placebo match rates equal those of the TF–IDF thresholds named in parentheses.
Entered submission
Near-verbatim
Entered unshared
Contrast (window)
Raw
Type FE
Raw
Type FE
Raw
Type FE
TF–IDF, threshold 0.25
C2 (S2) − C1 (S1)
+10.7 ***
+6.3 *
+6.1 **
+3.4
+4.5 ***
+3.8 **
C2 (S2) − C1 (S1), generation requests
+8.0 **
–
+2.2
–
+3.7 **
–
C2 (S2) − C3 (S2)
+0.8
+1.9
+6.6 **
+8.0 ***
−5.1 **
−4.9 *
C3 (S2) − C3 (S1)
+17.3 ***
+10.0 ***
+4.9 ***
+1.6
+10.1 ***
+8.8 ***
Table G.2: Robustness of the tracing results. Entries are differences in percentage points of idea units, from unit-level linear probability models with task fixed effects and team-clustered standard errors; “Type FE” adds fixed effects for the five request types. Near-verbatim uses a TF–IDF similarity of 0.70 (for embeddings, the threshold with the same placebo match rate). Generation rows restrict to units from requests for new options, so request-type fixed effects do not apply. ∗p<0.10 , ∗∗p<0.05 , ∗∗∗p<0.01 .
Outcome
Units
Slope per minute, search window (pp)
Difference in slope, selection window (pp)
Units
Entered submission
all units
+1.04 ( <0.001 )
+0.39 (0.259)
7,973
Entered submission
generation requests only
+0.48 (0.010)
+0.55 (0.510)
3,551
Near-verbatim
all units
+0.70 ( <0.001 )
+0.24 (0.413)
7,973
Near-verbatim
generation requests only
+0.37 (0.010)
−0.12 (0.794)
3,551
Entered unshared
all units
+0.38 (0.002)
+0.38 (0.088)
7,973
Entered unshared
generation requests only
+0.07 (0.361)
+0.93 (0.175)
3,551
Table G.3: Timing within the access window. Unit-level linear probability models of the outcome on minutes elapsed in the window, a selection-window indicator and its interaction with minutes, with team fixed effects and team-clustered standard errors ( p in parentheses). Units from the search windows of C1, C3 and C5 and the selection windows of C2 and C3. Lemma 1 implies a steeper slope in the selection window for options kept without comparison; the differences are small, all but one positive, and none is significant at the 5% level.
Panel A. Content shares (team means; Welch tests)
C2 − C1
C2 − C3
C1 − C4
C5 − C1
AI-derived share of submission sentences
+0.167 (0.008)
−0.110 (0.009)
+0.650 ( <0.001 )
−0.182 (0.020)
of which never shared in chat
+0.062 (0.236)
−0.201 (0.005)
+0.133 (0.001)
−0.049 (0.260)
of which shared in chat
+0.104 (0.088)
+0.091 (0.187)
+0.517 ( <0.001 )
−0.133 (0.058)
Panel B. Association with standardized team scores within conditions
Human panel
Language-model panel
AI − human
Table G.4: AI-derived content, its pricing by the two panels, and typicality. Panel A: each submission sentence is matched to its most similar assistant idea unit of the same team (TF–IDF, threshold 0.35); it is AI-derived if the match reaches the threshold and never shared if that unit did not reappear in the team chat; p values in parentheses. Matched against the assistant output of five other teams in the same task, the same rule classifies 1.4% of submission sentences as AI-derived (against 71.7% for the team’s own output). Panel B: OLS of standardized team scores on the three variables with condition and task fixed effects ( n=148 ; HC3 standard errors in parentheses); because condition is held fixed, these are descriptive associations. Panel C: mean TF–IDF similarity of a submission to the other submissions in its task, and to the pooled assistant output received by other teams in its task; defined for every team, so the contrasts preserve randomization ( p in parentheses).
Process variable
Indicator
n
a
pa
b
pb
Total
Direct
Indirect a×b [95% CI]
Semantic dispersion
Stage-1 AI
148
0.033
<0.001
4.191
0.020
0.486
0.346
0.140 [0.019, 0.268]
Stage-1 chat (1,000 characters)
Stage-1 AI
148
1.022
<0.001
0.115
0.031
0.486
0.368
0.117 [0.021, 0.247]
Rater-coded proposal count
Stage-2 AI
148
1.526
0.005
− 0.179
<0.001
0.124
0.398
− 0.273 [ − 0.462, − 0.097]
Chat messages, all stages
Stage-1 AI
148
− 15.9
0.109
0.0009
0.468
0.486
0.501
− 0.015 [ − 0.072, 0.028]
Chat messages, all stages
Stage-2 AI
148
− 13.3
0.242
0.0009
0.468
0.124
0.137
− 0.012 [ − 0.066, 0.030]
Chat characters, all stages (1,000)
Stage-1 AI
148
2.037
<0.001
0.030
0.360
0.486
0.425
0.060 [ − 0.041, 0.186]
Table H.1: Exploratory decomposition of condition effects through process variables. The a path is the task-adjusted effect of the access indicator on the process variable (in the variable’s units); the b path is the adjusted association between the process variable and creativity; total and direct effects are the indicator’s coefficients without and with the process variable; indirect effects use percentile intervals from 5,000 team bootstraps. Stage-1 and Stage-2 indicators follow the saturated coding (C1, C3, C5 and C2, C3, respectively). All process variables are post-treatment, so these are descriptive decompositions rather than mediation estimates. Rows below the line show that total interaction volume carries no indirect association.
Panel A. Share of submission sentences whose nearest source is an assistant output
Condition
Embedding
TF–IDF
Early-search (C1)
0.130 [0.073, 0.187]
0.116 [0.053, 0.179]
Later-selection (C2)
0.258 [0.176, 0.340]
0.260 [0.171, 0.350]
Full-process (C3)
0.391 [0.296, 0.487]
0.388 [0.265, 0.511]
Warning (C5)
0.125 [0.069, 0.180]
0.060 [0.027, 0.093]
Panel B. Condition differences in semantic proximity among assistant users ( n=118 ; reference C1)
Table H.2: Engagement with assistant outputs. Semantic proximity is the similarity between a team’s final submission and the assistant outputs it received (sentence embeddings, cosine similarity; robustness with TF–IDF vectors). Embedding- and TF–IDF-based team measures correlate at r=0.91 – 0.92 . All models use HC3 standard errors. Proximity is undefined for the no-AI control, so Panels B and C are observational comparisons among assistant users and do not preserve randomization; one assigned C5 team that never used the assistant is excluded.
Model
Family
Mean
s.d.
Share ≥6
Pearson r
Spearman ρ
Mean diff.
Claude Sonnet 4.6
Claude
4.53
0.80
0.010
0.483
0.427
0.11
Claude Opus 4.6
Claude
4.72
0.76
0.028
0.395
0.325
0.30
Claude Opus 4.7
Claude
5.42
0.75
0.208
0.516
0.430
0.99
GPT-5.5
GPT
5.17
0.78
0.117
0.565
0.514
0.74
GPT-5.5 Pro
GPT
5.16
0.81
0.137
0.548
0.499
0.74
Gemini 3.1 Flash
Gemini
4.86
0.81
0.048
0.562
0.503
0.44
Table I.1: The language-model evaluator panel. Each model scored the 394 proposal units of the locked template once at temperature 0.1, using the human rubric, the full submission text, the expected number of proposals, and a JSON response schema. Mean and s.d. refer to the 1–7 composite; share ≥6 is the fraction of proposals scored 6 or higher. Agreement statistics compare each model with the mean human composite for the same proposal; mean difference is AI minus human. Because ICC(2, k ) rises with the number of raters, the panel’s higher reliability is not evidence of validity.
Credit γ per proposal
C1 − C2 [95% CI]
p
Hedges g
C2 − C4
p
0.0
0.356 [0.064, 0.648]
0.018
0.62
0.130
0.425
0.1
0.279 [0.047, 0.512]
0.019
0.61
0.283
0.043
0.2
0.203 [ − 0.022, 0.428]
0.076
0.46
0.436
0.001
0.3
0.126 [ − 0.147, 0.399]
0.359
0.24
0.589
<0.001
0.4
0.049 [ − 0.306, 0.405]
0.782
0.07
0.742
<0.001
0.5
− 0.027 [ − 0.481, 0.427]
0.905
− 0.03
0.895
<0.001
Table I.2: Evaluators that credit breadth. Upper panel: the human team score plus γ points per rater-coded proposal (Corollary 1 ). The timing gap vanishes at γ∗=0.46 , the C1–C2 difference in the human score divided by the C2–C1 difference in proposals. Lower panel: originality =0.6× quality +0.4× a score for the number of proposals (1 =2 , 2 =4 , 3 =6 , ≥4=7 ), and creativity is the unweighted mean of originality, appropriateness and elaboration, computed per rater and averaged. Welch tests.
Panel A. AI-minus-human gap (standardized) regressed on condition and submission features
Specification
C1 − C2
C2
C3
Added variables
Adj. R2
Conditions only
−0.662 ***
+0.577 **
+0.642 ***
–
0.107
+ proposal count
−0.547 **
+0.354
+0.585 ***
proposals +0.146 ***
0.207
+ AI-derived sentence share
−0.591 ***
+0.434 *
+0.426 *
AI share +0.553
0.115
+ both
−0.527 **
+0.312
+0.514 **
proposals +0.142 ***; AI share +0.187
0.203
+ both + rated text quality
−0.455 **
+0.431 *
+0.629 **
proposals +0.086 *; AI share +0.283 ; text quality −0.322 **
0.228
Table I.3: Sources of the divergence between the human and language-model panels. Panel A: OLS with task fixed effects and HC3 standard errors ( n=148 ); the dependent variable is the difference between the standardized model-panel and human-panel team scores (Table 8 , column 4 of the main text). Proposal count is rater-coded; AI share is the share of submission sentences whose nearest source is an assistant output (sentence embeddings; zero without access); rated text quality is the human team mean, centered. Panel B: Welch contrasts of standardized team scores ( p in parentheses); model-panel dimension scores are standardized within model, averaged over models, and re-standardized; the last column is the correlation of the two panels’ team scores across all 148 teams. ∗p<0.10 , ∗∗p<0.05 , ∗∗∗p<0.01 .
Team-level moderator
Stage-1 AI × moderator
Stage-2 AI × moderator
Warning × moderator
AI-use frequency, mean
− 0.021 (0.844)
0.332 (0.002)
0.281 (0.105)
Conscientiousness, s.d.
0.031 (0.795)
− 0.141 (0.218)
− 0.306 (0.013)
Conscientiousness, range
0.032 (0.784)
− 0.141 (0.208)
− 0.311 (0.016)
AI optimism, mean
− 0.052 (0.576)
0.205 (0.026)
0.197 (0.246)
Prefer AI interaction, mean
− 0.062 (0.599)
− 0.241 (0.029)
− 0.220 (0.290)
Emotional stability, mean
− 0.250 (0.034)
0.017 (0.883)
0.144 (0.291)
Table J.1: Exploratory moderator screen (eight smallest p values). Each row is a separate model that adds the moderator and its interactions with the Stage-1, Stage-2 and warning indicators to the saturated condition model with task fixed effects (HC3); the moderator is standardized, and cells give b ( p ). The screen covered all team means, standard deviations and ranges of the registration-survey scales. No adjustment for multiplicity is applied, and none of these interactions would survive one; we report them for transparency and do not interpret them.
Ad creative optimization is increasingly constrained by evaluation rather than generation. Generative models can produce many plausible creatives, but reliable evaluation requires online experiments, in which only a limited slate can be tested. We study how to use data from historical A/B tests to generate and select the candidates in that slate. We developed and deployed a performance-driven offline-to-online workflow that guides creative generation with a predictive model as an inference-time critic. In the offline phase, we use a predictive model trained on historical experiments to rank and refine variants created by a generative model. A final test slate is then deployed in an online adaptive experiment. In a 50-arm field experiment, we found that the best creative generated with this method yielded 45.1% higher engagement than the best human-authored creative. Two additional experiments showed the same upper-tail pattern, with lifts of 46.7% and 36.2%. We found that despite the predictive model being too noisy to directly identify the best creative offline, it effectively guides the generative model toward creating strong candidates that can be efficiently evaluated in an adaptive experiment. The results suggest a design principle for creative optimization with generative models: use predictive models to guide generation of a slate to test, judge the slate by whether it contains high-performing candidates at a feasible test size, and use adaptive experiments to select among candidates while limiting traffic lost to weak arms.
Diverse human groups produce diverse ideas, the raw material of innovation. Generative AI challenges this engine twice over: everyday AI assistance may homogenize what diverse people create, and AI-simulated diversity may replace the people altogether. We tested both challenges in a preregistered creative metaphor experiment with native (L1) and non-native (L2) English writers, who wrote without AI, with AI-generated ideas (AI ideation), or with AI refining their own ideas (AI refinement). L2 writers contributed more collective diversity than L1 writers, with native-language ideation showing the most diverse pools. AI ideation compressed collective diversity for everyone and left the L2 advantage undetectable, whereas AI refinement preserved both. We then simulated the entire writer pool using personas built from participants' real backgrounds, three model families, native-language prompting, and elevated sampling temperatures. Every simulated pool fell below every human pool, and pushing models further induced diversity only through degenerate text. However, at the individual level, AI ideation raised writers' ratings, pitting private incentives against the collective good, except when L2 writers used their native language, which benefited both. Human diversity remains a valuable creative resource that current AI cannot simulate or sustain; the design of human-AI collaborative workflows determines whether it survives.
Mengchen Dong, Hiromu Yakura
Center for Humans and Machines, Max Planck Institute for Human Development, Berlin, Germany
AI-generated outputs are frequently perceived as highly creative, but also tend to be more similar to each other (homogenized) than human ideas. Does the human heterogeneity that extends beyond AI reflect valuable thinking that AI misses? Alternatively, is greater human heterogeneity driven by lower-quality ideas and less effective thinking that AI avoids? We developed Expansion of Divergent Generative Expression (EDGE) to map AI's semantic "blind spots" by assessing how strongly/weakly individual human ideas were represented in a task-matched AI semantic comparison set. Specifically, we compared human-generated vs. AI-generated ideas across three open-ended tasks: college admissions essays, alternative uses for everyday objects, and interpersonal advice. Across all three tasks, human responses extended into semantic regions sparsely covered by AI outputs. These distinctively human responses showed no meaningful deficits in creativity, quality, or effectiveness, were associated with equal-or-better academic performance, and with credibly superior ratings and achievement for some outcomes. These ideas were thus not merely poor-performing variation; they performed as well as or better than ideas that overlapped more strongly with AI ideas. More distinctive human ideas tended to receive higher creativity ratings, but this relationship was weak or reversed for AI ideas. Taken together, these results suggest that differentiation among human-generated ideas does not reflect poor thinking that AI can safely omit, and instead may afford both individual and population-level advantages.
Kibum Moon, Dan R. Johnson, Kostadin Kushlev +6
Department of Psychology, Georgetown University · Department of Cognitive and Behavioral Science, Washington and Lee University · The Wharton School, University of Pennsylvania +4