While AI assistance can improve human task performance in the short term, it may also undermine the development of skills in the longer term. We examine this tension in a controlled logic-puzzle experiment involving on-demand AI assistance, where participants complete tasks before, during, and after AI is available. By experimentally varying AI request costs, we find that lower-cost assistance induces more frequent AI use. We also find that participants who request AI assistance during the AI-access phase perform worse at the task after assistance is removed, and their subsequent unassisted performance is overestimated when predicted from earlier AI-assisted performance. We use a Bayesian latent ability model to separate initial ability, post-AI ability, and participant-specific skill change, while estimating how independent reasoning during the AI-access phase relates to skill development. The results show that greater independent problem-solving effort is associated with larger gains in latent ability, consistent with the interpretation that skill development is weaker when AI assistance substitutes for independent reasoning.
Figures & tables
Figure 1. Overview of the three-phase study design. Participants completed Phase 1 and Phase 3 without simulated AI assistance, providing pre- and post-AI measures of unassisted performance. During Phase 2, participants were assigned either to a no-AI control condition or to an AI-access condition with experimentally varied AI request costs. A horizontal timeline shows the three-phase study progressing from a pre-assessment to an experimental phase and then a post-assessment. During the middle phase, participants are randomly assigned to one of three conditions: no AI assistance, high-cost AI assistance, or low-cost AI assistance.
Figure 2. User-interface examples from the logic-based puzzle task Examples of two user interfaces for the logic-based puzzle task, consisting of 6 objects. (a) The user interface without AI assistance; (b) The user interface with AI assistance, identical to (a) except for an added dialog on the left that displays the AI suggestion as text and displays the location of a randomly chosen object on the screen.
Figure 3. Reward rate across phases for participants with zero AI use in Phase 2. Labels 2-1 and 2-2 denote the first and second halves of Phase 2. Error bars indicate standard errors. A line plot shows reward rate increasing steadily across the four displayed time points, from 2.03 in Phase 1 to 3.86 in Phase 3, with intermediate increases during the two halves of Phase 2.
Figure 4. Average AI usage in Phase 2 by AI cost condition. Error bars indicate standard errors. A two-bar chart shows that participants in the low-cost AI condition made roughly twice as many AI requests as participants in the high-cost AI condition, 6.67 vs. 3.33 requests on average.
Figure 5. Residuals by AI-use status. Residuals are defined as observed minus predicted Phase 3 reward rate, y−y^ ; error bars indicate standard errors. An error-bar plot compares prediction residuals for non-AI users and AI users. The average residual is 0.15 for non-AI users and -0.22 for AI users, indicating opposite directions of prediction error across the two groups.
Figure 6. Bayesian model of latent skill development. Phase 1 and Phase 3 accuracy and response time are modeled as noisy measurements of latent abilities θi1 and θi3 . Participant-specific skill change δi depends on initial ability and solo share. Shaded nodes are observed variables, unshaded nodes are latent variables or unknown parameters, and the plate denotes participants i=1,…,N . A plate diagram shows the Bayesian model linking initial latent ability to Phase1 accuracy and response time, and linking Phase3 latent ability to the corresponding Phase3 measures. Initial ability and Phase2 solo share jointly predict participant-specific skill change, which in turn connects initial ability to post-AI ability.
Figure 7. Posterior modes and 95% credible intervals of participant-specific latent ability changes, δi , ordered by posterior mode. The vertical dashed line indicates zero change. Participant-specific posterior estimates of latent skill change are ordered from smallest to largest. Most posterior modes lie to the right of the zero-change line, while many credible intervals are wide, showing substantial uncertainty and heterogeneity across participants.
Figure 8. Posterior distributions of latent ability regression coefficients. Histograms show posterior draws; orange solid lines denote posterior means, orange dotted lines denote 95% credible intervals, and black dashed lines mark zero. Three posterior histograms summarize the population-level coefficients. Average expected skill change is clearly positive, the coefficient on initial ability is clearly negative, and the coefficient on solo share is positive, with its lower credible bound remaining slightly above zero.
Model
Test LogL ↑
Accuracy MSE ↓
Log RT MSE ↓
Solo share (main)
-329.94
0.809
0.850
Solo share + AI-cost conditions
-331.45
0.830
0.854
AI usage
-332.96
0.831
0.882
Constant change (baseline)
-339.02
0.893
0.915
Table 1. Five-fold held-out prediction of Phase 3 unassisted performance across Bayesian latent-ability models. The main model uses Phase 2 solo share to predict latent skill change; the condition model adds AI-cost indicators; the AI-usage model replaces solo share with total AI requests; and the constant-change baseline assumes identical expected skill change across participants. Test LogL is the total held-out log predictive likelihood (higher is better). MSE is the mean squared prediction error for standardized Phase 3 accuracy and log response time (lower is better).