While AI assistance can improve human task performance in the short term, it may also undermine the development of skills in the longer term. We examine this tension in a controlled logic-puzzle experiment involving on-demand AI assistance, where participants complete tasks before, during, and after AI is available. By experimentally varying AI request costs, we find that lower-cost assistance induces more frequent AI use. We also find that participants who request AI assistance during the AI-access phase perform worse at the task after assistance is removed, and their subsequent unassisted performance is overestimated when predicted from earlier AI-assisted performance. We use a Bayesian latent ability model to separate initial ability, post-AI ability, and participant-specific skill change, while estimating how independent reasoning during the AI-access phase relates to skill development. The results show that greater independent problem-solving effort is associated with larger gains in latent ability, consistent with the interpretation that skill development is weaker when AI assistance substitutes for independent reasoning.
Figures & tables
Figure 1. Overview of the three-phase study design. Participants completed Phase 1 and Phase 3 without simulated AI assistance, providing pre- and post-AI measures of unassisted performance. During Phase 2, participants were assigned either to a no-AI control condition or to an AI-access condition with experimentally varied AI request costs. A horizontal timeline shows the three-phase study progressing from a pre-assessment to an experimental phase and then a post-assessment. During the middle phase, participants are randomly assigned to one of three conditions: no AI assistance, high-cost AI assistance, or low-cost AI assistance.
Figure 2. User-interface examples from the logic-based puzzle task Examples of two user interfaces for the logic-based puzzle task, consisting of 6 objects. (a) The user interface without AI assistance; (b) The user interface with AI assistance, identical to (a) except for an added dialog on the left that displays the AI suggestion as text and displays the location of a randomly chosen object on the screen.
Figure 3. Reward rate across phases for participants with zero AI use in Phase 2. Labels 2-1 and 2-2 denote the first and second halves of Phase 2. Error bars indicate standard errors. A line plot shows reward rate increasing steadily across the four displayed time points, from 2.03 in Phase 1 to 3.86 in Phase 3, with intermediate increases during the two halves of Phase 2.
Figure 4. Average AI usage in Phase 2 by AI cost condition. Error bars indicate standard errors. A two-bar chart shows that participants in the low-cost AI condition made roughly twice as many AI requests as participants in the high-cost AI condition, 6.67 vs. 3.33 requests on average.
Figure 5. Residuals by AI-use status. Residuals are defined as observed minus predicted Phase 3 reward rate, y−y^ ; error bars indicate standard errors. An error-bar plot compares prediction residuals for non-AI users and AI users. The average residual is 0.15 for non-AI users and -0.22 for AI users, indicating opposite directions of prediction error across the two groups.
Figure 6. Bayesian model of latent skill development. Phase 1 and Phase 3 accuracy and response time are modeled as noisy measurements of latent abilities θi1 and θi3 . Participant-specific skill change δi depends on initial ability and solo share. Shaded nodes are observed variables, unshaded nodes are latent variables or unknown parameters, and the plate denotes participants i=1,…,N . A plate diagram shows the Bayesian model linking initial latent ability to Phase1 accuracy and response time, and linking Phase3 latent ability to the corresponding Phase3 measures. Initial ability and Phase2 solo share jointly predict participant-specific skill change, which in turn connects initial ability to post-AI ability.
Figure 7. Posterior modes and 95% credible intervals of participant-specific latent ability changes, δi , ordered by posterior mode. The vertical dashed line indicates zero change. Participant-specific posterior estimates of latent skill change are ordered from smallest to largest. Most posterior modes lie to the right of the zero-change line, while many credible intervals are wide, showing substantial uncertainty and heterogeneity across participants.
Figure 8. Posterior distributions of latent ability regression coefficients. Histograms show posterior draws; orange solid lines denote posterior means, orange dotted lines denote 95% credible intervals, and black dashed lines mark zero. Three posterior histograms summarize the population-level coefficients. Average expected skill change is clearly positive, the coefficient on initial ability is clearly negative, and the coefficient on solo share is positive, with its lower credible bound remaining slightly above zero.
Model
Test LogL ↑
Accuracy MSE ↓
Log RT MSE ↓
Solo share (main)
-329.94
0.809
0.850
Solo share + AI-cost conditions
-331.45
0.830
0.854
AI usage
-332.96
0.831
0.882
Constant change (baseline)
-339.02
0.893
0.915
Table 1. Five-fold held-out prediction of Phase 3 unassisted performance across Bayesian latent-ability models. The main model uses Phase 2 solo share to predict latent skill change; the condition model adds AI-cost indicators; the AI-usage model replaces solo share with total AI requests; and the constant-change baseline assumes identical expected skill change across participants. Test LogL is the total held-out log predictive likelihood (higher is better). MSE is the mean squared prediction error for standardized Phase 3 accuracy and log response time (lower is better).
Artificial intelligence (AI) is being increasingly integrated into human problem-solving, yet its effects on individual skill development remain unclear. We examine how both AI usage and informativeness can shape learning in the context of a controlled logical reasoning task with on-demand access to AI assistance. We find that greater AI usage is associated with weaker skill development: heavy AI users underperform relative to comparable peers, whereas light AI users perform similarly to matched users who do not use AI. We also find in our study that these patterns are mediated by AI informativeness. Low-information AI neither improves immediate performance nor preserves performance after AI assistance is removed, and is linked to weaker learning overall. On the other hand, high-information AI was found to improve short-run performance without reducing post-AI outcomes on average in our experiments, but with heterogeneous effects. Our findings in general suggest that AI can, depending on context, either complement human skill development by amplifying independent reasoning or can act as a substitute that undermines such reasoning, with the implication that regulating AI access and usage will be important for promoting skill development in the presence of AI assistance.
Shang Wu, Hongyu Yao, Catarina Belem +3
University of California, Irvine · Massachusetts Institute of Technology
People often optimize for long-term goals in collaboration: A mentor or companion doesn't just answer questions, but also scaffolds learning, tracks progress, and prioritizes the other person's growth over immediate results. In contrast, current AI systems are fundamentally short-sighted collaborators - optimized for providing instant and complete responses, without ever saying no (unless for safety reasons). What are the consequences of this dynamic? Here, through a series of randomized controlled trials on human-AI interactions (N = 1,222), we provide causal evidence for two key consequences of AI assistance: reduced persistence and impairment of unassisted performance. Across a variety of tasks, including mathematical reasoning and reading comprehension, we find that although AI assistance improves performance in the short-term, people perform significantly worse without AI and are more likely to give up. Notably, these effects emerge after only brief interactions with AI (approximately 10 minutes). These findings are particularly concerning because persistence is foundational to skill acquisition and is one of the strongest predictors of long-term learning. We posit that persistence is reduced because AI conditions people to expect immediate answers, thereby denying them the experience of working through challenges on their own. These results suggest the need for AI model development to prioritize scaffolding long-term competence alongside immediate task completion.
Grace Liu, Brian Christian, Tsvetomira Dumbalska +2
Carnegie Mellon University · UC Berkeley · University of Oxford +2
Repeated AI assistance can improve immediate task performance while reducing the skill available for future independent work. We develop a mathematical framework for this long-run tradeoff. The model tracks two state variables: a latent human skill level governing expected independent performance, and a delegation level representing the learner's evolving tendency to rely on AI. Skill changes through error-driven learning under practice and decay under delegation; delegation responds to observed performance, increasing when AI-assisted work appears to outperform independent work. We analyze the resulting dynamics and contrast them with fixed delegation. With fixed delegation, skill follows a one-dimensional learning-decay process with a single stable equilibrium. With adaptive delegation, the coupled system has two attracting equilibria separated by the stable manifold of an interior saddle. The existence and geometry of this separatrix require a global phase-plane analysis of the coupled dynamics. The system is path-dependent: small differences in initial skill or reliance can lead to different long-run outcomes. We use this characterization to show that AI assistance can improve short-run performance while producing worse long-run performance than a no-AI baseline. Increasing AI capability can enlarge the basin of attraction of the low-skill equilibrium, making delegation appear beneficial for longer while increasing the risk of eventual skill loss. The qualitative picture is observed to persist across alternative specifications. Together, these results show that the risk is not AI assistance itself, but the coupling between performance-driven reliance and use-dependent skill change.