Toward Measuring AI's Effects on Skill Formation: The Stock-Formation Gap
Organizations: University of Illinois Urbana-Champaign, Urbana, IL, USA
Abstract
Large-scale AI deployment data and controlled learning experiments characterize different consequences of the same technology. Deployment telemetry shows that AI use is concentrated in skilled work and frequently supports immediate task performance. It observes tasks, interaction patterns, and outputs, however, not whether users become more capable of performing those tasks independently. Controlled studies measure independent capability more directly, but only in narrower populations and settings, with outcomes that vary substantially by interaction design. We formulate this discrepancy as a stock--formation measurement gap: current systems observe the use of existing expertise more readily than the formation of future expertise. Because formation has historically been society's recovery mechanism through technological change, the gap matters well beyond any single classroom. We synthesize the experimental and observational evidence by identification strength, use public deployment data as a descriptive illustration of the gap, and identify the missing bridge between interaction traces and unassisted retention and transfer. We then propose a research program that links consented usage records to independent assessments while experimentally varying whether AI supplies answers, hints, feedback, or evaluation. The claim is not that AI has been shown to erode skill formation at population scale. It is that existing measurement cannot determine whether it does, and that this question is both measurable and designable.
Figures & tables
| Deployment telemetry | Controlled learning studies | |
|---|---|---|
| Population | Mostly stock ( ): skilled workers | Mostly formation ( ): students, novices |
| Scale | Millions of conversations | Tens to 1,000 participants |
| Observes | Task, mode labels (noisy correlates of ), output | Unassisted capability change |
| Does not observe | ; who is learning what | Population-scale usage; long-run field behavior |
| Can support | Descriptions of use of existing expertise | Causal effects of assigned conditions in specific settings |
| Study | Pop. | Scale | Design | Alloc. | Learn. | Headline finding |
|---|---|---|---|---|---|---|
| Deployment telemetry (four datasets) ( 2025b ; 2025a ; 2025 ; 2026 ) | Both | 9.8K–4M conv. | Tel. | — | — | usage patterns only |
| Fan et al. ( 2025 ) | Form. | 117 students | RCT | Obs. | performance up, learning flat | |
| Shen & Tamkin ( 2026 ) | Bridge | 64 professionals | RCT | Obs. | quiz deficit; pattern-dependent | |
| Bastani et al. ( 2025 ) | Form. | 1K students | RCT | Yes | deficit in base arm; not detected in scaffolded arm | |
| Alsaiari et al. ( 2026 ) | Form. | 329 students | RCT | Yes | hybrid feedback best; pure metacognitive worst | |
| Kumar et al. ( 2026 ) | Bridge | 968 adults | RCT | Yes | gains under learner-works design |
AI in Education
AI in Education studies how artificial intelligence can support teaching, learning, assessment, and educational decision-making.
See all 262 papers