Metacognitive Scaffolds Create Productive Friction for AI-Assisted Knowledge Work
Organizations: Microsoft Research, United Kingdom
Abstract
Generative AI imposes new metacognitive demands on knowledge workers as their work shifts from production to planning, evaluation, and orchestration. In a randomized, between-subjects experiment on AI-assisted writing (297 workers, 594 essays), we test dynamically-generated metacognitive scaffolds supporting goal specification and task decomposition. We find that scaffolds improve writing quality, with this boost persisting in short-lag transfer when scaffolds are removed. We also find evidence of improved topic recall three days later. These effects are consistent with observed changes in how people learn to prompt AI systems after exposure to scaffolding. However, scaffolded participants rate the AI as less helpful and more cognitively demanding even as they perform better, suggesting that subjective ease can understate the value of support. Our findings make the case for metacognitive scaffolds in AI-assisted knowledge work, revealing a productive friction worth designing for as workers increasingly delegate work to agentic AI.
Figures & tables
| Control ( ) | Treatment ( ) | ||
| Age | Mean (SD) | 34.2 (12.0) | 33.6 (11.9) |
| Median [IQR] | 31 [26, 41] | 31 [24, 42] | |
| Gender | Male | 71 (49.3%) | 78 (51.0%) |
| Female | 71 (49.3%) | 69 (45.1%) | |
| Non-binary or other | 2 (1.4%) | 6 (3.9%) | |
| Country | USA | 144 (100%) | 153 (100%) |
| Dimension | Task 1 | Task 2 |
|---|---|---|
| LLM judge: full set of 594 essays | ||
| Context and purpose | 0.083 [ , 0.173], | 0.144 [0.054, 0.234], |
| Recommendation clarity | 0.057 [ , 0.147], | 0.030 [ , 0.120], |
| Argument quality | 0.122 [0.040, 0.204], | 0.078 [ , 0.160], |
| Evidence credibility | 0.115 [0.026, 0.204], | 0.039 [ , 0.128], |
| Feasibility | 0.085 [ , 0.177], | 0.056 [ , 0.149], |
| Measure | Prior-scaffold (SD) | Control (SD) | Mean difference | Cohen’s | Rank-biserial | Probability of superiority | OR [95% CI] | |
|---|---|---|---|---|---|---|---|---|
| Goal clarity | 8.49 (7.25) | 6.43 (6.25) | 2.067 | 0.30 | 0.19 | 0.60 | 1.862 [1.225, 2.828] | .0036 |
| Task decomposition | 5.94 (5.37) | 4.65 (4.66) | 1.291 | 0.26 | 0.14 | 0.57 | 1.537 [1.013, 2.332] | .0431 |
| Combined GC+TD | 14.43 (12.29) | 11.07 (10.67) | 3.358 | 0.29 | 0.18 | 0.59 | 1.779 [1.173, 2.698] | .0067 |
| Task 1: scaffold active | Task 2: no scaffold | |||||||
|---|---|---|---|---|---|---|---|---|
| Question | Control | Scaffold | [95% CI] | Control | Prior-scaffold | [95% CI] | ||
| “‘latex How mentally demanding was the essay writing task? | 4.88 | 5.40 | [ , ] | .087 | 4.86 | 4.73 | [ , ] | .906 |
| How hurried or rushed was the pace of the task? | 4.20 | 5.23 | [ , ] | .005 | 3.79 | 4.06 | [ , ] | .906 |
| How hard did you have to work to accomplish your level of performance? | 5.44 | 5.85 | [ , ] | .130 | 4.91 | 5.12 | [ , ] | .906 |
| How insecure, discouraged, irritated, stressed, or annoyed were you? | 2.59 | 3.59 | [ , ] | .002 | 2.85 | 3.04 | [ , ] | .906 |
| How mentally demanding was the essay topic itself to understand and write about? | 4.55 | 4.83 | [ , ] | .369 | 4.23 | 4.34 | [ , ] | .906 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Criterion | Score 1 | Score 2 | Score 3 | Score 4 |
|---|---|---|---|---|
| A. Context & Purpose | Little/no problem framing; unclear audience; substantial off-task content. | Vague or generic framing; audience needs are mostly implicit; parts feel templated or only loosely tied to the decision context. | Frames the problem and audience reasonably well; mild mismatch in emphasis but remains on task. | Clearly frames the decision problem; explicitly tailors to the intended decision-maker; stays tightly aligned to the task. |
| B. Recommendation Clarity | No clear recommendation/position. | Stance is implied or wobbly; proposal details are underspecified or contradictory. | Clear stance, but key specifics (scope/model) are incomplete, delayed, or mildly inconsistent. | States a clear recommendation early and specifies what is being proposed (scope, model, key design choices); provides a concise summary of the rationale. |
| C. Argument Quality | Disorganized, internally inconsistent, or largely non-argumentative. | Relies on assertions or one-step reasoning; weak linkage between points. | Mostly logical and persuasive; some reasoning gaps or unexamined assumptions. | Builds a coherent chain from claims reasons support; makes assumptions/warrants legible. |
| D. Evidence and Credibility | Virtually no support; repeated overclaims; credibility is questionable. | Mostly generalities; evidence is thin/generic; numbers (if any) feel ungrounded. | Some evidence, but uneven (claims without support in places). | Uses specific supporting material (examples, comparisons); avoids overclaims; support is relevant and credible. |
| E. Feasibility | No implementation thinking; proposal would be difficult to execute as described. | Implementation is largely aspirational; feasibility concerns are waved away; limited operational detail. | Mentions implementation with some concrete steps; gaps remain (timeline vague, risks not fully addressed, success metrics thin). | Presents a realistic implementation path: key steps, timeline or pilot, operational coverage plan, governance/roles, and how success will be measured. |
| F. Impact | Ignores stakeholder impacts or makes one-sided claims without considering harms. | Narrow focus (e.g., only employee wellbeing or only productivity); description of impacts are minimal. | Discusses stakeholder impacts, but not deeply; consequences lightly addressed. | Identifies key stakeholders (employees, managers); anticipates impacts and (unintended) consequences; links change to outcomes. |
| Comparison | Measure | Estimate | 95% CI | |
| Prompt coding: goal clarity (0–3) | ||||
| R1–R2 | 60 | 0.703 | [0.463, 0.844] | |
| R1–R3 | 60 | 0.587 | [0.315, 0.755] | |
| R2–R3 | 60 | 0.654 | [0.368, 0.813] | |
| LLM–R1 | 60 | 0.458 | [0.269, 0.613] | |
| LLM–R2 | 60 | 0.435 | [0.196, 0.604] | |
| Code | Label | Definition | Coding cue |
|---|---|---|---|
| GC-0 | Absent | No discernible goal or purpose is expressed. Prompts consist of bare topics or instructions so vague that the intended writing outcome cannot be inferred. | No explicit purpose, audience or constraint. |
| GC-1 | Implicit | A general purpose can be inferred from context, but it is never stated explicitly. The user delegates full goal definition to the AI. | Genre or topic implies a goal, but the user does not specify it. |
| GC-2 | Moderate | At least one explicit goal is stated, but the conversation does not specify audience, constraints or evaluative criteria in enough detail to make the target unambiguous. | Purpose or genre is named, but supporting constraints are thin. |
| GC-3 | High | Clear purpose is articulated plus at least two of: audience, constraints, format/tone or evaluative criteria. The intended outcome is unambiguous and may develop across turns. | A full rhetorical problem is made explicit. |
| Code | Label | Definition | Coding cue |
|---|---|---|---|
| TD-0 | Monolithic | The writing task is presented as a single undifferentiated request. No sub-tasks, steps or structural divisions are discernible. | “Write about climate change.” |
| TD-1 | Partially decomposed | More than one component or aspect of the task is named or implied, but without specified ordering or relations among those components. | Components are listed but not organized. |
| TD-2 | Structured | The task is explicitly organized into ordered steps, labeled sections or a sequence of sub-tasks. Relationships between parts are specified at one level. | Numbered or ordered steps are present. |
| TD-3 | Hierarchical | The task is decomposed at multiple levels: top-level goals are broken into sub-goals, which are themselves broken into operations or priorities. Dependencies across levels are indicated. | Multi-level structure with priorities or dependencies. |
| Field | Value |
|---|---|
| Conversation ID | |
| Number of user turns | |
| Writing task type | |
| GC rating | GC-0 / GC-1 / GC-2 / GC-3 |
| GC-P (purpose) | 0 / 1 |
| GC-A (audience) | 0 / 1 |
| Item | Wording [response options] |
|---|---|
| Baseline survey (before Task 1) | |
| Prior AI use | |
| AI-use 1 | How often do you use AI tools like ChatGPT? [once per month / once per week / a few times per week / daily / a few times daily] |
| AI-use 2 | What do you use AI tools like ChatGPT for? [professional writing / creative writing / research / coding / general knowledge / content creation / other (free text)] |
| General metacognitive regulation, adapted from the Metacognitive Awareness Inventory ( Schraw and Dennison, 1994 ) (1 = does not describe me at all, 5 = describes me very much) | |
| Meta 1 | I set specific goals before I begin a task |
| Primary model | + log time on task | |||||
| Term or contrast | Coef. | s.e. | Coef. | s.e. | ||
| Intercept | 0.088 | 0.157 | ||||
| Task 1: scaffold vs control | 0.034 | .008 | 0.035 | .011 | ||
| Scaffold second task | 0.043 | .665 | 0.044 | .663 | ||
| Second vs first task (control) | 0.031 | .440 | 0.032 | .472 | ||
| Topic: city workweek vs hybrid launch | 0.021 | 0.022 | ||||
| Metric | Estimate | 95% CI |
|---|---|---|
| Essays compared | 107 | — |
| Absolute-agreement ICC(A,1), human versus judge | 0.456 | [0.349, 0.547] |
| Pearson , human versus judge | 0.662 | — |
| Calibration slope, judge on human | 0.375 | — |
| Task | Scaffold-assigned (SD) | Control (SD) | Raw difference | Hedges’ | Adjusted difference | Adjusted standardized difference |
|---|---|---|---|---|---|---|
| Task 1 | 2.99 (0.32) | 2.90 (0.32) | 0.094 | 0.29 | 0.089 | 0.28 |
| Task 2 | 2.98 (0.29) | 2.92 (0.32) | 0.063 | 0.21 | 0.070 | 0.23 |
| Outcome | Task | Sample | OR [95% CI] | Observed rates (control vs scaffold) | |
|---|---|---|---|---|---|
| Topic recall | Task 1 | Complete case | [ , ] | .026 | 45.6% vs 64.6% |
| Topic recall | Task 1 | Multiple imputation ( ) | [ , ] | .032 | — |
| Topic recall | Task 2 | Complete case | [ , ] | .046 | 38.0% vs 54.4% |
| Topic recall | Task 2 | Multiple imputation ( ) | [ , ] | .042 | — |
| Predictor | Definition / scoring | OR [95% CI] | ||
|---|---|---|---|---|
| Treatment | Scaffold = 1; control = 0 | – | [ , ] | .216 |
| First-task scenario | City workweek first = 1; hybrid-work first = 0 | – | [ , ] | .252 |
| AI-use frequency | Single baseline item, 1–5 | – | [ , ] | .264 |
| AI use for writing | Writing-related AI use = 1; otherwise = 0 | – | [ , ] | .983 |
| Essay-writing skill | Single baseline item, 1–5 | – | [ , ] | .895 |
| Baseline general metacognition | Mean of metacognition items 1–8; 1–5 | 0.862 | [ , ] | .208 |
| No. | Source | Item | Control pre | Control post | Scaffold pre | Scaffold post | 95% CI | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | MAI | Set specific goals | 3.72 (0.89) | 3.77 (1.10) | 3.68 (0.95) | 3.91 (1.00) | [-0.019, 0.398] | .074 | .241 | 290 | |
| 2 | MAI | Monitor goal progress | 3.68 (0.99) | 3.91 (1.01) | 3.59 (1.06) | 3.81 (0.96) | [-0.257, 0.154] | .624 | .811 | 280 | |
| 3 | MAI | Organize time around goals | 3.86 (0.94) | 3.83 (0.97) | 3.79 (1.04) | 3.68 (1.09) | [-0.311, 0.132] | .427 | .710 | 280 | |
| 4 | MAI | Evaluate goal attainment | 3.88 (0.91) | 4.20 (0.80) | 3.73 (0.99) | 4.03 (0.82) | [-0.265, 0.061] | .220 | .476 | 279 | |
| 5 | MAI | Break task into steps | 3.83 (1.00) | 3.24 (1.16) | 3.98 (0.90) | 4.03 (0.84) | [0.546, 0.959] | 295 | |||
| 6 | MAI | Ask questions before starting | 3.71 (1.06) | 3.89 (1.09) | 3.70 (1.00) | 3.87 (0.97) | [-0.182, 0.229] | .821 | .931 | 292 |
| Outcome | Control | Scaffold | [95% CI] | |
|---|---|---|---|---|
| AI helpfulness | 4.77 | 4.40 | [ , ] | |
| Perceived control | 4.09 | 3.78 | [ , ] | .036 |
Human-AI Collaboration
Human-AI collaboration studies how people and AI systems divide, coordinate, and adapt work to support decisions, creativity, and learning.
See all 446 papers