Metacognitive Scaffolds Create Productive Friction for AI-Assisted Knowledge Work

Viktor Kewenig; Lev Tankelevitch; Sean Rintel; Advait Sarkar; Abigail Sellen

cs.HCOct 22, 2026

Metacognitive Scaffolds Create Productive Friction for AI-Assisted Knowledge Work

Authors: Viktor Kewenig, Lev Tankelevitch, Sean Rintel, Advait Sarkar, Abigail Sellen

Organizations: Microsoft Research, United Kingdom

Abstract

Generative AI imposes new metacognitive demands on knowledge workers as their work shifts from production to planning, evaluation, and orchestration. In a randomized, between-subjects experiment on AI-assisted writing (297 workers, 594 essays), we test dynamically-generated metacognitive scaffolds supporting goal specification and task decomposition. We find that scaffolds improve writing quality, with this boost persisting in short-lag transfer when scaffolds are removed. We also find evidence of improved topic recall three days later. These effects are consistent with observed changes in how people learn to prompt AI systems after exposure to scaffolding. However, scaffolded participants rate the AI as less helpful and more cognitively demanding even as they perform better, suggesting that subjective ease can understate the value of support. Our findings make the case for metacognitive scaffolds in AI-assisted knowledge work, revealing a productive friction worth designing for as workers increasingly delegate work to agentic AI.

Figures & tables

Figure 1. A , The AI-assisted writing interface with the metacognitive scaffold: after an initial request (1), dynamically generated controls prompt goal specification (2) and task decomposition (3) before a draft is generated (4); controls could be revised in natural language (5). B , Randomized experimental design: 297 knowledge workers wrote two briefs, with the scaffold available only for one group in Task 1, followed by a recall test three days later. C , Key findings: better briefs with the scaffold, at the cost of higher load and lower perceived helpfulness; the quality advantage and more deliberate prompting persisted after removal; evidence of better topic recall three days later. Description.

Figure 2. Participant flow in the study.

		Control ( $n=144$ )	Treatment ( $n=153$ )
Age	Mean (SD)	34.2 (12.0)	33.6 (11.9)
	Median [IQR]	31 [26, 41]	31 [24, 42]
Gender	Male	71 (49.3%)	78 (51.0%)
	Female	71 (49.3%)	69 (45.1%)
	Non-binary or other	2 (1.4%)	6 (3.9%)
Country	USA	144 (100%)	153 (100%)

Table 1. Participant characteristics of final analytic sample. Values are

n

(%) unless otherwise indicated. IQR, interquartile range; SD, standard deviation.

Figure 3. Essay quality by condition and task. Scores are means across the seven 1–4 rubric ratings. Faint points show observed essay-level scores; larger markers show condition means with 95% confidence intervals. Neither group had the scaffold in Task 2. Both planned adjusted contrasts were statistically reliable.

{}^{**}P<.01

;

{}^{*}P<.05

Dimension	Task 1	Task 2
LLM judge: full set of 594 essays
Context and purpose	0.083 [ $-0.007$ , 0.173], $P=.071$	0.144 [0.054, 0.234], $P=.002$
Recommendation clarity	0.057 [ $-0.033$ , 0.147], $P=.216$	0.030 [ $-0.060$ , 0.120], $P=.514$
Argument quality	0.122 [0.040, 0.204], $P=.004$	0.078 [ $-0.004$ , 0.160], $P=.062$
Evidence credibility	0.115 [0.026, 0.204], $P=.011$	0.039 [ $-0.050$ , 0.128], $P=.390$
Feasibility	0.085 [ $-0.008$ , 0.177], $P=.074$	0.056 [ $-0.036$ , 0.149], $P=.234$

Table 2. Rubric-level scaffold contrasts, reported as

b

[95% CI],

P

. Full-set models include a participant random intercept, condition

\times

task, topic and the common set of six baseline covariates ( Section 3.4.6 ); models for the 107-essay benchmark include the same fixed effects without a random intercept, with participant-clustered standard errors. Task 1: scaffold minus control; Task 2: previously scaffolded minus previously unscaffolded. Rubric-level analyses are exploratory; all

P

values are uncorrected.

Measure	Prior-scaffold $M$ (SD)	Control $M$ (SD)	Mean difference	Cohen’s $d$	Rank-biserial $r$	Probability of superiority	OR [95% CI]	$P$
Goal clarity	8.49 (7.25)	6.43 (6.25)	2.067	0.30	0.19	0.60	1.862 [1.225, 2.828]	.0036
Task decomposition	5.94 (5.37)	4.65 (4.66)	1.291	0.26	0.14	0.57	1.537 [1.013, 2.332]	.0431
Combined GC+TD	14.43 (12.29)	11.07 (10.67)	3.358	0.29	0.18	0.59	1.779 [1.173, 2.698]	.0067

Table 3. Task 2 prompt behavior by condition: descriptive statistics, standardized effect sizes and ordinal logistic regression estimates with the six common covariates (Section 3.4.6 ). OR is the treatment effect of prior scaffolding versus control on the odds of a higher score;

N=297

Figure 4. Blind coding of prompt behavior in the unscaffolded Task 2. A , Combined goal-clarity and task-decomposition score (0–39); B , goal clarity (seven indicators; 0–21); C , task decomposition (six indicators; 0–18). Faint points: participants’ mean scores across five LLM coder runs (

N=297

); circles: condition means with 95% participant-bootstrap CIs (10,000 resamples). Brackets: ordinal regressions with the six common covariates ( Section 3.4.6 ); Combined

P=.0067

, GC

P=.0036

, TD

P=.0431

Figure 5. Correct topic recall by condition and task. Bars show observed recall percentages with 95% Wilson confidence intervals. Asterisks indicate adjusted logistic mixed-model contrasts (Task 1, $P=.026$ ; Task 2, $P=.046$ ).

	Task 1: scaffold active				Task 2: no scaffold
Question	Control $M$	Scaffold $M$	$b$ [95% CI]	$P_{\mathrm{BH}}$	Control $M$	Prior-scaffold $M$	$b$ [95% CI]	$P_{\mathrm{BH}}$
“‘latex How mentally demanding was the essay writing task?	4.88	5.40	$0.526$ [ $-0.036$ , $1.087$ ]	.087	4.86	4.73	$-0.154$ [ $-0.730$ , $0.421$ ]	.906
How hurried or rushed was the pace of the task?	4.20	5.23	$0.983$ [ $0.359$ , $1.607$ ]	.005	3.79	4.06	$0.215$ [ $-0.353$ , $0.782$ ]	.906
How hard did you have to work to accomplish your level of performance?	5.44	5.85	$0.441$ [ $-0.115$ , $0.997$ ]	.130	4.91	5.12	$0.215$ [ $-0.380$ , $0.811$ ]	.906
How insecure, discouraged, irritated, stressed, or annoyed were you?	2.59	3.59	$0.960$ [ $0.425$ , $1.495$ ]	.002	2.85	3.04	$0.146$ [ $-0.373$ , $0.665$ ]	.906
How mentally demanding was the essay topic itself to understand and write about?	4.55	4.83	$0.272$ [ $-0.322$ , $0.867$ ]	.369	4.23	4.34	$0.062$ [ $-0.523$ , $0.647$ ]	.906

Table 4. Item-level cognitive-load outcomes during the scaffolded and subsequent unscaffolded tasks

Figure 6. Subjective experience by condition. A , AI helpfulness and B , perceived control after Task 1; C , unweighted NASA-TLX load after Tasks 1 and 2 (higher, more load). Faint points: participant ratings with slight vertical jitter (up to

\pm 0.04

scale units); markers: condition means with 95% confidence intervals from the unjittered data.

{}^{***}P<.001

;

{}^{*}P<.05

; n.s., not significant.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1. Goal Clarity interface. After the participant entered an initial request, the system rendered editable controls for message, audience, intended reader outcome, primary writing objective and desired scope/depth. This phase operationalized goal specification as an explicit, inspectable interface step rather than leaving it embedded only in free-form prompting.

Figure A2. Task Decomposition interface. Once goal-clarity information was specified, the system generated task-specific planning controls for analytical approaches, essay structure and critical considerations. A dedicated natural-language box allowed participants to ask for adjustments or replacement controls, making the decomposition adaptive to their evolving intent.

Figure A3. Scenario brief 1 | 4-Day Work-Week, Yes or No? Participants were cast as a senior policy analyst in a city-planning department and asked to write a proposal about whether a city should adopt a 4-day work week for municipal employees. The brief framed the task as a persuasive knowledge-work assignment requiring trade-off analysis, audience awareness and justification of recommendations.

Figure A4. Scenario brief 2 | Hybrid-Work Launch on the Line. Participants were cast as a strategic operations lead at a consulting firm and asked to write a persuasive brief for a client considering a major hybrid-work rollout. The brief emphasized decision quality, productivity, employee well-being, talent attraction and client outcomes.

Criterion	Score 1	Score 2	Score 3	Score 4
A. Context & Purpose	Little/no problem framing; unclear audience; substantial off-task content.	Vague or generic framing; audience needs are mostly implicit; parts feel templated or only loosely tied to the decision context.	Frames the problem and audience reasonably well; mild mismatch in emphasis but remains on task.	Clearly frames the decision problem; explicitly tailors to the intended decision-maker; stays tightly aligned to the task.
B. Recommendation Clarity	No clear recommendation/position.	Stance is implied or wobbly; proposal details are underspecified or contradictory.	Clear stance, but key specifics (scope/model) are incomplete, delayed, or mildly inconsistent.	States a clear recommendation early and specifies what is being proposed (scope, model, key design choices); provides a concise summary of the rationale.
C. Argument Quality	Disorganized, internally inconsistent, or largely non-argumentative.	Relies on assertions or one-step reasoning; weak linkage between points.	Mostly logical and persuasive; some reasoning gaps or unexamined assumptions.	Builds a coherent chain from claims $\rightarrow$ reasons $\rightarrow$ support; makes assumptions/warrants legible.
D. Evidence and Credibility	Virtually no support; repeated overclaims; credibility is questionable.	Mostly generalities; evidence is thin/generic; numbers (if any) feel ungrounded.	Some evidence, but uneven (claims without support in places).	Uses specific supporting material (examples, comparisons); avoids overclaims; support is relevant and credible.
E. Feasibility	No implementation thinking; proposal would be difficult to execute as described.	Implementation is largely aspirational; feasibility concerns are waved away; limited operational detail.	Mentions implementation with some concrete steps; gaps remain (timeline vague, risks not fully addressed, success metrics thin).	Presents a realistic implementation path: key steps, timeline or pilot, operational coverage plan, governance/roles, and how success will be measured.
F. Impact	Ignores stakeholder impacts or makes one-sided claims without considering harms.	Narrow focus (e.g., only employee wellbeing or only productivity); description of impacts are minimal.	Discusses stakeholder impacts, but not deeply; consequences lightly addressed.	Identifies key stakeholders (employees, managers); anticipates impacts and (unintended) consequences; links change to outcomes.

Appendix

Table A1. Essay-scoring rubric for the seven shared criteria (A–G), used by the human raters and supplied to the LLM judge.

Comparison	Measure	$N$	Estimate	95% CI
Prompt coding: goal clarity (0–3)
R1–R2	$\kappa_{w}$	60	0.703	[0.463, 0.844]
R1–R3	$\kappa_{w}$	60	0.587	[0.315, 0.755]
R2–R3	$\kappa_{w}$	60	0.654	[0.368, 0.813]
LLM–R1	$\kappa_{w}$	60	0.458	[0.269, 0.613]
LLM–R2	$\kappa_{w}$	60	0.435	[0.196, 0.604]

Appendix

Table A2. Prompt-coding reliability on the 60-sequence subset (original categorical codebook ratings), human–essay-judge agreement, and reliability across five essay-judge runs. R1–R3 are the three human coders; LLM is the automated coder applying the same codebook.

Code	Label	Definition	Coding cue
GC-0	Absent	No discernible goal or purpose is expressed. Prompts consist of bare topics or instructions so vague that the intended writing outcome cannot be inferred.	No explicit purpose, audience or constraint.
GC-1	Implicit	A general purpose can be inferred from context, but it is never stated explicitly. The user delegates full goal definition to the AI.	Genre or topic implies a goal, but the user does not specify it.
GC-2	Moderate	At least one explicit goal is stated, but the conversation does not specify audience, constraints or evaluative criteria in enough detail to make the target unambiguous.	Purpose or genre is named, but supporting constraints are thin.
GC-3	High	Clear purpose is articulated plus at least two of: audience, constraints, format/tone or evaluative criteria. The intended outcome is unambiguous and may develop across turns.	A full rhetorical problem is made explicit.

Appendix

Table A3. Goal-clarity dimension-level codes

Code	Label	Definition	Coding cue
TD-0	Monolithic	The writing task is presented as a single undifferentiated request. No sub-tasks, steps or structural divisions are discernible.	“Write about climate change.”
TD-1	Partially decomposed	More than one component or aspect of the task is named or implied, but without specified ordering or relations among those components.	Components are listed but not organized.
TD-2	Structured	The task is explicitly organized into ordered steps, labeled sections or a sequence of sub-tasks. Relationships between parts are specified at one level.	Numbered or ordered steps are present.
TD-3	Hierarchical	The task is decomposed at multiple levels: top-level goals are broken into sub-goals, which are themselves broken into operations or priorities. Dependencies across levels are indicated.	Multi-level structure with priorities or dependencies.

Appendix

Table A4. Task-decomposition dimension-level codes

Field	Value
Conversation ID
Number of user turns
Writing task type
GC rating	GC-0 / GC-1 / GC-2 / GC-3
GC-P (purpose)	0 / 1
GC-A (audience)	0 / 1

Appendix

Table A5. Blank coding sheet

Item	Wording [response options]
Baseline survey (before Task 1)
Prior AI use
AI-use 1	How often do you use AI tools like ChatGPT? [once per month / once per week / a few times per week / daily / a few times daily]
AI-use 2	What do you use AI tools like ChatGPT for? [professional writing / creative writing / research / coding / general knowledge / content creation / other (free text)]
General metacognitive regulation, adapted from the Metacognitive Awareness Inventory ( Schraw and Dennison, 1994 ) (1 = does not describe me at all, 5 = describes me very much)
Meta 1	I set specific goals before I begin a task

Appendix

Table A6. Survey items by stage, in the order administered. Response scales are given in the block headings or in square brackets after the item. Items shown with a leading ellipsis were presented under a shared introductory stem in the survey form. Item numbers within blocks match those used in Table A12 and Table A11 . Participant-identifier fields and the completion-code screen are omitted.

	Primary model			+ log time on task
Term or contrast	Coef.	s.e.	$P$	Coef.	s.e.	$P$
Intercept	$2.753$	0.088	$<.001$	$2.761$	0.157	$<.001$
Task 1: scaffold vs control	$0.089$	0.034	.008	$0.089$	0.035	.011
Scaffold $\times$ second task	$-0.019$	0.043	.665	$-0.019$	0.044	.663
Second vs first task (control)	$0.024$	0.031	.440	$0.023$	0.032	.472
Topic: city workweek vs hybrid launch	$0.251$	0.021	$<.001$	$0.251$	0.022	$<.001$

Appendix

Table A7. Mixed-effects models for primary and time-on-task robustness models, adjusted for the six common baseline covariates.

Metric	Estimate	95% CI
Essays compared	107	—
Absolute-agreement ICC(A,1), human versus judge	0.456	[0.349, 0.547]
Pearson $r$ , human versus judge	0.662	—
Calibration slope, judge on human	0.375	—

Appendix

Table A8. Agreement between the LLM essay judge and the human benchmark on the 107 human-rated essays not used as calibration examples. Human scores are the mean of two raters on the formatted essays; judge scores are the mean of five runs on the de-formatted essays. All statistics are computed on essay-level means across the seven shared rubric dimensions (A–G). The calibration slope is from a regression of judge on human scores; a slope below 1 indicates that the judge compresses the range of human scores.

Task	Scaffold-assigned $M$ (SD)	Control $M$ (SD)	Raw difference	Hedges’ $g$	Adjusted difference	Adjusted standardized difference
Task 1	2.99 (0.32)	2.90 (0.32)	0.094	0.29	0.089	0.28
Task 2	2.98 (0.29)	2.92 (0.32)	0.063	0.21	0.070	0.23

Appendix

Table A9. Standardized essay-quality effect sizes (de-formatted judge;

n=297

)

Outcome	Task	Sample	OR [95% CI]	$P$	Observed rates (control vs scaffold)
Topic recall	Task 1	Complete case	$5.07$ [ $1.21$ , $21.25$ ]	.026	45.6% vs 64.6%
Topic recall	Task 1	Multiple imputation ( $m=50$ )	$3.24$ [ $1.11$ , $9.49$ ]	.032	—
Topic recall	Task 2	Complete case	$4.24$ [ $1.02$ , $17.57$ ]	.046	38.0% vs 54.4%
Topic recall	Task 2	Multiple imputation ( $m=50$ )	$3.06$ [ $1.04$ , $9.02$ ]	.042	—

Appendix

Table A10. Topic-recall model summaries. OR is the treatment effect on the odds of correct topic recall (scaffold versus control for Task 1; previously scaffolded versus control for Task 2), from logistic mixed models with a participant random intercept, condition, task position, their interaction, scenario and the six common covariates ( Section 3.4.6 ). Observed rates are control versus scaffold.

Predictor	Definition / scoring	$\alpha$	OR [95% CI]	$P$
Treatment	Scaffold = 1; control = 0	–	$1.65$ [ $0.75$ , $3.73$ ]	.216
First-task scenario	City workweek first = 1; hybrid-work first = 0	–	$1.55$ [ $0.73$ , $3.32$ ]	.252
AI-use frequency	Single baseline item, 1–5	–	$1.18$ [ $0.88$ , $1.59$ ]	.264
AI use for writing	Writing-related AI use = 1; otherwise = 0	–	$1.01$ [ $0.49$ , $2.05$ ]	.983
Essay-writing skill	Single baseline item, 1–5	–	$1.02$ [ $0.76$ , $1.36$ ]	.895
Baseline general metacognition	Mean of metacognition items 1–8; 1–5	0.862	$1.27$ [ $0.88$ , $1.85$ ]	.208

Appendix

Table A11. Firth-penalized logistic model of missing scored recall records: predictors, scoring and estimates. OR is the change in the odds of a participant’s recall record being missing, per SD of each standardized continuous predictor or for the coded category of each binary predictor, with all predictors entered jointly (

N=234

participants with complete predictor data); OR

>1

indicates higher odds of missingness.

\alpha

: Cronbach’s alpha of composite predictors; NE: not estimated. Six further candidate indicators (writing-cohort membership, ambiguous exposure, missing post-task form, missing debrief form, Task 1 and Task 2 essay submitted) were constant across participants and were excluded.

No.	Source	Item	Control pre	Control post	Scaffold pre	Scaffold post	$b$	95% CI	$P$	$P_{\mathrm{BH}}$	$N$
1	MAI	Set specific goals	3.72 (0.89)	3.77 (1.10)	3.68 (0.95)	3.91 (1.00)	$0.190$	[-0.019, 0.398]	.074	.241	290
2	MAI	Monitor goal progress	3.68 (0.99)	3.91 (1.01)	3.59 (1.06)	3.81 (0.96)	$-0.051$	[-0.257, 0.154]	.624	.811	280
3	MAI	Organize time around goals	3.86 (0.94)	3.83 (0.97)	3.79 (1.04)	3.68 (1.09)	$-0.090$	[-0.311, 0.132]	.427	.710	280
4	MAI	Evaluate goal attainment	3.88 (0.91)	4.20 (0.80)	3.73 (0.99)	4.03 (0.82)	$-0.102$	[-0.265, 0.061]	.220	.476	279
5	MAI	Break task into steps	3.83 (1.00)	3.24 (1.16)	3.98 (0.90)	4.03 (0.84)	$0.753$	[0.546, 0.959]	$<.001$	$<.001$	295
6	MAI	Ask questions before starting	3.71 (1.06)	3.89 (1.09)	3.70 (1.00)	3.87 (0.97)	$0.024$	[-0.182, 0.229]	.821	.931	292

Appendix

Table A12. Item-level self-reported metacognition before and after Task 1

Outcome	Control $M$	Scaffold $M$	$b$ [95% CI]	$P$
AI helpfulness	4.77	4.40	$-0.342$ [ $-0.510$ , $-0.174$ ]	$<.001$
Perceived control	4.09	3.78	$-0.251$ [ $-0.486$ , $-0.017$ ]	.036

Appendix

Table A13. Subjective-outcome summaries after Task 1

Human-AI Collaboration

Human-AI collaboration studies how people and AI systems divide, coordinate, and adapt work to support decisions, creativity, and learning.

See all 446 papers

Structured pre-generation elicitation versus single-shot prompting in AI-assisted enterprise decision-making: a randomised online experiment
Oct 7, 2026William Scott-JacksonAI-Assisted Decision Making Generative AI Evaluation
Oxford Centre for Impact Research
When AI performs, what remains? An integrative review of generative AI and cognitive capital in individuals and organizations
Oct 28, 2026Giuseppe Riva, Stefania La Rocca, Brenda K. Wiederhold +1Generative AI Adoption Human-AI Collaboration
Humane Technology Lab., Catholic University of Sacred Heart, Milan, Italy · Applied Technology for Neuro-Psychology Lab., IRCCS Istituto Auxologico Italiano, Milan, Italy · Virtual Reality Medical Center, Scripps Memorial Hospital, La Jolla, California, USA +1
Beyond "ChatGPT Can Make Mistakes": Designing Interventions to Support Metacognitive Monitoring in AI-Assisted Work
Sep 15, 2026Manuel A. D. Santos, Paul Thiesse, Steeven Villa +4Human-AI Interaction Human-AI Decision Making
Aalto University, Espoo, Finland · LMU Munich, Munich, Germany · Department of Computer Science, Aalto University, Espoo, Finland

Oct 7, 2026J/K move · Enter open · S save

Structured pre-generation elicitation versus single-shot prompting in AI-assisted enterprise decision-making: a randomised online experiment

William Scott-Jackson

Oxford Centre for Impact Research

AI-Assisted Decision Making Generative AI Evaluation

Open paper AskPDF arXiv

Generative AI speeds, and mostly improves, professional work, but there is concern that users who delegate both the production and the evaluation of an answer may accept weak output and engage less with the underlying reasoning (cognitive surrender). Interventions proposed so far, such as unassisted practice or slowing adoption, sit outside the working task. We tested a different approach: an interactive metacognitive scaffolding layer (Cognistance, a prototype developed at the Oxford Centre for Impact Research (OCIR) that asks users to clarify context, choose a strategic direction and explain their reasoning before the AI generates a deliverable). Mean composite quality was 32% higher with the scaffold, with the same direction for every rater. Gains were largest for trade-off articulation and strategic coherence and absent for technical specificity. A large part of the aggregate effect reflected rescue of weak prompts: floor-scored (off-task) deliverables fell from 34% to 5%. Among participants whose own prompt already stated the data-localisation problem, the advantage was 21%. Treatment participants reported greater involvement and took about 2.4 minutes longer on average (10.46 minutes). Immediate recall scores were higher, which tentatively suggests better retention, but in this limited experiment, was not robust to sensitivity analyses. Self-ratings of quality did not track rated quality in either condition. Structured elicitation before generation improved the rated quality and task relevance of AI-assisted strategy documents at modest cost in time. Delayed retention, error detection and effects in live organisations are the priorities for the next stage of research.

Abstract

Human-AI Collaboration

Explore similar work

Structured pre-generation elicitation versus single-shot prompting in AI-assisted enterprise decision-making: a randomised online experiment

When AI performs, what remains? An integrative review of generative AI and cognitive capital in individuals and organizations

Beyond "ChatGPT Can Make Mistakes": Designing Interventions to Support Metacognitive Monitoring in AI-Assisted Work