Feeling Easier, Working Harder: Cognitive Load in AI-Assisted Code Comprehension
Authors: Maryangela White, Maximilian Rose, Matthew Buckman, Amy McDonnell, Francisco Ortega, Bianca Trinkenreich
Organizations: Computer Science Department, Colorado State University, Fort Collins, United States · Psychology Department, Colorado State University, Fort Collins, United States
Software developers spend significant time reading and understanding code, increasingly supplemented by generative AI. While these tools aim to reduce the difficulty of reading code, interacting with them introduces a dual-task environment that may impose extraneous cognitive load that is challenging to capture. This study investigates how access to generative AI affects developers' real-time cognitive workload, code comprehension performance, and self-perceived code comprehension strategies. We conducted a 3x2 within-subjects study with 20 novice CS students reviewing Java code snippets of varying lengths with or without AI assistance, followed by an AI-unassisted comprehension quiz. Cognitive workload was continuously captured using a Detection-Response Task (tDRT) during both reading and quiz phases, and compared to post-quiz NASA-TLX and SEQ scores. AI assistance significantly increased objective cognitive workload, raising tDRT reaction times by 11.2%, yet participants' perceived workload remained unchanged. Code comprehension scores were high under both conditions, with no evidence that AI assistance improved comprehension at the difficulty levels evaluated; a ceiling effect limited power to detect benefits on harder tasks. Qualitatively, participants reported trusting the AI conditionally -verifying its output against their own reading for more complex snippets, but directly trusting it for simpler code- and, while many felt their reading process was unchanged, a notable subset shifted toward skimming or reading the AI's explanations in place of the code itself. These findings reveal that developers do not accurately perceive the cognitive resources they expend when managing AI tools. This study provides evidence of added cognitive effort associated with using generative AI, pointing towards a need to account for this dual-task penalty in software engineering tools and education.
Figures & tables
Attribute
Category
Percentage (%)
Age
18–20
45%
21–23
30%
24–26
10%
27–29
10%
30+
5%
Java Experience
< 1 year
5%
TABLE I: Participant Demographic and Experience Summary ( N=20 )
Fig. 1: tDRT Set Up
Fig. 2: Study Procedure Diagram
Fig. 3: Study Set Up
Measurement
Variable Type
Source
tDRT reaction times
Dependent
tDRT Device [ 20 ]
Quiz scores
Dependent
Comprehension Quiz [ 2 ]
TLX and SEQ scores
Dependent
Post Quiz Survey [ 17 , 36 ]
Background questions
Independent
Post Study Survey [ 2 ]
TABLE II: Study Measurements
Task Order
Code Size & ID
AI Assist.
1
Small1
No
2
Small2
Yes
3
Medium2
Yes
4
Medium1
No
5
Large2
Yes
6
Large1
No
TABLE III: Experimental Protocol for Participant #1
Model
DV
Tested Effect
χ2
DF
p-value
Model 1
RT
AI Assistance
17.37
1
<.001∗∗
Model 2
RT
AI × Task Phase
7.10
1
0.008∗∗
Model 3
RT
AI × Code Size
8.22
2
0.016∗
Model 4
RT
AI × Java Exp
0.89
3
0.828
Model 5
TLX Score
AI Assistance
1.80
1
0.180
Model 6
SEQ Score
AI Assistance
1.96
1
0.161
TABLE IV: tDRT & Questionnaire LRT Results
Fig. 4: Main Effect of AI Assistance on Cognitive Workload
Fig. 5: Interaction between AI Assistance and Task Phase on Cognitive Workload
Fig. 6: Interaction between AI Assistance and Code Size on Cognitive Workload
Measure
AI Assisted
No AI
Mean accuracy, 0–4 (SD)
3.22 (0.99)
3.23 (1.08)
Complete comprehension (all 4 facets) ∗
50.0%
58.3%
TABLE V: Code Comprehension Performance by Condition
AI-based code assistants are transforming software development, yet we lack empirical evidence on how they affect developers' cognitive processes. We present a multisite study investigating the neurophysiological correlates of AI-assisted programming through a within-subjects crossover design. We recruited participants at two universities (Bari, Italy, and Copenhagen, Denmark) and collected electroencephalography, eye-tracking, electrodermal activity, and heart rate variability data alongside a rubric-based performance score and self-reported workload across six dimensions using the NASA Task Load Index (NASA-TLX). We tested four hypotheses addressing physiological differences between AI-assisted and non-assisted conditions, the moderating role of developer experience, the association between physiology and performance, and the alignment between subjective perceptions and objective measures. Under AI assistance, the EEG θ/α ratio was lower during the first task and the gaze blink rate was higher during the second, both consistent with reduced cognitive engagement when developers offload generative effort to the model. This pattern did not differ between undergraduate and graduate students. Electrodermal activity correlated with performance under the non-AI condition but not under AI. Among the six NASA-TLX dimensions of self-reported workload, only Physical demand was associated with performance under the non-AI condition but not under AI. These findings suggest that AI-assisted programming is not a faster version of solo coding but a cognitively distinct activity, with implications for the design of AI assistants and for biometric monitoring in AI-augmented development.
Paolo Burelli, Fabio Calefato, Daniela Grassi +5
IT University of Copenhagen, Denmark · University of Bari, Italy
AI is changing how software engineers work, but it often comes with hidden burdens and costs. In this paper, we characterize two such often-overlooked burdens: (1) the constant need for human oversight and inspection of AI-generated artifacts; and (2) the growing cognitive overload on software engineers from receiving large amounts of suggestions from AI tools. The need for human oversight is not optional-engineers must review, validate, and sometimes rework what AI produces. At the same time, the flood of AI suggestions, prompts, and possible solutions can leave developers mentally stretched. By blending evidence from recent opinions from practitioners, we highlight these often-overlooked challenges and open a conversation about how teams can handle them in day-to-day AI-assisted software engineering.
Vahid Garousi
Queen’s University Belfast, UK · Azerbaijan Technical University, Azerbaijan
The democratization of Large Language Models has given rise to vibe coding, where novice programmers prioritize semantic intent over syntactic implementation. Without pedagogical guardrails, we argue this is fundamentally misaligned with cognitive skill acquisition. Drawing on Kirschner's distinction between cognitive offloading and outsourcing, unrestricted AI encourages novices to outsource the intrinsic cognitive load required for schema formation rather than merely offloading extraneous load. This accumulation of epistemic debt creates fragile experts: developers whose high functional utility masks critically low corrective competence. To quantify and mitigate this debt, we conducted a between-subjects experiment (N=78) using a custom Cursor IDE plugin backed by Claude 3.5 Sonnet. Participants were recruited via Prolific and UserInterviews.com to represent AI-native learners. We compared three conditions: manual (control), unrestricted AI (outsourcing), and scaffolded AI (offloading). The scaffolded condition employed a novel Explanation Gate -- a real-time LLM-as-a-Judge framework enforcing a teach-back protocol before generated code could be integrated. Results reveal a collapse of competence: both AI groups significantly outperformed the manual control on functional utility (p < .001) and did not differ from each other (p = .64), yet unrestricted AI users suffered a 77% failure rate on a subsequent 30-minute AI-blackout maintenance task, vs. only 39% in the scaffolded group. Qualitative analysis suggests successful vibe coders naturally self-scaffold, treating AI as a consultant rather than a contractor. We discuss implications for AI-generated software maintainability and propose that future learning systems must enforce metacognitive friction to prevent mass production of unmaintainable code. Replication package: https://github.com/sreecharansankaranarayanan/vibecheck
Sreecharan Sankaranarayanan
Extuitive Inc. (Flagship Pioneering) Cambridge, Massachusetts, United States of America