Feeling Easier, Working Harder: Cognitive Load in AI-Assisted Code Comprehension
Authors: Maryangela White, Maximilian Rose, Matthew Buckman, Amy McDonnell, Francisco Ortega, Bianca Trinkenreich
Organizations: Computer Science Department, Colorado State University, Fort Collins, United States · Psychology Department, Colorado State University, Fort Collins, United States
Software developers spend significant time reading and understanding code, increasingly supplemented by generative AI. While these tools aim to reduce the difficulty of reading code, interacting with them introduces a dual-task environment that may impose extraneous cognitive load that is challenging to capture. This study investigates how access to generative AI affects developers' real-time cognitive workload, code comprehension performance, and self-perceived code comprehension strategies. We conducted a 3x2 within-subjects study with 20 novice CS students reviewing Java code snippets of varying lengths with or without AI assistance, followed by an AI-unassisted comprehension quiz. Cognitive workload was continuously captured using a Detection-Response Task (tDRT) during both reading and quiz phases, and compared to post-quiz NASA-TLX and SEQ scores. AI assistance significantly increased objective cognitive workload, raising tDRT reaction times by 11.2%, yet participants' perceived workload remained unchanged. Code comprehension scores were high under both conditions, with no evidence that AI assistance improved comprehension at the difficulty levels evaluated; a ceiling effect limited power to detect benefits on harder tasks. Qualitatively, participants reported trusting the AI conditionally -verifying its output against their own reading for more complex snippets, but directly trusting it for simpler code- and, while many felt their reading process was unchanged, a notable subset shifted toward skimming or reading the AI's explanations in place of the code itself. These findings reveal that developers do not accurately perceive the cognitive resources they expend when managing AI tools. This study provides evidence of added cognitive effort associated with using generative AI, pointing towards a need to account for this dual-task penalty in software engineering tools and education.
Figures & tables
Attribute
Category
Percentage (%)
Age
18–20
45%
21–23
30%
24–26
10%
27–29
10%
30+
5%
Java Experience
< 1 year
5%
TABLE I: Participant Demographic and Experience Summary ( N=20 )
Fig. 1: tDRT Set Up
Fig. 2: Study Procedure Diagram
Fig. 3: Study Set Up
Measurement
Variable Type
Source
tDRT reaction times
Dependent
tDRT Device [ 20 ]
Quiz scores
Dependent
Comprehension Quiz [ 2 ]
TLX and SEQ scores
Dependent
Post Quiz Survey [ 17 , 36 ]
Background questions
Independent
Post Study Survey [ 2 ]
TABLE II: Study Measurements
Task Order
Code Size & ID
AI Assist.
1
Small1
No
2
Small2
Yes
3
Medium2
Yes
4
Medium1
No
5
Large2
Yes
6
Large1
No
TABLE III: Experimental Protocol for Participant #1
Model
DV
Tested Effect
χ2
DF
p-value
Model 1
RT
AI Assistance
17.37
1
<.001∗∗
Model 2
RT
AI × Task Phase
7.10
1
0.008∗∗
Model 3
RT
AI × Code Size
8.22
2
0.016∗
Model 4
RT
AI × Java Exp
0.89
3
0.828
Model 5
TLX Score
AI Assistance
1.80
1
0.180
Model 6
SEQ Score
AI Assistance
1.96
1
0.161
TABLE IV: tDRT & Questionnaire LRT Results
Fig. 4: Main Effect of AI Assistance on Cognitive Workload
Fig. 5: Interaction between AI Assistance and Task Phase on Cognitive Workload
Fig. 6: Interaction between AI Assistance and Code Size on Cognitive Workload
Measure
AI Assisted
No AI
Mean accuracy, 0–4 (SD)
3.22 (0.99)
3.23 (1.08)
Complete comprehension (all 4 facets) ∗
50.0%
58.3%
TABLE V: Code Comprehension Performance by Condition