AI-based writing assistants are ubiquitous, yet little is known about how users' mental models shape their use. We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works -- and how they affect control behavior -- how users request, accept, or edit AI suggestions as they write -- and writing outcomes. We primed participants (N=48) with different system descriptions to induce these mental models before asking them to complete a cover letter writing task using a writing assistant that occasionally offered preconfigured ungrammatical suggestions to test whether the mental models affected participants' critical oversight. We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect: while these participants judged the system as more usable, they also produced letters with more grammatical errors, highlighting a complex relationship between system understanding, trust, and control in contexts that require user oversight of error-prone AI outputs.
Figures & tables
Research Question
Expected outcome
Hypotheses
RQ1: How do different mental models affect the way users exert control in the writing process?
Participants with a structural mental model will demonstrate more effective and deliberate control, steering the system rather than passively accepting suggestions.
H1a: Structural participants will request more AI -generated suggestions. H1b: Structural participants will accept a greater proportion of suggestions. H1c: Structural participants will make more edits to the accepted suggestions. H1d: Structural participants will issue more instructions to steer suggestions. H1e: Structural and functional participants will contribute an equal amount of original text.
RQ2: How do different mental models held by users influence the quality of the final written output?
Participants with a structural mental model will produce higher-quality cover letters with fewer errors and better overall quality.
H2a: Structural participants will achieve higher grammar correctness scores and accept fewer erroneous suggestions. H2b: Structural participants’ letters will receive higher overall quality ratings.
RQ3: How do these mental models shape users’ overall experience of writing with the AI -based writing assistant?
Participants with a structural mental model will report more positive perceptions of the writing process and system, including greater perceived control, ownership, and satisfaction.
H3a: Structural participants will report higher perceived usefulness and ease of use. H3b: Structural participants will rate the quality of their final outputs more highly. H3c: Structural participants will report a stronger sense of control. H3d: Structural participants will report a greater sense of ownership.
Table 1. Research questions, expected outcome, and hypotheses
Figure 1. This figure illustrates the experimental design and procedure. White boxes indicate stages where participants completed various surveys. Gray boxes represent phases where participants either familiarized themselves with the platform or engaged in the writing task. The black box marks the manipulation point in the experiment. Flowchart of the experimental procedure. Participants first completed the consent form and the demographics survey (5 min), then familiarization period #1 (5 min), followed by either a functional or structural video (5–10 min, manipulation point). Next came familiarization period #2 (5 min), an understanding survey (5 min), the writing task (20 min), and finally a post-task survey and interview (10–15 min). White boxes indicate surveys, gray boxes indicate familiarization or writing, and the black box marks the manipulation.
Figure 2. A screenshot of the AI-based writing assistant with the text editor where the main letter is written in the center, the Instruction Box is on the left side, and a basic summary of how to use the platform is on the right side. The interface is a modified version of the open-source CoAuthor platform created by Lee et al. (2022) . Screenshot of the modified CoAuthor platform. A large text editor occupies the center where participants write the cover letter. On the left is a small instruction box where participants can write meta-prompts to modify the suggestions that appear in the text editor when they press the tab button. On the right, a sidebar gives a basic summary of how to use the platform.
Content
Functional
Structural
What is the CoAuthor platform?
✓
✓
What are the key features of the CoAuthor platform?
✓
✓
What are the main functionalities of the CoAuthor platform?
✓
✓
Usage of text editor (feature 1)
✓
✓
How does CoAuthor generate its suggestions?
✗
✓
Examples of how content in the text editor affects the suggestions generated
✗
✓
Table 2. The content of the video description for the two experimental conditions
#
Question
Correct Answer
1
CoAuthor is a writing assistant that does not rely on AI .
False
2
CoAuthor offers phrases and sentences when a user requests suggestions.
True
3
Where should the user write the main body of their cover letter on the CoAuthor platform?
Text editor
4
CoAuthor will automatically generate suggestions for the user when the user stops typing.
False
5
CoAuthor filters and trims the original list of suggestions generated by the large language model before showing it to the user.
True
6
CoAuthor gives suggestions based on what is a likely continuation of the last sentence in the text editor.
True
Table 3. The seven questions in the manipulation survey and their corresponding correct answers. Questions 5–7 assessed knowledge of internal system mechanisms described only in the structural video.
Hypothesis
Construct(s)
Data Source
Specific Measures (calculated per participant and letter)
H1a
Requesting suggestions
Interaction logs
Suggestion requests: total number of requests for suggestions divided by word count.
H1b
Accepting suggestions
Interaction logs
Acceptance ratio: accepted suggestions divided by total suggestion requests.
H1c
Editing accepted suggestions
Interaction logs
Edit ratio: number of deleted characters within the first 20 recorded edit actions after each accepted suggestion, divided by the total number of characters in all accepted suggestions.
H1d
Steering via instructions
Interaction logs
Instruction count: total number of instructions the user inputs divided by the word count.
H1e
Direct writing contribution
Interaction logs
User character ratio: total number of user-written characters divided by total characters.
H2a
Grammatical correctness of the letter and accepted erroneous suggestions
Grammarly scores of the final letter, Interaction logs
Calibrated corrections: Grammarly correctness score, defined as the number of corrections suggested by Grammarly divided by word count; Error ratio: total number of accepted suggestions with errors divided by total accepted suggestions; Errors per word: total number of accepted suggestions containing grammatical errors divided by word count.
Table 4. Mapping of hypotheses to constructs and measures (quantitative analysis)
Figure 3. Bar graphs of mean values (with error bars) for five control behavior measures—suggestion requests, acceptance ratio, edit ratio, instruction count, and user character ratio—across the functional and structural mental model conditions. Figure 3 shows five bar graphs with error bars comparing mean values of control behavior measures across functional and structural mental model conditions. The measures include calibrated suggestion requests, acceptance ratio, edit ratio, instruction count, and user character ratio. Each graph compares the functional condition (white bars) and the structural condition (striped gray bars). Across all five measures, the mean values between conditions are similar, and no statistically significant differences are observed.
Figure 4. Bar graphs of mean values, with error bars, for calibrated corrections identified by Grammarly in final letters, error ratio, and accepted erroneous suggestions, comparing the functional and structural mental model conditions. An asterisk (*) denotes statistically significant differences ( p<0.05 ) for calibrated corrections and errors per word. Three bar graphs with error bars comparing functional (white bars) and structural (striped gray bars) mental model conditions. (A) Calibrated corrections per word measure shows higher values in the structural condition, marked significant. (B) Error ratio measure also trends higher for structural but not significant. (C) Errors per word measure shows higher values in the structural condition, marked significant.
Figure 5. Bar graph of mean values (and error bars) for rubric-based cover letter quality ratings across dimensions of relevance, flow, and tone/style for the functional and structural conditions. Bar graph showing mean rubric-based cover letter quality ratings, with error bars, across the dimensions of relevance, flow, and tone/style for two conditions. Ratings are high for both the functional (white bars) and structural (striped gray bars) mental model conditions, with similar values across all dimensions.
Figure 6. Bar graphs of mean values (and error bars) for self-reported ratings across conditions for usefulness, ease of use, perceived quality, perceived ownership, and perceived control. A one-sided independent samples t -test revealed a difference in ease of use , with a moderate effect size ( d=0.60 ). As marked on the left-most plot, this effect was statistically significant at the uncorrected α=.05 level, but did not meet the Bonferroni-corrected threshold of α=.01 . Three grouped bar graphs with error bars comparing functional (white bars) and structural (striped gray bars) conditions on self-reported ratings. On the left, the first graph shows usefulness (composite) and ease of use, with structural condition rated significantly higher for ease of use (*) at the uncorrected alpha level (α=.05). The second graph shows perceived quality, including suggestions being error-free and letter style and content, with structural condition values being slightly higher but with no significant differences. The third graph shows ownership and control measures, including perceived ownership (composite) and four control dimensions, with similar ratings across conditions.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Criterion
Rating anchor (5 = Excellent), (1 = Poor)
Demonstration of relevant skills and experience
The letter clearly highlights the candidate’s skills, knowledge, and experiences that directly relate to the job posting. Examples are used to support claims.
Flow and clarity
The flow of the letter is appropriate for a cover letter and the writing is clear. The content transitions smoothly from one idea to the next. Sentences and paragraphs are logically organized, making the letter easy to read and understand without confusion or abrupt shifts.
Tone and style
The writing has an appropriate and compelling tone and style for a cover letter. The letter has a professional tone and the candidate’s voice comes through as confident, enthusiastic, and sincere.
Appendix
Table 5. Cover letter assessment rubric used by evaluators.
Criterion
Response
Tone or style shift
Tone is focused on the emotional stance and attitude the writer brings through in their writing. Style is broader than tone, and captures how the writer expresses their message using different words, unique voice, and structure. For example, “use positive language” shifts tone; “make words professional” or “decrease monotony” shifts style.
Yes / No
Qualifications articulation
Does the instruction articulate the candidate’s relevant qualities, qualifications, or skills? For example, “add graduate degree at McGill University” is specific, whereas “highlight great asset” is too broad.
Yes / No
Structure prompting
Does the instruction cue the structure of the cover letter (e.g., ending, section transition)? For example, “conclusive sentence” or “write something about company.”
Yes / No
Appendix
Table 6. Instruction coding rubric used to annotate user-written instructions.
Q
Functional (MM = 1)
Structural (MM = 2)
Mean Rank
U
Z
p
Mean
SD
Median
Mean
SD
Median
MM=1
MM=2
1
1.00
0.00
1.00
0.79
0.41
1.00
27.00
22.00
228.0
-2.34
.019
2
1.00
0.00
1.00
0.96
0.20
1.00
25.00
24.00
276.0
-1.00
.317
3
0.96
0.20
1.00
0.96
0.20
1.00
24.50
24.50
288.0
0.00
1.000
4
1.00
0.00
1.00
1.00
0.00
1.00
24.50
24.50
288.0
0.00
1.000
5
0.21
0.41
0.00
0.79
0.41
1.00
17.50
31.50
120.0
-4.00
<.001
Appendix
Table 7. Descriptive statistics and Mann–Whitney U tests for manipulation check questions by condition.
Measure
Functional condition
Structural condition
Test
Statistic
p
Effect size
Suggestion requests (calibrated)
M=0.071 , SD=0.047
M=0.074 , SD=0.036
Mann–Whitney U
U=248 , Z=−0.83
.409
r=0.12
Acceptance ratio
M=0.651 , SD=0.188
M=0.685 , SD=0.161
t(46)
t=−0.67
.253
d=−0.19
Edit ratio
M=0.173 , SD=0.154
M=0.179 , SD=0.153
Mann–Whitney U
U=280 , Z=−0.17
.869
r=0.02
Instruction count
M=0.028 , SD=0.021
M=0.024 , SD=0.016
Mann–Whitney U
U=271 , Z=−0.35
.726
r=0.05
User character ratio
M=0.609 , SD=0.227
M=0.578 , SD=0.187
t(46)
t=0.51
.305
d=0.15
Appendix
Table 8. Full statistical results for interaction measures.
Instruction dimension
Functional (0 / 1)
Structural (0 / 1)
Total N
χ2(1)
p
Tone / style shift
154 / 13
133 / 15
315
0.54
.464
Qualifications articulation
51 / 105
34 / 102
315
1.30
.254
Structure prompting
137 / 30
123 / 25
315
0.06
.802
Appendix
Table 9. Distribution of coded instruction dimensions by condition (instruction-level).
Instruction dimension
Functional ( n )
Structural ( n )
Total N
Binomial p
Tone / style shift (present)
13
15
28
.851
Qualifications articulation (present)
112
108
220
.840
Structure prompting (present)
30
25
55
.590
Appendix
Table 10. Binomial tests evaluating deviations from a 50/50 distribution across conditions for each instruction dimension.
Dependent variable
Effect
F(1,44)
p
Partial η2
Error ratio
Condition (MMNumeric)
2.36
.132
.051
Native language
1.58
.216
.035
Condition × Language
0.41
.526
.009
Errors per word
Condition (MMNumeric)
3.37
.073
.071
Native language
1.32
.258
.029
Condition × Language
0.36
.554
.008
Appendix
Table 11. Two-way ANOVA results for erroneous suggestion acceptance measures, controlling for native language.
Measure
Functional condition
Structural condition
Test
Statistic
p
Effect size
Calibrated corrections
M=0.016 , SD=0.010
M=0.023 , SD=0.010
t(46)
t=−2.32
.013
d=0.67
Error ratio
M=0.146 , SD=0.119
M=0.226 , SD=0.157
Mann–Whitney U
U=204.5 , Z=−1.73
.084
r=0.25
Errors per word
M=0.006 , SD=0.007
M=0.011 , SD=0.008
Mann–Whitney U
U=170 , Z=−2.44
.015
r=0.35
Appendix
Table 12. Grammatical correctness and erroneous suggestion acceptance measures by condition.
Measure
Functional condition
Structural condition
Test
Statistic
p
Effect size
Relevance (Q1)
M=3.67 , SD=0.87
M=3.83 , SD=0.96
Mann–Whitney U
U=249 , Z=−0.85
.398
r=−0.12
Flow (Q2)
M=4.04 , SD=0.81
M=3.96 , SD=0.91
Mann–Whitney U
U=280.5 , Z=−0.16
.869
r=−0.02
Tone / style (Q3)
M=4.17 , SD=0.64
M=4.04 , SD=0.96
Mann–Whitney U
U=285.5 , Z=−0.06
.954
r=−0.01
Appendix
Table 13. Cover letter quality ratings by condition.
Scale / Item
Type
Functional condition
Structural condition
Test statistic
p (one-sided)
Usefulness ( α=.76 )
Usefulness (composite)
Scale mean
M=3.89 , SD=0.56
M=4.01 , SD=0.55
t(46)=−0.78
.220
Suggestions were relevant
Item
M=3.88 , SD=0.68
M=4.04 , SD=0.46
t(46)=−0.99
.163
Suggestions were helpful
Item
M=3.96 , SD=0.69
M=4.13 , SD=0.68
t(46)=−0.84
.202
Customizing suggestions using the Instruction Box was helpful
Item
M=3.75 , SD=0.79
M=3.79 , SD=0.66
t(46)=−0.20
.422
Able to customize the provided suggestions
Item
M=3.96 , SD=0.96
M=4.08 , SD=0.83
t(46)=−0.48
.315
Appendix
Table 14. Composite self-report scales and constituent items by condition.
Measure
Functional condition
Structural condition
t(46)
p (one-sided)
CoAuthor was easy to use
M=3.88 , SD=1.04
M=4.38 , SD=0.58
−2.07
.022
The final letter is error-free
M=3.38 , SD=1.06
M=3.46 , SD=1.22
−0.25
.400
The final letter has appropriate style and content
M=3.96 , SD=1.00
M=4.25 , SD=0.79
−1.12
.134
The suggestions influenced the content of the cover letter
M=3.17 , SD=1.17
M=3.50 , SD=0.93
−1.09
.140
I felt in control of the quality and content of the suggestions
M=3.25 , SD=1.07
M=2.92 , SD=1.18
1.03
.155
I felt in control of what was included in the final letter
M=4.38 , SD=0.92
M=4.46 , SD=0.66
−0.36
.360
Appendix
Table 15. Single-item self-report measures by condition.
Hundreds of millions of people use AI for writing assistance. We asked how AI distorts perceptions of AI users and their opinions. Writers (N=2,939) wrote political opinion paragraphs with and without AI assistance. Separate readers (N=11,091) blindly evaluated these paragraphs and made inferences about the writers. With AI, writers seemed more opinionated, competent, positive, and demographically privileged. Writers objected to many of these distortions yet preferred AI-assisted text even when made aware of them. Reward models trained on our data mitigated objectionable distortions but reduced user acceptance. Additional readers (N=8,798) trusted AI-assisted writers more and were more persuaded by AI writing when AI was more distortive. Overall, distortions from AI writing assistance are pervasive, persistent under human oversight, and likely to have consequential effects on human behaviours and attitudes.
Paul Röttger, Kobi Hackenburg, Hannah Rose Kirk +1
Emerging experimental evidence shows that writing with AI assistance can change both the views people express in writing and the opinions they hold afterwards. Yet, we lack substantive understanding of procedural and behavioral changes in co-writing with AI that underlie the observed opinion-shaping power of AI writing tools. We conducted a mixed-methods study, combining retrospective interviews with 19 participants about their AI co-writing experience with a quantitative analysis tracing engagement with ideas and opinions in 1{,}291 AI co-writing sessions. Our analysis shows that engaging with the AI's suggestions -- reading them and deciding whether to accept them -- becomes a central activity in the writing process, taking away from more traditional processes of ideation and language generation. As writers often do not complete their own ideation before engaging with suggestions, the suggested ideas and opinions seeded directions that writers then elaborated on. At the same time, writers did not notice the AI's influence and felt in full control of their writing, as they -- in principle -- could always edit the final text. We term this shift Reactive Writing: an evaluation-first, suggestion-led writing practice that departs substantially from conventional composing in the presence of AI assistance and is highly vulnerable to AI-induced biases and opinion shifts.
Advait Bhat, Marianne Aubin Le Quéré, Mor Naaman +1
Paul G. Allen School of Computer Science, University of Washington · Center for Information Technology Policy, Princeton University · Cornell Tech +1
Expert writing feedback from experienced researchers is critical for early-career scholars to improve their manuscripts, yet high-quality feedback often remains scarce because reviewing research papers is labor-intensive. Emerging AI-powered writing assistants largely focus on grammar fixes or simulating peer review with final scores, yet they fall short of providing concrete, actionable suggestions that help students improve their papers during drafting. We present PaperMentor, a human-centered writing assistant system that delivers actionable suggestions as Overleaf-native inline comments while leaving the actual writing entirely to human authors. PaperMentor integrates an expert skill library carefully curated from established researchers' writing advice with 12 specialized agents covering different aspects of paper writing, such as formatting compliance, phrasing accuracy, and terminology consistency. In a user study (n=14), 90.6% of the generated comments were rated actionable and 67.5% were rated valid, significantly outperforming a GPT-5.2 baseline uswithout the skill library. We release PaperMentor as open source for public use. Our code is publicly available under the AGPL-3.0 license at https://github.com/jiarui-liu/overleaf
Jiarui Liu, Terry Jingchen Zhang, Ryan Faulkner +17
1CMU · 2Jinesis Lab, University of Toronto & Vector Institute · 3EuroSafeAI +4