From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants

Shalaleh Rismani; Su Lin Blodgett; Q. Vera Liao; Alexandra Olteanu; AJung Moon

cs.HCApr 6, 2026

From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants

Authors: Shalaleh Rismani, Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, AJung Moon

Organizations: McGill University Montréal, Canada · University of Michigan Ann Arbor, Michigan, USA

Abstract

AI-based writing assistants are ubiquitous, yet little is known about how users' mental models shape their use. We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works -- and how they affect control behavior -- how users request, accept, or edit AI suggestions as they write -- and writing outcomes. We primed participants ( $N = 48$ ) with different system descriptions to induce these mental models before asking them to complete a cover letter writing task using a writing assistant that occasionally offered preconfigured ungrammatical suggestions to test whether the mental models affected participants' critical oversight. We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect: while these participants judged the system as more usable, they also produced letters with more grammatical errors, highlighting a complex relationship between system understanding, trust, and control in contexts that require user oversight of error-prone AI outputs.

Figures & tables

Research Question	Expected outcome	Hypotheses
RQ1: How do different mental models affect the way users exert control in the writing process?	Participants with a structural mental model will demonstrate more effective and deliberate control, steering the system rather than passively accepting suggestions.	H1a: Structural participants will request more AI -generated suggestions. H1b: Structural participants will accept a greater proportion of suggestions. H1c: Structural participants will make more edits to the accepted suggestions. H1d: Structural participants will issue more instructions to steer suggestions. H1e: Structural and functional participants will contribute an equal amount of original text.
RQ2: How do different mental models held by users influence the quality of the final written output?	Participants with a structural mental model will produce higher-quality cover letters with fewer errors and better overall quality.	H2a: Structural participants will achieve higher grammar correctness scores and accept fewer erroneous suggestions. H2b: Structural participants’ letters will receive higher overall quality ratings.
RQ3: How do these mental models shape users’ overall experience of writing with the AI -based writing assistant?	Participants with a structural mental model will report more positive perceptions of the writing process and system, including greater perceived control, ownership, and satisfaction.	H3a: Structural participants will report higher perceived usefulness and ease of use. H3b: Structural participants will rate the quality of their final outputs more highly. H3c: Structural participants will report a stronger sense of control. H3d: Structural participants will report a greater sense of ownership.

Table 1. Research questions, expected outcome, and hypotheses

Figure 1. This figure illustrates the experimental design and procedure. White boxes indicate stages where participants completed various surveys. Gray boxes represent phases where participants either familiarized themselves with the platform or engaged in the writing task. The black box marks the manipulation point in the experiment. Flowchart of the experimental procedure. Participants first completed the consent form and the demographics survey (5 min), then familiarization period #1 (5 min), followed by either a functional or structural video (5–10 min, manipulation point). Next came familiarization period #2 (5 min), an understanding survey (5 min), the writing task (20 min), and finally a post-task survey and interview (10–15 min). White boxes indicate surveys, gray boxes indicate familiarization or writing, and the black box marks the manipulation.

Figure 2. A screenshot of the AI-based writing assistant with the text editor where the main letter is written in the center, the Instruction Box is on the left side, and a basic summary of how to use the platform is on the right side. The interface is a modified version of the open-source CoAuthor platform created by Lee et al. (2022) . Screenshot of the modified CoAuthor platform. A large text editor occupies the center where participants write the cover letter. On the left is a small instruction box where participants can write meta-prompts to modify the suggestions that appear in the text editor when they press the tab button. On the right, a sidebar gives a basic summary of how to use the platform.

Content	Functional	Structural
What is the CoAuthor platform?	✓	✓
What are the key features of the CoAuthor platform?	✓	✓
What are the main functionalities of the CoAuthor platform?	✓	✓
Usage of text editor (feature 1)	✓	✓
How does CoAuthor generate its suggestions?	✗	✓
Examples of how content in the text editor affects the suggestions generated	✗	✓

Table 2. The content of the video description for the two experimental conditions

#	Question	Correct Answer
1	CoAuthor is a writing assistant that does not rely on AI .	False
2	CoAuthor offers phrases and sentences when a user requests suggestions.	True
3	Where should the user write the main body of their cover letter on the CoAuthor platform?	Text editor
4	CoAuthor will automatically generate suggestions for the user when the user stops typing.	False
5	CoAuthor filters and trims the original list of suggestions generated by the large language model before showing it to the user.	True
6	CoAuthor gives suggestions based on what is a likely continuation of the last sentence in the text editor.	True

Table 3. The seven questions in the manipulation survey and their corresponding correct answers. Questions 5–7 assessed knowledge of internal system mechanisms described only in the structural video.

Hypothesis	Construct(s)	Data Source	Specific Measures (calculated per participant and letter)
H1a	Requesting suggestions	Interaction logs	Suggestion requests: total number of requests for suggestions divided by word count.
H1b	Accepting suggestions	Interaction logs	Acceptance ratio: accepted suggestions divided by total suggestion requests.
H1c	Editing accepted suggestions	Interaction logs	Edit ratio: number of deleted characters within the first 20 recorded edit actions after each accepted suggestion, divided by the total number of characters in all accepted suggestions.
H1d	Steering via instructions	Interaction logs	Instruction count: total number of instructions the user inputs divided by the word count.
H1e	Direct writing contribution	Interaction logs	User character ratio: total number of user-written characters divided by total characters.
H2a	Grammatical correctness of the letter and accepted erroneous suggestions	Grammarly scores of the final letter, Interaction logs	Calibrated corrections: Grammarly correctness score, defined as the number of corrections suggested by Grammarly divided by word count; Error ratio: total number of accepted suggestions with errors divided by total accepted suggestions; Errors per word: total number of accepted suggestions containing grammatical errors divided by word count.

Table 4. Mapping of hypotheses to constructs and measures (quantitative analysis)

Figure 3. Bar graphs of mean values (with error bars) for five control behavior measures—suggestion requests, acceptance ratio, edit ratio, instruction count, and user character ratio—across the functional and structural mental model conditions. Figure 3 shows five bar graphs with error bars comparing mean values of control behavior measures across functional and structural mental model conditions. The measures include calibrated suggestion requests, acceptance ratio, edit ratio, instruction count, and user character ratio. Each graph compares the functional condition (white bars) and the structural condition (striped gray bars). Across all five measures, the mean values between conditions are similar, and no statistically significant differences are observed.

Figure 4. Bar graphs of mean values, with error bars, for calibrated corrections identified by Grammarly in final letters, error ratio, and accepted erroneous suggestions, comparing the functional and structural mental model conditions. An asterisk (*) denotes statistically significant differences ( $p<0.05$ ) for calibrated corrections and errors per word. Three bar graphs with error bars comparing functional (white bars) and structural (striped gray bars) mental model conditions. (A) Calibrated corrections per word measure shows higher values in the structural condition, marked significant. (B) Error ratio measure also trends higher for structural but not significant. (C) Errors per word measure shows higher values in the structural condition, marked significant.

Figure 5. Bar graph of mean values (and error bars) for rubric-based cover letter quality ratings across dimensions of relevance, flow, and tone/style for the functional and structural conditions. Bar graph showing mean rubric-based cover letter quality ratings, with error bars, across the dimensions of relevance, flow, and tone/style for two conditions. Ratings are high for both the functional (white bars) and structural (striped gray bars) mental model conditions, with similar values across all dimensions.

Figure 6. Bar graphs of mean values (and error bars) for self-reported ratings across conditions for usefulness, ease of use, perceived quality, perceived ownership, and perceived control. A one-sided independent samples $t$ -test revealed a difference in ease of use , with a moderate effect size ( $d=0.60$ ). As marked on the left-most plot, this effect was statistically significant at the uncorrected $\alpha=.05$ level, but did not meet the Bonferroni-corrected threshold of $\alpha=.01$ . Three grouped bar graphs with error bars comparing functional (white bars) and structural (striped gray bars) conditions on self-reported ratings. On the left, the first graph shows usefulness (composite) and ease of use, with structural condition rated significantly higher for ease of use (*) at the uncorrected alpha level ($\alpha= .05$). The second graph shows perceived quality, including suggestions being error-free and letter style and content, with structural condition values being slightly higher but with no significant differences. The third graph shows ownership and control measures, including perceived ownership (composite) and four control dimensions, with similar ratings across conditions. — Figure 6. Bar graphs of mean values (and error bars) for self-reported ratings across conditions for usefulness, ease of use, perceived quality, perceived ownership, and perceived control. A one-sided independent samples $t$ -test revealed a difference in ease of use , with a moderate effect size ( $d=0.60$ ). As marked on the left-most plot, this effect was statistically significant at the uncorrected $\alpha=.05$ level, but did not meet the Bonferroni-corrected threshold of $\alpha=.01$ . Three grouped bar graphs with error bars comparing functional (white bars) and structural (striped gray bars) conditions on self-reported ratings. On the left, the first graph shows usefulness (composite) and ease of use, with structural condition rated significantly higher for ease of use (*) at the uncorrected alpha level ( $\alpha= .05$ ). The second graph shows perceived quality, including suggestions being error-free and letter style and content, with structural condition values being slightly higher but with no significant differences. The third graph shows ownership and control measures, including perceived ownership (composite) and four control dimensions, with similar ratings across conditions.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Criterion	Rating anchor (5 = Excellent), (1 = Poor)
Demonstration of relevant skills and experience	The letter clearly highlights the candidate’s skills, knowledge, and experiences that directly relate to the job posting. Examples are used to support claims.
Flow and clarity	The flow of the letter is appropriate for a cover letter and the writing is clear. The content transitions smoothly from one idea to the next. Sentences and paragraphs are logically organized, making the letter easy to read and understand without confusion or abrupt shifts.
Tone and style	The writing has an appropriate and compelling tone and style for a cover letter. The letter has a professional tone and the candidate’s voice comes through as confident, enthusiastic, and sincere.

Appendix

Table 5. Cover letter assessment rubric used by evaluators.

Criterion	Response
Tone or style shift
Tone is focused on the emotional stance and attitude the writer brings through in their writing. Style is broader than tone, and captures how the writer expresses their message using different words, unique voice, and structure. For example, “use positive language” shifts tone; “make words professional” or “decrease monotony” shifts style.	Yes / No
Qualifications articulation
Does the instruction articulate the candidate’s relevant qualities, qualifications, or skills? For example, “add graduate degree at McGill University” is specific, whereas “highlight great asset” is too broad.	Yes / No
Structure prompting
Does the instruction cue the structure of the cover letter (e.g., ending, section transition)? For example, “conclusive sentence” or “write something about company.”	Yes / No

Appendix

Table 6. Instruction coding rubric used to annotate user-written instructions.

Q	Functional (MM = 1)			Structural (MM = 2)			Mean Rank		U	Z	p
	Mean	SD	Median	Mean	SD	Median	MM=1	MM=2
1	1.00	0.00	1.00	0.79	0.41	1.00	27.00	22.00	228.0	-2.34	.019
2	1.00	0.00	1.00	0.96	0.20	1.00	25.00	24.00	276.0	-1.00	.317
3	0.96	0.20	1.00	0.96	0.20	1.00	24.50	24.50	288.0	0.00	1.000
4	1.00	0.00	1.00	1.00	0.00	1.00	24.50	24.50	288.0	0.00	1.000
5	0.21	0.41	0.00	0.79	0.41	1.00	17.50	31.50	120.0	-4.00	<.001

Appendix

Table 7. Descriptive statistics and Mann–Whitney U tests for manipulation check questions by condition.

Measure	Functional condition	Structural condition	Test	Statistic	$p$	Effect size
Suggestion requests (calibrated)	$M=0.071$ , $SD=0.047$	$M=0.074$ , $SD=0.036$	Mann–Whitney $U$	$U=248$ , $Z=-0.83$	.409	$r=0.12$
Acceptance ratio	$M=0.651$ , $SD=0.188$	$M=0.685$ , $SD=0.161$	$t(46)$	$t=-0.67$	.253	$d=-0.19$
Edit ratio	$M=0.173$ , $SD=0.154$	$M=0.179$ , $SD=0.153$	Mann–Whitney $U$	$U=280$ , $Z=-0.17$	.869	$r=0.02$
Instruction count	$M=0.028$ , $SD=0.021$	$M=0.024$ , $SD=0.016$	Mann–Whitney $U$	$U=271$ , $Z=-0.35$	.726	$r=0.05$
User character ratio	$M=0.609$ , $SD=0.227$	$M=0.578$ , $SD=0.187$	$t(46)$	$t=0.51$	.305	$d=0.15$

Appendix

Table 8. Full statistical results for interaction measures.

Instruction dimension	Functional (0 / 1)	Structural (0 / 1)	Total $N$	$\chi^{2}(1)$	$p$
Tone / style shift	154 / 13	133 / 15	315	0.54	.464
Qualifications articulation	51 / 105	34 / 102	315	1.30	.254
Structure prompting	137 / 30	123 / 25	315	0.06	.802

Appendix

Table 9. Distribution of coded instruction dimensions by condition (instruction-level).

Instruction dimension	Functional ( $n$ )	Structural ( $n$ )	Total $N$	Binomial $p$
Tone / style shift (present)	13	15	28	.851
Qualifications articulation (present)	112	108	220	.840
Structure prompting (present)	30	25	55	.590

Appendix

Table 10. Binomial tests evaluating deviations from a 50/50 distribution across conditions for each instruction dimension.

Dependent variable	Effect	$F(1,44)$	$p$	Partial $\eta^{2}$
Error ratio	Condition (MMNumeric)	2.36	.132	.051
	Native language	1.58	.216	.035
	Condition $\times$ Language	0.41	.526	.009
Errors per word	Condition (MMNumeric)	3.37	.073	.071
	Native language	1.32	.258	.029
	Condition $\times$ Language	0.36	.554	.008

Appendix

Table 11. Two-way ANOVA results for erroneous suggestion acceptance measures, controlling for native language.

Measure	Functional condition	Structural condition	Test	Statistic	$p$	Effect size
Calibrated corrections	$M=0.016$ , $SD=0.010$	$M=0.023$ , $SD=0.010$	$t(46)$	$t=-2.32$	.013	$d=0.67$
Error ratio	$M=0.146$ , $SD=0.119$	$M=0.226$ , $SD=0.157$	Mann–Whitney $U$	$U=204.5$ , $Z=-1.73$	.084	$r=0.25$
Errors per word	$M=0.006$ , $SD=0.007$	$M=0.011$ , $SD=0.008$	Mann–Whitney $U$	$U=170$ , $Z=-2.44$	.015	$r=0.35$

Appendix

Table 12. Grammatical correctness and erroneous suggestion acceptance measures by condition.

Measure	Functional condition	Structural condition	Test	Statistic	$p$	Effect size
Relevance (Q1)	$M=3.67$ , $SD=0.87$	$M=3.83$ , $SD=0.96$	Mann–Whitney $U$	$U=249$ , $Z=-0.85$	.398	$r=-0.12$
Flow (Q2)	$M=4.04$ , $SD=0.81$	$M=3.96$ , $SD=0.91$	Mann–Whitney $U$	$U=280.5$ , $Z=-0.16$	.869	$r=-0.02$
Tone / style (Q3)	$M=4.17$ , $SD=0.64$	$M=4.04$ , $SD=0.96$	Mann–Whitney $U$	$U=285.5$ , $Z=-0.06$	.954	$r=-0.01$

Appendix

Table 13. Cover letter quality ratings by condition.

Scale / Item	Type	Functional condition	Structural condition	Test statistic	$p$ (one-sided)
Usefulness ( $\alpha=.76$ )
Usefulness (composite)	Scale mean	$M=3.89$ , $SD=0.56$	$M=4.01$ , $SD=0.55$	$t(46)=-0.78$	.220
Suggestions were relevant	Item	$M=3.88$ , $SD=0.68$	$M=4.04$ , $SD=0.46$	$t(46)=-0.99$	.163
Suggestions were helpful	Item	$M=3.96$ , $SD=0.69$	$M=4.13$ , $SD=0.68$	$t(46)=-0.84$	.202
Customizing suggestions using the Instruction Box was helpful	Item	$M=3.75$ , $SD=0.79$	$M=3.79$ , $SD=0.66$	$t(46)=-0.20$	.422
Able to customize the provided suggestions	Item	$M=3.96$ , $SD=0.96$	$M=4.08$ , $SD=0.83$	$t(46)=-0.48$	.315

Appendix

Table 14. Composite self-report scales and constituent items by condition.

Measure	Functional condition	Structural condition	$t(46)$	$p$ (one-sided)
CoAuthor was easy to use	$M=3.88$ , $SD=1.04$	$M=4.38$ , $SD=0.58$	$-2.07$	.022
The final letter is error-free	$M=3.38$ , $SD=1.06$	$M=3.46$ , $SD=1.22$	$-0.25$	.400
The final letter has appropriate style and content	$M=3.96$ , $SD=1.00$	$M=4.25$ , $SD=0.79$	$-1.12$	.134
The suggestions influenced the content of the cover letter	$M=3.17$ , $SD=1.17$	$M=3.50$ , $SD=0.93$	$-1.09$	.140
I felt in control of the quality and content of the suggestions	$M=3.25$ , $SD=1.07$	$M=2.92$ , $SD=1.18$	$1.03$	.155
I felt in control of what was included in the final letter	$M=4.38$ , $SD=0.92$	$M=4.46$ , $SD=0.66$	$-0.36$	.360

Appendix

Table 15. Single-item self-report measures by condition.

Human Oversight of AI

Human oversight of AI studies how people supervise, verify, and control AI decisions, especially as systems become more autonomous and consequential.

See all 172 papers

How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinions
Apr 24, 2026Paul Röttger, Kobi Hackenburg, Hannah Rose Kirk +1Trust in AI Demographic Bias in Language Models
University of Oxford · UK AI Security Institute
Reactive Writers: How Co-Writing with AI Changes How We Engage with Ideas
Mar 11, 2026Advait Bhat, Marianne Aubin Le Quéré, Mor Naaman +1Human-AI Co-Creation Opinion Dynamics
Paul G. Allen School of Computer Science, University of Washington · Center for Information Technology Policy, Princeton University · Cornell Tech +1
PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf
Jun 7, 2026Jiarui Liu, Terry Jingchen Zhang, Ryan Faulkner +17Intelligent Tutoring Systems LLM Agents
1CMU · 2Jinesis Lab, University of Toronto & Vector Institute · 3EuroSafeAI +4

Apr 24, 2026J/K move · Enter open · S save

How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinions

Paul Röttger, Kobi Hackenburg, Hannah Rose Kirk, Christopher Summerfield

University of Oxford · UK AI Security Institute

Trust in AI Demographic Bias in Language Models Human-AI Co-Creation

Open paper AskPDF arXiv

Hundreds of millions of people use AI for writing assistance. We asked how AI distorts perceptions of AI users and their opinions. Writers (N=2,939) wrote political opinion paragraphs with and without AI assistance. Separate readers (N=11,091) blindly evaluated these paragraphs and made inferences about the writers. With AI, writers seemed more opinionated, competent, positive, and demographically privileged. Writers objected to many of these distortions yet preferred AI-assisted text even when made aware of them. Reward models trained on our data mitigated objectionable distortions but reduced user acceptance. Additional readers (N=8,798) trusted AI-assisted writers more and were more persuaded by AI writing when AI was more distortive. Overall, distortions from AI writing assistance are pervasive, persistent under human oversight, and likely to have consequential effects on human behaviours and attitudes.

Abstract

Human Oversight of AI

Explore similar work

How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinions

Reactive Writers: How Co-Writing with AI Changes How We Engage with Ideas

PaperMentor: A Human-Centered Multi-Agent Writing Tutor for AI Research Papers on Overleaf