CoWriter: Guiding Methodological and Synergetic Human-LLM Co-Authoring
Authors: Shmuel Amar, Benjamin Lieber, Ori Shapira, Debanjana Kar, Sebastian Maximilian Dennerlein, Yufang Hou, Ido Dagan
Organizations: Bar-Ilan University · OriginAI · IT:U Interdisciplinary Transformation University Austria · IBM Software Innovation Lab, India · University of Twente, the Netherlands
While LLMs have transformed text generation, standard chat-and-canvas interfaces often reduce human authoring to quite passive supervision, causing review fatigue, over-reliance on AI, and the sidelining of the human role in long-form, source-grounded writing. Prior systems typically address isolated writing phases, leaving comprehensive end-to-end human-LLM collaboration largely unexplored. In this paper, we introduce CoWriter, a proactive agentic scheme that adapts expert human (co-)authoring methodologies from cognitive science to flexibly steer the entire writing workflow while actively encouraging the user to engage. In an exploratory user study, compared to a standard LLM baseline, authors using CoWriter better expressed their perspectives and maintained a higher sense of document ownership, supported quantitatively by 2.2× more successful agent engagements. Furthermore, external reader judges rated the resulting texts higher in "human touch" qualities like storytelling and integrating unique personal experiences. Overall, our results demonstrate that proactively engaging users through established human practices is beneficial for effective LLM-supported co-authoring, providing useful methodologies, techniques and insights for future research on collaborative agents.
Figures & tables
Figure 1: The CoWriter platform (§ 3 ). The environment comprises a UI that exposes functionality to the user and the agentic system. The Leader proactively guides the iterative authoring workflow ( Table 1 ), outlined within the Leader ’s box. Monitor tracks UI events to update the Writing Log and trigger Leader with system-reminders to maintain the authoring workflow. Reviewer simulates an external reader and provides feedback.
Table 1: The iterative co-authoring workflow steps (§ 3.3.1 ) encoded in the Leader system prompt (adapted from the writing activities of Lowry et al., 2004 ). The per-step invite instruction specifies how the agent encourages engagement. Colors denote workflow phases ( Figure 1 ): planning , production , finalization , and ad-hoc requests .
Figure 2: Percentage of sessions in which each agent (baseline vs. CoWriter ) discussed each of the seven writing activities. Bold bars indicate p<0.05 significance of CoWriter over the baseline.
Figure 3: Comparative preferences between the baseline and CoWriter by 14 authors (normalized counts). ∗ , ∗∗ , ∗∗∗ denote p<0.05,0.01,0.001 , respectively.
Figure 4: Percentage of pairs preferred per system, rated by five human annotators for the four key dimensions. LLM-as-a-judge counterpart in Figure 11 .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The user interface for CoWriter , at the end of a session. 1 The task description. 2 A source with its title and short LLM-generated summary; clicking opens the full source text. 3 The title of the document being authored; the body is co-edited by the user and CoWriter via the Yjs-synchronized canvas. 4 An inline link referring to a span in a source document, to support the written claim. 5 Message-level buttons including a diff view of CoWriter ’s edits. 6 CoWriter ’s thought process (expandable), which shows the reasoning and sequence of actions of Leader in real time. 7 When clicked, opens the Writing Log (see an example of a full log in Figure 21 ).
Tool (signature)
Description
Sources
list_source_files() → SourceFile[]
List metadata for all available source files ( id , type, title, description, status).
get_source_file( id ) → SourceFile
Return a source file’s full content by id or title.
Document
read_document() → XML
Return the current shared document as XML ( <heading> , <paragraph> , inline tags).
update_document( xpath , xml , mode )
Insert, replace, or delete a block node using XPath; mode ∈ { sibling-after , sibling-before , first-child , last-child , replace-node }.
Appendix
Table 2: Tools exposed to the Leader (main) agent. All tool responses are formatted to text for easier consumption by the LLM. The rules for usage of the tools are detailed in the system prompt ( Figure 28 ).
Tool (signature)
Description
get_source_file( id , start =1, end =-1 ) → lines
Read a source file (by id or title), line-numbered and sliced to the inclusive range [start, end] ; used to verify a <deeplink> ’s cited range.
message( content )
Final review, posted by Leader into chat as a quoted block (one per invocation).
Appendix
Table 3: Tools available to the Reviewer sub-agent. The Reviewer’s ReAct loop interleaves get_source_file lookups (to spot-check cited spans against the actual source lines) and a single final message that returns the structured review ( Appendix E ).
Source document
DOI / ERIC ID
Topic 1: The Influence of Family and Friends on Young Adults (Family)
Family and peer influences on adolescent psychological inflexibility: a regression mixture analysis
10.3389/fpsyg.2025.1650930
Parent-child attachment and mental health in young adolescents: a moderated mediation analysis
10.3389/fpsyg.2023.1298485
Family intimacy and adolescent peer relationships: investigating the mediating role of psychological capital and the moderating role of self-identity
10.3389/fpsyg.2023.1165830
Effects of peer relationships on parent-youth relationships and self-differentiation
10.17220/ijpes.2018.03.002
Parenting, Peer Relationships, Academic Self-efficacy, and Academic Achievement: Direct and Mediating Effects
10.3389/fpsyg.2017.02120
Appendix
Table 4: The 40 research papers used as writing sources in the user study, 10 per topic. Each title links to the paper; the DOI (or ERIC identifier, where no DOI is registered) is given for reference.
Figure 9: Pre-study survey responses of the 14 study participants.
Participant
Session 1
Session 2
P1
B ⋅ Family
C ⋅ Smartphones
P2
B ⋅ Family
C ⋅ Education
P3
B ⋅ Smartphones
C ⋅ Family
P4
B ⋅ Education
C ⋅ Family
P5
B ⋅ Education
C ⋅ WLB
P6
C ⋅ Education
B ⋅ WLB
Appendix
Table 5: Per-participant assignment for the 14 study participants (P1–P14). Each participant completed one B aseline session and one CoWriter session ( C ), each on a different topic. System order and the topic–system pairing were counterbalanced across participants. Topics are given by short code. Full titles and sources are in Table 4 .
Construct
Per-session questionnaire statement
Engagement
I felt the system made me actively engaged in writing this document.
The system enabled me to express my perspectives during the authoring process.
The system enabled me to use my choice of content during the authoring process.
The system offered me useful suggestions throughout the authoring process.
I felt I am not part of the writing process. †
Task Load (NASA-TLX)
How mentally demanding was the task? †
Appendix
Table 6: The 37 statements of the per-system questionnaire (§ 4.1 ), grouped by construct. After each writing session, participants rated every statement on a 7-point Likert scale (1–7): strongly disagree , disagree , somewhat disagree , neither agree nor disagree , somewhat agree , agree , strongly agree ; the five Task Load items use a 7-point numeric scale adapted from NASA-TLX ( Hart and Staveland, 1988 ) . Statements marked † are reverse-coded — their scores are inverted <8−score> so that a higher value always denotes a more positive response.
Construct
Comparative questionnaire question
Engagement
Which system made you more engaged in the writing process?
Source Understanding
Which system enabled you to better understand the sources for the needs of the writing?
Cognitive Load
Which system was less mentally demanding to use?
System Guidance
Which system provided clearer steps for writing the document?
Document Factuality
Which system made it easier to verify cited claims against the sources?
Self-Expression
Which system’s final document feels more like your own tone and writing style?
Appendix
Table 7: The 13 questions of the comparative questionnaire, grouped by construct. After completing both writing sessions, participants answered each question on a 5-point comparative scale: System A more , System A slightly more , tied / no preference , System B slightly more , System B more , where System A and System B denote the participant’s first- and second-session systems, respectively.
Figure 10: Preference counts for the five questions with the highest average preference toward CoWriter in the human reader evaluation. LLM-as-a-judge counterpart in Figure 12 .
Figure 11: Percentage of pairs preferred per system across the four key dimensions, as rated by the three LLM judges over the 48 evaluated pairs. Human-rater counterpart in Figure 4 .
Figure 12: Preference counts for the five questions with the highest average preference toward CoWriter in the LLM-as-a-judge reader evaluation.
Aspect
Reader-evaluation question
Interestingness
Which of the two was written in a more appealing storytelling approach?
Which of the two was written in a more friendly tone for the readers?
Which of the two contains more unique ideas or perspectives from the author?
Which of the two uses the personal experience of its writer that produces more unique writing?
Which of the two was written in a more convincing manner?
Which of the two was more enjoyable to read?
Appendix
Table 8: The 19 questions of the human reader questionnaire adapted from ( Lee et al., 2025 ) , grouped by aspect (§ 5.3 ). Annotators answered each question on a 5-point comparative scale: Document A more , Document A slightly more , Tied / No preference , Document B slightly more , Document B more .
System
P
R
Rcited
Runcited
F1
CL
Baseline
0.988 (0.030)
0.741 (0.123)
0.699 (0.111)
0.760 (0.159)
0.841 (0.091)
12754 (925)
CoWriter
0.996 (0.017)
0.743 (0.105)
0.622 (0.117)
0.802 (0.107)
0.847 (0.073)
205 (41)
Appendix
Table 9: Citation correctness of the final documents, evaluated with LongCite ( Zhang et al., 2025 ) : mean over the 14 baseline and 14 CoWriter documents, standard deviation in parentheses. P : citation precision; R : citation recall; F1=2PR/(P+R) ; CL : citation length (mean number of words per cited snippet; a lower CL marks more localized citations). Rcited is recall over statements that cite at least one snippet (judged fully / partially / not supported =1/0.5/0 ); Runcited is recall over statements with no citation ( 1 if the statement is a functional sentence requiring no citation, else 0 ).
Figure 14: Workflow adherence per writing activity (§ 5.1 ). Left: Total chat turns per activity, split into agent and user turns. Right: Percentage of sessions discussing each activity at least once, categorized by participation (both parties, agent only, or user only). While 54% of the baseline agent turns concentrate in localized reviewing & revising (vs. 29% for CoWriter ) with low agent involvement in early stages, CoWriter distributes the conversation more uniformly and successfully drives higher user participation across most writing activities.
Figure 15: Aggregated writing activity transitions for the baseline (top) and CoWriter (bottom) across the 28 co-authoring sessions (§ 5.1 ). Arrows indicate transitions between activities, labeled with their total counts. Activity nodes display the percentage of their incoming transitions that are adjacent (i.e., from an immediately preceding or succeeding activity). While both systems are iterative, each featuring over 30% backward transitions, CoWriter transitions are predominantly to adjacent activities (74% vs. 56% for the baseline; p=0.015 ).
Figure 16: Agent-initiated (left) and user-initiated (right) engagement expressions by type (§ 5.1 ). Light bars show total expressions, while darker subsets denote the portion of successful engagements (i.e., followed up by the other collaborator). While user-initiated engagements are dominated by action requests and remain comparable across systems, CoWriter yields more agent-initiated engagements across all types.
Figure 17: Agent-initiated (left) and user-initiated (right) engagement expressions per writing activity (§ 5.1 ). Light bars show total expressions, while darker subsets denote the portion of successful engagements (i.e., followed up by the other collaborator). While baseline agent-initiated engagements concentrate in localized reviewing & revising (48%), CoWriter distributes engagements much more uniformly across the workflow and achieves higher success rates.
Figure 18: Per-system averaged absolute ratings of the first 19 of the 37 statements ( Table 6 ) on the 7-point Likert scale. The per-statement paired-test significance marker is shown above each pair.
Figure 19: Per-system averaged absolute ratings of the remaining 18 of the 37 statements ( Table 6 ) on the 7-point Likert scale. The per-statement paired-test significance marker is shown above each pair.
Figure 20: Per-question distributions of the 13 comparative questions ( Table 7 ).
P#
System
Moment the system helped
Moment the system got in the way
P1
Baseline
It helped concise the blog post without losing meaningful information
The way I interacted with the system is to let the system take lead in formulating everything upfront and I did the editing. Due to this nature of interaction, there was no way it conflicted with what I did.
P1
CoWriter
It reminded me the missing tasks that I am supposed to do - at the end of the review.Though i didn’t follow through, it was a very important help or reminder.
Nothing
P2
Baseline
help me to write a text within the time limits
used the title of the writing task instead of making a new one
P2
CoWriter
highlight relevant citations
by suggesting a topic ideas that were not factual but match the title, like the importance of formal education on children
P3
Baseline
Focus on the subject
Editing
P3
CoWriter
It helped me aggregate my thoughts in the right direction
No interruption
Appendix
Table 10: Free-text qualitative responses from the per-session questionnaire, one row per (participant, system) and one column per prompt. All 14 participants answered both prompts for both systems. Responses are verbatim.
Metric
Baseline
CoWriter
Sig.
Session duration (min)
67.6 (22.0)
61.6 (19.6)
ns
LLM token cost (USD)
5.57 (5.03)
5.21 (3.80)
ns
User messages count
16.6 (9.7)
22.1 (11.0)
ns
Agent messages count
16.3 (9.9)
25.0 (12.1)
**
Agent total words count
6988 (3897)
5120 (3102)
ns
User revisions count
23.6 (31.5)
14.7 (21.9)
ns
Appendix
Table 11: Per-system automatic measurements across the 14 writing sessions per system. Each cell shows average (standard deviation) across all sessions. Significance (Sig.) markers are from paired within-subject tests on the < proactive − baseline > differences ( p<0.01 ** , p<0.05 * ; ns otherwise); the t-test / Wilcoxon selection rule is described in Appendix N . CoWriter -only metrics report CoWriter sessions.
The rise of human-AI collaborative writing has created a growing need for fine-grained detection methods that support localizing likely LLM-generated content in mixed-authorship documents. Existing methods for detecting LLM-generated text mainly focus on document-level classification and cannot identify which parts of the text are generated by LLMs. This paper introduces a new method to address this urgent need. Our method operates at the token level, the natural unit of modern language models, and builds on existing token-level detection scores. The key idea is to smooth adjacent token scores to reduce their variability, while using an adaptive Lepski-type rule to select the bandwidth according to the local authorship structure. Our method is simple to implement and does not require token-level labeled data for training. Theoretically, we characterize this trade-off and show that the proposed method achieves favorable mean square error performance in estimating the underlying signal. Empirically, we demonstrate strong performance of our method against a wide range of baselines in both synthetic datasets and a realistic dataset. We deploy a publicly accessible website that implements the methods as well.
Yangjun Lu, Hongyi Zhou, Fabian Spill +3
School of Mathematics, University of Birmingham, Birmingham, UK · School of Statistics and Data Science, Shanghai University of Finance and Economics, Shanghai, China · Department of Statistics, The London School of Economics and Political Science, London, UK
LLMs have been rapidly adopted across writing tasks, prompting the development of tools for detecting LLM-generated text. Yet, these tools largely measure how much of a document's surface text was written by an LLM and aren't fundamentally designed to measure how much of the information content or ideas originated from the LLM itself rather than being supplied by the user in the prompt. In this work, we design a framework that measures how much value a person adds on top of what a language model could have easily produced by itself. The method requires no training or labeled data and never scores the document's surface text, insulating it from stylistic confounders. Instead, it extracts the document's content at increasing levels of granularity, uses an LLM to reconstruct the document from each partial representation, and compares these reconstructions with those produced from the task description alone. We call this framework Value Over Language Model (VOLM), which measures a document's contribution relative to a replacement-level document that an LLM could produce from the task description alone. We evaluate VOLM with a specific instantiation of this framework across three domains: news articles, ICLR peer reviews, and argumentative essays. VOLM separates human-authored documents from matched LLM-generated documents produced from generic task descriptions, while remaining substantially invariant to content-preserving transformations, including LLM-based reconstruction and round-trip translation. We further find that increasingly constrained content extractors reduce residual differences between LLM-generated and humanized text, demonstrating the importance of disentangling informational content from stylistic variation. We hope these results encourage further work on specialized instantiations of the framework and on assessing human contributions in LLM-assisted writing more generally.
Large language models (LLMs) are used by over a billion people globally, most often to assist with writing. In this work, we demonstrate that LLMs not only alter the voice and tone of human writing but also consistently alter the intended meaning. First, we conduct a human user study to understand how people actually interact with LLMs when using them for writing. Our findings reveal that extensive LLM use led to a nearly 70% increase in essays that remained neutral in answering the topic question. Significantly more heavy LLM users reported that the writing was less creative and not in their voice. Next, using a dataset of human-written essays that was collected in 2021 before the widespread release of LLMs, we study how asking an LLM to revise the essay based on the human-written feedback in the dataset induces large changes in the resulting content and meaning. We find that even when LLMs are prompted with expert feedback and asked to only make grammar edits, they still change the text in a way that significantly alters its semantic meaning. We then examine LLM-generated text in the wild, specifically focusing on the 21% of AI-generated scientific peer reviews at a recent top AI conference. We find that LLM-generated reviews place significantly less weight on clarity and significance of the research, and assign scores that, on average, are a full point higher. These findings highlight a misalignment between the perceived benefit of AI use and an implicit, consistent effect on the semantics of human writing, motivating future work on how widespread AI writing will affect our cultural and scientific institutions.
Marwa Abdulhai, Isadora White, Yanming Wan +4
UC Berkeley · UC San Diego · University of Washington +2