Does AI-Generated Scientific Text Follow Human Argumentation Patterns? A CARS-Based Comparison of Research Article Introductions

Abdelrahman Sadallah; Narjes Sheikh Asadi; Lonneke van der Plas

cs.CLOct 1, 2026

Does AI-Generated Scientific Text Follow Human Argumentation Patterns? A CARS-Based Comparison of Research Article Introductions

Authors: Abdelrahman Sadallah, Narjes Sheikh Asadi, Lonneke van der Plas

Organizations: Università della Svizzera italiana (USI)

Abstract

Large language models are moving from helping write up research to helping do it, which makes it important to know how the scientific text they produce differs from human writing. Work on this question has stayed mostly at the surface, using lexical and stylistic cues that light paraphrasing erases. We look instead at rhetorical structure, the sequence of argumentative moves through which a text makes its case. We study research-article introductions under Swales' CARS model, and compare original introductions from published linguistics articles with generated counterparts of the same papers. We find that human-written introductions are more flexible in which moves they use and in what order, while the generated ones are more uniform. Giving the models the CARS definitions makes them more rigid.

Figures & tables

Figure 1: Swales’ CARS steps on this paper’s own introduction, cut into consecutive segments , each labeled with one CARS step. A segment’s color hue represents the parent move, and the shade represents the step within the move. Move 1, establishing a territory (purple): claiming centrality , topic generalizations , reviewing previous research . Move 2, establishing a niche (magenta): indicating a gap . Move 3, occupying the niche (blue): announcing present research , outlining purposes , announcing principal findings .

Figure 2: Overview of our framework. From each of the

76

corpus papers we reconstruct the content and the works cited in the original introduction. Four LLMs write each paper’s introduction under a basic and a CARS-guided prompt, giving

608

generated introductions. An automatic annotator then labels all original and generated introductions with CARS moves and steps. These labels feed the comparison of the original and generated introductions and the manual analysis.

	Original	ChatGPT b	ChatGPT c	Claude b	Claude c	Gemma b	Gemma c	Qwen b	Qwen c
Move 1: Establishing a Territory	59.5	70.1*	62.9	57.5	51.1*	60.6	46.8*	54.4	48.3*
Claiming centrality (CC)	18.5	21.9*	20.4	19.2	17.3	19.9	17.1	16.0	15.8
Topic generalization (MTG)	14.5	22.3*	19.3*	14.0	11.5	15.7	9.9*	13.0	10.0*
Reviewing prev. research (RR)	26.5	25.9	23.1*	24.3	22.3*	25.0	19.9*	25.4	22.5*
Move 2: Establishing a Niche	16.0	14.6	17.9*	19.9*	22.8*	21.1*	22.1*	22.3*	26.6*
Continuing a tradition (CT)	3.6	3.8	4.3	3.8	4.1	6.6*	6.0*	4.6	4.3

Table 1: Distribution of the moves and steps: each cell is the percentage of all segments produced by the source (pooled over its

76

introductions) that carry the given label. Original is the human-written introduction. Subscripts:

b

= basic,

c

= CARS-guided prompting. * marks a value that differs significantly from the original’s (Appendix F ).

Source	Top step 3-gram (%)	Cyclic %	Uniq. moves	Uniq. steps
Original	RR ${\rightarrow}$ MTG ${\rightarrow}$ RR (25)	66	2.67	5.3
ChatGPT b	CC ${\rightarrow}$ MTG ${\rightarrow}$ RR (57)	70	2.93*	6.3*
ChatGPT c	CC ${\rightarrow}$ MTG ${\rightarrow}$ RR (61)	63	3.00*	6.9*
Claude b	RR ${\rightarrow}$ CC ${\rightarrow}$ RR (33)	74	2.95*	6.2*
Claude c	CC ${\rightarrow}$ RR ${\rightarrow}$ Gap (42)	51	3.00*	6.6*
Gemma b	RR ${\rightarrow}$ CC ${\rightarrow}$ RR (37)	67	2.96*	6.1*

Table 2: Step ordering profile. Top step 3-gram : the source’s most frequent sequence of three consecutive steps, followed in brackets by the percentage of the source’s introductions that contain it. Cyclic % : percentage of introductions that return to a move they had already left. Uniq. moves and Uniq. steps : mean number of distinct moves (out of

3

) and of distinct steps (out of

10

) that one introduction draws on.

	Similarity to the original				Within source
Source	Edit m	Edit s	Bag m	Bag s	Self s	Str m
Original	–	–	–	–	0.33	36
ChatGPT b	0.62	0.41	0.83	0.63	0.43	18
ChatGPT c	0.61	0.41	0.81	0.62	0.44	9
Claude b	0.61	0.42	0.81	0.63	0.40	24
Claude c	0.61	0.43	0.80	0.61	0.46	11

Table 3: Graph comparison of move structures (Appendix E.2 defines the measures). Left: mean similarity of each generated RAIs to the same paper’s original, as edit similarity (Edit) and bag similarity (Bag), at the move (

m

) and step (

s

) level. Right: mean pairwise step edit similarity among the source’s own introductions (Self s ) and its number of distinct move structures (Str m ).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Move	Steps
M1 : Establishing a Territory	Claiming centrality (CC); Topic generalization (MTG); Reviewing previous research (RR)
M2 : Establishing a Niche	Continuing a tradition (CT); Indicating a gap (Gap); Question raising (QR)
M3 : Occupying the Niche	Announcing present research (APR); Announcing principal findings (APF); Indicating structure (IS); Outlining purpose (OP)

Appendix

Table 4: The Swales CARS move/step taxonomy used for annotation (

3

moves,

10

steps). The acronym after each step is the short form used in the other tables, written as move-step (e.g. M1-CC).

	Classification			Segmentation
Annotator	step-acc	step-F1	move-acc	bound-F1
HB	0.752	0.699	0.911	0.756
AA	0.694	0.664	0.901	0.725

Appendix

Table 5: The automatic annotator (AA) vs. the human baseline (HB), both scored against the expert gold on the same

32

introductions. The two column groups score the two decisions separately. Classification scores which rhetorical function a piece of text realises: step-acc and step-F1 are the accuracy and the macro-averaged F1 of the per-sentence step label, and move-acc is the same accuracy after aggregating each sentence’s step to its parent move. Segmentation scores where the rhetorical function changes, independently of the labels: bound-F1 is boundary F1 with a

\pm 5

-character tolerance. Appendix E.1 gives the full definitions.

Class	Human	Annotator
M1-CC	0.70	0.46
M1-MTG	0.63	0.55
M1-RR	0.86	0.84
M2-CT	0.23	0.32
M2-Gap	0.77	0.77
M2-QR	0.42	0.35

Appendix

Table 6: Per-class F1 on the

32

-paper validation set, scored against the expert gold. Human is the second-annotator baseline; Annotator is our automatic annotator. Classes are named move-step as in Table 4 .

Source	Words /intro	Segments /intro	Words /segment
Original (human)	615	9.4	65.1
ChatGPT, basic	743	13.9	53.5
ChatGPT, CARS	745	13.7	54.4
Claude, basic	655	11.4	57.3
Claude, CARS	653	9.9	66.1
Gemma, basic	563	10.5	53.7

Appendix

Table 7: Descriptive statistics of the original and generated introductions (

76

each): mean words per introduction, mean number of segments, and mean words per segment.

	Source	Uniq. moves	Uniq. steps	Cyclic %
Q1	Original	2.88	5.0	75
Q1	Generated	2.99	6.1	73
Q2	Original	2.65	5.7	70
Q2	Generated	2.97	7.2	74
Q3	Original	2.50	5.3	55
Q3	Generated	2.97	6.7	66

Appendix

Table 8: CARS repertoire and cyclicity by journal quartile. Uniq. moves : mean number of distinct moves (of

3

) per introduction; Uniq. steps : mean number of distinct steps (of

10

) per introduction; Cyclic % : percentage of introductions that return to a move they had already left. Generated pools the eight generated sets.

Figure 3: The annotation interface used in the human detection study (§ 5.2 ). A single introduction is shown, with its automatic CARS move annotations as coloured highlights in the labelled condition, above the five-point provenance question and the required decision-basis question. The colour key at the top lists only the moves that occur in the item on screen. The judge sees the whole introduction by scrolling; the text is cut off at the foot of the screen here.

Condition	Items	Correct
All items	120	108 ( $90\%$ )
By move annotation (all items)
Labelled	60	54 ( $90\%$ )
Unlabelled	60	54 ( $90\%$ )
Generated items, by prompting setting
Basic	30	24 ( $80\%$ )

Appendix

Table 9: Results of the human detection study (§ 5.2 ). Labelled / Unlabelled : whether the item was shown with its automatic CARS move annotations as coloured highlights. Basic / CARS-guided : the prompting setting of the generated item. Correct counts judgements that match the true provenance, so the percentage is the accuracy.

Explore similar work

Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation
Aug 19, 2025Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay +1Ai-Assisted Writing Narratives
University of Illinois Chicago · Adobe Research
LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation
May 25, 2026Jiabei Xiao, Yizhou Wang, Chen Tang +3Artificial Intelligence Scientists Ai-Assisted Writing
1The Chinese University of Hong Kong · 2Shanghai Artificial Intelligence Laboratory · 3Fudan University
Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers
Sep 30, 2026Yerim Oh, Young-Jun Lee, Jaewoo Ahn +2Scientific Discovery
Seoul National University · University of Minnesota

Aug 19, 2025, cs.CLJ/K move · Enter open · S save

Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay, Cornelia Caragea

University of Illinois Chicago · Adobe Research

Ai-Assisted Writing Narratives

Open paper AskPDF arXiv

As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential. We introduce Scientific Introduction Generation (SciIG), a task that evaluates LLMs' ability to produce coherent introductions from titles, abstracts, and related works. Curating new datasets from NAACL 2025 and ICLR 2025 papers, we assess five state-of-the-art models, including both open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems, across multiple dimensions: lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality. Our comprehensive framework combines automated metrics with LLM-as-a-judge evaluations. Results demonstrate LLaMA-4 Maverick's superior performance on most metrics, particularly in semantic similarity and faithfulness. Moreover, three-shot prompting consistently outperforms fewer-shot approaches. These findings provide practical insights into developing effective research writing assistants and set realistic expectations for LLM-assisted academic writing. To foster re- producibility and future research, we publicly release all code and datasets.

Abstract

Explore similar work

Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers