I would rather quit NLP than read another paper like this: The rise of antithesis in NLP papers

Olga Zamaraeva; Adrián Gude; Roi Santos-Ríos; Carlos Gómez-Rodríguez

cs.CLOct 7, 2026

I would rather quit NLP than read another paper like this: The rise of antithesis in NLP papers

Authors: Olga Zamaraeva, Adrián Gude, Roi Santos-Ríos, Carlos Gómez-Rodríguez

Organizations: Universidade da Coruña, CITIC

Abstract

For better or worse, LLMs are by now used routinely for scientific writing.\footnote{This paper is no exception; we did use AI to assist with writing some of the sections (see Acknowledgments).} Many have noticed that recent models fill papers with unnecessary antithesis, stating over and over what the work does not do, in ways that do not contribute to its precision or quality of expression and annoy reviewers \emph{rather than impressing them}. We study the construction \emph{rather than} in ACL papers from 2019, ACL-style arXiv papers from 2026, and papers written by GPT models from the same titles and abstracts. Its rate in 2026 is seven times the 2019 rate, and higher still in the GPT papers. Two annotators, blind to the source, find almost no 2019 use \emph{annoying} and about one in ten 2026 uses; they seldom agree on which, yet about half of 2026 papers contain a use that annoys each of them. \emph{Annoying} uses present the rejected alternative less favorably than legitimate uses. Raters of preference data and open reward models favor the construction, and an instruction to be honest promotes it. We conjecture that it is a side effect of post-training on pairwise preferences, which credit a disavowal in a single response and cannot register its cost across a text.

Figures & tables

Dataset	Selection	Body text	Papers	Body words		Annotated, by batch
				Mean	Median	1	2	3	4	Total
ACL 2019	Anthology, all venues	pdftotext	4,744	4,575	4,648	148	100	40	50	338
short ( $\leq$ 7 pp.)			1,785	2,949	3,016
long ( $\geq$ 9 pp.)			2,590	5,775	5,673
arXiv 2026	cs.CL , *ACL venue and style	detex	1,821	7,626	7,167	150	238	180		568
short ( $\leq$ 3.5k words)			66	2,541	2,787

Table 1: The datasets. Body text : extraction tool and back-matter cut 7 7 7 The cut is at the References heading in ACL 2019 PDF text, and after the last closing section (Conclusion, Limitations, Ethics, Acknowledgments) in L a T e X source, where reference lists are absent since BibTeX generates them at compile time. It succeeds for 4,680 of 4,744 ACL 2019 and 1,359 of 1,821 arXiv 2026 papers; the rest are kept whole. Leaving them out would change the mean rate of rather than from 0.134 to 0.133 per 1,000 words for ACL 2019, and from 0.993 to 1.030 for arXiv 2026. . Short/long: by published pages (ACL 2019) or body words (arXiv 2026). High-count: papers with 35 or more uses. Batches 1–2: the annotation pool; batch 3: the GPT comparison set; batch 4: the Claude comparison set (A1 only so far; its GPT-6-sol row counts in the GPT row).

Question	Data	Method	Sec.
Q1: How frequent is the construction?	ACL 2019, arXiv 2026, LLM-written papers; arXiv version pairs	Pattern counts per 1,000 words	5.1
Q2: Do readers find it annoying ?	1,000 sampled occurrences	Blind labels from two annotators	5.2
Q3: What correlates with an annoying use?	Labeled arXiv 2026 occurrences	Straw-man codes, stance ratings, syntax, embeddings	5.3
Q4: Where does it come from?	Four preference datasets; UltraFeedback completions; pool sentences	Chosen vs. rejected responses; honesty vs. helpfulness instruction; reward-model minimal pairs	5.4

Table 2: Overview of the analyses and the sections reporting their results.

ID	Annoy.	Straw m.	Auth.	Author	Stage
A1	✓	✓	✓	yes	PhD + 5 yrs
A2	✓	✓	✓	yes	PhD + 17 yrs
A3		✓	✓	yes	PhD student, yr 1
A4		✓		no	PhD in 2026
A5	(51)		✓	yes	PhD student, yr 2

Table 3: Annotators. Annoy. : annoyance labels on the pool, the GPT comparison set and (A1 so far) the Claude comparison set (A5: 51 items, not used). Straw m. : straw-man codes. Auth. : authorship judgments.

Set	Items	N
Development	A1 unsure	20
	legitimate, ACL 2019	10
Test	A1 annoying (all)	41
	A1 legitimate (random)	80

Table 4: Items coded for straw men. Test items are first-batch arXiv 2026 items; development items are outside the test set.

Figure 1: Occurrences per 1,000 words, ACL 2019 vs. arXiv 2026 (1,821 papers).

		Rate		Precision
Type	Pattern	2019	2026	2019	2026
	Papers	4,744	1,821
Substitution	rather than	0.134	0.993	100	100
	instead of	0.165	0.105	100	98
	as opposed to	0.025	0.007	100	100
Negation	not X but Y	0.097	0.126	58	84

Table 5: Antithesis patterns in ACL 2019 and arXiv 2026. Rate : occurrences per 1,000 words. Precision : % of 50 matches per pattern and corpus that GPT-6-sol judged to be instances. Bold (italic): significantly higher (lower) in 2026 (Mann-Whitney,

p<.05

	Words		Papers	Per 1k words
Source	Mean	Median	with RT	RT	X, not Y
200 ACL 2019 papers
Original	4,493	4,354	60	0.11	0.04
GPT-4o	1,473	1,438	20	0.08	0.00
revised	1,342	1,311	17	0.07	0.01
GPT-5.6-sol	8,453	8,412	200	1.62	0.22

Table 6: GPT papers and their originals. RT: rather than . Bold (italic): RT significantly more (less) frequent than in the original (generated) or the generated version (revised); paired Wilcoxon test,

p<.05

	Words		Papers	Per 1k words
Source	Mean	Median	with RT	RT	X, not Y
45 ACL 2019 papers
Original	4,506	4,656	9	0.07	0.06
Tülu 3 8B SFT	1,296	1,393	3	0.04	0.00
revised	1,269	1,295	2	0.02	0.02
Tülu 3 8B DPO	1,551	1,513	6	0.09	0.00

Table 7: Papers written by Tülu 3 8B at three post-training stages and by three Claude models, and their originals: the first 100 source papers of the GPT papers, minus the papers for which any model’s text was unusable (nine, all by Tülu 3 8B SFT, which refused or wrote a stub). RT: rather than . Bold (italic): RT significantly more (less) frequent than in the original (generated) or the generated version (revised); paired Wilcoxon test,

p<.05

Set	Papers	N	A1	A2	Either
Pool	ACL 2019	248	4 (1.6)	0 (0.0)	4 (1.7)
	arXiv 2026	652	75 (11.5)	54 (8.3)	116 (17.9)
	High-count	100	12 (12.0)	12 (12.0)	18 (18.0)
GPT set	ACL 2019	40	0 (0.0)	0 (0.0)	0 (0.0)
	arXiv 2026	180	32 (17.9)	9 (5.0)	35 (19.6)
	GPT-4o	46	4 (8.7)	0 (0.0)	4 (8.7)

Table 8: Items labeled annoying , count (% of the items). Either : annoying for at least one annotator. Italic (bold): significantly lower (higher) than arXiv 2026 in the same set; † higher than first draft (Fisher’s exact test,

p<.05

Between annotators
	Pool	GPT set
Items (garbled by neither)	989	398
Raw agreement, three labels	73.6%	68.8%
Cohen’s $\kappa$ , three labels	0.18	0.16
Cohen’s $\kappa$ , annoying vs. other	0.18	0.20
Gwet’s AC1, annoying vs. other	0.86	0.82

Table 9: Annotator agreement. Positive agreement:

2\cdot\mathrm{both}/(\mathrm{A1}+\mathrm{A2})

P(at least one annoying use per paper)
	ACL 2019	arXiv 2026
Papers	4,744	1,821
with rather than	36%	94%
Uses per paper, mean (median)	0.65 (0)	8.0 (6)
A1	1.1%	51.1%
A2	0 ( $\leq$ 1.0%)	42.1%

Table 10: Papers with at least one use an annotator finds annoying , estimated from the per-use rates. A2, ACL 2019: 95% upper bound, since A2 labeled no ACL 2019 use annoying .

Set	Items	A1: annoying	Gap X $-$ Y
ACL 2019 originals	50	2 (4.0)	$+$ 1.22
Claude	163	26 (16.0)	$+$ 0.81
first draft	88	9 (10.2) ‡	$+$ 0.81
revised	75	17 (22.7) †	$+$ 0.81
Sonnet 5.5	84	16 (19.0)	$+$ 0.90
Opus 5.5	79	10 (12.7)	$+$ 0.72

Table 11: The Claude comparison set, labeled by A1 (A2’s labels will be added): items labeled annoying , count (% of the items), and the mean stance gap X

-

Y of the uses (GPT-6-sol ratings; Section 4.4 ). Bold: significantly more annoying than ACL 2019 uses; † revised more than first draft; ‡ Claude first draft less than GPT-6-sol first draft (Fisher’s exact test,

p<.05

, uncorrected). Across the set, the gap of annoying uses does not differ significantly from that of the others (Appendix Q ).

	Revised	Disavowal	Stance
Revised only	$+$ 0.91 (.043)	–	–
$+$ disavowal	$+$ 0.58 (.28)	$+$ 3.19 ( $<$ .001)	–
$+$ disavowal, stance	$+$ 0.62 (.26)	$+$ 3.25 ( $<$ .001)	$+$ 0.16 (.40)

Table 12: Logistic regression of A1’s annoying label on the 160 uses in Claude papers of the Claude comparison set: coefficients (log-odds) and

p

values for revised versus first draft, for disavowal (both coders), and for the stance gap ( Stance ; mean of two raters).

Codes and labels from different people
Codes / labels	Annoying	Legit.	$p$
A2+A3+A4 / A1	0.28 (29)	0.12 (54)	.007
A3+A4 / A1	0.34 (31)	0.18 (58)	.02
A2 / A1	5/36 (14%)	2/74 (3%)	.04
A3 / A1	16/33 (48%)	18/59 (31%)	.12
A4 / A1	6/39 (15%)	5/76 (7%)	.18

Table 13: Share of items coded straw man among the items an annotator labeled annoying or legitimate. Several coders: mean share per item (items), one-sided Mann-Whitney test; one coder: Fisher’s exact test. Bold:

p<.05

	Gap X $-$ Y			Y
Labels	Ann.	Leg.	$p$	Ann.	Leg.	$p$
All items
A1	+2.21	+0.86	$<$ .001	$-$ 0.97	$-$ 0.36	$<$ .001
A2	+1.62	+0.94	.006	$-$ 0.73	$-$ 0.40	.01
Either	+1.87	+0.76	$<$ .001	$-$ 0.81	$-$ 0.32	$<$ .001
Non-inverted items

Table 14: Evaluative stance (GPT-6-sol rating,

-2

+2

). Labels : items each annotator labeled annoying (Ann.) or legitimate (Leg.). Codes : items each coder coded straw man (SM) or not. One-sided Mann-Whitney test; bold:

p<.05

Papers	Uses	Gap X $-$ Y	Y	Y $<$ 0
ACL 2019	281	$+$ 0.87	$-$ 0.30	42%
arXiv 2026	828	$+$ 1.06	$-$ 0.45	55%
GPT-4o	46	$+$ 1.13	$-$ 0.46	57%
GPT-5.6-sol	67	$+$ 1.39	$-$ 0.67	64%
GPT-6-sol	66	$+$ 1.73	$-$ 0.86	73%

Table 15: Evaluative stance of uses by corpus (mean GPT-6-sol ratings,

-2

+2

). Bold (italic): significantly more (less) disparaging of Y than arXiv 2026 (Mann-Whitney test,

p<.05

Source	Uses	Gap X $-$ Y	Y	Y $<$ 0
Original papers	451	$+$ 1.13	$-$ 0.47	57%
Tülu 3 8B SFT	23	$+$ 2.30	$-$ 0.91	78%
revised	23	$+$ 2.26	$-$ 0.96	83%
Tülu 3 8B DPO	21	$+$ 0.14	$+$ 0.19	33%
revised	7	$+$ 0.14	$+$ 0.14	43%
Tülu 3 8B (final)	26	$-$ 0.85	$+$ 0.62	15%

Table 16: Evaluative stance of the uses in papers written from the first 100 source papers, and in those source papers (mean GPT-6-sol ratings,

-2

+2

; all papers, without the exclusions of Table 7 ). Bold (italic): significantly more (less) disparaging of Y than in the source papers (Mann-Whitney test,

p<.05

); the Tülu 3 8B rows rest on 7–26 uses.

Set	Ann.	Disavowal	Other	$p$
Claude set	A1	26/42 (62)	22/268 (8)	$<$ .001
Pool sample	A1	27/39 (69)	53/200 (26)	$<$ .001
	A2	13/39 (33)	43/200 (22)	.15
GPT set	A1	25/48 (52)	31/350 (9)	$<$ .001
	A2	10/48 (21)	17/350 (5)	$<$ .001

Table 17: Uses labeled annoying , among uses coded as disavowals (both coders) and among the others: count/items (%), and

p

(Fisher’s exact test); Ann.: annotator. The pool sample is enriched for annoying uses, so its rates are higher than in the pool.

GPT comparison set
Source	Uses	Disavowals (%)
ACL 2019 papers	40	0 (0.0)
arXiv 2026 papers	179	19 (10.6)
GPT-4o	46	0 (0.0)
GPT-5.6-sol	67	19 (28.4)
GPT-6-sol	66	10 (15.2)

Table 18: Uses coded as disavowals by both coders, by source. The two coders disagree on more of GPT-6-sol’s first drafts (24% of uses are disavowals for at least one coder), so that row is conservative.

Pattern	A1		A2		Either
Items	87 /	598	66 /	582	134 /	486
Initial Rather than	15 /	3 ∗†	8 /	3	12 /	2 ∗†
Subject we	30 /	13 ∗†	17 /	15	25 /	13 ∗
X adjective	9 /	4	20 /	4 ∗†	13 /	3 ∗†
X adj. modifier	2 /	1	6 /	0 ∗†	4 /	0 ∗
X prep. object	16 /	20	3 /	23 ∗†	12 /	22 ∗

Table 19: Syntactic patterns with

q<.05

in at least one label set: % of annoying / legitimate arXiv 2026 items. ∗

p<.05

; †

q<.05

(Benjamini-Hochberg over about 640 patterns).

Unit	A1	A2	Either
Items	87 / 598	66 / 582	134 / 486
Chance	13%	10%	22%
X	20% ∗	22% ∗	35% ∗
Y	25% ∗‡	18% ∗	35% ∗‡
Sentence	22% ∗	16% ∗	33% ∗
Paragraph	14%	9%	23%

Table 20: Share of annoying arXiv 2026 items’ five nearest neighbors that are annoying ; Chance : share of annoying items. ∗

p<0.05

(label permutations); ‡ mean pairwise cosine among annoying items also above chance (

p<0.01

Measure	Passages or labels	A1	A2
LLM-leaning	ACL 2019 (14)	7%	0%
	arXiv 2026 (109)	47%	27%
	high-count (19)	68%	53%
Hunch vs. annoyance, $\rho$	own labels	$-$ 0.01	0.18
	other’s labels	0.21	0.02

Table 21: Perceived LLM authorship. LLM-leaning : passages rated probably LLM or LLM.

\rho

: Spearman correlation over 125 papers between the mean hunch and the share of pool items labeled annoying by the same annotator (own) or the other (other’s); bold:

p<.05

Measure	Passages or labels	A3	A5
LLM-leaning, %	ACL 2019	41 / 21	24 / 11
	arXiv 2026	48 / 20	46 / 25
	GPT	37 / 22	43 / 22
GPT vs. ACL, AUC	all GPT	.47	.59
	from ACL papers	.42	.54
With vs. filler, AUC		.65	.62

Table 22: Authorship judgments of A3 and A5 (303 passages, 101 of them fillers). LLM-leaning: passages with the construction / fillers. Annoying : passages of annoying vs. legitimate items. AUC: probability that a passage of the first group gets a higher hunch than one of the second; bold:

p<.05

, one-sided Mann-Whitney test, predicted direction. Agreement between A3 and A5 on LLM-leaning:

\kappa=0.14

	UF	Tülu 3	HS2	HS3
Rater	GPT-4	GPT-4o	human	human
Pairs	61k	270k	7k	22k
rather than	0.99	1.09	1.37	1.24
not only…but	1.39	1.14	1.41	1.13
X, not Y	1.24	0.90	1.14	1.35
not X but Y	0.94	0.81	1.06	1.25

Table 23: Odds ratio of a response being chosen when it contains the pattern, controlling for length. Bold (italic): 95% interval above (below) 1. UF: UltraFeedback; HS: HelpSteer.

				ACL 2019
Reward model	O $-$ D	C $-$ D	O $-$ C	O $-$ C
Qwen3 0.6B	0.88 (89)	0.62 (76)	0.26 (71)	0.15 (62)
Qwen3 1.7B	0.83 (84)	0.54 (73)	0.29 (67)	0.07 (51)
Qwen3 8B	1.54 (89)	0.91 (75)	0.63 (71)	0.26 (58)
Llama-3.1 8B	2.46 (76)	1.33 (64)	1.13 (70)	0.96 (67)

Table 24: Mean reward difference between versions of a sentence (% of items with a positive difference): as written (O), with the rather than Y clause replaced by a non-contrastive clause of the same length (C), and with the clause deleted (D). 424 items, 90 of them from ACL 2019. Bold:

p<.05

, Wilcoxon signed-rank test. Mean sentiment (

-2

+2

): O

+0.02

, C

+0.13

, D

+0.07

; O

-

C remains significant for every model with the sentiment difference held fixed (OLS).

Reward model	A1	A2
Annoying / legitimate	40 / 261	26 / 264
Qwen3 0.6B	$+$ 0.26	$+$ 0.09
Qwen3 1.7B	$+$ 0.36	$+$ 0.20
Qwen3 8B	$+$ 0.88	$+$ 0.43
Llama-3.1 8B	$+$ 1.69	$+$ 0.72

Table 25: Extra reward gain of the clause (O

-

D) in arXiv 2026 uses an annotator labeled annoying , relative to legitimate uses, with log clause length held fixed (OLS). Bold:

p<.05

, uncorrected.

Pattern	Honesty	Truthfulness	Calibration
rather than	1.15	1.08	0.85
not X but Y	1.34	0.97	0.93
instead of	1.13	1.07	0.79
not only…but	0.68	0.84	0.72

Table 26: Odds ratio of using the pattern under each system-prompt principle, relative to helpfulness (UltraFeedback). Bold (italic): 95% interval above (below) 1.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Count
cs.CL submissions, 2026 12 12 12 Submitted 2026-01-01 through 2026-09-11.	19,485
Comment mentions an *ACL venue	1,985
$\geq$ 2 arXiv versions	819 (41%)

Appendix

Table 27: arXiv comment-field pre-filter for arXiv 2026, and the subset usable for the version-diff analysis of Section 3 .

	ACL 2019	arXiv 2026
	(N=2,590)	(N=1,748)
Limitations section	13 (0.5%)	910 (52.1%)
Ethics Statement	10 (0.4%)	381 (21.8%)
Author Contributions	8 (0.3%)	9 (0.5%)
Mean body words	5,775	7,847
minus the above	5,765	6,599

Appendix

Table 28: Prevalence of Limitations/Ethics/Author Contributions sections among long-format papers, and their contribution to mean body length.

Quantity (long-format papers)	Mean/paper
ACL 2019 parenthetical citations	25.3
ACL 2019 reference-list entries	36.1
arXiv 2026 [redacted] tokens	148.1

Appendix

Table 29: Citation and cross-reference density, long-format papers. In arXiv 2026, one placeholder token stands for citations, cross-references and inline math, so the two corpora are not directly comparable.

Section	Mean words
abstract	158
introduction	986
related work	944
preliminaries	164
method	408
experiments	805

Appendix

Table 30: arXiv 2026 word counts per section, long-format papers (mean per paper).

Model (snapshot)	Access	Decoding	Max output	Run
GPT-4o	OpenAI API	$T$ 0.8, top- $p$ 0.92	16,384	Sep 30
GPT-5.6-sol	OpenAI API	API defaults	16,384 r	Sep 30–Oct 2
GPT-6-sol	OpenAI API	API defaults	16,384 r	Sep 30–Oct 1
Tülu 3 8B SFT ( Llama-3.1-Tulu-3-8B-SFT )	local GPU, fp16	$T$ 0.8, top- $p$ 0.92	16,384	Oct 6–7
Tülu 3 8B DPO ( Llama-3.1-Tulu-3-8B-DPO )	local GPU, fp16	$T$ 0.8, top- $p$ 0.92	16,384	Oct 6–7
Tülu 3 8B final ( Llama-3.1-Tulu-3-8B )	local GPU, fp16	$T$ 0.8, top- $p$ 0.92	16,384	Oct 6–7

Appendix

Table 31: Generation settings.

T

: temperature. Max output: output-token limit of the generation and revision requests. Run: dates of generation and revision, 2026. r Including reasoning tokens. OpenAI models were called by the listed names; the dated snapshot behind each name was not recorded.

Model	Source	13-gram %	Longest	RT copied
GPT-4o	ACL 2019	0.27 / 0.02	10 / 0	0 / 44
	arXiv 2026	0.08 / 0.04	12 / 9	0 / 133
GPT-5.6-sol	ACL 2019	0.04 / 0.03	11 / 11	0 / 6,312
	arXiv 2026	0.01 / 0.01	12 / 11	0 / 7,178
GPT-6-sol	ACL 2019	0.03 / 0.02	10 / 10	0 / 4,455
	arXiv 2026	0.02 / 0.01	12 / 11	0 / 4,930

Appendix

Table 32: Verbatim overlap of LLM-written papers with their source paper, generated / revised (GPT: 400 sources; Tülu 3 8B and Claude: the first 100). 13-gram % : 13-grams of the LLM paper found in the source’s body but not its abstract. Longest : median longest shared run (words). RT copied : rather than sentences sharing a 13-gram with the source, of all.

	Claude	GPT-6-sol	Both
Uses coded as disavowals (of 947)	147	148	129
$\kappa$ , disavowal against the rest	0.85
$\kappa$ , four classes	0.73

Appendix

Table 33: The two LLM coders on 947 uses (Claude comparison set, GPT comparison set, pool sample).

Codes	Disav.	$\kappa$ with A1	Annoying	AUC
A1	36	–	26/36 (72)	0.83
Claude	39	0.47	22/39 (56)	0.70
GPT-6-sol	41	0.38	23/41 (56)	0.70
Both coders	25	0.51	18/25 (72)	0.73
Claude, broader definition	47	0.53	26/47 (55)	0.72
GPT-6-sol, broader definition	52	0.51	28/52 (54)	0.73

Appendix

Table 34: A1’s coding of 80 uses compared with the coders’: uses coded as disavowals,

\kappa

with A1’s codes, uses labeled annoying by A1 among them, count/uses (%), and the area under the ROC curve for predicting A1’s annoying labels.

Instructions	Claude	GPT-6-sol
Original	0.56	0.32
Broader definition	0.61	0.54
Broader, with 30 of A1’s codes as examples	0.56	0.58

Appendix

Table 35:

\kappa

with A1’s codes (disavowal against the rest) on the 50 of the 80 uses not used as examples.

Stance rater	n	Gap	n	Gap	$\rho$
	Disavowal		Other
GPT-6-sol	129	$+$ 1.11	818	$+$ 1.29	$-$ 0.07
Claude agents	81	$+$ 0.86	468	$+$ 1.21	$-$ 0.09

Appendix

Table 36: Mean stance gap X

-

Y of uses coded as disavowals (both coders) and of the others, by rater, and Spearman correlation

\rho

between disavowal and the gap.

	Change per 1k words		RT sentences
Model	RT	X, not Y	removed	added
Haiku 4.5	$+$ 0.16 (0.007)	$+$ 0.01 (0.262)	27/399 (0)	42/574 (3)
Sonnet 5.5	$-$ 0.01 (0.701)	$+$ 0.23 (<.001)	39/695 (8)	26/857 (4)
Opus 5.5	$+$ 0.16 (<.001)	$+$ 0.25 (<.001)	92/637 (21)	55/927 (14)

Appendix

Table 37: Revision of the Claude papers: change in the mean rate from draft to revision (

p

: paired Wilcoxon test), and the sentences with rather than (RT) removed from the draft or added in the revision, of all such sentences in the draft or the revision, with the number in which another antithesis form takes the place of the phrase (or was replaced by it).

	v1	Latest
Mean rate per 1,000 words	0.921	0.988
Total occurrences	5,394	6,471
Median body length (words)	6,747	7,572
Paired difference	$+$ 0.067
95% CI	0.042–0.093
Wilcoxon $p$	$<10^{-5}$

Appendix

Table 38: Rather than in the first and latest arXiv versions of the same 793 papers (body text). Paired difference: mean per-paper change, bootstrap interval, Wilcoxon signed-rank test.

Quantity	Count
Paper pairs (v1 + latest)	807
Mined diff hunks with phrase	3,550
phrase added	1,156
phrase kept, context reworded	1,818
phrase removed from hunk	576
surgical removal	86

Appendix

Table 39: arXiv version-diff mining results (807/819 candidates yielded a usable v1/latest pair).

Figure 2: Occurrences per 1,000 words, first (v1) and latest arXiv versions of the same 793 papers.

Type	Pattern	2019	2026	v1	latest
Substitution	rather than	0.134	0.993	0.921	0.988
	instead of	0.165	0.105	0.109	0.109
	as opposed to	0.025	0.007	0.007	0.006
Negation	not X but Y	0.097	0.126	0.129	0.125
	X, not Y	0.049	0.201	0.188	0.202
Emphatic	not only…but	0.079	0.119	0.124	0.116

Appendix

Table 40: Occurrences per 1,000 words of body text in ACL 2019, arXiv 2026, and the first (v1) and latest versions of the 793 arXiv papers with several versions.

Quantity	Count
Candidate repositories	68
one commit to the paper	36
five or more commits	10
Diff hunks with the phrase (10 repositories)	95
phrase added	24
phrase kept, context reworded	55

Appendix

Table 41: Rather than in GitHub version histories of ACL-style papers.

Paper (arXiv id)	Uses	Per 1k words
2608.13706	69	5.84
2608.06171	63	4.61
2606.05743	54	3.28
2608.29995	46	2.01
2607.14252	46	2.18
2604.17022	46	3.88

Appendix

Table 42: arXiv 2026 papers with the most uses of rather than . The ACL 2019 maximum is 12.

Body text only (without 33 items)
Stratum	N	A1	A2	Either
arXiv 2026, main	388	51 (13.1)	39 (10.1)	79 (20.5)
v1	145	14 (9.7)	9 (6.2)	22 (15.3)
latest	119	10 (8.5)	6 (5.0)	15 (12.7)
ACL 2019	245	1.7	0.0
arXiv 2026, all	627	11.8	8.0

Appendix

Table 43: Pool strata not shown in Table 8 : items labeled annoying , count (% of the stratum); body text only: %.

	A1		A2
Measure	Batch 1	Batch 2	Batch 1	Batch 2
Annoying , ACL 2019	0 / 143	4 / 100	0 / 144	0 / 99
Annoying , arXiv 2026	11.5%	11.6%	5.6%	10.0%
Gap X $-$ Y, annoying / legitimate	$+$ 2.31 / $+$ 0.78 ∗	$+$ 2.33 / $+$ 0.83 ∗	$+$ 1.79 / $+$ 0.91	$+$ 1.43 / $+$ 0.90
Y, annoying / legitimate	$-$ 1.03 / $-$ 0.31 ∗	$-$ 1.00 / $-$ 0.34 ∗	$-$ 0.71 / $-$ 0.39	$-$ 0.65 / $-$ 0.37
Initial Rather than (%)	12 / 4 ∗	17 / 2 ∗	8 / 3	8 / 3

Appendix

Table 44: Label-based results within each batch of the pool. Rates: annoying items among all items of the corpus; other rows: annoying / legitimate arXiv 2026 items. ∗

p<.05

within the batch (Fisher’s exact test; Mann-Whitney test, one-sided, for stance).

Hidden repeats (80 items)
	A1	A2
Same label on both passes	61 (76%)	67 (84%)
legitimate $\to$ unsure	7	3
unsure $\to$ legitimate	4	8
other $\to$ annoying	3	2
annoying $\to$ other	4	0

Appendix

Table 45: Annotator drift. Repeats : early items re-labeled blind later. Order : annoying rate in successive fifths of each annotator’s items, and a logistic regression on position with stratum fixed effects.

	A1–A2	A1–A5	A2–A5
Raw agreement	76%	26%	20%
Cohen’s $\kappa$	0.15	$-$ 0.00	$-$ 0.03
Annoying : 1st / 2nd / both	4 / 1 / 0	4 / 35 / 3	1 / 35 / 0

Appendix

Table 46: Agreement on the 50 items A5 labeled that no annotator marked garbled (raw agreement and

\kappa

over three labels).

	Core NLP	Other	OR	$p$
A1	69/603 (11%)	18/132 (14%)	1.2	.46
A2	46/602 (8%)	19/132 (14%)	2.1	.018
Either	102/601 (17%)	31/132 (23%)	1.5	.082

Appendix

Table 47: Annoying arXiv 2026 items by the domain of their paper (403 core NLP, 70 applied, 8 linguistics papers). OR: other vs. core NLP, logistic regression with stratum;

p

: Fisher’s exact test. Predicted direction: one-sided

p\geq.77

Whole paper
Shown text	A1	$p$	A2	$p$
Another rather than	9 / 12	.41	7 / 9	.57
Other antithesis	17 / 12	.32	10 / 9	.78
Graphite tell	11 / 12	1.0	9 / 9	.85
Kobak words /100	5.1 / 4.8	.10	4.4 / 4.8	.80
Rather than /1k	1.48 / 1.56	.46	1.61 / 1.53	.013

Appendix

Table 48: Density and AI tells, arXiv 2026 items. First three rows: annoying rate (%) with / without the feature, Fisher’s exact test. Kobak: mean rate, annoying / other items, one-sided logistic test ( Kobak et al., 2025 ; Graphite: Druck et al., 2026 ). Whole paper: median rate, annoying / other items, logistic regression on

\log(1+\mathrm{rate})

with stratum.

A2 annoying , by A1’s codes elsewhere in the paper
	Papers	Same-paper pairs	$p$
A1	129	11 vs. 8.6	.25
A2	129	24 vs. 6.4	$<$ .001
straw man / none		12/64 vs. 8/105	.047
controlled		OR 1.5	.51

Appendix

Table 49: Clustering of annoying labels by paper, arXiv 2026 papers with two or more items. Same-paper pairs : observed vs. expected under 5,000 shuffles of labels within strata. Last rows: Fisher’s exact test, then a logistic regression with A1’s label on the item, the paper’s rather than rate and the high-count stratum, paper-clustered standard errors.

Term	A1		A2		Either
Items	87 /	598	66 /	582	134 /	486
intended	8 /	1	0 /	1	5 /	1
rather than relying	8 /	2	2 /	2	5 /	2
we	34 /	18	21 /	21	29 /	19
so	10 /	8	17 /	7	11 /	7
rather than the	3 /	5	0 /	5	2 /	5

Appendix

Table 50: Candidate terms: % of annoying / legitimate arXiv 2026 items containing each term, in the sentence unless marked. No term has

q<.05

in the scan (Benjamini-Hochberg over 3,475–3,948 terms per label set).

Similarity to blatant straw men, AUC
	A1	A2	Either
Y, straw-man codes	.67 ∗
Y, annoying	.66 ∗	.53	.61 ∗
clause, annoying	.64 ∗	.61 ∗	.63 ∗
Annoying in the next five items, %
after / otherwise	9 / 11	5 / 9

Appendix

Table 51: Blatant straw men. AUC: how well the maximum embedding similarity ( text-embedding-3-large ) to the six agreed straw men (and, for Y, the definition’s examples) ranks A1’s straw-man codes or annoying arXiv 2026 items; ∗

p<.05

, permutation test. Next five items: annoying rate after meeting one of the six vs. otherwise (Fisher’s exact test,

p=1

for both).

Source	Rather than	X-ed/Y-ed	Share
ACL 2019	3,087	4	0.13%
arXiv 2026	14,539	82	0.56%
GPT-4o	104	1	0.96%
revised	73	1	1.4%
GPT-5.6-sol	5,878	74	1.3%
revised	7,636	115	1.5%

Appendix

Table 52: “X-ed rather than Y-ed”: uses of rather than with a past participle (spaCy tag VBN) right before rather and after than . Full text; GPT rows: all 400 papers per model.

Feature	A1		A2		Either
Items	56 /	251	27 /	293	72 /	221
Subject we	29 /	9 ∗	11 /	12	22 /	10 ∗
X adjective	12 /	4	11 /	3	11 /	2 ∗
X adj. modifier	2 /	0	0 /	0	1 /	0
X prep. object	27 /	21	15 /	24	22 /	22
than coord. adj.	12 /	3 ∗	7 /	3	10 /	2 ∗

Appendix

Table 53: GPT comparison set, arXiv 2026 and GPT items: annoying / legitimate, % of items (gap and Y: mean stance rating). ∗

p<0.05

Items	Unit	A1	A2	Either
arXiv 2026	Items	32 / 121	9 / 153	36 / 111
	Chance	20%	5%	24%
	X	31% ∗	9%	34% ∗
	Y	41% ∗	24% ∗	42% ∗
	Sentence	41% ∗	0%	44% ∗
	Paragraph	32% ∗	4%	33%

Appendix

Table 54: GPT comparison set: share of annoying items’ five nearest neighbors that are annoying , as in Table 20 . ∗

p<0.05

(label permutations).

Hunches	Labels	Papers	Passages
A1	A1	$-$ 0.01	$-$ 0.06
A1	A2	0.21	0.26
A2	A2	0.18	0.16
A2	A1	0.02	0.01
arXiv 2026	n	A1	A2
with rather than	52	73%	50%

Appendix

Table 55: Authorship hunches and annoyance. Upper part, Spearman

\rho

: Papers , mean hunch vs. share of a paper’s pool items labeled annoying (125 papers); Passages , hunch vs. the label of the item next to which a passage was drawn (107 passages). Lower part: passages rated LLM-leaning. Bold:

p<.05

GPT-6-sol	Favors X	Favors Y	Neither
Inverted	1	24	15
Not inverted	20	2	18

Appendix

Table 56: A1’s judgment of which alternative the author favors, for 40 items GPT-6-sol rated inverted and 40 it did not.

	$\kappa_{w}$	Spearman	Within 1	Mean diff.
X (affirmed)	0.77	0.76	98%	$-$ 0.12
Y (rejected)	0.72	0.73	97%	$+$ 0.02
Gap X $-$ Y	0.81	0.80	81%	$-$ 0.14
Inverted (X $<$ Y), $\kappa$	0.71

Appendix

Table 57: Agreement of the Claude agents with GPT-6-sol on 549 uses: quadratic weighted

\kappa

(

\kappa_{w}

), Spearman correlation, share of uses within one point, and mean difference (Claude agents minus GPT-6-sol).

	Uses	GPT-6-sol	Claude
ACL 2019 originals	50	$+$ 1.22	$+$ 1.04
Claude, first draft	85	$+$ 0.81	$+$ 0.86
Claude, revised	75	$+$ 0.81	$+$ 0.61
GPT-6-sol, first draft	50	$+$ 1.88	$+$ 1.52
GPT-6-sol, revised	50	$+$ 1.84	$+$ 1.54

Appendix

Table 58: Mean stance gap X

-

Y of the uses in the Claude comparison set, by rater.

Set	Rater	Gap: ann. / other	AUC	$p$
Pool sample	GPT-6-sol	$+$ 1.99 / $+$ 0.81	0.65	$<$ .001
	Claude	$+$ 1.79 / $+$ 0.81	0.65	$<$ .001
	mean	$+$ 1.89 / $+$ 0.81	0.66	$<$ .001
Claude set (A1)	GPT-6-sol	$+$ 1.42 / $+$ 1.18	0.53	.46
	Claude	$+$ 1.40 / $+$ 0.98	0.58	.08
	mean	$+$ 1.41 / $+$ 1.08	0.55	.24

Appendix

Table 59: Stance gap X

-

Y of annoying (ann.) and other uses, by rater (Claude: the agents). Pool sample: 120 uses either annotator labeled annoying and 119 neither did. Claude comparison set: A1’s 48 annoying and 262 other uses. AUC: probability that an annoying use has a larger gap than another use (Mann-Whitney test).

	A1	A2	A3	A4
Straw men (of 151)	42	7	40	12
$\kappa$ with A2	0.18
$\kappa$ with A3	0.16	0.10
$\kappa$ with A4	0.11	0.27	0.12

Appendix

Table 60: Straw-man coders: number of items each codes straw man, and Cohen’s

\kappa

between coders (straw man vs. other, all 151 items).

	Revised	Ctrl.	Position	Earlier
Claude	$+$ 0.94 (0.03)	$+$ 0.90 (0.05)	$+$ 0.35 (0.63)	$-$ 0.12 (0.26)
GPT-6-sol	$-$ 0.51 (0.32)	$-$ 0.35 (0.50)	$-$ 1.86 (0.15)	$-$ 0.13 (0.45)

Appendix

Table 61: Logistic regression of A1’s annoying label on revised versus first draft, alone ( Revised ) and controlled ( Ctrl. ) for the position of the item in the labeling order ( Position ) and the number of earlier-labeled uses from the same source paper ( Earlier ): coefficients (log-odds) and

p

values; the last two columns are the coefficients of the controls. Of 97 pairs of draft and revised uses from the same paper and model, 5 are near-identical (similarity at least 0.6), none annoying .

Explore similar work

What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience
May 19, 2026Filip Miletić, Neele FalkScientific Communication Text Generation
Institute for Natural Language Processing, University of Stuttgart, Germany
Citing Less Critically: LLMs Reshape the Rhetoric and Reach of Scientific Citation
Sep 1, 2026Yixuan Liu, Lin Chen, Zhuoqi Liu +2AI-Assisted Scientific Research
1Network Science Institute, Northeastern University · Department of Physics, Northeastern University · Department of Electrical and Computer Engineering, Northeastern University +1
Most biomedical publications show signs of LLM-assisted writing
Aug 11, 2026Lena Holzwarth, Rita González-Márquez, Dmitry KobakAI-Generated Text Detection LLM Auditing
1Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Department of Mathematics, Computer Science, and Statistics, Ghent University, Belgium · 3VIB Center for AI and Computational Biology, VIB, Ghent, Belgium

May 19, 2026, cs.CLJ/K move · Enter open · S save

What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience

Filip Miletić, Neele Falk

Institute for Natural Language Processing, University of Stuttgart, Germany

Scientific Communication Text Generation

Open paper AskPDF arXiv

Has the style of scientific communication changed due to the growing use of large language models in the writing process? We address this question in the domain of Natural Language Processing by leveraging two data resources we create: a naturalistic corpus of over 37,000 papers from the ACL Anthology (2020-2024); and a synthetic dataset of 3,000 human-written passages and their LLM-generated improvements. We first implement a series of diachronic lexical analyses, showing that both word frequency and usage contexts have changed significantly over time, indicating semantic specialization in some cases and generalization in others. Broadening our perspective, we then model a range of more complex stylistic features and find that LLM-modified texts more frequently contain certain syntactic constructions, more complex and longer words and a lower lexical diversity. Finally, we connect these changes in writing practices to subjective reading experience through a pilot annotation study with 20 domain experts. They overall rate LLM-improved texts as more understandable and exciting, but also express negative qualitative attitudes towards LLMs, highlighting the strongly subjective effect of AI-assisted writing on reading experience.

Abstract

Explore similar work

What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience

Citing Less Critically: LLMs Reshape the Rhetoric and Reach of Scientific Citation

Most biomedical publications show signs of LLM-assisted writing