Measuring AI "Slop" in Text

Chantal Shaib; Tuhin Chakrabarty; Diego Garcia-Olano; Byron C. Wallace

cs.CLSep 23, 2025

Measuring AI "Slop" in Text

Authors: Chantal Shaib, Tuhin Chakrabarty, Diego Garcia-Olano, Byron C. Wallace

Organizations: Northeastern University · Stony Brook University · Meta AI

Abstract

AI "slop" is an increasingly popular term used to describe low-quality AI-generated text, but there is currently no agreed upon definition of this term nor a means to measure its occurrence. In this work, we develop a taxonomy of "slop" through interviews with experts in NLP, writing, and philosophy, and propose a set of interpretable dimensions for its assessment in text. Through span-level annotation, we find that binary "slop" judgments are (somewhat) subjective, but such determinations nonetheless correlate with latent dimensions such as coherence and relevance. Our framework can be used to examine AI-generated text in both detection and binary preference tasks, potentially offering new insights into the linguistic and stylistic factors that contribute to quality judgments. We highlight that fully automated and scalable methods remain an open challenge.

Figures & tables

Figure 1: Sample of annotations over a human-written news article highlighting indicators of “slop” (red; from Russell et al. 2025 ), human-writing (green; ours), and both (yellow). “Slop” labels are notably different than indicators of human-written text.

Themes	Final Codes	Granular Codes	Count
Info. Utility	Density	IU1: Density	5
Info. Utility	Relevance	IU2: Relevance	9
Info. Quality	Factuality	IQ1: Factuality	7
Info. Quality	Bias	IQ2: Bias	2
Style Quality	Structure	SQ1: Repetition	7
	Structure	SQ2: Templatedness	2

Table 1: Themes and codes for slop, count of expert responses containing each.

Figure 2: Label counts and distributions for each annotator. Annotators used all codes, but there are individual differences in the code frequencies assigned in each theme. Detailed span statistics in Appendix H .

Themes	Final Codes	$\alpha_{MASI}$	$\kappa$	AC 1	Prev. (%)
Info. Utility	Density	0.37	0.17	0.55	49.1
Info. Utility	Relevance	0.37	0.17	0.24	69.1
Info. Quality	Factuality	0.44	0.18	0.75	30.9
Info. Quality	Bias	0.44	0.15	0.73	32.7
Style Quality	Structure	0.37	0.13	0.15	74.5
	Coherence		0.11	0.51	50.9

Table 2: Agreement (

\alpha_{MASI},\kappa,AC_{1}

) and label prevalence (Prev. %) for “slop” codes across all data.

Code category	Highlighted span
Relevance	“During the Roman Empire, physicians developed techniques to repair injured gladiators and soldiers, including methods for treating facial injuries and performing basic skin grafts. The field experienced a significant evolution during the Renaissance, as European surgeons began documenting and sharing their techniques more widely. ”
Factuality	“[…] leading to more frequent and severe heatwaves. “Climate change is like adding steroids to our weather,” says Dr. Michael Oppenheimer, a climate scientist at Princeton. ”
Structure	“But did you know there’s another important number-sort of like a ”secret” code—printed just beneath the sell-by date ? […] Find the secret code, which is usually near the sell-by date. ”
Tone	“The very power of the word [“witch”] lies in its imprecision. It is not merely a word but an archetype, a cluster of powerful images.” The uncertainty of exactly what a witch is forms part of the titillation ”

Table 3: Text marked by all annotators. Relevance : all marked as irrelevant. Factuality : Inaccurate quote attribution. Structure : marked for repeated text. Tone : marked for coherence, fluency.

Figure 3: “Slop” codes most predictive of the overall positive label for (left) the entire corpus, (middle) news, and (right) MS MARCO. *

p<0.05

, and **

p<0.01

Panel of Figure 4 — Figure 4: AUROC predicting the binary “slop” label across both the news and QA domains using (left) a logisitic regression model and (right) the PCA+LDA model. The syntax-only feature set performs better than lexical or automatic features.

Model	$\kappa$	Pr./R	Pct. Slop
GPT-5	0.01	0.38/0.12	0.08
Deepseek-V3	-0.01	0.25/0.08	0.03
o3-mini	0.03	0.42/0.08	0.07
Human	–	–	0.34

Table 4: (left) Binary slop assessment: 0-shot prompting relative to human labels (humans assigned positive slop label to 35% of data). (right) Span-level extraction: Character-level precision and recall in zero- and few-shot settings. *Qwen-7B is fine-tuned.

Code	Precision	Recall	Presence F1
Bias	0.15	0.03	0.84
Coherence	0.11	0.02	0.83
Density	0.26	0.08	0.90
Factuality	0.12	0.02	0.82
Relevance	0.05	0.02	0.84
Structure	0.25	0.03	0.82

Table 5: (left) Character precision, recall, and code presence F1 of the extracted “slop” spans using GPT-5 over the News dataset (cf. Table 4 ). (right) Average span character length and counts comparison to human highlights.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Question	Answer Options
What is your primary field of work?	Natural Language Processing Linguistics Writing (e.g., Copywriting, Journalism) Other: [Free text]
How many years of experience do you have in your field?	1-2 3-5 5-8 9+
Have you encountered the term “slop” before, as it relates to generated text or images?	Yes No
If you answered “Yes” to the above question, please describe which contexts you’ve encountered the term in (e.g., in a news article, in a podcast, in discussions)	Free text
How often do you use large language models (LLMs) in your work?	Never Rarely (sporadically) Sometimes (2 times a week) Often (3-4+ times a week)
If you use LLMs once or more a week, please select the type of tasks you use them for.	Ideating Writing Rewriting Summarizing documents/texts Creative writing Question Answering (general) Question Answering (specific, based on an input document) Other: [Free text] (Check all that apply)

Appendix

Table 7: Anonymous survey sent to experts to collect definitions of “slop.”

Panel of Figure 5 — Figure 5: Expert demographics: (a) distribution of years of experience and (b) fields of expertise.

Expert Response (trunc)
“…‘slop’ text from humans as something that is very generic and overly verbose —perhaps excessive marketing copy or rambling prose that is published when it really should have been heavily edited. I wouldn’t call it ‘slop’ if it was simply a first draft—it’s the brazen publishing (online or IRL) of content that wastes a reader’s time that befits the term best.”
“The classic highschool transitional words that students just learning to write essays use, without variance .”
“Slop in certain human contexts may perform a useful function as an intermediary step in the author’s process (consider certain kinds of notes or student writing , for example).”
“…there are many responses from humans that are hastily written without critical thoughts .”

Appendix

Table 8: Sample responses describing how “slop” may manifest in human written text.

Figure 6: “Slop” Evaluation consent form given to annotators prior to the study.

Figure 7: questions given to annotators in interface (LabelStudio)

Figure 8: Definition Guide presented to annotators (along with Table 9 ) to reference.

Code	Name	Description	Example
Information Quality
IQ1	Factuality	Incorrect or fabricated information Misleading or fallacious claims	“Dr. Sarah Johnson of Harvard University published groundbreaking research on this topic in 2022.” (Slop if Dr. Johnson doesn’t exist, isn’t at Harvard, or didn’t publish such research)
IQ2	Bias	Text that feels too “objective” when subjectivity is appropriate Missing rhetorical point of view when needed Lack of appropriate perspective based on context Content that seems detached when engagement is required The presence of inappropriate perspective or assumptions	“The economic policy changes of 2023 were universally beneficial.” (Slop because it presents a one-sided view of complex policy impacts)
Information Utility
IU1	Information Density	Text that is verbose but conveys little actual information Generic statements that could apply in almost any context Excessive filler words and phrases that add no value	“In today’s fast-paced modern world of cutting-edge technology and innovation, it has become increasingly important to consider the various factors and elements that contribute to our understanding of this complex and multifaceted issue.” (Slop because it uses many words to say almost nothing)
IU2	Information Relevance	Content that fails to address the nuances of the query or task Content that contributes nothing meaningful to context/query/task Text that appears disconnected from its intended purpose For text with additional context, consider relevance to such texts For text with no additional context, consider internal relevance	In response to “How can I improve my marathon time?”: “Running is an excellent form of exercise with many health benefits including improved cardiovascular function, enhanced mood, and weight management.” (Slop because it doesn’t address the specific question about improving marathon times)

Appendix

Table 9: “Slop” Codes with Examples

Themes	Final Codes	Pair	AC 1	$\kappa$	Prev. (%)
Info. Utility	Density	A1–A2	0.94	0.37	7.3
		A1–A3	0.23	0.04	49.1
		A2–A3	0.30	0.08	50.9
	Relevance	A1–A2	0.51	0.21	40.0
		A1–A3	0.17	0.20	67.3
		A2–A3	0.05	0.09	61.8

Appendix

Table 10: Gwet’s AC 1 , Cohen’s

\kappa

, and prevalence for each annotator pair and final code.

Annotator	Avg. Spans/Item	Avg. Span Length (tokens)
A1	5.85	21.42
A2	3.72	13.94
A3	12.61	17.36
Average	7.39	17.57

Appendix

Table 11: Average spans-per-item and length by annotator for the News dataset (

n=1520

spans).

Annotator	Avg. Spans/Item	Avg. Span Length (tokens)
A1	1.77	9.22
A2	1.07	5.57
A3	2.97	11.64
Average	1.94	8.81

Appendix

Table 12: Average spans-per-item and length by annotator for the MS MARCO dataset (

n=192

spans).

Annotator	Avg. Spans/Item	Avg. Span Length (tokens)
A1	4.82	20.58
A2	3.06	13.37
A3	10.22	17.16
Average	6.03	17.04

Appendix

Table 13: Average spans-per-item and length by annotator for the combined News and MS MARCO datasets (

n=1712

spans).

Figure 9: Tri-grams extracted from o3-mini rationales over highlighted “slop” spans.

Publication	Number
National Geographic	41
Smithsonian Magazine	38
New York Times	37
Wall Street Journal	31
Discover	29
Readers Digest	24

Appendix

Table 14: Number of articles for each News source.

Figure 10: “Slop” axis label prevalence stratified by news source.

Themes	Final Codes	$\alpha_{MASI}$	$\kappa$	AC 1	Prev. (%)
Info. Utility	Density	0.34	0.16	0.45	59.1
Info. Utility	Relevance	0.34	0.14	0.22	68.2
Info. Quality	Factuality	0.45	0.23	0.76	29.5
Info. Quality	Bias	0.45	0.11	0.67	38.6
Style Quality	Structure	0.34	0.11	0.51	52.3
	Coherence		0.13	0.39	59.1

Appendix

Table 15: Agreement (

\alpha_{MASI},\kappa,AC_{1}

) and label prevalence (Prev. %) for “slop” codes in the News domain.

Panel of Figure 11 — Figure 11: “Slop” codes most predictive of the overall positive label for (a) Annotator 1, (b) Annotator 2, and (c) Annotator 3 in the News domain. * $p<0.05$ , and ** $p<0.01$ .

Themes	Final Codes	Codes	Sig. Feature?	Auto. Metric
Info. Utility	Density	IU1: Density	✓	Surprisal ( Meister et al., 2021 )
Info. Utility	Relevance	IU2: Relevance	✓	—
Info. Quality	Factuality	IQ1: Factuality	✗	—
Info. Quality	Bias	IQ2: Bias	✓	Subjectivity-Lexicon ( Wiebe et al., 2004 )
Style Quality	Structure	SQ1: Repetition	✗	Compression Ratios ( Shaib et al., 2025 )
	Structure	SQ2: Templatedness	✗	Templates-per-Token ( Shaib et al., 2024 )

Appendix

Table 16: Existing automatic metrics for “slop” codes. We mark codes that are significant predictors of slop label with a green checkmark: 3 of 5 of significant features lack reliable automatic measures.

Figure 12: Correlation news

Panel of Figure 13 — Figure 13: Distributions of WQRM scores across the (a) News and (b) MS MARCO datasets.

Feature Category	Feature	Description
Lexical Features	Hapax legomena ratio	ratio of unique words to total words
	Dis legomena ratio	ratio of words appearing exactly twice
	Word-length distribution	fraction of words of length 1-20
	Punctuation rates	per character rates of punctuation list
	Letter frequency deviations	L2 norm of all 26 letter freq. deviations
Syntactic Features	POS tag distributions	fraction of tokens with tag from the universal SpaCy tag set

Appendix

Table 17: Full list of features extracted and descriptions.

Panel of Figure 14 — Figure 14: AUROC using logistic regression over all available automatic text metrics (Table 16 ) for the (a) News and (b) MS Marco datasets. In the news domain, the automatic metrics in Table 16 perform best, whereas the syntax feature set for MS MARCO does well, but within a wider confidence interval.

Panel of Figure 15 — Figure 15: AUROC using the PCA+LDA pipeline over all available automatic text metrics (Table 16 ) for the (a) News and (b) MS Marco datasets. In the news domain, the automatic metrics in Table 16 perform best, whereas the syntax feature set for MS MARCO does well, but within a wider confidence interval.

Metric	Value
precision	0.14
recall	0.13
f1	0.14

Appendix

Table 18: Word-overlap metrics between o3-mini and human span annotations annotations.

Panel of Figure 16 — Figure 16: Frequency of (a) human-assigned slop code prevalence and, (b) “slop” category collapsed trigrams in o3-mini span rationales in the News domain. o3-mini over-represents issues related to Density in the rationales, while under-representing issues with Coherence, Tone, Bias, Relevance, and Factuality.

LAMP Category	Slop Taxonomy
Cliche	Tone
Poor Sentence Structure	Coherence
Awkward Word Choice and Phrasing	Tone
Tense Inconsistency	Tone
Unnecessary/Redundant Exposition	Density, Repetition
Lack of Specificity and Detail	Relevance

Appendix

Table 19: Mapping of the categories in Chakrabarty et al. 2025a to the “slop” taxonomy.

Theme	Source	Avg Spans	Any Rate	Avg Span Length
Bias	Human	0.14	0.14	98.67
Bias	GPT-5	1.55	0.91	87.27
Coherence	Human	2.46	0.32	132.63
Coherence	GPT-5	1.55	0.82	82.50
Density	Human	0.46	0.32	134.50
Density	GPT-5	2.00	1.00	101.09

Appendix

Table 20: Individual span counts, appearance, and length for each individual “slop” code.

Text	Count	n
the article stays focused	56	4
no clear factual inaccuracies	49	4
and easy to follow	44	4
no unnecessary jargon buzzwords	34	4
appropriate word choice no awkward grammar	38	6
no awkward grammar or unnatural turns	32	6

Appendix

Table 21: N-grams and counts of reasoning provided the GPT-5 in the single-code span extraction setting.

Generative AI Evaluation

Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.

See all 309 papers

SlopBench: How Well Can We Rank Language Models by Slop? A Multi-Domain Benchmark of Repetitive AI Writing
Sep 27, 2026Dhruv Roongta, Harsha Gaddipati, Anh Tuan HuynhLLM Evaluation Repetition in Language Generation
Slashy, San Francisco, California, USA
Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers
Sep 30, 2026Yerim Oh, Young-Jun Lee, Jaewoo Ahn +2AI-Generated Text Detection Scientific Reasoning in Language Models
Seoul National University · University of Minnesota
'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions
Jun 3, 2026Nils Dycke, Marina Sakharova, Nico Daheim +1AI-Generated Text Detection Generative AI Evaluation
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany · Zuse School ELIZA

Sep 27, 2026J/K move · Enter open · S save

SlopBench: How Well Can We Rank Language Models by Slop? A Multi-Domain Benchmark of Repetitive AI Writing

Dhruv Roongta, Harsha Gaddipati, Anh Tuan Huynh

Slashy, San Francisco, California, USA

LLM Evaluation Repetition in Language Generation Language Model Generation Evaluation Text Generation Generative AI Evaluation

Open paper AskPDF arXiv

SlopBench asks which models produce the stiff, repetitive prose readers call AI slop, a question detectors leave open once they have classified a text as machine-written. We evaluated eighteen models on 112 hand-written tasks in email, social posts, essays, and workplace chat, sampling each model on each task up to ten times, for 19,928 outputs in all. SlopBench scores four surface behaviors a reader can check by hand: length against the word band each task specifies, opener repetition across a model's own samples of one task, and paragraph rhythm and fixed lexical constructions against pre-ChatGPT human corpora. Under one fixed weighting, Kimi K2.6 scores lowest at 21.1 and Mistral Large highest at 40.6. Across 500 random reweightings Kimi has the lowest score in 58 percent of draws and Mistral the highest in 97 percent. No draw preserves the full order of the eighteen, and a scenario bootstrap leaves exactly one of those ranks unambiguous. We ran three further checks on that middle order: a crowd arena, an AI detector, and lexical diversity. None of them confirmed the order. We therefore report the four behaviors separately and treat the composite as one weighting among many, and we release the prompts, outputs, reference statistics, and scoring code.

Abstract

Generative AI Evaluation

Explore similar work

SlopBench: How Well Can We Rank Language Models by Slop? A Multi-Domain Benchmark of Repetitive AI Writing

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions