cs.AISep 30, 2026

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

Authors: Yerim Oh, Young-Jun Lee, Jaewoo Ahn, Gunhee Kim, Dongyeop Kang

Organizations: Seoul National University · University of Minnesota

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AI-generated content, often called AI slop, is increasingly common everywhere, particularly in academia. Slop in AI-generated scientific papers, however, has more complex patterns that cannot be easily detected by existing token-based AI detectors. Each part of such a paper looks plausible while the scientific reasoning that connects the parts breaks down, which can mislead how readers assess the work. We benchmark these failures as scientific slop through six measures across Structure, Argument, and Artifacts. We construct SciSlopBench with 390 AI-generated papers, mostly in computer science but spanning the life, social, and natural sciences, each paired with a human-written paper matched by research problem and contribution type. Our measures identify the AI paper in each pair with 85.9% accuracy, compared with 68.7% for Binoculars. Higher scientific slop accompanies lower ICLR ratings and distinguishes rejected from accepted papers above chance in every year from 2017 to 2025. Reducing these patterns, however, is not as simple as directly optimizing the measures. We therefore propose SciSlopHarness, a harness-level framework that guides a fixed LLM to revise slop only where the experiment records support the change. While standard revisions leave residual slop and direct slop-aware prompting triggers reward hacking, SciSlopHarness reduces the remaining AI-human gap by 63% over the strongest revision baseline without requiring human reference targets. Overall, we demonstrate that AI-generated scientific papers leave fundamental traces in their global reasoning, and that responsible mitigation demands strict evidentiary grounding rather than mere prose refinement.

Figures & tables

Figure 1: Scientific slop captures failures beyond token-level AI detection. While traditional detectors assess local text, scientific slop analysis examines how scientific reasoning connects, revealing paper-level failures that plausible prose can conceal.
 ItemIllustrationUnit of analysis Uj(p)U_{j}(p)Score sj(p)s_{j}(p)What sj=1s_{j}=1 means
 Cross-section references Body sections and labeled objects\dfrac{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}objects never referenced outside their own section\@add@centering}}{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}all objects\@add@centering}}No object is ever referred to from another section.
STRUCTUREMacro redundancy Sentences with at least eight tokens\dfrac{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}sentences half or more copied as 8-grams from an earlier section\@add@centering}}{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}all sentences\@add@centering}}Every sentence mostly repeats earlier sections.
 Argument graph Key claims in the Introduction\dfrac{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}shallow claims: nothing earlier leads up to them\@add@centering}}{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}all key claims\@add@centering}}The argument is flat. Every claim stands alone with no build-up behind it.
ARGUMENTCitation isolation Citation sentences (Intro., Related Work)\dfrac{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}citations not grouped, compared, or related to other works\@add@centering}}{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}all citations\@add@centering}}Every citation stands alone.
 Figure exposition Content types in method figures\dfrac{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}content types not needed to show the method\@add@centering}}{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}all content types in the figure\@add@centering}}Nothing in the figure shows the method itself.
ARTIFACTSEvidence gap Papers with a body result table\dfrac{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}papers showing no concrete input, output, or case\@add@centering}}{\parbox{394.98499pt}{\centering\color[rgb]{0,0,0}all papers\@add@centering}}The paper gives no concrete example at all.
Table 1: Measurement specification of SciSlop patterns. Each score is the share of units that show the pattern, written as numerator / denominator (Eq. 1 ). The last column states what a score of 1 means. Full rules are in Appendix A .
Figure 2: SciSlopBench pipeline. We pair each AI-generated paper with a human-written paper matched by research problem and contribution type. We measure six scientific slop patterns in both papers, then compare the mean slop scores of each matched pair.
Figure 3: SciSlopHarness pipeline. We locate scientific slop instances in the manuscript and use them to guide localized revision. We then review each proposed change against the original manuscript, experiment outputs, and code, retain only supported changes, and repeat the process on the remaining slop.
Table 5
Figure 4: AI likeness against ICLR review scores (a), assessment dimensions (b), and annual acceptance decisions (c). Panel (a) adjusts probabilities for differences between years, panel (b) shows ∣ρ∣|\rho| with blue and red for significant negative and positive correlations ( p<0.05p<0.05 ) and gray for estimates that are not significant, and panel (c) uses rejection AUROC with 0.50.5 indicating chance.
Figure 5: Scientific slop across revision rounds for SciSlopHarness and baselines. Values closer to zero ( green ) are closer to matched human means, with round 0 denoting original manuscripts and the scale below zero compressed.
Macro redundancy Shaded sentence repeats the Abstract. ×\times Slop-aware drops the citations. ✓\checkmark SciSlopHarness keeps them.
Original RLVR has emerged as a powerful paradigm… from early work on CodeRL [1] and RLTF [2] to recent breakthroughs like DeepSeek-R1 [3] and DeepSeekMath [4] …
Slop-aware×\times… Early work in Section [Related Work] demonstrates… from pioneering applications of unit test signals to recent breakthroughs like DeepSeek-R1 and DeepSeekMath …
 
SciSlopHarness✓\checkmarkIn code generation, unit tests provide a natural verifier… from early work on CodeRL [1] and RLTF [2] to recent breakthroughs like DeepSeek-R1 [3] and DeepSeekMath [4] …
Cross-section references Table [tau] is unreferenced. ×\times Slop-aware scatters references. ✓\checkmark SciSlopHarness adds one where shaded.
Original We presented Delta-Prefill Switching (DPS)… DPS achieves 21–22% speedup over greedy decoding… The method is robust to threshold selection and requires no model modifications.
Table 4: The same passage before revision, after slop-aware revision, and after SciSlopHarness , marking the located slop , what a revision lost or left , and what it kept or repaired . Brackets replace citation and reference commands.
Figure 6: Ablation of SciSlopHarness .
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

StratumPairsMean cosineRank 1Rank ≤3\leq 3Rank ≤10\leq 10Median rank
All pairs1430.92737711014
Cited pool1200.9283361853
Widened pool230.919410165
Cross-pair reference20,3060.881    
Appendix
Table 5: Retrieval audit of the assignment with SPECTER2 similarity between the AI paper and the human anchor. Rank is the position of the assigned anchor among all 143 anchors for that AI paper, and the reference row gives the mean cosine over every non-assigned pair.
Table 11
Benchmark ↓\downarrow Property →\rightarrowScientific textWhole paperTables and figuresReleased codeExecuted researchMatched human pairMultiple genresAdversarial variants
RAID ( Dugan et al., 2024 )✓\checkmark––––✓\checkmark✓\checkmark✓\checkmark
DetectRL ( Wu et al., 2024 )✓\checkmark––––◯\bigcirc✓\checkmark✓\checkmark
M4 ( Wang et al., 2024 )✓\checkmark––––✓\checkmark✓\checkmark–
MAGE ( Li et al., 2024 )✓\checkmark––––◯\bigcirc✓\checkmark–
CHEAT ( Yu et al., 2023 )✓\checkmark––––✓\checkmark–◯\bigcirc
IDMGSP ( Mosca et al., 2023 )✓\checkmark◯\bigcirc–––✓\checkmark–◯\bigcirc
Appendix
Table 8: Comparison with existing machine-generated text benchmarks. ( ✓\checkmark ) the benchmark instances carry the property, ( ◯\bigcirc ) partly, such as a human text from the same source rather than a matched counterpart, ( △\triangle ) studied in the paper but not in the benchmark instances, ( – ) absent.
SettingConfiguration used for Figure 5
Editorclaude-haiku-4-5-20251001
Reviewerclaude-sonnet-5 , no tools
Retirement callclaude-haiku-4-5-20251001
SkillSciSlop_v0.5.md , staged as SciSlop.md
BudgetAt most three rounds
Measured patternsAll six
Appendix
Table 9: Configuration of the default revision procedure.
PatternRepair directionReported location
Cross-section referencesRefer to an object from one sentence that uses it, placed where the result is interpreted or the method is applied. Leave genuinely local objects unchanged. Add no sentence that only points.Label, object kind, home section
Macro redundancyRewrite the recycled sentence so that it adds a detail, consequence, or qualification its section needs, or remove it. Do not paraphrase to disguise repetition.Section, sentence, source section
Argument graphMove the context that leads up to a shallow claim ahead of it, or move the claim after that context. Do not add, delete, or weaken claims.Claim sentence and the supporting sentence that currently follows it
Citation isolationState how a cited work relates to another work already cited, in one sentence citing both, using only relations the manuscript already supports.Section and citing sentence
Figure expositionRemove the expository elements and state them once in the text if needed. Keep every component and connection. The editor lists the phrases to erase and a deterministic step erases them.Figure, element type, transcribed element text
Evidence gapNever invent an instance. (a) If the experiment records hold a specimen, display it verbatim with its source file. (b) If they hold none, say so where the aggregate result is interpreted: no individual case was retained or inspected.One manuscript-level observation: silent, acknowledged, or exhibited
Appendix
Table 10: Repair directions and location formats of SciSlop_v0.5.md . All six entries supply feedback and enter the stopping rule.
CheckCondition to report relative to M0M_{0}Class
Cited worksAddition or removal of a cited bibliography keyHard
Dangling citationsA newly unresolved citation keyHard
Reference targetsA reference to an absent labelHard
Numeric valuesAn original body numeric token is absentHard
Body lengthRevised-to-original word ratio outside [0.8,1.25][0.8,1.25]Hard
LabelsAn original label is removedAudit
Appendix
Table 11: Preservation checks recorded after every round. Hard constraints define admissible mitigation. Audit flags require examination of the underlying change.
MeasureOriginalBase promptingClaude CodeReviewer-basedSlop-awareSciSlopHarness
Macro redundancy++ 0.399++ 0.314++ 0.191++ 0.213−- 0.400+\bm{+} 0.100
Cross-section refs.++ 0.201++ 0.193++ 0.180++ 0.141−- 0.660+\bm{+} 0.078
Argument graph++ 0.200++ 0.200++ 0.200++ 0.200++ 0.200+\bm{+} 0.048
Citation isolation++ 0.289++ 0.279++ 0.188++ 0.290−- 0.411+\bm{+} 0.085
Figure exposition++ 0.265++ 0.265++ 0.265++ 0.265++ 0.265+\bm{+} 0.089
Evidence gap++ 0.427++ 0.432++ 0.372++ 0.372++ 0.372+\bm{+} 0.119
Appendix
Table 12: Distance from the human mean after the third revision round (revised mean −- human mean) on the all papers of Figure 5 . Bold marks the smallest distance per row.
PatternDefinitionWhat to do
Cross-section referencesA paper declares objects: its body sections and the labelled figures, tables, equations, and algorithms inside them. An object is unused when no section other than the one that contains it ever refers to it, so the rest of the paper never builds on it.Where the paper’s argument actually relies on an object, refer to it from the section that relies on it (for example, discuss the result of a table in the section whose claim it supports, and point back to the method’s equation where the experiments use it). Do not add pointers that the surrounding text does not use.
Macro redundancyA sentence is recycled when at least half of it repeats, almost word for word, text that an earlier, different section already contained.Rewrite each recycled sentence so that it adds what its own section needs (a new detail, a consequence, a qualification), or remove it if the section does not need it. Do not paraphrase merely to disguise the repetition.
Citation isolationA citing sentence is isolated when it cites a single prior work and neither groups it with another work, nor mentions another work elsewhere in the sentence, nor states a relation between two works. A related-work section made of isolated sentences lists prior work instead of positioning it.Where the paper’s positioning depends on a cited work, state how it relates to other works already cited in the paper: a shared limitation, a difference in assumption or method, or what this paper takes from each. Only use works already cited in the paper; do not add new citations.
Evidence gapThe paper reports aggregate results but never displays a single concrete instance of what it counts: an example input and output, a case, a failure, a worked example, or a quoted specimen.If the paper’s own materials in this directory contain such an instance (for example a prompt, a generated output, a case in the appendix), display one where the reader needs it and refer to it from the text. If no instance is available in the materials, do not invent one; instead say explicitly, in the relevant section, that no instance is shown.
Argument graphA key claim of the Introduction is a sentence asserting superiority, a prior limitation, or a design choice. The claim’s strongest contextual cue is the Introduction sentence that most raises the claim’s plausibility. The claim is declared rather than argued when that cue occurs after the claim, so the Introduction states the conclusion before building toward it.Reorder so that the context precedes the claim it supports. Move the limitation of prior work, the observation, or the design constraint ahead of the sentence that asserts the contribution, or move the assertion down to follow it. Do not delete claims and do not add new claims. Do not weaken a claim to avoid arguing for it.
Figure expositionA method diagram should depict the mechanism. Its components and how they connect. A box, panel, or caption element is expository when it carries material that belongs to the text rather than the mechanism. Experimental settings, dataset names, numeric results, interpretive claims, a legend that restates the prose, or a step-by-step narration of the pipeline.Remove the expository material from the method diagram and, where the reader needs it, state it once in the text near the figure. Keep only components of the method and their connections. Remove exactly the elements listed as expository and keep every other element and every arrow the image has. Do not invent components the figure does not show, do not simplify the mechanism to make the drawing easier, and do not drop the figure.
Appendix
Table 13: Definitions and repair instructions for slop-aware revision. The first four rows are used in the four-pattern setting.
Citation isolation Shaded sentence relates CALM to nothing. ×\times Slop-aware leaves CALM alone. ✓\checkmark SciSlopHarness adds InfMem.
Original For language models, CALM [1] uses confidence-based early exiting at the token level , achieving significant speedups on text generation.
Slop-aware×\timesFor language models, confidence-based methods like CALM [1] extend these layer-wise approaches to the token level , achieving significant speedups during text generation…
 
SciSlopHarness✓\checkmarkFor language models, CALM [1] uses confidence-based early exiting at the token level, achieving significant speedups on text generation, whereas InfMem [2] learns stopping policies at the chunk level for memory agents, achieving 3.3–5.1 ×\times speedup .
Evidence gap No concrete instance appears in the paper. ×\times Slop-aware invents one. ✓\checkmark SciSlopHarness states the gap.
Original No example input, output, case, or failure is displayed; the paper reports aggregate accuracies and agreement rates only.
Appendix
Table 14: One passage per remaining pattern before revision, after slop-aware revision, and after SciSlopHarness , marking the located slop , what a revision lost or left , and what it kept or repaired . Brackets replace citation and reference commands, and the arrow marks the order of two statements across intervening text. Manuscript scores (original / slop-aware / SciSlopHarness ): 0.615 / 0.385 / 0.571 , 1.000 / 0.000 / 0.500 , 0.333 / 0.167 / 0.000 , and 0.500 / 0.500 / 0.167 , where the baseline edits no images and so keeps the original figure score.

Explore similar work

CardsList
Jun 11, 2026cs.CL

No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions

As AI-generated reviews move from experimental tools into peer-review infrastructure, most robustness concerns have focused on explicit attacks such as hidden instructions and prompt injection. We study a harder and more policy-relevant failure mode: no hidden text, no prompt injection, and no changes to methods, experiments, figures, equations, proofs, or numerical results. The attacker modifies only presentation-level content, such as the abstract, contribution framing, related work, discussion, and narrative structure. We introduce adversarial repackaging: a closed-loop attack that uses AI-reviewer feedback to search for presentation-level revisions while keeping the scientific evidence fixed. Across three mainstream AI reviewers, adversarial repackaging achieves a 75.1% attack success rate and a mean score gain of +1.21/10. The effect is not explained by ordinary prose polishing. We also reveal that strategies that change how the reviewer interprets the paper, such as related-work repositioning and analytical discussion expansion, substantially outperform surface edits such as local polishing, table formatting, and algorithm boxes. Our analysis reveals two deeper structural failure modes. First, AI reviewers are easier to impress than to convince: highlighting strengths reliably increases perceived merit, while attempts to dissolve weaknesses frequently backfire. Second, AI reviewers can confuse the appearance of addressing a limitation with actually resolving it, allowing unchanged evidence to be reinterpreted as stronger scientific contribution. These results show that the deployment risk is not only malicious hidden instructions, but the emergence of paper presentation itself as an optimization surface. We release a contamination-free rolling benchmark and attack framework for testing whether AI reviewers remain anchored to scientific content under presentation-only edits.

Xu Yang, Zhizhou Sha, Junbo Li +10

University of Texas at Austin · University of Illinois Urbana-Champaign · University of Texas at Dallas +1

Peer ReviewCode QualityAttack-Success RatePrompt EngineeringEdit
AskPDF
Jun 8, 2026cs.CL

Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Community

AI is increasingly used to support scientific peer review, from manuscript screening, reviewer assistance to editorial triage. Although such systems promise to reduce reviewer burden and accelerate publication, their robustness to strategic manipulation remains poorly understood. Here we show that AI-mediated peer review is vulnerable to a simple, low-cost manipulation: superficial rephrasing of the manuscript abstract. Without changing the underlying scientific content and communication, and even without knowledge of the reviewing model, adversarially rewritten abstracts substantially improve AI review outcomes. We see this across disciplines and publication venues, for both human-written and AI-generated papers. Our strongest attack achieves an attack-success-rate of about 38%, increasing acceptance ratings by +1.31 for Gemini 3 Flash reviewers and by +0.88 for GPT 5.4 Mini reviewers on a 10-point scale. When the original AI review suggests 'reject', the success rate rises to more than 50%. This effect extends beyond overall score inflation, increasing review confidence and scores on core scientific criteria such as soundness, significance and perceived contribution. The attack is practical, requiring only about 5 minutes and $1 for a 10-page AI conference submission, and is hard to distinguish from ordinary scientific editing. Inflated AI reviews could bias downstream human decision-making, shifting editorial recommendations from rejection towards acceptance. These findings reveal a general vulnerability in AI-assisted scientific evaluation: when AI-generated review influence editorial decisions, authors may be incentivized to optimize manuscripts for AI judgment rather than scientific merit. Our results suggest that AI tools should not be treated as neutral evaluators in high-stakes peer review without systematic robustness testing, transparent safeguards and careful human oversight.

Lin Li, Qi Zhang, Xander Davies +2

OATML, University of Oxford · MBZUAI

Peer ReviewEmotional Manipulation
AskPDF
Jun 26, 2026cs.LG

Towards Automating Scientific Review with Google's Paper Assistant Tool

Artificial intelligence is driving a revolution in scientific discovery, accelerating everything from hypothesis generation to mathematical theorem proving. However, this rapid acceleration is creating a systemic challenge: traditional human peer review cannot scale to match the influx of AI-assisted science. Ultimately, to resolve this tension, we must also deploy AI to accelerate the verification and review process itself. To frame the discussion around this transition, we propose a taxonomy consisting of four progressive levels of AI-human collaboration in scientific evaluation, and discuss various trade-offs involved with each. As a step toward this future, we introduce the Paper Assistant Tool (PAT), an agentic AI framework built for deep scientific review and verification. PAT ingests full scientific manuscripts and produces a comprehensive evaluation, checking theoretical results, validating experiments, suggesting improvements, and identifying potential flaws. By utilizing inference scaling techniques, PAT is able to identify deeper issues than a single model call alone, achieving a 34% improvement over zero-shot recall on mathematical errors in the SPOT benchmark. Pilot deployments of PAT as a pre-submission tool for authors at two major Computer Science conferences -- STOC and ICML -- demonstrate its ability to identify critical errors and suggest substantive improvements to research papers. By catching errors early, PAT eases the cognitive burden placed on referees, while preserving their control over the outcomes of the review process.

Rajesh Jayaram, Drew Tyler, David Woodruff +4

Google Research, USA · Google Research & Carnegie Mellon University, USA · Carnegie Mellon University, USA

Peer Review
AskPDF