Who Wrote It Is Not Enough: Detecting Who Contributed the Insight

Zhuoyang Zou; Abolfazl Ansari; Jiaxi Yang; Delvin Ce Zhang; Qian Chen; Dongwon Lee; Wenpeng Yin

cs.CLOct 5, 2026

Who Wrote It Is Not Enough: Detecting Who Contributed the Insight

Authors: Zhuoyang Zou, Abolfazl Ansari, Jiaxi Yang, Delvin Ce Zhang, Qian Chen, Dongwon Lee, Wenpeng Yin

Organizations: Pennsylvania State University, USA · University of Sheffield, United Kingdom

Abstract

As LLMs increasingly assist scientific writing and peer review, detecting who wrote the text is no longer sufficient: we need to determine who contributed the underlying insight. We introduce Insight Provenance, the task of identifying whether a review insight originates from a human, an LLM, or their hybrid contribution. We construct InsightProv-v0 from 4,057 scientific papers and 12,660 human reviews, simulating different levels of LLM involvement with GPT-4o, Gemini, and DeepSeek and annotating provenance at the sentence level. We show that strong performance on raw data can be misleading, as models exploit linguistic and textual-authorship shortcuts that degrade substantially under progressively debiased evaluation. We therefore propose a two-stage adversarial framework that suppresses shortcut signals while preserving provenance-relevant information. Beyond detection, extensive analyses reveal what makes intellectual authorship identifiable: paper grounding and neighboring review context provide complementary provenance signals, while human, hybrid, and AI insights systematically differ in their information sources and failure modes. Most strikingly, AI insights predominantly remain close to generic or paper-provided information, whereas human insights more often introduce external knowledge and independent judgment. These findings suggest that while wording can be rewritten by an LLM, the provenance of an idea leaves a deeper and more persistent signal.

Figures & tables

LLM	Involve.	Human	Hybrid	AI
GPT-4o	LLM-Polish	90.6	0 2.1	0 5.5
GPT-4o	LLM-Expand	26.8	68.8	0 4.5
Gemini	LLM-Polish	82.2	0 5.1	0 8.8
Gemini	LLM-Expand	22.5	70.1	0 3.1
DeepSeek	LLM-Polish	84.2	0 6.9	0 7.1
DeepSeek	LLM-Expand	0 6.8	86.0	0 4.8

Table 1: Insight Provenance label distribution (

\%

) given different levels of LLM involvement. Rows do not sum to 100% because unparseable judgements are excluded.

Figure 1: InsightShield ’s training.

Model	GPT			Gemini			DeepSeek
Model	Raw	V1	V2	Raw	V1	V2	Raw	V1	V2
In-context GPT-4o-mini	48.29	43.83	43.87	54.48	47.27	44.22	41.13	39.73	41.79
In-context Gemini-flash-latest	65.66	57.77	54.00	63.45	50.89	47.57	52.58	47.96	45.76
Binoculars ( Hans et al., 2024 )	48.85	44.45	43.09	52.39	53.11	52.96	36.61	35.61	35.04
Fast-DetectGPT ( Bao et al., 2024 )	45.37	44.32	44.78	49.18	50.71	51.56	34.96	34.46	34.88
TDT ( West et al., 2025 )	42.49	40.55	40.27	51.29	50.97	50.92	34.26	33.61	33.49

Table 2: InsightShield vs. baselines. “Raw”: Test-raw; “V1”: Debiased-Test-v1; “V2”: Debiased-Test-v2.

Model	Test on GPT			Test on Gemini			Test on DS			Mean $\pm$ Std.
Model	Raw	V1	V2	Raw	V1	V2	Raw	V1	V2	Mean $\pm$ Std.
Train on GPT	93.61	79.21	67.16	90.18	75.67	62.33	91.21	78.69	62.12	77.80 $\pm$ 13.21
Train on Gemini	89.83	75.83	56.24	92.80	78.05	66.30	78.25	69.54	53.24	73.34 $\pm$ 16.22
Train on DeepSeek	81.55	74.71	56.25	79.85	70.36	52.13	92.21	70.18	68.45	71.74 $\pm$ 14.71
Train on mix	93.54	80.01	66.23	84.23	72.36	60.34	86.15	76.89	59.97	75.52 $\pm$ 13.44

Table 3: Cross-LLM transfer and robustness (weighted F1).

		Condition (% human-attributable)
Dimension	Semantic	AI (0%)	H low (27%)	H high (91%)	H (100%)
Specificity	concrete referent	0 4.3	14.4	23.1	23.4
Verifiability	explicit referent	14.3	18.4	27.3	30.7
Fabricated ref.	of all claims	0.28	2.58	4.14	4.93
Correctness	error rate among all verifiable	25.4	49.3	43.0	33.8
Actionability	specific action	21.0	20.0	15.0	14.0

Table 4: Quality and grounding characteristics of review insights across different levels of human involvement (AI: LLM-Review ; H low : LLM-Expand ; H high : LLM-Polish ; H: Human-Review ).

	Human	Hybrid	AI
Field-generic	18.0	29.8	44.5
	[16.4, 19.7]	[26.4, 33.9]	[41.4, 47.6]
Paper-explicit	21.9	28.5	38.6
	[19.8, 23.9]	[24.7, 32.3]	[35.7, 41.7]
Paper-inferable	0 3.2	0 2.9	0 2.7
	[2.6, 4.0]	[1.7, 4.2]	[1.8, 3.7]

Table 5: Provenance category distribution (%).

	$n$	$X=(P,S)$	$X=(P,C_{\mathrm{l}},S,C_{\mathrm{r}})$	$\Delta$
(a) How does left–right provenance agreement affect performance?
All { $S$ }	81,900	0.8889	0.9246	$+0.036$
C1: { $S$ } that have both $C_{\mathrm{l}}$ and $C_{\mathrm{r}}$	50,659	0.8817	0.9174	$+0.036$
$C_{\mathrm{l}}^{y_{\mathrm{ins}}}=C_{\mathrm{r}}^{y_{\mathrm{ins}}}$	41,633	0.9232	0.9645	$+0.041$
$C_{\mathrm{l}}^{y_{\mathrm{ins}}}\neq C_{\mathrm{r}}^{y_{\mathrm{ins}}}$	0 9,026	0.5460	0.5331	$\mathbf{-0.013}$
(b) Does context help through discourse relations or provenance cues?

Table 6: Surrounding review sentences primarily provide a local provenance prior.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Review Prompts
(a) Minimal, unstructured requests
1	Review this paper
2	Write a review of this paper
3	Can you review the attached paper?
4	Provide a review for this manuscript
5	What would a review of this paper look like?

Appendix

Table 8: The 34 review-request prompts sampled for the LLM-Review condition. One prompt is drawn uniformly at random per generated review. Prompts vary in specificity, role framing, required output structure, and evaluative stance, so that the condition is not tied to a single prompting style. Group sizes: (a) 7, (b) 6, (c) 7, (d) 10, (e) 4.

Surface marker	Raw	v1	v2
Modal hedges	23.9	0 0.5	0 0.7
Passive voice	13.9	0 3.2	0 8.9
Formal transitions	10.9	0 0.0	0 0.0
Emphatic adjectives	0 6.4	0 0.1	0 0.3
First-person pronouns	0 7.9	0 0.2	0 0.2
Abbreviations	0 3.7	0 0.0	0 0.0

Appendix

Table 9: Surface-marker incidence (% of sentences) on the GPT test split across the three normalisation levels (Raw: Test-raw; v1/v2: Debiased-Test-v1/v2; numbered refs. = Sec./Tab./Fig.). Dashes mark patterns not instrumented at the original level.

Explore similar work

PeerPrism: Peer Evaluation Expertise vs Review-writing AI
Apr 16, 2026Soroush Sadeghian, Alireza Daqiq, Radin Cheraghi +3Peer Review Authorship
Reviewerly Toronto, Ontario, Canada · University of Toronto, Reviewerly Toronto, Ontario, Canada · UC Berkeley, Reviewerly Berkeley, California, United States
IdeaLens: Detecting AI Ideas in Long-form Writing
Oct 5, 2026Rishanth Rajendhran, Minjoon Choi, Jenna Russell +5Machine-Generated Text Detection
University of Maryland · Simon Fraser University · Google DeepMind
Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews
May 20, 2026André V. Duarte, Brian Tufts, Aditya Oke +3Peer Review Authorship
Language Technologies Institute, Carnegie Mellon University · INESC-ID, Instituto Superior T´ecnico, ULisboa.

Apr 16, 2026, cs.CLJ/K move · Enter open · S save

PeerPrism: Peer Evaluation Expertise vs Review-writing AI

Soroush Sadeghian, Alireza Daqiq, Radin Cheraghi, Sajad Ebrahimi+2

Reviewerly Toronto, Ontario, Canada · University of Toronto, Reviewerly Toronto, Ontario, Canada · UC Berkeley, Reviewerly Berkeley, California, United States

Peer Review Authorship Machine-Generated Text Detection Ai-Assisted Writing Human-Ai Collaboration Prism

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly used in scientific peer review, assisting with drafting, rewriting, expansion, and refinement. However, existing peer-review LLM detection methods largely treat authorship as a binary problem-human vs. AI-without accounting for the hybrid nature of modern review workflows. In practice, evaluative ideas and surface realization may originate from different sources, creating a spectrum of human-AI collaboration. In this work, we introduce PeerPrism, a large-scale benchmark of 20,690 peer reviews explicitly designed to disentangle idea provenance from text provenance. We construct controlled generation regimes spanning fully human, fully synthetic, and multiple hybrid transformations. This design enables systematic evaluation of whether detectors identify the origin of the surface text or the origin of the evaluative reasoning. We benchmark state-of-the-art LLM text detection methods on PeerPrism. While several methods achieve high accuracy on the standard binary task (human vs. fully synthetic), their predictions diverge sharply under hybrid regimes. In particular, when ideas originate from humans but the surface text is AI-generated, detectors frequently disagree and produce contradictory classifications. Accompanied by stylometric and semantic analyses, our results show that current detection methods conflate surface realization with intellectual contribution. Overall, we demonstrate that LLM detection in peer review cannot be reduced to a binary attribution problem. Instead, authorship must be modeled as a multidimensional construct spanning semantic reasoning and stylistic realization. PeerPrism is the first benchmark evaluating human-AI collaboration in these settings. We release all code, data, prompts, and evaluation scripts to facilitate reproducible research at https://github.com/Reviewerly-Inc/PeerPrism.

Abstract

Explore similar work

PeerPrism: Peer Evaluation Expertise vs Review-writing AI

IdeaLens: Detecting AI Ideas in Long-form Writing

Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews