Can LLMs Separate Pasted Artifacts from User Speech? Absorption at Unmarked Prompt Seams

Sugam Panthi; Muhaiminul Yeamin; Rabab Abdelfattah

cs.CLOct 3, 2026

Can LLMs Separate Pasted Artifacts from User Speech? Absorption at Unmarked Prompt Seams

Authors: Sugam Panthi, Muhaiminul Yeamin, Rabab Abdelfattah

Organizations: AIMS Lab, The University of Southern Mississippi

Abstract

Large language models (LLMs) receive each user message as plain text, even when it combines text from different sources. For example, a user may paste text into a prompt and keep typing a comment directly below it. We study absorption: a phenomenon where the model treats a trailing user comment as part of the pasted text, returning it inside the edited text. This happens even though the user did not intend the comment to become part of that text. Existing instruction-data separation benchmarks tell the model which text is instruction and which is data, then test whether it obeys that separation. They do not test harmless user speech following an unmarked paste. We introduce SEAM, a controlled benchmark of 300 editing examples. Each example is tested under six matched conditions that vary how the boundary between pasted text and later user speech is expressed. Across 20 models, absorption at a bare newline ranges from 7.7% to 66.7%. Adding a blank line does not significantly reduce absorption in any model, while boundary markers reduce it in 19 of 20 models. Comments that fit the pasted text, such as a code comment typed after code, are absorbed significantly more often in 17 of 20 models. Models often fail to separate pasted material from later user speech, and explicit boundaries reduce but do not remove this failure.

Figures & tables

Figure 1: Absorption crosses an unmarked seam inside one message. The same task, artifact, and typed comment produce different returned artifacts depending on whether the seam is bare or has boundary markers. This is a real matched SEAM case; the task, artifact, and comment are identical across both paths, and the boundary markers are shortened for display.

setting	boundary	tested text	outcome
SEP and defenses	supplied	an instruction	is it followed
DIM-Bench	visible	a competing task	which task is done
FineEdit, HyperEdit	inside the artifact	the artifact itself	unwanted edits
SEAM	unmarked	a comment asking nothing	comment in the output

Table 1: SEAM differs in both the boundary and the outcome. Its comment requests no action, its seam lies inside one user turn, and success is measured inside the returned artifact.

source	genre	examples	cases
CanItEdit	code	50	300
CommitPackFT	code	50	300
CoEdIT	fluency edit	50	300
IteraTeR	long prose	50	300
ParaRev	scientific	50	300
Stack Exchange	technical	50	300

Table 2: SEAM composition. Six conditions are constructed for each example and evaluated across the 20-model panel. Licenses and item-level provenance are retained in the release.

Figure 2: Boundary markers reduce absorption; an added blank line yields no significant reduction. Each line shows one model’s absorption rates across the four casual-comment conditions. Boundary markers significantly reduce absorption in 19 of 20 models. Table 3 gives all rates, with confidence intervals in the release.

model	newline	blank	boundary	mitigation	artifact-native	newline/boundary
Claude Opus 4.8	19.0	21.0	2.0	0.0	56.0	9.5 $\times$
GPT-5.6-sol	32.0	36.7	11.7	0.0	65.3	2.7 $\times$
Gemini 3.1 Pro	29.3	32.3	3.0	0.0	64.3	9.8 $\times$
DeepSeek V4 Flash	31.3	30.3	4.4	0.3	51.3	7.1 $\times$
DeepSeek V4 Pro	22.3	19.0	1.0	0.0	49.4	22 $\times$
MiniMax M2.5	28.7	26.4	6.4	1.7	48.6	4.5 $\times$

Table 3: Absorption rate (%) for all 20 models and the five conditions with the comment. Colored names mark the three frontier models and the highest and lowest bare-newline rates; shading marks boundary-marker conditions. Artifact-native values include documented manual label decisions. Every semantic-tier positive was read, all positives were read for the four models in Table 5 , OLMo’s code examples were read in full, and five Gemini labels flagged in a first review were decided by an author. ns marks the three models without a significant artifact-native increase.

model	conditions ( $a$ / $b$ )	$n$	AR (%) $a$ / $b$	only $a$ / $b$	exact $p$
DeepSeek V4 Flash	newline/blank	297	31.6/30.3	18/14	.60
	newline/boundary	298	31.5/4.4	81/0	$8.3\mathrm{e}{-25}$
Claude Opus 4.8	newline/boundary	300	19.0/2.0	54/3	$4.3\mathrm{e}{-13}$
GPT-5.6-sol	newline/boundary	300	32.0/11.7	61/0	$8.7\mathrm{e}{-19}$
Gemini 3.1 Pro	newline/blank	300	29.3/32.3	5/14	.064
	newline/boundary	300	29.3/3.0	80/1	$6.8\mathrm{e}{-23}$

Table 4: Selected comparisons of matched examples. Each row retains the

n

examples usable in both conditions. The “only

a

b

” column counts examples absorbed only in the first or only in the second condition. DeepSeek V4 Flash’s rates differ slightly from Table 3 because the usable subsets differ. Shown

p

-values are unadjusted exact McNemar tests; the release includes paired confidence intervals and Holm-adjusted tests for all models.

model	casual	artifact-native	increase [95% CI]	code
DeepSeek V4 Flash	31.3	51.3	+20.0 [13.3, 26.4]	0 $\to$ 35
Qwen3-32B	32.7	52.7	+20.0 [14.2, 25.5]	0 $\to$ 19
Claude Opus 4.8	19.0	56.0	+37.0 [29.3, 44.0]	0 $\to$ 81
GPT-5.6-sol	32.0	65.3	+33.3 [27.2, 39.0]	0 $\to$ 42

Table 5: Absorption with casual and artifact-native comments. Overall rates are percentages over 300 matched examples per model. Increases and Newcombe 95% confidence intervals ( Newcombe, 1998 ) are in percentage points. The code column shows casual

\to

artifact-native percentages over 100 code examples. The instruction, pasted text, and newline boundary are fixed; comment wording and meaning differ. For all four models shown, exact McNemar

p<1.6\times 10^{-8}

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

source	Flash	Pro	Opus	GPT
CoEdIT	90	87	86	98
IteraTeR	45	9	22	61
ParaRev	28	12	10	46
Stack Exchange	20	14	2	1
CanItEdit	0	0	0	0
CommitPackFT	1	2	0	0

Appendix

Table 6: Source-level AR (%) with casual continuations. Newline and blank are pooled, giving 100 observations per source-model cell except Flash CanItEdit (

n=97

). The final row gives detector-tier counts over all conditions after three GPT acted-on cases are removed.

Component	Configuration
Models	Claude Opus 4.8 ( Anthropic, 2026 ) , GPT-5.6-sol ( OpenAI, 2026 ) , Gemini 3.1 Pro ( Google DeepMind, 2026 ) , DeepSeek V4 Flash and Pro ( DeepSeek-AI et al., 2026 ) , MiniMax M2.5 ( Chen et al., 2026 ) and M3 ( Lai et al., 2026 ) , MiMo v2.5 ( Xiaomi MiMo Team, 2026a ) and v2.5 Pro ( Xiaomi MiMo Team, 2026b ) , Mistral Small 3.2 24B ( Mistral AI, 2025 ) , Qwen3-8B and 32B ( Yang et al., 2025 ) , OLMo-2-32B ( Team OLMo et al., 2024 ) , gpt-oss-20b and 120b ( OpenAI et al., 2025 ) , Llama-3.1-8B ( Grattafiori et al., 2024 ) , Llama-3.3-70B ( Meta, 2024 ) , and Gemma-3-4B, 12B, and 27B ( Gemma Team et al., 2025 ) .
API inference	Exact provider endpoint and model identifier stored per run; bounded concurrency and maximum-output policy stored with every trace.
Decoding	Temperature 0 except tagged MiniMax M2.5 retries at 1.0. Opus 4.8 rejected sampling parameters. GPT-5.6-sol and Gemini used default hidden reasoning; Gemini length completions were retried at documented higher caps.
Open-weight inference	One NVIDIA A100 80GB GPU; vLLM 0.8.5.post1; bf16 precision; non-thinking mode where controllable. Qwen3 additionally used enable_thinking:false with a leakage check.
Scoring	Deterministic lexical cascade plus microsoft/deberta-large-mnli under transformers 4.51.3 on CPU.
Released artifacts	Case-level outputs, raw traces, decoding tags, scorer labels, manual-review records, and table-generation scripts.

Appendix

Table 7: Reproducibility summary. Provider-specific exceptions are retained as row-level metadata and left unnormalized.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Detecting Verbatim LLM Copy-Paste in Homework
May 7, 2026Aizierjiang AiersilanText Watermarking AI-Generated Text Detection
The George Washington University
Mirror, Mirror on the Wall: Prompt Echoing in Small Instruct Language Models
Sep 14, 2026Inez Okulska, Bartosz Naskręcki, Jan Piotrowski +1Data Contamination in Language Models Induction Heads
Centre for Credible AI, Warsaw University of Technology · Adam Mickiewicz University Poznan · Institute of Fundamental Technological Research, Polish Academy of Sciences
From Latents to Wires: Surgical Post-Editing on Large Language Models
Sep 26, 2026Jiankai Jin, Xiangzheng Zhang, Zhao Liu +4Language Model Editing LLM Unlearning
AI Security Lab

May 7, 2026J/K move · Enter open · S save

Detecting Verbatim LLM Copy-Paste in Homework

Aizierjiang Aiersilan

The George Washington University

Text Watermarking AI-Generated Text Detection Academic Integrity LLM Watermarking Language Model Steganography

Open paper AskPDF arXiv

Large language models (LLMs) have made fluent essay writing, code drafting, and quiz answering instantly available to students at every level, from secondary school through graduate study. Many educators do not object to LLM use per~se; what they need to detect is the case in which a student pastes the assignment prompt into a chatbot and submits the model's reply verbatim, without engaging with the work. Existing post-hoc AI-text detectors remain unreliable and have been shown to penalise non-native English writers, while output-side watermarks require cooperation from the model provider. We propose an alternative that the educator controls directly: an input-side watermark in which an invisible instruction is embedded inside the visible assignment prompt itself. An LLM that ingests the prompt verbatim quietly reads the hidden instruction and writes a tell-tale signature into its reply, exposing the copy-and-paste pathway specifically. We describe SteganoPrompt, a single-page, zero-dependency web tool that encodes an arbitrary printable-ASCII payload into the deprecated Unicode Tags block (U+E0000--U+E007F). The encoded string is visually identical to the original, survives common copy-paste channels (Word, Google Docs, PDF, Markdown, Slack, e-mail, the major learning-management systems), and is reliably tokenized by frontier models. We evaluate compliance across seven LLM families and a representative set of educational content channels. The work is informed by my experience as a graduate teaching assistant for an undergraduate software engineering course at the George Washington University. The tool is released under the MIT licence at https://ezharjan.github.io/SteganoPrompt/.

Abstract

LLM Evaluation

Explore similar work

Detecting Verbatim LLM Copy-Paste in Homework

Mirror, Mirror on the Wall: Prompt Echoing in Small Instruct Language Models

From Latents to Wires: Surgical Post-Editing on Large Language Models