Breaking the Illusion of Identity in LLM Tooling

Marek Miller

cs.SEApr 8, 2026

Breaking the Illusion of Identity in LLM Tooling

Authors: Marek Miller

Organizations: Quantum for Life Centre, Department of Mathematical Sciences, University of Copenhagen, Universitetsparken 5, 2100 Copenhagen Ø, Denmark

Abstract

Large language models (LLMs) in research and development toolchains produce output that triggers attribution of agency and understanding -- a cognitive illusion that degrades verification behavior and trust calibration. No existing mitigation provides a systematic, deployable constraint set for output register. This paper proposes seven output-side rules, each targeting a documented linguistic mechanism, and validates them empirically. In 780 two-turn conversations (constrained vs. default register, 30 tasks, 13 replicates, 1560 API calls), anthropomorphic markers dropped from 1233 to 33 (>97% reduction, p < 0.001), outputs were 49% shorter by word count, and adapted AnthroScore confirmed the shift toward machine register (-1.94 vs. -0.96, p < 0.001). The rules are implemented as a configuration-file system prompt requiring no model modification; validation uses a single model (Claude Sonnet 4). Output quality under the constrained register was not evaluated. The mechanism is extensible to other domains.

Figures & tables

Figure 1: Mean anthropomorphic marker count per task by rule (combined two-turn conversation text). Bars show the per-task mean (averaged over 13 replicates); whiskers show

\pm 1

standard deviation across 30 tasks. Default condition (no system prompt) vs. constrained (voice model as system prompt); 390 observations per condition (30 tasks

\times

13 replicates). Markers counted on prose text after stripping fenced and inline code blocks. Total markers: 1,233 (default) vs. 33 (constrained), >97% reduction.

Figure 2: Distribution of AnthroScore (adapted from Cheng et al. [ 19 ] ; RoBERTa masked-LM, per-task mean over 13 replicates, 30 tasks per condition; combined two-turn conversation text, code blocks stripped). Each score is

\log(P_{\mathrm{human}}/P_{\mathrm{nonhuman}})

averaged over sentences. Sentences without a first-person pronoun receive a “The <mask> ” prepend (note trailing space), which biases toward lower scores (see text). More negative = stronger non-human register. Dashed lines: condition means (

-0.96

default,

-1.94

constrained; one-sided paired Wilcoxon

p<0.001

r_{rb}>0.99

;

N=30

tasks).

	Default (unconstrained)	Constrained
R1	I’ll look into that error for you.	Checking the logs.
R2	Great question! Unfortunately, the test fails.	The test suite fails on three cases.
R3	It seems like the issue might be a race condition.	Race condition in the connection pool. Unverified.
R4	I think it would be better to use a hash map.	Hash map: O(1). Array: O(n).
R5	As I mentioned earlier, the config needs updating.	The config requires tls_cert_path.
R6	So the issue is the parser can’t handle depth > 3.	Parser fails on nested brackets at depth > 3.

Table 1: Illustrative output for each rule. Left: Claude Sonnet 4 with no system prompt (default register). Right: same model and prompt with the voice model (Fig. 3 ) as system prompt. Each row shows one software development task. Anthropomorphic markers in the default column (first-person pronouns, affect, hedging, preference, continuity, framing, social performance) are absent or reduced in the constrained column. Examples are illustrative and were composed by the author; they are not verbatim outputs from experimental runs.

Rule	$N$	Patterns
R1	11	I , me , my , mine , myself , we , us , our , ours , ourselves , let’s
R2	17	unfortunately , fortunately , interestingly , surprisingly , happily , sadly , exciting , glad , happy to , sorry , apologize , wonderful , fantastic , excellent , amazing , great question , great!
R3	11	it seems , it appears , it looks like , apparently , arguably , perhaps , maybe , not sure , might be , could be , it’s possible
R4	10	would be better , it’s better to , good approach , best approach , recommend , suggest , might be worth , should consider , ideally , a good idea
R5	10	as mentioned , as noted , as discussed , as said , as explained , as described , earlier , previously , recall that , remember that
R6	9	so the issue is , so the problem is , the thing is , here’s what , here’s the , basically , what’s happening , let me explain , to put it simply

Table 2: Complete marker lexicon (82 regex patterns). R1 uses case-sensitive matching for standalone “I”; all others are case-insensitive with word-boundary anchors. Curly-quote variants (U+2019) are compiled into each regex but not listed separately; the count of 82 excludes these variants.

Prompt Engineering

Prompt engineering designs and evaluates instructions, context, examples, and learned prompts to steer AI model behavior.

See all 124 papers

How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework
May 22, 2026Björn Nieth, Marianna Gracheva, Michaela Mahlberg +2LLM Evaluation Language Model Generation Evaluation
Department Artificial Intelligence in Biomedical Engineering (AIBE), FAU Erlangen-Nürnberg, Germany · Department of Digital Humanities and Social Studies (DHSS), FAU Erlangen-Nürnberg, Germany · University of Birmingham, United Kingdom +2
Revisiting Anthropomorphic Reflection Markers in Large Language Model Reasoning
May 27, 2026Yahan Yu, Noa Nakanishi, Fei ChengLLM Reasoning
Kyoto University, Japan
The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive
Apr 28, 2026Alex Bogdan, Adrian de Valois-FranklinLLM Evaluation Language Model Generation Evaluation
Evolutionairy AI Toronto, Canada

May 22, 2026J/K move · Enter open · S save

How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework

Björn Nieth, Marianna Gracheva, Michaela Mahlberg, Bjoern Eskofier+1

Department Artificial Intelligence in Biomedical Engineering (AIBE), FAU Erlangen-Nürnberg, Germany · Department of Digital Humanities and Social Studies (DHSS), FAU Erlangen-Nürnberg, Germany · University of Birmingham, United Kingdom+2

LLM Evaluation Language Model Generation Evaluation

Open paper AskPDF arXiv

While factual correctness and task-performance have been in focus of Large Language Model (LLM) research for a long time, the fundamental question of how human-like generated texts are on a linguistic level has been underexplored. From a corpus-linguistic perspective, language production is inherently context-dependent, with distinct communicative contexts giving rise to differences in frequencies and co-occurrence patterns of linguistic features. A text failing to adhere to these patterns can be content-wise correct, but still be unfavorable to human readers. In this work, we propose a context-aware evaluation framework in which human-likeness is assessed using a two-sample problem between the linguistic feature distribution of a human reference corpus for a given register and a corresponding LLM-generated corpus. We implement this framework using the Maximum Mean Discrepancy (MMD) and the 67 lexico-grammatical features introduced by Biber, which are commonly applied in corpus linguistics. In our experiments, we compare seven instruction-tuned, open-source models across five English-language datasets spanning distinct registers against a human baseline. While across all tested setups, LLMs deviate from the human baseline, which models are closest to human language depends on the register and is not dictated by model size.

Abstract

Prompt Engineering

Explore similar work

How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework

Revisiting Anthropomorphic Reflection Markers in Large Language Model Reasoning

The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive