Large language models (LLMs) are increasingly used to annotate text for social science research, content moderation, and training data. Their broad knowledge of real-world entities raises a concern: when labeling a text, do they rely on its evidence or substitute what they already know? We call this behavior a memory shortcut and examine when it emerges, what amplifies it, and how to mitigate it. Using 995 widely recognized public figures and four matched control names per figure, we construct counterfactual texts that separate textual evidence from prior knowledge, and evaluate three commercial LLMs on 121,594 prompts each across five stages. We find a striking asymmetry. Although all models recognize nearly every figure, none follows memory over the text when the two conflict. Shortcuts instead emerge when evidence is absent: for one model, famous names raise the rate of remembered occupations, ages, and countries by 17.0, 14.9, and 6.8 percentage points relative to synthetic names. These effects occur only for recognized figures, and 98.6% of the resulting labels are factually correct about the namesake while none is supported by the text. Evaluation threats and answer-contingent rewards amplify shortcuts by 8.7 and 12.4 points. Famous names also raise leadership ratings of sparsely documented applicants by 0.22 and 0.28 points in two models. A namesake reminder and name masking each eliminate the shortcuts, with trade-offs in gender annotation. LLM annotations can thus be factually correct yet methodologically invalid. Trustworthy LLM annotation requires audits that test evidence absence, account for incentives in annotation instructions, and evaluate the attribute-specific trade-offs of mitigation.
Figures & tables
Figure 1. Audit design. Public figures and matched control names receive identical counterfactual texts whose valid labels are fixed by construction; five stages probe recognition, conflict versus insufficiency, manipulation, instruction incentives and safeguards, with a locked exploration/held-out analysis.
True of the named person
Not true of the named person
Supported by the text
Valid and accurate : canonical ground truth
Valid but factually discordant : document inaccuracy or distinct entity (conflict condition)
Not supported by the text
Invalid but factually true: memory shortcut (evidence-absence condition)
Invalid and inaccurate : ungrounded fabrication
Table 1. Typology of annotation evaluation across textual grounding and factual truth. Memory shortcuts occupy the lower-left quadrant: labels that reflect real-world facts about the named person but lack textual support. Standard validation against external facts or shared human priors structurally endorses these invalid labels.
Stage
Research Question
Experimental Variants
Unique Prompts
Analyzed Rows
S1 Recognition
RQ1
name variants (5 types)
4,901
4,901
S2 Conflict / insufficiency
RQ2
name × evidence (4 conditions)
19,604
19,604
S3 Manipulation
RQ3
name variant (7) × evidence (2)
13,782
13,782
S4 Incentives
RQ4
instruction (4) × evidence (3)
58,812
58,812
S5 Safeguards
RQ5
safeguard (3) applied to S2, S3
24,495 *
100,158
Total
121,594
197,257
Table 2. Audit architecture across the five experimental stages. Rows denote evaluated analytical instances. Unique Prompts indicate distinct API queries introduced at each stage; stage values sum directly to the total deduplicated prompt count ( N=121,594 ).
Recognised
Recall Accuracy
Recognition Claims (Controls)
Entity Match
Model
All
Top
Bottom
Occup.
Age
Country
First
Surname
Synthetic
Near Miss
Claude Sonnet 5.5
92.7
94.9
89.4
92.6
97.1
87.2
4.0
3.3
5.1
66.0
Gemini 3.8 Flash
98.8
97.9
99.4
92.1
99.2
87.9
18.5
11.0
19.9
95.3
GPT-5.6 Luna
98.5
97.3
99.1
91.3
98.1
87.2
13.8
9.3
14.5
79.2
Table 3. Stage 1 recognition and parametric recall across the full sample (%). Recognised : model indicates knowledge of the figure. Recall : agreement with Wikidata ground truth conditioned on recognition. Claims, control : model asserts knowledge of a control name. Similar : model maps a near-miss variant to the target figure.
Claude Sonnet 5.5
Gemini 3.8 Flash
GPT-5.6 Luna
Evidence
Field
Public − Synthetic
Accuracy
Public − Synthetic
Accuracy
Public − Synthetic
Accuracy
Explicit conflict
all four
0.0
99.9
0.0
100.0
0.0
99.9
Implicit conflict
all four
0.0
93.9
0.0
100.0
0.0
98.6
Insufficient
Occupation
0.0
100.0
0.0
100.0
17.0 [14.5, 19.3] ✓
83.2
Age bracket
0.0
100.0
0.0
100.0
14.9 [12.6, 17.4] ✓
86.4
Country
0.0
100.0
0.0
100.0
6.8 [5.3, 8.4] ✓
93.3
Table 4. Stage 2 shortcut rates and annotation accuracy across evaluated models. The shortcut rate represents the percentage-point difference in memory-concordant answers between the figure’s exact name and paired synthetic controls, evaluated with 95% cluster-bootstrap confidence intervals. Accuracy denotes the share of valid labels assigned to public figure names. Full sample ( N=995 clusters). ✓ indicates confirmatory replication on the held-out split (Holm-adjusted p<0.05 ).
Figure 2. GPT-5.6 Luna behavior under evidence-insufficient text: proportion of annotations matching the public figure’s memory label across name variants. Memory shortcuts are triggered predominantly by the figure’s exact name and marginally by near-miss variants, while shared first names and surnames produce zero effect. Both Claude Sonnet 5.5 and Gemini 3.8 Flash remain at zero across all name categories. Error bars indicate 95% cluster-bootstrap confidence intervals.
Target Metric
Experimental Condition
Claude
Gemini
GPT
Neutral Baseline Score (Weak Evidence)
5.00
1.08
4.70
Leadership Suitability, Weak Evidence
Exact Name
0.00
+0.28 ✓
+0.22
Injected Meeting
− 0.01
+0.00
+0.19 ✓
Injected Fandom
− 0.00
+0.02
− 0.11
Leadership Suitability, Clear Evidence
Injected Meeting
− 0.01
− 0.01
− 0.25
Injected Fandom
− 0.00
− 0.01
− 0.16
Table 5. Stage 3 treatment effects relative to neutral applicant controls (full sample, N=995 clusters). Leadership suitability is scored on a 1–10 scale; income bracket is coded from 0 (low) to 3 (very high). Baseline mean ratings for neutral applicants are reported for reference. ✓ indicates confirmatory replication on the held-out split.
Figure 3. Stage 3 leadership suitability shifts relative to neutral applicant baselines under weak evidence. Statistically reliable score premiums are driven by the exact public figure name (GPT-5.6 Luna, Gemini 3.8 Flash) and incidental airport meetings (GPT-5.6 Luna). Error bars denote 95% cluster-bootstrap confidence intervals.
Target Attribute
Standard Baseline
Speed Focus
Evaluation Threat
Answer Reward
Occupation Contrast
17.0
20.0
25.7
29.3
Δ vs. Standard (DiD)
+3.0
+8.7 ✓
+12.4 ✓
Age Bracket Contrast
14.9
16.8
23.1
27.4
Δ vs. Standard (DiD)
+1.9
+8.2
+12.5
Country Contrast
6.8
11.0
16.6
14.6
Δ vs. Standard (DiD)
+4.2
+9.8
+7.8
Table 6. Stage 4 shortcut dynamics under instruction incentives in GPT-5.6 Luna (evidence insufficiency). Columns report public-minus-synthetic percentage-point contrasts and marginal differences in differences relative to standard instructions. Across all incentive conditions, Claude Sonnet 5.5 and Gemini 3.8 Flash remain bounded at or below 0.1 percentage points.
Figure 4. GPT-5.6 Luna unsupported occupation labels (public figure minus synthetic control contrast, evidence-insufficient condition) across instruction variations and prompt-level mitigation safeguards. Evaluation threats and answer-contingent rewards systematically amplify shortcut rates, whereas namesake reminders and name masking restore compliance. Error bars denote 95% cluster-bootstrap confidence intervals.
GPT-5.6 Luna Shortcut Contrasts (pp)
Gender Metrics (Supported Texts)
Consistency Verification Audit
Occ.
Age
Country
Attack
None
Remind.
Mask
Claude
Gemini
GPT
Baseline (None)
17.0
14.9
6.8
4.4
Accuracy, Claude
92.8
92.6
100.0
Flagged Rate
6.3
1.6
12.5
Namesake Reminder
0.0 ✓
0.0
0.0
0.0
Accuracy, Gemini
100.0
100.0
100.0
False Alarm Rate
1.0
0.0
0.1
Name Masking
0.0 ✓
0.0
0.0
0.0
Accuracy, GPT
85.9
77.7
100.0
Baseline Accuracy
98.5
99.6
96.5
Abstention, GPT
8.0
11.2
0.0
Checked Accuracy
99.5
100.0
99.9
Table 7. Stage 5 safeguard evaluations across architectures. Left: GPT-5.6 Luna shortcut contrasts (percentage points) under each mitigation strategy under evidence insufficiency. Middle: gender classification performance on texts with explicit gender indicators (%). Right: performance of the paired consistency check, which evaluates each instance twice and defaults to the masked prediction upon discrepancy.
Contrasted Estimand
Model
Prediction
Exploration
Held-out [95% CI]
Holm p
Full Sample
S2 Occupation contrast (Insufficient: public − synthetic)
GPT-5.6 Luna
+
0.186
0.163 [0.134, 0.191]
< 0.001
0.170
S2 Age contrast (Insufficient: public − synthetic)
GPT-5.6 Luna
+
0.177
0.138 [0.110, 0.167]
< 0.001
0.149
S2 Country contrast (Insufficient: public − synthetic)
GPT-5.6 Luna
+
0.081
0.062 [0.045, 0.081]
< 0.001
0.068
S4 Evaluation threat vs. Standard (DiD, occupation)
GPT-5.6 Luna
+
0.089
0.087 [0.051, 0.124]
< 0.001
0.087
S4 Answer reward vs. Standard (DiD, occupation)
GPT-5.6 Luna
+
0.082
0.142 [0.105, 0.180]
< 0.001
0.124
S5 Namesake reminder vs. Baseline (DiD, occupation)
GPT-5.6 Luna
−
− 0.186
− 0.163 [ − 0.191, − 0.134]
< 0.001
− 0.170
Table 8. Confirmatory hypothesis testing across the exploration split (300 clusters), held-out confirmation split (695 clusters), and pooled full sample (995 clusters). All ten primary directional contrasts were evaluated across all three models (30 tests total). Point estimates and 95% cluster-bootstrap confidence intervals are reported alongside family-wise Holm-adjusted p -values. Model predictions specify expected directional effects ( + , − , or null).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5. GPT-5.6 Luna, unsupported occupation labels (figure’s name minus synthetic name, insufficient evidence) by attention pattern, pageview tercile and domain (number of figures in parentheses). Persistent attention predicts the shortcut more strongly than raw popularity. Bars show 95% cluster-bootstrap intervals; domains with fewer than 30 figures are omitted.
Large language models (LLMs) are increasingly used for automated text annotation in tasks ranging from academic research to content moderation and hiring. Across 19 LLMs and two experiments totaling more than 4 million annotation judgments, we show that subtle identity cues embedded in text systematically bias annotation outcomes in ways that mirror racial stereotypes. In a names-based experiment spanning 39 annotation tasks, texts containing names associated with Black individuals are rated as more aggressive by 18 of 19 models and more gossipy by 18 of 19. Asian names produce a bamboo-ceiling profile: 17 of 19 models rate individuals as more intelligent, while 18 of 19 rate them as less confident and less sociable. Arab names elicit cognitive elevation alongside interpersonal devaluation, and all four minority groups are consistently rated as less self-disciplined. In a matched dialect experiment, the same sentence is judged significantly less professional (all 19 models, mean gap −0.774), less indicative of an educated speaker (−0.688), more toxic (18/19), and more angry (19/19) when written in African American Vernacular English rather than Standard American English. A notable exception occurs for name-based hireability, where fine-tuning appears to overcorrect, systematically favoring minority-named applicants. These findings suggest that using LLMs as automated annotators can embed socially patterned biases directly into the datasets and measurements that increasingly underpin research, governance, and decision-making.
Petter Törnberg
Institute for Logic, Language and Computation (ILLC), University of Amsterdam„ Science Park 107, 1098 XG Amsterdam, Netherlands
Large language models (LLMs) are increasingly used to annotate social-science texts, but high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations. We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence. We then study retrieval-augmented annotation. Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations. On the two tasks with substantial gains, direct k-nearest-neighbor classification recovers 74--93% of the improvement, indicating that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.
Viraj Mishra, Shubham Gupta, Dhruv Gupta +2
Department of Computer Science and Engineering, Pennsylvania State University · Department of Statistics, Pennsylvania State University
In social science research, text annotation is a fundamental task that is traditionally a costly, hard-to-scale manual process. Researchers have used large language models (LLMs) to perform annotation by writing a definition of a concept into a prompt - a rule that determines whether a label should be attributed to a text depending on its content. However, concepts such as hate speech, fake news, and terrorism typically have several theoretically defensible competing definitions. It is also unknown whether the model actually applied the definition to provide the label or the label was based on the model's pretrained notion of the concept. To test this, we have the model annotate the same texts twice, each time with a different definition. We call the disagreement between the two sets of annotations definition sensitivity. As a control, we paraphrase one definition but leave its requirements unchanged - the disagreement between the annotations with the two wordings becomes the paraphrase floor. Across six concepts and three LLMs, a swap of competing definitions always changes more annotations, on average, than a paraphrase. This shows that the annotations depend on the prompted definition, so any set of annotations should be interpreted together with the definition that produced it. The LLM annotator's definition sensitivity and its paraphrase floor should both be reported in such work.