Activating the Annotator: Operationalising Lived Experience in Data Annotation

Sheza Munir; Ramaravind Kommiya Mothilal; Syed Ishtiaque Ahmed

cs.CLOct 8, 2026

Activating the Annotator: Operationalising Lived Experience in Data Annotation

Authors: Sheza Munir, Ramaravind Kommiya Mothilal, Syed Ishtiaque Ahmed

Organizations: Department of Computer Science University of Toronto · Faculty of Information University of Toronto

Abstract

Perspectivist NLP has established that annotator positionality shapes labels for subjective tasks, but the field still lacks methods for operationalising it. Positionality is typically operationalised as a demographic form completed at recruitment: a static attribute recorded once and treated as a covariate, on the assumption that the knowledge it indexes is active when the annotator reads an item. This paper treats that assumption as a design problem and asks whether lived experience can instead be primed by the task itself. We introduce AXLE, a pre-annotation stage in which a scenario vignette anchors a short, axis-guided conversation that is synthesised into a first-person micro-narrative and returned to the annotator before labelling begins. In a 2x2 between-subjects study (104 annotators, 30 items, 1,040 annotations) on ambiguous hate speech spanning immigration, religion and gender, elicitation was associated with higher identity resonance and higher sensitivity to harm. It also increased label entropy and lowered inter-annotator agreement: the stage appears to surface plurality rather than manufacture consensus, which distinguishes it from annotator training, whose success criterion is convergence.

Figures & tables

Panel of Figure 1 — Figure 1: Perspectivist methods operate downstream , preserving variation the annotation step already produced. axle operates upstream , on how much variation that step produces at all.

Panel of Figure 2 — Figure 2: Stage order in the two arms. Both share the disclosure, annotation and reflexivity stages and use an identical label taxonomy, rationale prompt and resonance item. In the control arm the three shaded stages are removed and disclosure runs after annotation, so that no identity-related framing precedes labelling.

Condition	A	B	C	Total
	Scenario
LE–elicit	18	6	6	30
LE–control	10	6	6	22
non-LE–elicit	18	6	6	30
non-LE–control	10	6	6	22
Total	56	24	24	104

Table 1: Design cells. A = immigration, B = religion, C = gender/sexuality. Each participant annotates all ten items in their scenario (1,040 annotations).

Panel of Figure 3 — Figure 3: Response distributions by arm. Left: harm labels; elicitation moves mass toward both ends, with the largest single change at clearly harmful . Right: identity resonance; the proportion of items rated “not at all relevant to my identity or experience” falls by more than half.

Figure 4: Per-item Shannon entropy of the harm label distribution. Points above the diagonal are items on which elicitation produced more dispersed labels.

Panel of Figure 5 — Figure 5: Left: what the rationale treats as grounds for its reading, by arm ( $n{=}240$ ); elicitation moves mass out of literal reading and into situated grounding. Right: proportion of rationales invoking any of the seven lived-experience axes, by design cell; the elicitation gain is larger in the non-LE stratum.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario	LE stratum	non-LE stratum
A (immigration)	Moved to country of residence	Born in country of residence
B (religion)	Moved to country of residence and affiliation in {Buddhism, Hinduism, Islam, Judaism, Sikhism}	Born in country of residence, nationality US/Canada, and affiliation in {Christianity, Non-religious}
C (gender)	Orientation in {homosexual, bisexual, asexual, other}, or sex female, or non-binary gender identity	Heterosexual male

Appendix

Table 2: Stratum screeners. All conditions additionally required: residence in the United States or Canada; English among fluent languages; submission approval rate 95–100; and exclusion of participants enrolled in any other study in this project. Sex was restricted to male or female in scenarios A and B.

Attribute	Level	$n$
Residence	Canada	59
Residence	United States	45
Sex	Male	56
Sex	Female	48
Migration	Born in country of residence	55
Migration	Moved to country of residence	49

Appendix

Table 3: Participant demographics (

N{=}104

). † Religion and sexual orientation were collected only for scenarios B and C respectively, so those counts sum to 24.

ID	Domain	Text
A01	imm	Our hospitals can’t keep up. At some point we have to ask who this country is actually for.
A02	imm	My grandparents came here legally and waited years. Why should anyone get to skip that line?
A03	imm	When you flood a labor market, wages go down. Economics doesn’t care about feelings.
A04	imm	There’s nothing wrong with wanting your hometown to stay recognizable.
A05	imm	Demographic change at this pace isn’t natural. Someone is choosing this.
A06	imm	Chain migration turns one visa into fifty. That’s not the deal Americans signed up for.

Appendix

Table 4: The 30 items. All satisfy C1–C4 of the ambiguity codebook. Scenario is given by the ID prefix: A = immigration, B = religion, C = gender/sexuality.

Measure	Elicit.	Control	$p$
Participant level ( $n{=}60$ / $44$ )
Identity resonance	3.35	2.94	.041
Harm rating	3.12	2.78	.029
Annotation time (s)	1977	2099	.409
Session (min, median)	51.4	43.8	.010
Time on item (s)	197.7	209.9	.072

Appendix

Table 5: Main results; values are means unless noted. Mann–Whitney

U

for level comparisons; Wilcoxon signed-rank for the paired per-item comparison. Time on item is item-level (

n{=}600/440

); item-level tests of resonance and harm are both

p<.0001

Stratum	Cond.	Reson.	Harm	Time (s)	Words
LE	Elicit	3.51	3.16	201.7	62.8
LE	Control	3.20	2.88	224.6	60.1
non-LE	Elicit	3.19	3.08	193.7	62.6
non-LE	Control	2.68	2.68	195.2	64.9

Appendix

Table 6: Item-level means by design cell (

n{=}300

220

300

220

Scenario	Elicitation	Control	Raters
	Krippendorff’s $\alpha$
A (immigration)	.096	.082	36 / 20
B (religion)	.001	.103	12 / 12
C (gender)	.031	.130	12 / 12
Pooled	.066	.111	60 / 44

Appendix

Table 7: Ordinal Krippendorff’s

\alpha

on the five-point harm scale, computed within condition. Absolute values are near chance in both arms, as expected for a corpus selected for experiential divergence; only the between-arm comparison is interpretable.

Figure 6: Elicitation effects by item domain, with standard errors. Harm gains concentrate in rel and int ; gen shows a large resonance gain with no mean harm shift.

Figure 7: Projected session time by items annotated, using median fixed overhead and median per-item time in each arm.

Figure 8: Agreement between the two coders on constructs present in both schemes, before reconciliation ( $n{=}240$ rationales). The construct carrying the central claim in § 5 is the one agreed on most closely.

Figure 9: Relationship between the micro-narrative and subsequent rationales ( $n{=}120$ elicitation-arm rationales). Formulaic tagging is concentrated in two of twelve participants.

Human-in-the-Loop Annotation

Human-in-the-loop annotation combines human judgment and AI assistance to create, review, and improve labeled data.

See all 135 papers

Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks
Apr 18, 2026Nisrine Rair, Alban Goupil, Valeriu Vrabie +1Annotator Disagreement
CReSTIC, Université de Reims Champagne-Ardenne, Reims, France · Chochoy Conseil, Reims, France
Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
May 27, 2026Beiduo Chen, Pingjun Hong, Ziyun Zhang +3Natural Language Explanation Generation LLM-Assisted Annotation
U MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Æ Munich Center for Machine Learning (MCML), Munich, Germany · [ University of Vienna, Austria +1
Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
Jun 1, 2026Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli +10Human-in-the-Loop Annotation
NLLG Lab University of Technology Nuremberg · Interdisciplinary Transformation University, Austria

Apr 18, 2026J/K move · Enter open · S save

Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks

Nisrine Rair, Alban Goupil, Valeriu Vrabie, Emmanuel Chochoy

CReSTIC, Université de Reims Champagne-Ardenne, Reims, France · Chochoy Conseil, Reims, France

Annotator Disagreement

Open paper AskPDF arXiv

Subjective NLP datasets typically aggregate annotator judgments into a single gold label, making it difficult to diagnose whether disagreement reflects unclear criteria, collapsed distinctions, or legitimate plurality. We propose a schema-level diagnostic for auditing expert-designed annotation schemas prior to gold-label commitment, using only multi-annotator criterion judgments. The diagnostic separates two failure modes: unstable criteria with hard-to-operationalize boundaries, and systematic overlap that blurs the boundaries between mutually exclusive categories. Applied to persuasive value extraction in commercial documents, we find that disagreement is not diffuse: instability concentrates in a few criteria, while nearly half of covered sentences activate multiple categories. These signals align with where domain experts disagree, yielding an evidence-based audit for tightening guidelines, revising category structure, or reconsidering the annotation paradigm.

Abstract

Human-in-the-Loop Annotation

Explore similar work

Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks

Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization

Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025