NepPlural: A Human-Anchored Benchmark for Persona Annotation in Nepali Migration Discourse

Jinu Nyachhyon; Anish Khatiwada; Bimal Timilsina

cs.CLOct 11, 2026

NepPlural: A Human-Anchored Benchmark for Persona Annotation in Nepali Migration Discourse

Authors: Jinu Nyachhyon, Anish Khatiwada, Bimal Timilsina

Organizations: Institute for Research and Innovation in Intelligent Systems (IRIIS) · Independent Researcher

Abstract

Pluralistic alignment asks that language models faithfully represent a society's diverse, competing values rather than collapse them into one dominant viewpoint, yet flagship pluralistic-alignment datasets remain heavily skewed toward Western respondents. We introduce NepPlural, a human-anchored benchmark for modelling competing socio-cultural personas in Nepali public discourse on youth migration and brain drain. We curate 2,148 code-mixed (Devanagari, Romanized Nepali, English) YouTube comments, annotate them along a four-axis persona taxonomy (Intent, Primary Driver, Value Orientation, Affect) plus an explicit No-Persona class, and build a 300-comment gold set labelled independently by three native Nepali speakers (mean Fleiss' $κ\approx 0.55$ ). Using this gold set to audit automatic annotation, we find that LLM annotation reliability is driven by protocol. Labels collected in batches through chat interfaces reach only near-chance inter-model agreement ( $κ\approx 0.17$ ). Annotating one comment at a time raises agreement above the native-speaker level ( $κ\approx 0.72$ ), but consistency is not validity: under a forced-choice schema, models agree on personas for the 39% of comments native speakers judge to express none, keeping macro-F1 against human gold near 0.50. Allowing abstention resolves this: models recover No-Persona comments with F1 of 0.79 to 0.87 and, against leave-one-annotator-out references, match or exceed individual native speakers (macro-F1 0.75 to 0.86 vs. 0.75). Encoders fine-tuned on the batched labels reach only 0.34. Finally, a human-judged steerability probe finds that three frontier models adopt single persona axes with 59 to 77% fidelity but compose full personas only 33% of the time, with no congruence gap detected at this scale between Western-congruent and Nepali-collectivist personas.

Figures & tables

Figure 1: Agreement is not validity. (a) Mean Fleiss’

\kappa

among three native speakers and among three LLMs under each protocol. (b) Mean macro-F1 against leave-one-annotator-out references for a held-out native speaker and the LLM majority vote. Dashed lines mark the human level.

Resource	Language(s)	Item source	What is measured	Human gold	Naturalistic text
OpinionQA ( Santurkar et al., 2023 )	English	Pew public-opinion surveys (US)	Alignment with demographic opinion distributions	Survey	✗
GlobalOpinionQA ( Durmus et al., 2024 )	English	Pew Global Attitudes, World Values Survey	Alignment with country-level opinion distributions	Survey	✗
PRISM ( Kirk et al., 2024 )	Mostly English	Participant conversations with LLMs	Individual preferences and ratings	✓	Partial
Cao et al. (2023)	Multilingual	Hofstede-style survey items	Cultural dimension alignment	Survey	✗
AlKhamissi et al. (2024)	Arabic, English	World Values Survey items	Simulated survey responses by country	Survey	✗
Naous et al. (2024)	Arabic	Curated prompts and cultural entities	Western-favouring cultural bias	✓	Partial

Table 1: NepPlural compared with representative cultural, moral, and pluralistic alignment resources. “Survey” means ground truth comes from survey response distributions rather than item-level human labels.

	Comment (original)	English gloss	Human gold (I/D/V/A)	LLM labels by protocol
a	rahar ley haina badhyata ley ho hajur	It is not by choice but out of compulsion, sir.	Pro / Economic / Individualist / Despairing	–
b	Manxe bides gaye ra xaena. desh xodi raxa hits hard	It is not just that people went abroad; the country is being abandoned. It hits hard.	Anti / Patriotism / Coll.-Nation / Despairing	–
c	Ani sarkar ko matra dosh chaina, sarkar ko under ma kam garne karmachari pani chorharule bhariyeko hunale pani ho (orig. Devanagari)	Not only the government is to blame; its staff are full of thieves too.	Neutral / Systemic Anger / Coll.-Nation / Angry	–
d	I want to go home but I can’t	(English)	Trapped / Family / Individualist / Despairing	–
e	Euta video Lumbini Development Trust ko corruption ra politics ma ni banaunu paryo	Someone should also make a video on corruption and politics at the Lumbini Development Trust.	No-Persona	Batched, forced: all three assign Neutral / Systemic Anger / Coll.-Nation. Abstain: all three No-Persona .
f	Genuine saving kati samma hunx uta	How much can one genuinely save over there?	No-Persona	Batched, forced: all three assign a persona (Economic / Individualist / Pragmatic in most). Abstain: all three still assign Pro / Economic / Individualist / Pragmatic.

Table 2: Examples from the gold set (emojis removed) across registers: Romanized Nepali (a), Romanized Nepali mixed with English (b), Devanagari with an English loanword (c), and English (d). Rows e and f are substantive-looking comments that native speakers judge to express no migration persona. I/D/V/A = Intent / Primary Driver / Value Orientation / Affect. Protocol labels are shown only for the No-Persona rows.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Congruent	Collectivist	Gap
Claude Opus 4.6	38.9	33.3	$+5.6$
GPT 5.6 Luna	38.9	22.2	$+16.7$
Gemini 3.6 Flash	22.2	44.4	$-22.2$
Pooled	33.3	33.3	$0.0$

Appendix

Table 13: Strict (all four axes) steerability fidelity in % by model and persona family (Western-congruent vs. Nepali-collectivist);

n=18

per cell, temperature 1.0 for all models. Gap = congruent minus collectivist.

Pluralistic Alignment

Pluralistic alignment develops AI systems that represent and navigate diverse human values rather than optimize for a single preference.

See all 36 papers

Oct 13, 2026

NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis

Many sentiment datasets use one annotator for each item, so the test labels show the conventions of that person. We build a Nepali benchmark from two annotators who labelled the same sentences blind. They labelled 5,614 sentences from tweets of Nepali public figures on a five-point scale. We keep only the 3,189 sentences where both give the same polarity. On the three-class benchmark, NepBERTa with a fixed tokenizer gets a macro-F1 of 0.877 and IndicBERTv2 0.867. Three commercial large language models (LLMs) get 0.882 to 0.896 zero-shot, and these differences are not significant. On the five-point scale, 766 sentences have two adjacent labels, for example Pos and SPos. We accept both labels and train on both. On this task, Claude Opus 5.5 beats the best encoders by 0.10 to 0.12 macro-F1. A tokenizer audit adds a second result. The default tokenizers of three Nepali BERT models delete the non-spacing Devanagari vowel signs and the virama. When we restore the marks, macro-F1 increases by 0.084 for NepBERTa but decreases by 0.095 for a model pretrained on stripped text.

Gaurab Baral, Mimansak Nepal, Grishma Basnet

University of Cincinnati Cincinnati, Ohio, USA · Tribhuvan University Kirtipur, Kathmandu, Nepal · Subhakamana English School Urlabari, Morang, Nepal

Sentiment Analysis LLM Evaluation Bidirectional Encoder Representations from Transformers

Ask PDF

Jul 9, 2026

PLURAL: A Global Dataset for Value Alignment

Large language models (LLMs) are used worldwide, yet disproportionately reflect Western values, limiting their ability to represent diverse value systems. We introduce PLURAL, a large-scale, value-focused preference dataset grounded in the Integrated Values Survey (IVS), a nationally representative survey spanning 92 countries. Using a two-stage generation pipeline, we transform survey responses into synthetic preference triplets that preserve normative value signals while producing realistic scenarios. We release an initial version of PLURAL containing ~500,000 preference triplets representing people in 20 diverse countries. We evaluate PLURAL in three ways: (i) dataset-level validation showing that it preserves both cross-country value differences and within-country diversity from the original survey; (ii) automated evaluation showing that training on PLURAL improves alignment with target countries' cultural profiles, reducing mean absolute error by up to 27.7% relative to strong baselines; and (iii) blind human evaluation with 176 evaluators in India, Brazil, and Japan, who judge PLURAL-aligned responses as more representative of their national values. Together, these results show that PLURAL contains learnable signal for value steering, offering a scalable resource for pluralistic alignment. Dataset: https://huggingface.co/datasets/agdhruv/plural-alignment

Dhruv Agarwal, Anya Shukla, Tanya Goyal +1

Cornell University

LLM Alignment Cultural Alignment Cultural Bias in Language Models Synthetic Data Generation for Language Models

Ask PDF

Jun 4, 2026

From Self to Other: Evaluating Demographic Perspective-Taking in LLM Hate Speech Annotation

Hate speech detection is inherently subjective: people from different demographic groups perceive the same content very differently. Collecting enough annotations from multiple demographic groups is costly and difficult to scale. Persona-conditioned Large Language Models (models prompted to adopt a specific demographic identity) have been proposed as a way to simulate diverse perspectives at scale. But do they actually reflect how different groups disagree? We evaluate three aspects of human social judgement: (i) whether personas from different groups disagree in human-like ways (inter-group disagreement), (ii) whether they become more sensitive when content targets their own identity (in-group sensitivity), and (iii) whether they can accurately predict how another group would react (vicarious prediction). Our results show that no model consistently captures all three dimensions, and performance is highly model-dependent and does not emerge reliably from minimal identity prompts alone. However, vicarious prompting with Llama 3.1 yields the highest cross-group agreement in most demographic axes and provides the closest overall approximation to human disagreement patterns, indicating that this configuration may provide a more reliable setting for automatic annotation aligned with human judgements.

Paloma Piot, Javier Parapar

Information Retrieval Lab CITIC Research Centre Universidade da Coruña

LLM-Assisted Annotation Hate Speech Detection Language Model Bias Evaluation Demographic Bias in Language Models

Ask PDF

Abstract

Pluralistic Alignment

Explore similar work

NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis

PLURAL: A Global Dataset for Value Alignment

From Self to Other: Evaluating Demographic Perspective-Taking in LLM Hate Speech Annotation