NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis

Gaurab Baral; Mimansak Nepal; Grishma Basnet

cs.CLOct 13, 2026

NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis

Authors: Gaurab Baral, Mimansak Nepal, Grishma Basnet

Organizations: University of Cincinnati Cincinnati, Ohio, USA · Tribhuvan University Kirtipur, Kathmandu, Nepal · Subhakamana English School Urlabari, Morang, Nepal

Abstract

Many sentiment datasets use one annotator for each item, so the test labels show the conventions of that person. We build a Nepali benchmark from two annotators who labelled the same sentences blind. They labelled 5,614 sentences from tweets of Nepali public figures on a five-point scale. We keep only the 3,189 sentences where both give the same polarity. On the three-class benchmark, NepBERTa with a fixed tokenizer gets a macro-F1 of 0.877 and IndicBERTv2 0.867. Three commercial large language models (LLMs) get 0.882 to 0.896 zero-shot, and these differences are not significant. On the five-point scale, 766 sentences have two adjacent labels, for example Pos and SPos. We accept both labels and train on both. On this task, Claude Opus 5.5 beats the best encoders by 0.10 to 0.12 macro-F1. A tokenizer audit adds a second result. The default tokenizers of three Nepali BERT models delete the non-spacing Devanagari vowel signs and the virama. When we restore the marks, macro-F1 increases by 0.084 for NepBERTa but decreases by 0.095 for a model pretrained on stripped text.

Figures & tables

Scale	Exact %	Cohen $\kappa$	QWK	Krippendorff $\alpha$
Five points	49.5	0.289 [.27, .31]	0.627	0.625 [.61, .64]
Three classes	65.5	0.465 [.45, .48]	–	0.455 [.43, .47]

TABLE I: Agreement between annotators A and B on the 4,943 sentences with two polar labels. Five points: ordinal

\alpha

; three classes: nominal

\alpha

. Brackets: 95% bootstrap CI (1,000 resamples).

Five-point label	Train	Dev	Test	Total	3-class
SNeg	14	1	7	22	Neg
SNeg or Neg	106	22	11	139
Neg	323	58	66	447
Neu	710	161	167	1,038	Neu
Pos	602	127	124	853	Pos
Pos or SPos	437	97	93	627

TABLE II: Labels in each split. Rows with two labels: the annotators differ only in intensity, and both labels are correct. The splits keep each tweet together (1,668, 357 and 356 tweets).

		Three classes		Five points, lenient				Five points, strict
	Model	Macro-F1	Accuracy	Macro-F1	Accuracy	QWK	Severe % $\downarrow$	Macro-F1	Accuracy
Baselines	Majority class	0.215	0.476	0.125	0.453	–	–	–	–
Baselines	TF-IDF (best dev)	0.766	0.793	0.633	0.729	0.714	5.2	–	–
Fine-tuned encoders	NepBERTa (fixed tok.)	0.877 $\pm$ .018	0.891 $\pm$ .016	0.721 $\pm$ .043	0.863 $\pm$ .024	0.897 $\pm$ .026	0.7 $\pm$ 0.5	0.638 $\pm$ .031	0.830 $\pm$ .031
	IndicBERTv2	0.867 $\pm$ .003	0.881 $\pm$ .004	0.734 $\pm$ .001	0.859 $\pm$ .001	0.883 $\pm$ .002	1.3 $\pm$ 0.2	0.640 $\pm$ .004	0.826 $\pm$ .002
	DeBERTa-base Nepali	0.858 $\pm$ .016	0.875 $\pm$ .013	0.704 $\pm$ .031	0.842 $\pm$ .013	0.867 $\pm$ .017	2.1 $\pm$ 0.9	0.608 $\pm$ .024	0.808 $\pm$ .014

TABLE III: Test results on 479 sentences (encoders: mean

\pm

s.d. over three seeds, in order of three-class macro-F1). Five-point lenient: a prediction is correct if it matches either annotator. Strict: the 375 test sentences with one five-point label. Severe: predicted polarity opposite to the reference. For LLMs, QWK and Severe use only the sentences with a polar answer. Bold: best in each column.

Model	Tok.	Fert.	Lost	MLM	Ours	COV19	News
NepBERTa	default	2.04	44.5%	0.205	0.793	0.702	0.957
NepBERTa	fixed	1.27	0.0%	0.518	0.877	0.730	0.981
Rajan	default	1.95	44.6%	0.051 ∗	0.739	0.666	0.945
Rajan	fixed	1.20	0.05%	0.176 ∗	0.788	0.672	0.974
Shushant	default	2.02	44.5%	0.650	0.817	0.700	0.975
Shushant	fixed	1.26	0.0%	0.071	0.721	0.668	0.918

TABLE IV: Default and fixed tokenizer for the three Nepali BERTs. Fert.: subwords per word. Lost: combining marks deleted. MLM: top-1 probe accuracy. F1: test macro-F1 (mean of 3 seeds) on our three classes, NepCOV19Tweets and news topics. Bold: the better setting where the 95% CI of the difference excludes zero.

Fig. 1: (a) Three-class test macro-F1 of each seed with the default (hollow) and fixed (filled) tokenizer. The numbers give the mean change. (b) Accuracy of the MLM probe for each checkpoint. It predicts the direction of each effect.

Sentiment Analysis

Sentiment analysis detects and measures opinions, emotions, and evaluative stance in text and multimodal data.

See all 125 papers

NepPlural: A Human-Anchored Benchmark for Persona Annotation in Nepali Migration Discourse
Oct 11, 2026Jinu Nyachhyon, Anish Khatiwada, Bimal TimilsinaPluralistic Alignment Cross-Cultural Language Model Evaluation
Institute for Research and Innovation in Intelligent Systems (IRIIS) · Independent Researcher
Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language
Feb 27, 2026Nischal Karki, Bipesh Subedi, Prakash Poudyal +2Bidirectional Encoder Representations from Transformers Text Classification
Information and Language Processing Research Lab, Kathmandu University, Kavre, Nepal
Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali
Sep 27, 2026Kalash Shrestha, Nikhil PradhanLanguage Model Pretraining Low-Resource Language Processing
Kathmandu University · IOE Thapathali Campus

Oct 11, 2026J/K move · Enter open · S save

NepPlural: A Human-Anchored Benchmark for Persona Annotation in Nepali Migration Discourse

Jinu Nyachhyon, Anish Khatiwada, Bimal Timilsina

Institute for Research and Innovation in Intelligent Systems (IRIIS) · Independent Researcher

Pluralistic Alignment Cross-Cultural Language Model Evaluation Personality Modeling in Language Models Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Pluralistic alignment asks that language models faithfully represent a society's diverse, competing values rather than collapse them into one dominant viewpoint, yet flagship pluralistic-alignment datasets remain heavily skewed toward Western respondents. We introduce NepPlural, a human-anchored benchmark for modelling competing socio-cultural personas in Nepali public discourse on youth migration and brain drain. We curate 2,148 code-mixed (Devanagari, Romanized Nepali, English) YouTube comments, annotate them along a four-axis persona taxonomy (Intent, Primary Driver, Value Orientation, Affect) plus an explicit No-Persona class, and build a 300-comment gold set labelled independently by three native Nepali speakers (mean Fleiss'

κ\approx 0.55

). Using this gold set to audit automatic annotation, we find that LLM annotation reliability is driven by protocol. Labels collected in batches through chat interfaces reach only near-chance inter-model agreement (

κ\approx 0.17

). Annotating one comment at a time raises agreement above the native-speaker level (

κ\approx 0.72

), but consistency is not validity: under a forced-choice schema, models agree on personas for the 39% of comments native speakers judge to express none, keeping macro-F1 against human gold near 0.50. Allowing abstention resolves this: models recover No-Persona comments with F1 of 0.79 to 0.87 and, against leave-one-annotator-out references, match or exceed individual native speakers (macro-F1 0.75 to 0.86 vs. 0.75). Encoders fine-tuned on the batched labels reach only 0.34. Finally, a human-judged steerability probe finds that three frontier models adopt single persona axes with 59 to 77% fidelity but compose full personas only 33% of the time, with no congruence gap detected at this scale between Western-congruent and Nepali-collectivist personas.

Abstract

Sentiment Analysis

Explore similar work

NepPlural: A Human-Anchored Benchmark for Persona Annotation in Nepali Migration Discourse

Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language

Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali