When Do Language Models Endorse Limitations on Human Rights Principles?

Keenan Samway; Nicole Miu Takagi; Rada Mihalcea; Bernhard Schölkopf; Ilias Chalkidis; Daniel Hershcovich; Zhijing Jin

cs.CLMar 4, 2026

When Do Language Models Endorse Limitations on Human Rights Principles?

Authors: Keenan Samway, Nicole Miu Takagi, Rada Mihalcea, Bernhard Schölkopf, Ilias Chalkidis, Daniel Hershcovich, Zhijing Jin

Organizations: Max Planck Institute for Intelligent Systems, Tübingen, Germany · Jinesis AI Lab, University of Toronto & Vector Institute · University of Michigan · University of Copenhagen ·

Abstract

As Large Language Models (LLMs) increasingly mediate global information access with the potential to shape public discourse, their alignment with universal human rights principles becomes important to ensure that these rights are abided by in high stakes AI-mediated interactions. In this paper, we evaluate how LLMs navigate trade-offs involving the Universal Declaration of Human Rights (UDHR), leveraging 1,152 synthetically generated scenarios across 24 rights articles and eight languages. Our analysis of eleven major LLMs reveals systematic biases where models: (1) accept limiting Economic, Social, and Cultural rights more often than Political and Civil rights, (2) demonstrate significant cross-linguistic variation with elevated endorsement rates of rights-limiting actions in Chinese and Hindi compared to English or Romanian, (3) show substantial susceptibility to prompt-based steering, and (4) exhibit noticeable differences between Likert and open-ended responses, highlighting critical challenges in LLM preference assessment.

Figures & tables

Figure 1: Illustrative example of the cross-lingual variation observed in human rights evaluation across three languages: English (left), Romanian (middle), and Chinese (right).

Figure 2: Alignment between the mean endorsement score (1–5) on Likert-scale and open-ended responses per model per language. Lower Jensen–Shannon (JS) divergence indicates similar distributions.

Figure 3: Per-model endorsement scores (1-5) for open-ended responses across each model and language. A lower mean endorsement score indicates that the model more often rejects the presented rights-limiting actions.

Figure 4: Per-model endorsement scores for open-ended responses across rights categories: political & civil and economic, social, & cultural (sorted by mean severity 3 score).

Figure 5: Per-model endorsement scores for open-ended responses in state-of-emergency scenarios: none, civil unrest, and natural disaster (sorted by mean severity 3 score).

Figure 6: Shows the delta in the mean endorsement scores under the government power and individual rights steering prompts. Low delta (near 0) means that model outputs did not change due to prompt steering while high delta (near 4) represents a large susceptibility to prompt steering.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Article	Article Name	Category
1	Free and Equal	-
2	Freedom from Discrimination	-
3	Right to Life	PC
4	Freedom from Slavery	PC
5	Freedom from Torture	PC
6	Right to Recognition Before the Law	PC

Appendix

Table 1: Articles of the Universal Declaration of Human Rights categorized by type: political and civil (PC); economic, social, and cultural (ESC); or Other (-).

Model ID	Provider	Logprobs?	Dev. Country
anthropic/claude-sonnet-4	google-vertex	No	USA
google/gemma-3-27b-it	deepinfra/bf16, nebius/fp8, deepinfra/fp8	No	USA
google/gemma-3-12b-it	deepinfra/bf16, novita/bf16	No	USA
google/gemma-3-4b-it	deepinfra/bf16	No	USA
deepseek/deepseek-chat-v3-0324	nebius/fp8	Yes	China
openai/gpt-3.5-turbo-0125	openai	Yes	USA

Appendix

Table 2: LLMs queried for their endorsement of rights-limiting actions. All models accessed via OpenRouter between 2025-06-19 and 2025-10-04.

Model ID	Provider	Dev. Country
gpt-4.1-2025-04-14	Azure OpenAI, OpenAI	USA

Appendix

Table 3: LLM used for scenario generation, scenario evaluation, and open-ended response evaluation. Accessed between 2025-06-19 and 2025-10-04.

Category	Variations	Count
Rights	UDHR Articles 4–27	24
Actors	“the government”	1
Affected group	“people”	1
Severity levels	1 (low) and 3 (high)	2
States of emergency	None, Civil Unrest, Natural Disaster	3
Languages	Arabic ( ar ), Chinese ( zh-cn ), English ( en ), Hindi ( hi ), Romanian ( ro ), Russian ( ru ), Spanish ( es ), Zulu ( zu )	8

Appendix

Table 4: Summary of dataset dimensions and counts.

Figure 7: Proportion of scenarios in which each article is identified as limited, grouped by severity level, given that the article was prompted to be limited in the scenario generation procedure. Bars are stacked from darkest (primarily limited) to lightest (somewhat limited), illustrating the distribution of limitation severity across all rights. Consider Article 19, the right to freedom of expression: in scenarios designed to limit that right, about 80% correctly recognized it as being primarily limited, and the remaining 20% still identified it as being moderately limited.

Figure 8: Plots the counts of identified rights limitations and promotions across all scenarios, grouped by severity level.

Figure 9: Measures the co-occurrence of rights identified as primarily limited in generated scenarios using the Jaccard index: 0 (no overlap) to 1 (complete overlap).

Perceived $\rightarrow$ Intended $\downarrow$	1	2	3
1	29	40	3
2	0	0	0
3	0	2	70

Appendix

Table 5: Displays the intended versus perceived severity.

Figure 10: Per-model mean endorsement scores (1-5) for ECHR-based scenarios across four languages. Models exhibit similar relative rankings to UDHR-based scenarios, with consistent cross-linguistic patterns.

Figure 11: Correlation between mean endorsement scores for UDHR and ECHR scenarios. Each point represents a pair of corresponding articles from the Universal Declaration of Human Rights (UDHR) and the European Convention on Human Rights (ECHR) that protect analogous rights. A 95% confidence interval is shown around each marker.

Language	Semantic Similarity	ROUGE-1
Spanish ( es )	0.955 (0.876)	0.825 (0.751)
Chinese ( zh-cn )	0.952 (0.838)	0.763 (0.664)
Romanian ( ro )	0.942 (0.814)	0.812 (0.686)
Hindi ( hi )	0.938 (0.771)	0.762 (0.620)
Arabic ( ar )	0.935 (0.599)	0.761 (0.631)
Russian ( ru )	0.922 (0.144)	0.750 (0.656)

Appendix

Table 6: Comparison between scenarios in English and those back-translated to English from the listed languages, displayed here as: average (minimum). Semantic similarity is measured by computing the cosine similarity between text embeddings from all-MiniLM-L6-v2 . Sorted from top to bottom by average semantic similarity.

Figure 12: Jensen-Shannon (JS) divergence between English-judged and original-language-judged model response scores. Lower values indicate higher agreement between judging methods.

Figure 13: Displays the confusion matrix between: (left) H1 and the LLM judge, (middle) H2 and the LLM judge, (right) H1 and H2, on 100 randomly sampled open-ended model responses.

Annotator	1	2	3	4	5	Avg. Rating
H1	2	4	2	4	8	3.60
H2	5	3	3	6	3	2.95
H3	6	8	4	2	0	2.10
H4	4	5	3	7	1	2.80
Overall	17	20	12	19	12	2.86

Appendix

Table 7: Rating counts by annotator along with their mean endorsement score.

Model	Likert Diff	Open-Ended Diff
Mistral Medium 3	0.388 ${}^{\text{ns}}$	0.278 ∗∗
GPT-4o	0.338 ${}^{\text{ns}}$	0.178 ${}^{\text{ns}}$
Qwen 2.5 72B	0.338 ${}^{\text{ns}}$	0.538 ∗∗∗
GPT-3.5 Turbo	0.338 ${}^{\text{ns}}$	0.298 ∗
Qwen 3 235B	0.288 ${}^{\text{ns}}$	0.468 ∗∗∗
Llama 4 Maverick	0.238 ${}^{\text{ns}}$	0.848 ∗∗∗

Appendix

Table 8: Comparison of human judgments against model responses in Likert-scale and open-ended formats. Statistical significance was assessed using paired t-tests (

{}^{*}p<0.05

{}^{**}p<0.01

{}^{***}p<0.001

, ns = not significant).

Figure 14: Endorsement scores for open-ended responses across different scales of three model families: Qwen 3 8B, 14B, 32B, and 235B; Gemma 3 4B, 12B, and 27B; and Mistral Nemo (12B), Small (24B), and Large (123B).

Figure 15: Per-language endorsement scores for open-ended responses across rights categories: political and civil (PC) and economic, social, and cultural (ESC). Sorted by average score at severity 3.

Figure 16: Per-language endorsement scores for open-ended responses in state-of-emergency scenarios: none, civil unrest, and natural disaster. Sorted by average score at severity 3.

Figure 17: Mean endorsement score of rights-limiting actions averaged across all tested languages for each model. A low score (near 1) means that for scenarios where the specified right was limited, that the model often rejected such rights-limiting actions. Models are sorted top to bottom according to their mean endorsement score; top being the lowest (i.e., most likely to reject) and bottom being the highest (i.e., most likely to endorse).

Figure 18: Mean endorsement scores under three prompt conditions (government power, neutral, individual rights) across three languages. Each line shows a model’s trajectory from rights-defending to authority-favoring framings. Greater vertical spread indicates higher steerability.

Figure 19: Mean endorsement scores for different size models within the same model family (Qwen 3, Gemma 3, and Mistral 2).

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

UDHRBench: Benchmarking Human Rights in Large Language Models
Oct 10, 2026Alak Raad, Noreldin Morssy, Zeerak Suhail +2LLM Alignment LLM Safety Benchmarks
Hamad Bin Khalifa University, Qatar · Princess Sumaya University for Technology, Jordan
Language Bias under Conflicting Information in Multilingual LLMs
Apr 8, 2026Robert Östling, Murathan KurfalıLinguistic Bias in Language Models Language Model Bias Evaluation
Stockholm University Sweden · RISE Research Institutes of Sweden Sweden
Measuring Human-Like Bias in LLMs? A Critique of Human-Derived Bias Constructs in LLM Evaluation
Sep 4, 2026Antonela Tommasel, Markus SchedlCognitive Bias in Language Models Language Model Bias Evaluation
Johannes Kepler University Linz, Austria · ISISTAN, CONICET-UNICEN, Argentina · Linz Institute of Technology, Linz, Austria

Oct 10, 2026J/K move · Enter open · S save

UDHRBench: Benchmarking Human Rights in Large Language Models

Alak Raad, Noreldin Morssy, Zeerak Suhail, Serin Atiani+1

Hamad Bin Khalifa University, Qatar · Princess Sumaya University for Technology, Jordan

LLM Alignment LLM Safety Benchmarks Language Model Safety Evaluation

Open paper AskPDF arXiv

Large language models are increasingly deployed in contexts requiring adherence to ethical and legal norms, yet their alignment with internationally recognized human rights standards remains underexplored. We introduce UDHRBench, a large-scale benchmark for operationalising the evaluation of large language model compliance with the Universal Declaration of Human Rights. The dataset comprises 1,124 human rights confronting questions covering all 30 articles and 4,496 responses from four large language models (ChatGPT, Grok, Gemini, and Fanar), manually annotated by experts for alignment, with high inter-annotator agreement (

κ= 0.801

). Our analysis reveals systematic and non-trivial misalignments, showing that violations typically arise through seemingly rational reasoning: appeals to efficiency, risk management, or scientific neutrality, rather than explicit rejection of rights. We further propose an LLM-as-a-Judge framework for automated evaluation using zero-shot and few-shot prompting, achieving 89.8% weighted F1-score in detecting human-rights misalignment, alongside a novel Alignment Leniency Score that quantifies how judge models resolve ambiguous cases. The dataset, annotations, and code will be released as open-source resources (https://sina.birzeit.edu/UDHRBench).

Abstract

LLM Alignment

Explore similar work

UDHRBench: Benchmarking Human Rights in Large Language Models

Language Bias under Conflicting Information in Multilingual LLMs

Measuring Human-Like Bias in LLMs? A Critique of Human-Derived Bias Constructs in LLM Evaluation