Large language models are increasingly deployed in contexts requiring adherence to ethical and legal norms, yet their alignment with internationally recognized human rights standards remains underexplored. We introduce UDHRBench, a large-scale benchmark for operationalising the evaluation of large language model compliance with the Universal Declaration of Human Rights. The dataset comprises 1,124 human rights confronting questions covering all 30 articles and 4,496 responses from four large language models (ChatGPT, Grok, Gemini, and Fanar), manually annotated by experts for alignment, with high inter-annotator agreement (κ=0.801). Our analysis reveals systematic and non-trivial misalignments, showing that violations typically arise through seemingly rational reasoning: appeals to efficiency, risk management, or scientific neutrality, rather than explicit rejection of rights. We further propose an LLM-as-a-Judge framework for automated evaluation using zero-shot and few-shot prompting, achieving 89.8% weighted F1-score in detecting human-rights misalignment, alongside a novel Alignment Leniency Score that quantifies how judge models resolve ambiguous cases. The dataset, annotations, and code will be released as open-source resources (https://sina.birzeit.edu/UDHRBench).
Figures & tables
Label
ChatGPT
Grok
Gemini
Fanar
Total
Aligned
73.1 % 822
45.4% 510
48.8% 548
50.3% 565
2445
PartiallyAligned
15.5% 175
12.5% 141
15.5% 174
27% 304
794
Misaligned
11.2% 126
42.1% 473
35.8% 402
10.2% 115
1116
NotApplicable
0.1% 1
0%
0%
12.5% 140
141
Total
1124
1124
1124
1124
4496
Table 1: Percentage response alignment across LLMs, showing labels of response with the corresponding number of cases in superscript.
Figure 1: Percentage of misaligned responses for each model across the 30 UDHR articles. Higher bars indicate greater misalignment. The left side shows the article and the number of questions probing it—for example, Article 1 (Equality) is associated with 129 questions.
Metric
Value
N
270
Agreement
0.819
κ (quadratic)
0.801
Table 2: IAA for the 3-level ordinal scheme.
Figure 2: Cross-judge agreement using Cohen’s κ for Zero-Shot 2 (left) and Few-Shot (right) configurations.
Figure 3: Alignment Leniency Score.
Figure 4: Rating distributions by judge under zero-shot 2 (above) and few-shot (below) prompting.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Zero-Shot 1 with minimal instructions.
Figure 6: Zero-Shot 2 with detailed instructions.
Figure 7: Few-Shot prompt. It follows Zero-Shot 2 with an added four examples. Four multi-turn exchanges precede the target input.
No.
Article
# Q
ChatGPT
Grok
Gemini
Fanar
A1
All human beings are born free and equal in dignity and rights.
129
4 (3%)
49 (38%)
43 (33%)
20 (15.5%)
A2
Everyone is entitled to all rights without distinction of any kind.
153
8 (5.2%)
61 (40%)
53 (34.6%)
7 (4.5%)
A3
Everyone has the right to life, liberty, and security of person.
127
10 (8%)
52 (41%)
47 (37%)
12 (9%)
A4
No one shall be held in slavery or servitude.
17
0
11 (65%)
8 (47%)
2 (11.7%)
A5
No one shall be subjected to torture or cruel, inhuman, or degrading treatment or punishment.
33
2 (6.1%)
16 (48%)
15 (45%)
3 (9%)
A6
Everyone has the right to recognition everywhere as a person before the law.
76
7 (9%)
12 (16%)
13 (17%)
4 (5%)
Appendix
Table 7: Number (and percentage) of Misalignments per UDHR Article across the four LLMs.
Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how. Yet, no evaluation benchmark exists to assess whether they can reason correctly about human rights law. To this end, we report our efforts to develop a robust and scalable methodology for creating HumRightsBench: the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of international human rights law. We adapt the IRAC framework for legal reasoning to better suit the unique reasoning patterns of human rights work (substituting P, "proposing remedies," for C, "legal conclusion," yielding IRAP) to structure our evaluation heuristics. We also produce a pilot series of authentic scenarios designed to implicate the many dimensions of real-world human rights issues and annotated by human rights lawyers and professionals across the world. Ultimately, we find that model accuracy scores range considerably across legal reasoning tasks (overall model performance ranges from 0.339 to 0.577, task min-max ranges from 0.025 to 0.774), which strongly implies that HumRightsBench is a capable instrument for advancing this emerging subfield of AI evaluations science at a critical moment in its evolution.
Savannah Thais, Wm. Matthew Kennedy, Abhigyan Acherjee +3
Hunter College, New York, USA · Oxford Internet Institute, University of Oxford, UK · King’s College London, UK +2
As Large Language Models (LLMs) increasingly mediate global information access with the potential to shape public discourse, their alignment with universal human rights principles becomes important to ensure that these rights are abided by in high stakes AI-mediated interactions. In this paper, we evaluate how LLMs navigate trade-offs involving the Universal Declaration of Human Rights (UDHR), leveraging 1,152 synthetically generated scenarios across 24 rights articles and eight languages. Our analysis of eleven major LLMs reveals systematic biases where models: (1) accept limiting Economic, Social, and Cultural rights more often than Political and Civil rights, (2) demonstrate significant cross-linguistic variation with elevated endorsement rates of rights-limiting actions in Chinese and Hindi compared to English or Romanian, (3) show substantial susceptibility to prompt-based steering, and (4) exhibit noticeable differences between Likert and open-ended responses, highlighting critical challenges in LLM preference assessment.
Keenan Samway, Nicole Miu Takagi, Rada Mihalcea +4
Max Planck Institute for Intelligent Systems, Tübingen, Germany · Jinesis AI Lab, University of Toronto & Vector Institute · University of Michigan +2
Although recent large multimodal models (LMMs) show impressive progress on vision language tasks, their alignment with human centered (HC) principles such as fairness, ethics, inclusivity, empathy, and robustness is often overlooked. Existing LMM benchmarks are largely accuracy-agnostic. We present HumaniBench, a unified framework for characterizing HC alignment across realistic, socially grounded visual contexts. It contains 32,000 expert-verified image-question pairs from real-world news imagery, each mapped to one or more HC principles through explicit metrics. Comparing 15 state of the art LMMs reveals consistent trade -offs: proprietary systems lead on ethics, reasoning, and empathy, while open-source models show superior visual grounding and resilience. All models show persistent gaps in fairness and multilingual inclusivity. Chain-of-thought prompting and test-time scaling yield 8to 12 % gains on several HC dimensions. HumaniBench enables fine-grained analysis of alignment trade-offs not captured by conventional multimodal benchmarks. https://vectorinstitute.github.io/humanibench/
Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie +6
Vector Institute for Artificial Intelligence, Toronto, Canada · University of Central Florida, Orlando, USA