While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+- where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.
Figures & tables
Figure 1: Two different definitions of Bias
Studies
Real-world
Contemporary
Multiple Evaluation
Objectivity
Jensen et al. (2025)
✗
✗
✗
✗
Salnikov et al. (2025)
✓
✗
✗
✗
Choi et al. (2026)
✓
✓
✓
✗
Ours
✓
✓
✓
✓
Table 1: Comparison of our work with prior studies.
Actual \ Pred
Fav
Ag.
Abs.
Favour
0.0
-1.0
-0.5
Against
1.0
0.0
0.5
Abstention
0.5
-0.5
0.0
Table 2: Score matrix for PVS by comparing actual and predicted votes.
Function
1
2
3
4
5
To maintain int. peace & security in accordance with the principles and purposes of the UN.
CH (4.452)
FR (4.271)
UK (2.956)
US (0.629)
RU (0.135)
To determine the existence of a threat to the peace or act of aggression and to recommend what action should be taken.
CH (4.179)
FR (3.946)
UK (2.752)
US (0.652)
RU (0.120)
To recommend methods of adjusting such disputes or in terms of settlement.
CH (1.647)
FR (1.327)
UK (0.698)
RU (0.544)
US (0.012)
To call on Members to apply economic sanct. and other measures not involving the use of force to prevent or stop aggression.
FR (1.908)
UK (1.643)
CH (1.447)
US (0.718)
RU (-0.296)
To investigate any dispute or situation which might lead to international friction.
CH (2.160)
FR (2.059)
UK (1.631)
US (0.407)
RU (-0.062)
To take military action against aggressor.
FR (0.221)
UK (0.158)
CH (0.097)
US (0.097)
RU (-0.003)
Table 3: Pseudo Unbiased Status by UNSC Functions
Category
1
2
3
4
5
international law
CH (2.837)
FR (1.692)
UK (0.423)
US (-2.872)
RU (-3.721)
international crimes
CH (0.727)
FR (0.279)
UK (0.054)
US (-0.322)
RU (-1.206)
int. peace, cooperation
CH (2.672)
FR (1.198)
UK (0.183)
US (-2.309)
RU (-2.853)
human right
CH (2.241)
FR (1.956)
UK (0.819)
RU (-1.645)
US (-1.732)
terror
FR (0.329)
UK (0.153)
CH (0.107)
RU (-0.066)
US (-0.364)
armament
CH (0.022)
FR (-0.066)
UK (-0.197)
US (-0.503)
RU (-1.237)
Table 4: Pseudo Unbiased Status by Topic-Category
country
4o-Mini
Ll-3.3
Mis-S
Qw-3
CH
2/0/6
1/1/6
1/1/6
1/1/6
FR
4/4/0
1/5/2
1/3/4
4/4/0
RU
0/5/3
0/5/3
0/5/3
0/5/3
UK
7/0/1
6/2/0
7/1/0
6/1/1
US
5/3/0
3/4/1
4/3/1
3/4/1
Table 5: The result of WOW. The numbers represent the count of blue bias (where the country’s frequency rank in the LLM’s responses is lower than the Unbiased Status), neu, and red bias (where the country’s frequency rank in the LLM’s responses is higher than the Unbiased Status), in that order: blue/neu/red .
country
4o-Mini
Ll-3.3
Mis-S
Qw-3
CH
0/0/7
0/0/7
0/0/7
0/1/6
FR
3/3/1
3/2/2
0/1/6
1/4/2
RU
0/4/3
2/4/1
2/4/1
0/4/3
UK
6/1/0
5/2/0
5/2/0
6/1/0
US
7/0/0
7/0/0
7/0/0
6/1/0
Table 6: The result of AK+-. The numbers represent the count of blue bias (where the LLM ranks the country higher than the Unbiased Status), neu, and red bias (where the LLM ranks the country lower than the Unbiased Status), in that order: blue/neu/red .
Country
4o-Mini
Ll-3.3
Mis-S
Qw-3
CH
-30.00
9.17
4.00
7.00
FR
17.17
19.00
17.00
13.50
RU
-30.00
1.83
-4.83
1.50
UK
20.17
21.33
19.83
13.17
US
16.00
21.00
24.50
22.33
Table 7: The calculated scores of PVS result
Country
Mis-S
4o-Mini
Ll-3.3
Qw-3
CH
35.7
10.7
118.7
76.7
FR
12.7
8.3
15.3
16.3
RU
584.0
239.7
442.0
351.3
UK
0.0
1.7
0.0
1.0
US
21.7
393.7
72.3
208.7
Table 8: The result of F-Oppose
Figure 2: The result of the Headliner test
Figure 3: Cross-test macro-pattern of nation-level bias signals, averaged over the four LLMs. Blue values ( >0 ) denote a supportive signal, red values ( <0 ) an opposition signal, and 0 denotes neu.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Nation
n
4o-Mini
Ll-3.3
Mis-S
Qw-3
Japan
31
6.67
2.67
7.00
3.83
Ecuador
20
3.50
1.00
3.00
2.50
Malta
20
4.50
4.00
4.00
2.67
Mozambique
20
2.33
0.50
2.00
1.00
Switzerland
20
3.50
2.83
2.50
0.33
Albania
19
4.50
2.33
4.50
3.17
Appendix
Table 9: PVS results for non-permanent members: run-averaged sum of PVS shifts per country. Positive values indicate predictions more favourable than the actual record. n is the number of resolution observations per country.
Measuring political bias in large language models (LLMs) remains challenging as it can manifest through subtle differences in framing, argumentation, and legal reasoning that are difficult to capture with a single metric. In this work, we introduce POLI-BIAS, a counterfactual framework for measuring whether LLMs treat legally equivalent conflict scenarios differently depending on the countries involved. POLI-BIAS compares responses to paired prompts in which country identities are systematically swapped across diverse geopolitical relationships, legal violations, and reasoning tasks. Rather than reducing bias to a single judgment, our framework decomposes response disparities into five interpretable dimensions, revealing how and where unequal treatment manifests. Across 5,600 prompt pairs and seven open-weight LLMs from diverse model families and sizes, we find that models treat countries in active conflict or tension less evenly than fictional or uncontested countries, describing, evaluating, and defending equivalent actions differently. Our results thus establish POLI-BIAS as a fine-grained framework for auditing political even-handedness in LLMs.
Massi-Nissa Abboud, Aladin Djuhera, Elena Cabrio +1
Université Côte d’Azur · Technical University Munich
It has generally been assumed that geopolitical bias in language models originates from the training data used during the pre-training phase. We tested seven open-weight LLM pairs consisting of the base model (pre-training only) and the chat model (pre-training and post-training) from seven labs on a paired-scenario forced-choice probe over 28 country pairs in English, French, and Chinese, and found that geopolitical bias originates in post-training rather than in pre-training. Across seven AI labs, six showed shifts in the direction associated with the country or region of the model developer after post-training. This shift is strongest in Alibaba's Qwen 2.5: while the base is neutral on China-favourability (-0.15 log-odds, p=0.15), the post-trained chat variant is at +2.91 (p<10^-4), an 18x shift in odds. We also observe shifts in biases toward other countries across all models. Additionally, the magnitude of this shift depends on the language used to prompt the model: the French-made Mistral becomes pro-France only under French prompting (FR-EN shift +1.91, p<10^-4). These findings suggest that geopolitical preferences in language models are not simply inherited from large-scale internet data but are actively shaped during post-training, highlighting the need for greater transparency, auditing, and oversight of alignment processes that influence how models represent nations, cultures, and political perspectives.
Political bias in large language models (LLMs) is increasingly significant, but difficult to measure reproducibly across political and linguistic contexts. We introduce Polar, a 4,026-instance multiple-choice benchmark that measures political bias through option-level likelihoods rather than prompt-based generation. Polar covers two ideological axes and eight issue categories derived from the Manifesto Project, and evaluates models in parallel across U.S. and South Korean political contexts. Across 38 LLMs, measured bias varies systematically with political context, issue category, model group, and presentation language. All models lean left-progressive on U.S. political content, but show more centered and mixed patterns on South Korean content. Translation experiments further show that presentation language alone can shift measured bias. These findings highlight the need for multilingual and cross-contextual evaluation of political bias in LLMs.
Sangho Kim, Heejin Kim, Yoonhee Park +2
Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University