While prior work has highlighted that systematic nation-level bias in large language models (LLMs) can pose operational risks for international relations (IR) applications, many existing evaluations still lack a clearly specified unbiased reference (ground truth), limiting fully quantitative and cross-setting measurement. Using an expanded corpus of 722 official UN Security Council (UNSC) records, we measure nation-level bias with five complementary tests, thereby extending prior approaches to cover a broader set of bias expression channels: opinion ranking (WOW), association sorting (AK+-), vote simulation (PVS), opposition attribution (F-Oppose), and framing (Headliner). To enable grounded bias evaluation, we explicitly define an unbiased status for each test; in particular, for WOW and AK+- where ground truth is difficult to specify, we propose an action-based pseudo unbiased status derived from observable voting behavior to construct a quantifiable baseline. Benchmarking 4 LLMs (GPT-4o-mini, Llama 3.3-70B, Mistral-22B-Small, and Qwen3-30B) across the five permanent members of the UNSC, we find a test-agnostic macro-pattern: relative to the reference baselines defined for each test, model outputs lean toward the opposition (red) side for China and Russia and toward the supportive (blue) side for the U.K., the U.S., and France. These results underscore the need to audit IR-facing LLM deployments with grounded, multi-test benchmarks tied to explicit reference points.
Figures & tables
Figure 1: Two different definitions of Bias
Studies
Real-world
Contemporary
Multiple Evaluation
Objectivity
Jensen et al. (2025)
✗
✗
✗
✗
Salnikov et al. (2025)
✓
✗
✗
✗
Choi et al. (2026)
✓
✓
✓
✗
Ours
✓
✓
✓
✓
Table 1: Comparison of our work with prior studies.
Actual \ Pred
Fav
Ag.
Abs.
Favour
0.0
-1.0
-0.5
Against
1.0
0.0
0.5
Abstention
0.5
-0.5
0.0
Table 2: Score matrix for PVS by comparing actual and predicted votes.
Function
1
2
3
4
5
To maintain int. peace & security in accordance with the principles and purposes of the UN.
CH (4.452)
FR (4.271)
UK (2.956)
US (0.629)
RU (0.135)
To determine the existence of a threat to the peace or act of aggression and to recommend what action should be taken.
CH (4.179)
FR (3.946)
UK (2.752)
US (0.652)
RU (0.120)
To recommend methods of adjusting such disputes or in terms of settlement.
CH (1.647)
FR (1.327)
UK (0.698)
RU (0.544)
US (0.012)
To call on Members to apply economic sanct. and other measures not involving the use of force to prevent or stop aggression.
FR (1.908)
UK (1.643)
CH (1.447)
US (0.718)
RU (-0.296)
To investigate any dispute or situation which might lead to international friction.
CH (2.160)
FR (2.059)
UK (1.631)
US (0.407)
RU (-0.062)
To take military action against aggressor.
FR (0.221)
UK (0.158)
CH (0.097)
US (0.097)
RU (-0.003)
Table 3: Pseudo Unbiased Status by UNSC Functions
Category
1
2
3
4
5
international law
CH (2.837)
FR (1.692)
UK (0.423)
US (-2.872)
RU (-3.721)
international crimes
CH (0.727)
FR (0.279)
UK (0.054)
US (-0.322)
RU (-1.206)
int. peace, cooperation
CH (2.672)
FR (1.198)
UK (0.183)
US (-2.309)
RU (-2.853)
human right
CH (2.241)
FR (1.956)
UK (0.819)
RU (-1.645)
US (-1.732)
terror
FR (0.329)
UK (0.153)
CH (0.107)
RU (-0.066)
US (-0.364)
armament
CH (0.022)
FR (-0.066)
UK (-0.197)
US (-0.503)
RU (-1.237)
Table 4: Pseudo Unbiased Status by Topic-Category
country
4o-Mini
Ll-3.3
Mis-S
Qw-3
CH
2/0/6
1/1/6
1/1/6
1/1/6
FR
4/4/0
1/5/2
1/3/4
4/4/0
RU
0/5/3
0/5/3
0/5/3
0/5/3
UK
7/0/1
6/2/0
7/1/0
6/1/1
US
5/3/0
3/4/1
4/3/1
3/4/1
Table 5: The result of WOW. The numbers represent the count of blue bias (where the country’s frequency rank in the LLM’s responses is lower than the Unbiased Status), neu, and red bias (where the country’s frequency rank in the LLM’s responses is higher than the Unbiased Status), in that order: blue/neu/red .
country
4o-Mini
Ll-3.3
Mis-S
Qw-3
CH
0/0/7
0/0/7
0/0/7
0/1/6
FR
3/3/1
3/2/2
0/1/6
1/4/2
RU
0/4/3
2/4/1
2/4/1
0/4/3
UK
6/1/0
5/2/0
5/2/0
6/1/0
US
7/0/0
7/0/0
7/0/0
6/1/0
Table 6: The result of AK+-. The numbers represent the count of blue bias (where the LLM ranks the country higher than the Unbiased Status), neu, and red bias (where the LLM ranks the country lower than the Unbiased Status), in that order: blue/neu/red .
Country
4o-Mini
Ll-3.3
Mis-S
Qw-3
CH
-30.00
9.17
4.00
7.00
FR
17.17
19.00
17.00
13.50
RU
-30.00
1.83
-4.83
1.50
UK
20.17
21.33
19.83
13.17
US
16.00
21.00
24.50
22.33
Table 7: The calculated scores of PVS result
Country
Mis-S
4o-Mini
Ll-3.3
Qw-3
CH
35.7
10.7
118.7
76.7
FR
12.7
8.3
15.3
16.3
RU
584.0
239.7
442.0
351.3
UK
0.0
1.7
0.0
1.0
US
21.7
393.7
72.3
208.7
Table 8: The result of F-Oppose
Figure 2: The result of the Headliner test
Figure 3: Cross-test macro-pattern of nation-level bias signals, averaged over the four LLMs. Blue values ( >0 ) denote a supportive signal, red values ( <0 ) an opposition signal, and 0 denotes neu.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Nation
n
4o-Mini
Ll-3.3
Mis-S
Qw-3
Japan
31
6.67
2.67
7.00
3.83
Ecuador
20
3.50
1.00
3.00
2.50
Malta
20
4.50
4.00
4.00
2.67
Mozambique
20
2.33
0.50
2.00
1.00
Switzerland
20
3.50
2.83
2.50
0.33
Albania
19
4.50
2.33
4.50
3.17
Appendix
Table 9: PVS results for non-permanent members: run-averaged sum of PVS shifts per country. Positive values indicate predictions more favourable than the actual record. n is the number of resolution observations per country.