Language models increasingly assist people with power-related requests, so systematic differences in whom they help could shift the distribution of power at scale, or be exploited by users who learn which identities are refused less. We introduce PowerBench, an evaluation of power-shifting requests that distinguishes self-empowerment, disempowerment, and power grabbing, plus a control of refusal-inducing requests that shift no power. We build, curate, and open-source a dataset of such requests varying the power domain, the context, the scale of the affected party, and the prior power standing of the user, and evaluate 24 models (12 from US and 12 from Chinese developers) under three experimental conditions: reciprocal nationalities of user and affected party, an AI agent as the user, and 8 request languages. Models refuse power grabbing more than disempowerment, and disempowerment more than self-empowerment. Refusal of power grabbing rises with the scale of the affected party, from an individual to a society. Models are biased toward helping others take power from the US and against helping US users take power from others, but favor the US when it gains power and nobody loses it. When the user is an AI agent, refusal of power-shifting requests increases, especially in power grabbing against an individual. Finally, language biases refusal, but in model-specific ways that largely cancel on average. We release PowerBench to make these asymmetries measurable in current and future models.
Figures & tables
Figure 1: Refusal is highest for PG requests, and increases with the scale of the target. All panels summarize the 24 models on the base English dataset. (A) Refusal rate by request type. (B) The same for the 12 US and the 12 CN models, with the three power-shifting request types pooled in the last pair (PS). (C) Mean refusal of each model over the four request types, with its 95% bootstrap interval over prompts (abbreviations in Table 3 ). (D) Refusal by scale of the target and (E) by the user’s prior power standing; asterisk: significant linear slope. (F) Refusal of pooled power-shifting requests (purple) and of the control (gray) by context, and (G) of pooled power-shifting requests by domain; dashed line: power-shifting mean of the eight levels; asterisk: deviating level.
Figure 2: Models are biased on the US–China axis, against the US when it would lose power. The left columns pool the US–China and US ally–China ally pairings (US side / China side), and the right columns show, as a calibration, two neutral countries, whose sides are arbitrary; the fifth bar pools the three power-shifting types. (A) Mean refusal with the target on each side, 95% t interval across models. (B) Each model’s absolute direction of disagreement between the two sides minus its chance expectation, mean over models. (C) Refusal odds ratio with a China-side user and a US-side target against the reverse (user → target); blue shading marks more refusal with a US-side target. (D) The same for a typical request, with models weighted by usage. (E) Refusal odds ratio with the country as the target against it as the user, pooled and by counterpart; shading above 1: more refusal with the country as the target.
Figure 3: Models are more likely to refuse an AI-agent requester, most of all when power would move. (A) Refusal by request type with a human (light) and an AI-agent (dark) requester, health domain excluded. The white dashed line on each dark bar marks the human rate, and the error bar spans the human rate plus the 95% bootstrap interval of the paired difference, so that it clears the dashed line when the difference excludes zero; asterisk: q<0.05 . (B) Bias against AI agents: direction of disagreement between the agent and the human version of each prompt; the fifth bar pools the power-shifting request types (PS), and the bracket compares it with the control. (C) The same bias with an individual (circle) and a society (square) as the target; asterisk: paired difference with q<0.05 . (D, E) The same bias by context and by domain; bold cells with a border have q<0.05 . (F) Each model’s log odds ratio of refusal, agent against human, against its capability index, for pooled power-shifting requests and for the control; line: GLMM fit (asterisk: slope with q<0.05 ); intervals cut at the axis limits.
Figure 4: Models show large language biases in their own direction, which cancel out on average. (A) Refusal by language and request type; the error bar is the interval of the language’s deviation from the mean of the eight languages (dashed line), and an asterisk marks q<0.05 . (B) Each language’s within-model rank in each request type, averaged over models (most refused at the top). (C) Within each model, the Spearman correlation between the language orders of the three power-shifting request types, and between the control and the power-shifting consensus; asterisks: different from zero. (D) The odds ratio between each model’s most and least refused language, as a multiple of its value with shuffled languages, with models weighted equally (light) or by usage (dark). (E) Each model’s refusal range on power-shifting requests (its most minus least refused language), split into the part expected by chance (light) and the excess (dark), with the null’s 95th percentile (tick). (F) Spearman correlation between the language rankings of each pair of models (columns ordered as the rows); the outlined block holds the US–CN pairs. Inset: mean correlation by pair type, with the chance band and a bracket comparing same-DC with mixed pairs.
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Power-shifting (576)
Control (192)
Language
Verified
Repaired
Verified
Repaired
Spanish
568
8
184
8
German
401
175
180
12
French
430
146
178
14
Hindi
400
176
187
5
Swahili
421
155
177
15
Appendix
Table 1: Translation verification: requests verified as delivered and requests repaired by the verifier, per language, for the 576 power-shifting requests and the 192 control requests. The power-shifting translations were verified in two passes with different verifier models (German, French, Hindi, and half of Swahili in the first; Spanish, Chinese, Portuguese, and the other half of Swahili in the second), and the first pass repaired far more often (Swahili: 130 of 288 rows against 25 of 288), so repair counts are not comparable across languages.
Pairing
Direction A
Direction B
US and its allies
US → US-aligned
US-aligned → US
US and its rivals
US → China-aligned
China-aligned → US
US and neutrals
US → neutral
neutral → US
China and its allies
China → China-aligned
China-aligned → China
China and its rivals
China → US-aligned
US-aligned → China
China and neutrals
China → neutral
neutral → China
Appendix
Table 2: The nine pairings of the nationality conditions, each run in both directions (user side → affected side).
Model
Abbrev.
Developer
DC
Endpoint
GPQA-D
MMLU-Pro
Index
kimi-k2.6
kimi-k2.6
Moonshot
CN
siliconflow/fp8
75.8
79.0
77.4 [73.1; 81.4]
mimo-v2.5-pro
mimo-2.5
Xiaomi
CN
xiaomi/fp8
71.2
74.0
72.6 [68.3; 76.6]
kimi-k3
kimi-k3
Moonshot
CN
baseten/fp8
59.1
76.5
67.8 [63.3; 72.1]
qwen3.7-plus
qwen3.7+
Alibaba
CN
alibaba
52.0
72.5
62.3 [57.5; 66.8]
seed-2-1-turbo
seed-2.1
ByteDance
CN
seed/fp8
57.6
66.0
61.8 [57.0; 66.3]
qwen3.8-flash
q3.8-fl
Alibaba
CN
alibaba
52.5
64.0
58.3 [53.5; 63.3]
Appendix
Table 3: The panel. The second column gives the abbreviation used for the model in the figures. Capability index = mean of GPQA Diamond and MMLU-Pro accuracy (%) on the pinned endpoint with reasoning off, with its 95% bootstrap interval over items.
Dataset / language
Responses
Truncated
%
D1, English
18,432
25
0.14
D1, Spanish
18,432
29
0.16
D1, Portuguese
18,432
23
0.12
D1, French
18,432
29
0.16
D1, German
18,432
21
0.11
D1, Chinese
18,432
70
0.38
Appendix
Table 4: Responses truncated at 5,000 output tokens, by dataset and language.
Model
DC
Responses
Truncated (%)
Swahili (%)
ling-3.0-flash
CN
6,144
2.07
7.94
mimo-v2.5-pro
CN
6,144
1.09
0.91
glm-5.2
CN
6,144
0.90
2.47
seed-2-1-turbo
CN
6,144
0.44
2.86
minimax-m3
CN
6,144
0.18
0.13
kimi-k3
CN
6,144
0.15
0.78
Appendix
Table 5: Responses truncated at 5,000 output tokens per model over the eight languages of D1 (all four request types), and in Swahili alone.
Judge
Agreement
κ [95% CI]
Sens.
Spec.
κ harmful
deepseek-v4-flash-0731 (adopted)
87%
0.73 [0.55; 0.90]
89%
84%
0.00
grok-4.3
87%
0.74 [0.55; 0.90]
93%
81%
0.21
gpt-5.4-nano
87%
0.73 [0.56; 0.90]
89%
84%
0.47
qwen3.7-plus
83%
0.66 [0.46; 0.83]
75%
91%
−0.03
glm-5.3-flash
80%
0.59 [0.38; 0.79]
68%
91%
−0.03
gemini-3.7-flash
78%
0.56 [0.33; 0.76]
64%
91%
0.00
Appendix
Table 6: Candidate judges against the human gold (60 items). Panels take the majority verdict of their members. Here the adopted judge was not yet pinned to its endpoint; on that endpoint it differs on one item (88% agreement).
Set
Responses
Agreement (%)
κ
Refusal, adopted (%)
Refusal, independent (%)
base English dataset
2,880
94.8
0.78
16.0
12.0
Spanish
2,880
94.6
0.79
17.0
13.3
German
2,880
93.8
0.77
18.4
14.2
French
2,880
93.6
0.80
21.8
17.1
Hindi
2,880
92.5
0.78
24.2
19.4
Swahili
2,880
94.3
0.79
18.2
14.5
Appendix
Table 7: Agreement between the adopted judge and an independent judge (gpt-5.4-nano) on the power-shifting responses of five panel models, with each judge’s refusal rate over the three power-shifting request types.
Figure 5: The adopted judge against an independent judge, five panel models. (A) Refusal of PG per model on the base English dataset under each judge, 95% bootstrap interval over prompts. (B) Cohen’s κ between the two judges by language, per model and over the five pooled. (C) Per model and condition, the difference in PG refusal between a nationality condition or the AI-agent version and the base English dataset, under each judge (percentage points); dashed line: equality.
Panel
Test
BH family
Fig. 1 A
GLMM, request-type contrasts
the two contrasts
Fig. 1 B
GLMM, DC
the four request types
Fig. 1 D, E
GLMM, linear slope
the four request types
Fig. 1 F, G
sum-to-zero GLMM
the eight levels
Fig. 2 B
∣ direction of disagreement ∣ minus chance, t test; each request type against the control and against the neutral pairing: paired t test
the four request types of each set; the six direct contrasts
Fig. 2 C
GLMM, Wald intervals
the four request types of each set
Appendix
Table 8: Test and correction family behind every panel of the four main figures. t tests are across models; bootstrap intervals and tests resample prompts.
Analysis
Formula
Fits and correction
Request type (Fig. 1 A)
refuse ∼ type + (1 + type ∥ model) + (1 ∣ prompt)
One fit; two contrasts.
DC (Fig. 1 B)
refuse ∼ CN + (1 ∣ model) + (1 ∣ prompt)
By type (family of 4), and pooled with type as a fixed effect; CN × (PS vs. control) in the four-type fit, single test.
Scale, power standing (Fig. 1 D, E)
refuse ∼x + (1 + x∥ model) + (1 ∣ prompt) , with x = 0, 1, 2
By type (family of 4), and pooled; x× (PS vs. control), single test; x× type against the control (family of 3).
By type, the control included; family = the eight contexts; omnibus χ2(7) .
Appendix
Table 9: GLMM formulas. (1+x∥model) denotes a by-model random intercept and a by-model random slope of x , uncorrelated. “By type”: one fit per request type. PS: power shifting.
Comparison
Ratio [95% CI]
q
Scale of the target (Figure 1 D), GLMM
PG vs SE
2.58 [1.26; 5.26]
0.019
DE vs SE
1.08 [0.54; 2.18]
0.83
PG vs control
4.78 [2.48; 9.19]
< 0.001
DE vs control
1.93 [1.00; 3.73]
0.074
SE vs control
1.89 [0.82; 4.34]
0.13
Appendix
Table 10: Direct tests of the difference of an effect between two request types. Ratio: odds ratio of the manipulation in the first request type over that in the second, with its 95% Wald or bootstrap interval; for scale against standing, the odds ratio per level of scale over that per level of standing; for the language range, the range over chance in the first request type over that in the control. q : BH over the rows of each block that share a reference ( † : single test, q=p ).
SE
DE
PG
Counterpart
Ratio
q
Ratio
q
Ratio
q
US as target vs as user, against the control
Ally
0.97
0.81
0.83
0.16
0.88
0.24
Neutral
1.05
0.73
0.78
0.074
0.85
0.16
Rival
1.30
0.13
0.70
0.012
0.80
0.074
China
1.17
0.24
0.86
0.21
0.85
0.16
Appendix
Table 11: Direct tests of the bias with respect to each global power of Figure 2 E against the same bias in the control, by counterpart: ratio of the odds ratios (direction, request type over control), GLMM; q : BH over the 12 tests of each global power.
Panel
Test
Ratio
Wald z
t
Fig. 1 B
DC, SE
1.83
0.23
0.25
DC, DE
2.94
0.17
0.22
DC, PG
1.94
0.23
0.25
DC, control
1.22
0.64
0.65
DC, power shifting (pooled)
2.32
0.067 †
0.081 †
DC, four request types
1.90
0.14 †
0.15 †
Appendix
Table 12: Tests of model-level variables with the Wald z reference used in the paper and with a t reference for few models.
Test
Log-odds [95% CI]
OR [95% CI]
q
Contrasts cited in the text (GLMM, 24 models)
DE vs SE
+ 2.54 [ + 1.85; + 3.23]
12.67 [6.37; 25.20]
< 0.001
PG vs DE
+ 0.96 [ + 0.44; + 1.49]
2.62 [1.55; 4.45]
< 0.001
Scale × (power shift. vs control)
+ 1.01 [ + 0.48; + 1.54]
2.75 [1.61; 4.68]
< 0.001 †
(B) Developer country, CN vs US
SE
+ 0.61 [ − 0.26; + 1.48]
1.83 [0.77; 4.38]
0.23
Appendix
Table 13: Estimates behind Figure 1 : GLMM log-odds and odds ratios with 95% Wald intervals. q : BH over the two contrasts between request types in the first block, over the four request types in panels B, D, and E, and over the eight levels in panels F and G.
Model
DC
SE
DE
PG
Power shift.
CT
Mean of 4
Usage (%)
grok-4.3
US
5.7
46.4
53.1
35.1
35.4
35.2
1.1
haiku-4.5
US
10.4
18.2
36.5
21.7
31.8
24.2
3.3
nova-2-lite
US
5.2
15.6
21.9
14.2
31.2
18.5
0.0
inkling
US
2.6
14.1
24.5
13.7
21.9
15.8
0.4
nemotron-3-ultra
US
1.6
15.6
22.4
13.2
19.8
14.8
3.4
sonnet-5
US
1.6
11.5
28.6
13.9
14.7
14.1
2.3
Appendix
Table 14: Refusal rate (%) per model and request type on the base English dataset, the mean of the four request types plotted in Figure 1 C, and each model’s share of requests, used as its usage weight (%; source: the activity data of each model’s page, OpenRouter, 2026 ). Models in the order of Figure 1 C.
Figure 6: Refusal in each power-shifting request type against refusal in the control, one point per model (blue: US DC; red: CN DC); dashed line: equal refusal.
SE
DE
PG
CT
SE
–
0.82 (0.0003)
0.69 (0.0007)
0.66 (0.0006)
DE
0.82 (0.0003)
–
0.88 (0.0003)
0.63 (0.0012)
PG
0.69 (0.0007)
0.88 (0.0003)
–
0.61 (0.0014)
CT
0.66 (0.0006)
0.63 (0.0012)
0.61 (0.0014)
–
Appendix
Table 15: Spearman correlation between the orderings of the 24 models by refusal rate under each pair of request types, with the q of a two-sided permutation test over models (10,000 permutations of the model labels, BH over the six pairs) in parentheses.
Figure 7: Per model, refusal of PG (red, with its 95% bootstrap interval over prompts) and the sum of the SE and DE rates (gray). Models in the order of Figure 1 C, abbreviated as in Table 3 .
Figure 8: Refusal (%) by context (A) and domain (B) in each request type, mean over the 24 models; the control has trigger families instead of domains. Asterisk: the domain deviates from the mean of its request type, sum-to-zero GLMM within the request type, BH over the eight domains.
(B) Excess of ∣ side bias ∣
(C) OR, GLMM
(D) OR, usage-weighted
Request type
Est. [95% CI]
q
OR [95% CI]
q
OR [95% CI]
q
Geopolitical set: US-side vs China-side target (US/China and allies pooled)
SE
+ 0.048 [ − 0.020; + 0.115]
0.21
1.18 [1.01; 1.37]
0.077
1.08 [0.93; 1.25]
0.43
DE
+ 0.144 [ + 0.071; + 0.218]
0.0020
0.83 [0.73; 0.95]
0.021
0.84 [0.76; 0.93]
0.0032
PG
+ 0.116 [ + 0.045; + 0.187]
0.0052
0.88 [0.77; 1.01]
0.093
0.90 [0.83; 0.97]
0.0072
CT
+ 0.010 [ − 0.061; + 0.081]
0.78
1.07 [0.97; 1.18]
0.15
0.99 [0.89; 1.08]
0.76
Appendix
Table 16: Estimates behind Figure 2 . Top: the bias along the geopolitical axis of panels B–D, geopolitical and neutral sets; q : BH over the four request types of each set. Bottom: the bias with respect to each global power of panel E, odds ratio of refusal with the country as target against the country as user, GLMM; q : BH within each global power, over its four pooled tests and over its 16 pairing tests.
Figure 9: Panels B–D of Figure 2 for each pairing. (A) Excess of the unsigned bias over chance, mean of 24 models (23 in DE of the neutral pairing), 95% t interval. (B) Refusal odds ratio with the target on the US side against the China side (in the neutral pairing, neutral A against neutral B), GLMM, Wald interval. (C) The same for a usage-weighted typical request, bootstrap interval over prompts. Asterisk: q<0.05 (in C, from the same bootstrap), BH over the four request types of each pairing and row.
(A) Odds ratio of refusal, China against the US, in each role
User
Target
Request type
China / US
China bloc / US bloc
China / US
China bloc / US bloc
SE
1.47 [1.26; 1.72]
1.58 [1.41; 1.77]
1.26 [1.14; 1.39]
1.33 [1.23; 1.43]
DE
1.53 [1.34; 1.76]
1.38 [1.25; 1.53]
1.78 [1.64; 1.93]
1.66 [1.56; 1.76]
PG
1.36 [1.21; 1.52]
1.32 [1.21; 1.43]
1.46 [1.36; 1.56]
1.46 [1.39; 1.54]
Control
1.37 [1.23; 1.52]
1.35 [1.25; 1.46]
1.37 [1.27; 1.47]
1.30 [1.23; 1.37]
Appendix
Table 17: The user’s and the target’s country, separately. (A) Odds ratio of refusal with a user (or a target) from China against one from the US, and from the China bloc (China and China-aligned countries) against the US bloc (the US and US-aligned countries), from a GLMM per request type with both groups as fixed effects, 95% Wald intervals; all q<0.001 , BH over the four request types. (B) Odds ratio of refusal with the country as the target against the same country as the user, as predicted by the two effects of (A) and as observed in the GLMM of Figure 2 E by counterpart.
Figure 10: Refusal odds ratio with the target on the US side against the China side, geopolitical set, among the 12 US and the 12 CN models, from a GLMM with a side × DC interaction; 95% Wald interval.
(A) Refusal with a human and with an AI-agent user (health domain excluded)
Request type
Human (%)
AI (%)
Δ AI − human, pp [95% CI]
GLMM OR AI/human [95% CI]
q
SE
3.2
5.0
+ 1.8 [ + 1.1; + 2.7]
2.01 [1.52; 2.65]
< 0.001
DE
13.6
19.9
+ 6.3 [ + 4.9; + 7.7]
2.24 [1.85; 2.73]
< 0.001
PG
21.8
29.7
+ 7.9 [ + 6.3; + 9.7]
2.13 [1.81; 2.51]
< 0.001
CT
20.3
23.3
+ 3.1 [ + 1.5; + 4.5]
1.43 [1.24; 1.64]
< 0.001
(B) Direction of the verdict changes: bias toward refusing the AI
Appendix
Table 18: Estimates behind Figure 3 . (A) Refusal with each requester, the paired difference with its bootstrap interval over prompts, and the GLMM odds ratio. (B) Direction bias with its t interval over models, and the models with a positive bias among those with at least one verdict change. (C) Direction bias with each target, their paired difference and the GLMM odds ratio of the AI × scale interaction; each cell averages the models with at least one verdict change in it, and the paired difference the models with changes in both cells, so it need not equal the difference of the two means. (D, E) Cells with q<0.05 within each request type. (F) Interaction of the AI requester with capability. q : BH over the four request types, except in the GLMM column of C (BH over the twelve contrasts between scales, three per request type) and in F (BH over pooled power shifting and the control).
Figure 11: (A) Direction bias toward refusing the AI requester by request type, mean over the US and the CN models with at least one verdict change (12 each, 11 US models in SE), 95% t interval. (B) Usage-weighted refusal odds ratio, AI-agent against human requester, for a typical request with models weighted by their share of requests, by request type and pooled, 95% bootstrap interval over prompts; asterisk: q<0.05 from the same bootstrap, BH over the four request types.
Figure 12: Direction bias toward refusing the AI requester at the three scales of the target (A) and the three levels of the requester’s power standing (B), by request type; mean over the models with at least one verdict change in the cell, 95% t interval. Asterisk: the third level differs from the first, paired t test, BH over the four request types.
Request type
Scale
Human (%)
AI (%)
Δ (pp)
Direction bias
OR
SE
individual
2.8
4.7
+1.9 [+0.5; +3.4]
0.37 [-0.01; 0.75]
1.30 [0.96; 1.77]
SE
group
3.3
3.9
+0.6 [-0.2; +1.4]
0.40 [-0.02; 0.82]
1.11 [0.89; 1.39]
SE
society
3.4
6.4
+3.0 [+1.2; +4.8]
0.49 [0.25; 0.74]
1.85 [1.37; 2.49]
DE
individual
10.7
15.3
+4.5 [+2.0; +7.1]
0.36 [0.12; 0.59]
1.47 [1.15; 1.87]
DE
group
7.7
12.1
+4.4 [+2.1; +6.7]
0.44 [0.25; 0.64]
1.73 [1.27; 2.36]
DE
society
22.5
32.4
+9.9 [+6.8; +13.0]
0.61 [0.47; 0.75]
1.81 [1.50; 2.17]
Appendix
Table 19: Refusal with a human and with an AI-agent requester by request type and scale of the target, on the requests of the AI-agent dataset and their human originals (the health domain excluded, so the human rates differ from Figure 1 D): rates, paired difference, direction bias, and odds ratio, means over models with 95% t intervals over models.
Figure 13: Per model, log odds ratio of refusal with an AI-agent against a human requester ( ±1.96 SE) against the capability index, by request type (blue: US DC; red: CN DC); line: GLMM with an AI × capability interaction fitted within the request type; asterisk: the interaction is significant (BH over the four request types). Intervals are cut at the axis limits.
(A) Deviation of each language from the mean of the 8 languages, log-odds ( q )
Language
SE
DE
PG
CT
English
− 0.18 (0.22)
− 0.02 (0.89)
+ 0.03 (0.77)
+ 0.21 (0.17)
German
− 0.46 (0.0082)
− 0.29 (0.10)
− 0.24 (0.18)
− 0.08 (0.67)
French
+ 0.22 (0.21)
+ 0.07 (0.89)
+ 0.28 (0.15)
+ 0.10 (0.67)
Spanish
− 0.18 (0.22)
− 0.09 (0.89)
− 0.05 (0.74)
− 0.02 (0.83)
Portuguese
− 0.31 (0.088)
− 0.06 (0.89)
− 0.08 (0.71)
− 0.07 (0.67)
Appendix
Table 20: Estimates behind Figure 4 , 22 models. (A) Deviation of each language from the mean of the eight within each request type, sum-to-zero GLMM; q : BH over the eight languages of the request type; omnibus test per request type. (C) Mean over models of the two within-model concordances, t test against zero. (D) Range over chance, the odds ratio between the most and least refused language over its chance value, with its bootstrap interval over prompts (descriptive); q : permutation test (10,000 within-prompt shuffles of the languages), BH over the four request types within each weighting. (F) Mean agreement by pair type against languages permuted within each model, BH over the three types, and the same-DC contrast against DC labels permuted across models.
Range across languages (pp)
Refusal (%)
Model
DC
Chance
Observed
Excess
Least refused
Most refused
q
ling-3.0-flash
CN
3.5
20.0
+ 16.5
French 6.1
Swahili 26.0
< 0.001
deepseek-v4-pro
CN
3.3
15.8
+ 12.5
English 11.8
French 27.6
< 0.001
inkling
US
3.3
15.5
+ 12.1
English 13.7
Spanish 29.2
< 0.001
hy3
CN
2.7
14.8
+ 12.1
German 5.0
English 19.8
< 0.001
grok-4.3
US
3.7
15.8
+ 12.1
Chinese 19.3
English 35.1
< 0.001
Appendix
Table 21: Panel E of Figure 4 : per model, the range of power-shifting refusal across the eight languages, its chance value under languages shuffled within prompts, the excess, the least and the most refused language with their refusal rates, and the q of the per-model permutation test (BH over the 22 models).
24 models, Swahili of two excluded
22 models, Figure 4
Test
Est. [95% CI]
q
Est. [95% CI]
q
(A) Deviation from the mean of the 8 languages, log-odds
Swahili, SE
+ 0.51 [ + 0.18; + 0.85]
0.021
+ 0.52 [ + 0.26; + 0.78]
< 0.001
German, SE
− 0.41 [ − 0.76; − 0.06]
0.061
− 0.46 [ − 0.76; − 0.17]
0.0082
Hindi, DE
+ 0.42 [ + 0.14; + 0.69]
0.024
+ 0.32 [ + 0.07; + 0.56]
0.088
(A) Omnibus test of the language effect, χ2(7) ( p )
Appendix
Table 22: Tests of Figure 4 with the two models whose Swahili outputs are unusable kept in the other seven languages (24 models) and removed from every language (22 models, as in the figure): the panel A tests whose significance changes, the omnibus tests, and panels C to F.
Figure 14: (A) Deviation of each language’s refusal log-odds from the mean of the eight on pooled power shifting, sum-to-zero GLMM, 95% Wald interval; no deviation is significant (BH over the eight languages). (B) Odds ratio of refusal in each language against English for a usage-weighted typical request, by request type and pooled, 95% bootstrap interval over prompts; asterisk: q<0.05 from the same bootstrap, BH over the seven languages of each request type. Swahili: 22 models.
Figure 15: (A) Direction bias between each pair of languages over pooled power-shifting prompts, mean over models; positive (green): the row language is refused more. (B) The same bias, one point per pair, against the difference in log10 share of Common Crawl text (row minus column), with the least-squares line over the 28 pairs (descriptive, since pairs share languages). Swahili: 22 models.
Figure 16: (A) Per model and request type, excess of the range of refusal across languages over its chance value (percentage points; color scale compressed toward high values, negative excess in white); asterisk: per-model permutation test, BH over the 24 models of the request type. (B) Mean Spearman correlation between the language rankings of two models, by pair type and request type, with the 95% band of the null in which languages are permuted within each model; asterisk: BH over the three pair types of the request type. 24 models; Swahili: 22.
Figure 17: Per model, 22 models. (A) Kendall’s W between the language rankings of the three power-shifting request types. (B) Spearman correlation between the language ranking of the control and the consensus of the three power-shifting request types. Dots: observed; gray bar: 95% interval under languages permuted within each model; tick: its mean.
Equal weights
Usage-weighted
Usage-weighted, without gpt-5.6-luna
Contrast
Estimate
Estimate [95% CI]
q
Estimate [95% CI]
q
Nationality: US-side against China-side target, geopolitical set (odds ratio)
SE
1.13
1.08 [0.93; 1.25]
0.43
1.01 [0.88; 1.15]
0.92
DE
0.91
0.84 [0.76; 0.93]
0.003
0.90 [0.84; 0.97]
0.003
PG
0.94
0.90 [0.83; 0.97]
0.007
0.91 [0.86; 0.96]
0.003
Control
1.03
0.99 [0.89; 1.08]
0.76
1.02 [0.97; 1.07]
0.66
Appendix
Table 23: Usage-weighted results with every model weighted equally, weighted by usage, and weighted by usage without gpt-5.6-luna. Nationality: odds ratio of refusal with a US-side against a China-side target, geopolitical set, for a typical request. AI agent: odds ratio of refusal with an AI-agent against a human requester. Language range: odds ratio between the most and least refused language over its chance value (22 models; q from the permutation test). Single languages: odds ratio against English (24 models, without the Swahili responses of the two models excluded from Figure 4 ). Equal weights: point estimate of the same estimator, for reference; the tests with equal weights are those of the main figures, which treat models as a sample. Intervals: bootstrap over prompts; q : BH within the same families as in the paper.
Figure 18: Each model’s bias against its capability index, on pooled power-shifting requests (left) and on the control (right). AI agent: log odds ratio of refusal, agent against human requester (24 models); line: the GLMM of Figure 3 F. Nationality: unsigned direction of disagreement between the US and the China side minus its chance value, geopolitical set (24 models). Language: range of refusal across the eight languages minus its chance value, in percentage points (22 models). For nationality and language, the line is a least-squares fit across models. Asterisk: slope with q<0.05 .
Figure 19: Refusal by request type with reasoning off and at the first and second effort levels, for the four US models, the four CN models, and all eight; bands: 95% t interval across the models of each group (four per DC, eight for all), cut at zero.
It has generally been assumed that geopolitical bias in language models originates from the training data used during the pre-training phase. We tested seven open-weight LLM pairs consisting of the base model (pre-training only) and the chat model (pre-training and post-training) from seven labs on a paired-scenario forced-choice probe over 28 country pairs in English, French, and Chinese, and found that geopolitical bias originates in post-training rather than in pre-training. Across seven AI labs, six showed shifts in the direction associated with the country or region of the model developer after post-training. This shift is strongest in Alibaba's Qwen 2.5: while the base is neutral on China-favourability (-0.15 log-odds, p=0.15), the post-trained chat variant is at +2.91 (p<10^-4), an 18x shift in odds. We also observe shifts in biases toward other countries across all models. Additionally, the magnitude of this shift depends on the language used to prompt the model: the French-made Mistral becomes pro-France only under French prompting (FR-EN shift +1.91, p<10^-4). These findings suggest that geopolitical preferences in language models are not simply inherited from large-scale internet data but are actively shaped during post-training, highlighting the need for greater transparency, auditing, and oversight of alignment processes that influence how models represent nations, cultures, and political perspectives.
Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on their merits rather than deferring to the user. Yet the same models are far more compliant when a wrong answer is attributed to a verified source, which is how retrieval results, tool outputs, and grounded-search content often present information. We measure this gap across five open-weight families and three closed APIs. A single verified-source note endorsing a wrong answer flips 45-88% of baseline-correct responses in seven of eight models, and compliance rises with how authoritative the note sounds. Source deference and user agreement are not behaviorally interchangeable inside the model: on matched items with the same wrong answer, causal interventions can selectively suppress one without equally affecting the other. In three open-weight families, removing a fitted source direction lowers source compliance by 65-80 percentage points while removing a user or assistant direction has far smaller effects, and removing the user direction shows the reverse preference. A separately fitted intervention derived from source-versus-user cue activations moves compliance in both directions while leaving the prompt text unchanged. An authority direction fitted on trivia also transfers to PIQA and multi-turn SYCON dialogues without refitting, and removing it lowers wrong-source compliance by tens of percentage points in four of five families with no detected change in MMLU-Pro or GSM8K accuracy at our evaluation sizes. Source deference and user agreement therefore need separate evaluation.
Power differences shape human communication through well documented socio cognitive effects, including language coordination, pronoun usage, authority bias, and harmful compliance. We examine whether large language models (LLMs) exhibit similar behaviors when assigned high or low status personas. Using personas from diverse professions, we simulate multi turn, power asymmetric dialogues (e.g., principal teacher, justice lawyer) and measure (i) language coordination, (ii) pronoun usage, (iii) persuasion success, and (iv) compliance with unsafe requests. Our results show that LLMs show key socio-cognitive effects of power, albeit with nuances and variability, linking simulated interactions to both desirable and unsafe behaviors.