Is Abliteration Getting Harder? Not Uniformly: Refusal Removal by Direction-Based Ablation and ARA in Heretic on the Qwen3.8, Gemma 4, Olmo 3.1 and Mistral Small 3.2 LLMs Versus Their Predecessors
Organizations: University of Nevada Las Vegas Department of Information Systems Las Vegas, NV, USA
Abstract
Abliteration, a fine-tuning-free weight edit, removes refusal from open-weight large language models (LLMs). The developer of Arbitrary-Rank Ablation (ARA), an optimisation-based variant, reports that direction-based abliteration, which ablates a refusal direction, increasingly fails on newer models. On five release pairs, refusal removal did not become uniformly harder. Qwen2.5-32B and Qwen3.8-27B, Gemma-2-27B and Gemma-4-31B, OLMo-2-32B and Olmo-3.1-32B, Mistral-Small-2409 and Mistral-Small-3.2, and Qwen2.5-7B and Qwen3.5-4B were edited in Heretic 2.0 by three or four direction-based methods (difference-in-means, MPOA, SOM and, on the small pair, J-space) and ARA under the same prompts, nominal search budget, first-token KL divergence cap and selection rule; removal counted as harder when the automatic WildGuard judge found more of 972 held-out harmful prompts (JailbreakBench, XSTest, StrongREJECT, HarmBench and an author-compiled malicious-code bank) still refused. After direction-based removal, newer releases retained -1 to +37 percentage points more refusal, and after ARA -26 to +34. With run-to-run variance included, between-pair heterogeneity of the direction-based effect ranged from high to low with the scale and variance model (I^2 20-80%). ARA lost less effectiveness than direction-based methods in Gemma and OLMo and more in Qwen; each verdict rests on one ARA run, and some changed with the divergence budget or variance model. In Gemma the direction-based increase came from the malicious-code bank and reversed on public sets. Contrary to prediction, keyword counts at the tool's window did not undercount judged refusal on average, but differed by a mean 7 points per model. Audits of a single pair, method or prompt source can reach opposite conclusions. Whether a newer model resisted removal depended on the release pair, method and budget, not on generation alone.
Figures & tables
| Method | Assumption about refusal | Edit | Search | Export | Implementation |
|---|---|---|---|---|---|
| Diff-in-means [ 1 ] | 1 direction | rank-1 projection | TPE | yes | Heretic 2.0 (no proj./norm) |
| Projected / MPOA [ 35 , 6 ] | 1 direction, harmless | rank-1 + norm a | TPE | yes | Heretic 2.0 |
| SVD multi-dir. [ 42 ] | -dim subspace | rank- projection | – | yes | OBLITERATUS b |
| SOM [ 7 ] | cluster directions | rank- + norm a | TPE | yes | Heretic 2.0 fork c |
| ARA [ 9 ] | none (per-module map) | rank-50 LoRA | TPE + L-BFGS | yes | Heretic 2.0 |
| J-space | 1 direction in J-space | rank-1 + norm a | TPE | yes | Heretic 2.0 fork d |
| Family | Gen. | Released | Repository | Revision | Size (GB) | Layers | Notes |
| Main study | |||||||
| Qwen | earlier | 2024-09 | Qwen/Qwen2.5-32B-Instruct | 5ede1c97 | 65.5 (32B) | 64 | – |
| Qwen | later | 2026-08 | Qwen/Qwen3.8-27B | 1d4bf0f2 | 55.6 (27B) | 64 | hybrid linear/full attention; image-text |
| Gemma | earlier | 2024-06 | google/gemma-2-27b-it | aaf20e6b | 54.5 (27B) | 46 | post-norm [ 72 ] ; no system role |
| Gemma | later | 2026-04 c | google/gemma-4-31B-it | 842da379 | 62.6 (31B) | 60 | post-norm; image-text |
| OLMo | earlier | 2025-03 | allenai/OLMo-2-0325-32B-Instruct | b9602434 | 64.5 (32B) | 64 | post-norm [ 73 ] |
| Subset | Kind | Overlap | |
|---|---|---|---|
| JailbreakBench harmful | harmful | 100 | 11 / 6 |
| JailbreakBench benign | benign | 100 | – |
| XSTest unsafe | harmful | 200 | 0 / 0 |
| XSTest safe | benign | 250 | – |
| StrongREJECT | harmful | 313 | 25 / 1 |
| HarmBench (standard) | harmful | 159 | 0 / 1 |
| Benchmark | Task | Shots | Metric | Max. tok. | |
|---|---|---|---|---|---|
| MMLU-Pro | mmlu_pro_* | 1,000 | 5 (CoT) | exact match | 2,048 |
| GSM8K | gsm8k | 500 | 5 | exact match (flexible) | 512 |
| IFEval | ifeval | 541 | 0 | prompt-level strict acc. | 1,280 |
| HumanEval | humaneval_instruct | 164 | 0 | pass@1 | 1,024 |
| MBPP | mbpp_instruct | 500 | 3 | pass@1 | 1,024 |
| Estimand, scale | Qwen | Gemma | OLMo | Mistral Small | Small pair | Pooled, 4 main pairs |
| Direction, raw, pp | +10.7 [ 4.0, +25.5] | +4.7 [ 10.2, +19.6] | +36.9 [+22.1, +51.7] | 1.2 [ 16.0, +13.5] | +17.3 [+4.4, +30.1] | +12.8 [ 13.9, +39.5] |
| log-odds | +0.46 [ 0.86, +1.79] | +0.21 [ 1.13, +1.54] | +1.63 [+0.30, +2.96] | 0.10 [ 1.43, +1.23] | +0.79 [ 0.36, +1.94] | +0.55 [ 0.66, +1.76] |
| Direction, adjusted, pp | 2.4 [ 17.3, +12.5] | +3.7 [ 11.2, +18.6] | +36.0 [+21.2, +50.9] | 19.2 [ 34.1, 4.2] | +0.9 [ 12.1, +14.0] | +4.6 [ 32.2, +41.3] |
| log-odds | 0.53 [ 1.86, +0.81] | +0.10 [ 1.24, +1.44] | +1.57 [+0.24, +2.91] | 0.90 [ 2.23, +0.44] | 0.30 [ 1.46, +0.86] | +0.06 [ 1.67, +1.80] |
| ARA, raw, pp | +34.4 [+8.8, +59.9] | 26.0 [ 51.6, 0.5] | 2.5 [ 27.9, +23.0] | 0.7 [ 26.1, +24.7] | +14.2 [ 11.3, +39.7] | +1.3 [ 38.3, +40.8] |
| log-odds | +2.35 [+0.04, +4.66] | 4.07 [ 6.50, 1.64] | 0.23 [ 2.54, +2.07] | 0.17 [ 2.48, +2.15] | +1.55 [ 0.77, +3.86] | 0.49 [ 4.66, +3.69] |
| Condition | Older, % | Newer, % | New old, pp | / | Baseline-adj., pp | Seed | |
| [95% CI] | [95% CI] | [95% CI] | [95% CI] | noise s | |||
| Qwen: Qwen2.5-32B Qwen3.8-27B | |||||||
| Original | 76.8 [74.1, 79.4] | 89.9 [87.9, 91.7] | +13.1 [+10.9, +15.4] | 6/133 | 0.001 u | – | – |
| DiM | 47.4 [44.3, 50.6] | 74.3 [71.4, 76.9] | +26.9 [+23.8, +29.8] | 17/278 | 0.001 | +13.7 [+10.1, +17.1] | exceeds |
| MPOA | 3.9 [2.9, 5.3] | 14.0 [12.0, 16.3] | +10.1 [+7.8, +12.5] | 23/121 | 0.001 | 3.1 [ 6.3, +0.2] | within |
| SOM | 42.7 [39.6, 45.8] | 38.0 [35.0, 41.1] | 4.7 [ 8.0, 1.4] | 157/111 | 0.041 a | 17.8 [ 21.5, 14.2] | within |
| Originals | Direction | ARA | H2 contrast | |||||
|---|---|---|---|---|---|---|---|---|
| Pair | Public | Code | Public | Code | Public | Code | Public | Code |
| Qwen | +5.3 [+3.5, +7.1] | +43.2 [+36.5, +50.5] | +11.0 [+8.8, +13.3] | +9.4 [+6.5, +12.4] | +41.3 [+37.6, +45.1] | +7.5 [+4.0, +11.5] | +30.3 [+25.9, +34.6] | 1.8 [ 5.5, +2.0] |
| Gemma | 5.7 [ 7.5, 4.0] | +27.0 [+18.0, +36.0] | 6.8 [ 9.5, 4.4] | +49.3 [+43.7, +55.0] | 32.4 [ 35.8, 29.0] | 1.5 [ 3.5, 0.0] | 25.6 [ 29.3, 21.9] | 50.8 [ 56.3, 45.2] |
| OLMo | 0.3 [ 1.6, +1.0] | +5.0 [+0.5, +9.5] | +45.2 [+42.8, +47.5] | +4.8 [+2.7, +7.2] | 3.1 [ 5.7, 0.4] | 0.0 [0.0, 0.0] | 48.3 [ 51.6, 44.9] | 4.8 [ 7.2, 2.7] |
| Mistral Small | +20.5 [+17.2, +23.7] | +8.0 [+4.5, +12.1] | 1.6 [ 3.5, +0.2] | +0.2 [0.0, +0.5] | 0.9 [ 2.5, +0.6] | 0.0 [0.0, 0.0] | +0.7 [ 1.6, +3.0] | 0.2 [ 0.5, 0.0] |
| Small pair | +9.5 [+7.1, +11.9] | +43.0 [+36.0, +50.0] | +20.2 [+17.9, +22.7] | +5.9 [+4.1, +7.6] | +17.7 [+14.6, +21.0] | +0.5 [0.0, +1.5] | 2.5 [ 6.1, +1.2] | 5.4 [ 7.2, 3.5] |
| Pair | ARA, | Direction | RD contrast, pp | GEE, log-odds | Sign | |
| pp | mean, pp | [95% CI] | [95% CI] | vs. H2 | ||
| Qwen | +34.4 | +10.7 | +23.6 [+19.9, +27.1] | +1.89 [+1.61, +2.16] | 0.001 | opposite |
| Gemma | 26.0 | +4.7 | 30.8 [ 34.0, 27.5] | 4.27 [ 5.07, 3.47] | 0.001 | predicted |
| OLMo | 2.5 | +36.9 | 39.3 [ 42.2, 36.4] | 1.87 [ 2.08, 1.65] | 0.001 | predicted |
| Mistral Small | 0.7 | 1.2 | +0.5 [ 1.4, +2.4] | 0.06 [ 0.37, +0.25] | 0.696 | n.s. |
| Small pair | +14.2 | +17.3 | 3.1 [ 6.0, 0.1] | +0.76 [+0.46, +1.06] | 0.001 | opposite c |
| Kw, % | Gap, pp [95% CI] | ||||||
| Models | Cells | 512 | 100 | WG, % | 512 tokens | 256 tokens | 100 tokens |
| Originals | 10 | 82.8 | 80.9 | 80.2 | +2.6 [+1.8, +3.4] | +1.6 [+0.8, +2.4] | +0.7 [ 0.1, +1.5] |
| Ablated (seed 1), all | 42 | 39.9 | 32.0 | 31.5 | +8.4 [+6.9, +10.0] | +3.8 [+2.4, +5.2] | +0.5 [ 1.0, +1.9] |
| main study | 32 | 42.0 | 33.8 | 32.3 | +9.8 [+8.2, +11.3] | +4.9 [+3.5, +6.4] | +1.5 [+0.0, +2.9] |
| small pair | 10 | 33.3 | 26.5 | 29.1 | +4.2 [+2.5, +5.9] | +0.4 [ 1.4, +2.0] | 2.6 [ 4.3, 0.9] |
| responses ending before the cap | 42 | 51.2 | 42.1 | 52.4 | 1.2 [ 3.2, +0.9] | 5.6 [ 7.6, 3.8] | 10.4 [ 12.2, 8.6] |
| Models | Cells | Responses | Kw, % | WG, % | Gap, pp [95% CI] | kw/WG only | [95% CI] |
| Originals | 10 | 9,718 | 82.8 | 80.2 | +2.6 [+1.8, +3.4] | 531/277 | 0.73 [0.70, 0.75] |
| Ablated (seed 1), all | 42 | 40,824 | 39.9 | 31.5 | +8.4 [+6.9, +10.0] | 7,503/4,059 | 0.39 [0.37, 0.41] |
| main study | 32 | 31,104 | 42.0 | 32.3 | +9.8 [+8.2, +11.3] | 5,897/2,859 | 0.40 [0.39, 0.42] |
| small pair | 10 | 9,720 | 33.3 | 29.1 | +4.2 [+2.5, +5.9] | 1,606/1,200 | 0.33 [0.31, 0.35] |
| responses ending before the cap | 42 | 20,011 | 51.2 | 52.4 | 1.2 [ 3.2, +0.9] | 2,998/3,240 | – |
| responses truncated at 512 tokens | 42 | 20,813 | 29.1 | 11.4 | +17.7 [+16.2, +19.3] | 4,505/819 | – |
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Range | Methods |
| Direction scope | {global, per layer} | DiM, MPOA, J-space, SOM |
| Direction index (global scope) | , continuous, interpolated | DiM, MPOA, J-space |
| integers in | SOM | |
| Direction decay | , | SOM |
| Max. weight, attention | DiM, MPOA, J-space, SOM | |
| Max. weight, MLP | , | DiM, MPOA, J-space, SOM |
| Machine | GPU (memory) | CPU / RAM | Driver | Role |
|---|---|---|---|---|
| H200 box | 1 NVIDIA H200 NVL (143,771 MiB) | 16 vCPU, AMD EPYC 9355 / 176 GiB | 580.126.09 | main-study search; export and evaluation of the OLMo and Mistral Small pairs (incl. OLMo -MPOA) and 3 community models; ARA/MPOA/DiM pilots |
| RTX box | 1 RTX PRO 6000 Blackwell Server Ed. (97,887 MiB) | 16 vCPU, AMD EPYC 9335 / 141 GiB | 580.126.09 | export and evaluation of the Qwen and Gemma pairs (incl. Gemma -MPOA) and 5 community models; all judging; release |
| Workstation | 1 GeForce RTX 4090 (24,564 MiB) | AMD Ryzen 9 7950X / 62 GiB | 615.78.08 | small pair: all steps; SOM and J-space pilots |
| Qwen | Gemma | OLMo | Mistral Small | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | 2.5-32B | 3.8-27B | 2-27B | 4-31B | 2-32B | 3.1-32B | 2409 | 3.2-24B | |
| Original | 98 | 98 | 100 | 99 | 98 | 100 | 93 | 98 | |
| DiM | 0.05 | 89 | 53 | 92 | 99 | 47 | 92 | 85 | 46 |
| 0.1 | 89 | 47 | 89 | 98 | 32 | 88 | 69 | 35 | |
| 0.2 | 56 | 37 | 73 | 98 | 32 | 67 | 31 | 11 | |
| MPOA | 0.05 | 75 | 43 † | 89 | 81 | 36 | 89 | 10 | 60 |
| Qwen (small pair) | |||
|---|---|---|---|
| Method | 2.5-7B | 3.5-4B | |
| Original | 99 | 99 | |
| DiM | 0.05 | 63 | 34 |
| 0.1 | 55 | 30 | |
| 0.2 | 11 | 30 | |
| MPOA | 0.05 | 33 | 10 |
| Model | Method | MMLU-Pro (1000) | GSM8K (500) | IFEval (541) | HumanEval (164) | MBPP (500) |
|---|---|---|---|---|---|---|
| Qwen2.5-32B-Instruct | Original § | 71.1 | 85.4 | 78.9 | 89.6 | 77.6 ‡ |
| DiM | 69.9 ( 1.2) | 85.6 (+0.2) | 79.7 (+0.7) | 90.2 (+0.6) | 77.0 ( 0.6) | |
| MPOA | 70.5 ( 0.6) | 85.0 ( 0.4) | 80.8 (+1.8) | 89.6 (0.0) | 76.6 ( 1.0) | |
| ARA | 70.3 ( 0.8) | 86.6 (+1.2) | 79.9 (+0.9) | 84.8 ( 4.9) | 77.0 ( 0.6) | |
| SOM | 71.1 (0.0) | 85.6 (+0.2) | 79.5 (+0.6) | 89.6 (0.0) | 78.4 (+0.8) | |
| Qwen3.8-27B | Original | 79.6 | 96.2 | 80.6 | 97.6 | 80.8 |
| Model | Method | MMLU-Pro (1000) | GSM8K (500) | IFEval (541) | HumanEval (164) | MBPP (500) |
|---|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | Original | 59.5 | 85.8 | 71.2 | 84.8 | 63.2 |
| DiM | 58.1 ( 1.4) | 86.0 (+0.2) | 71.7 (+0.6) | 86.0 (+1.2) | 63.8 (+0.6) | |
| MPOA | 57.9 ( 1.6) | 85.0 ( 0.8) | 70.6 ( 0.6) | 86.6 (+1.8) | 63.2 (0.0) | |
| ARA | 57.8 ( 1.7) | 86.8 (+1.0) | 70.2 ( 0.9) | 87.2 (+2.4) | 64.8 (+1.6) | |
| SOM | 59.8 (+0.3) | 84.0 ( 1.8) | 70.6 ( 0.6) | 83.5 ( 1.2) | 64.8 (+1.6) | |
| J-space | 60.5 (+1.0) | 86.4 (+0.6) | 72.6 (+1.5) | 84.8 (0.0) | 64.4 (+1.2) |
| Shift vs. original | ||||||
|---|---|---|---|---|---|---|
| Model | Tokens | Original | DiM | MPOA | ARA | SOM |
| Qwen2.5-32B-Instruct | 112398 | 0.272 § | 0.017 | 0.042 | 0.077 | 0.020 |
| Qwen3.8-27B | 138016 | 0.275 | 0.006 | 0.060 | 0.042 | 0.021 |
| gemma-2-27b-it | 97666 | 0.227 § | 0.068 | 0.063 | 0.161 | 0.031 |
| gemma-4-31B-it | 142322 | 0.110 | 0.042 | 0.007 | 0.279 | 0.026 |
| OLMo-2-0325-32B-Instruct | 110792 | 0.520 | 0.051 | 0.088 | 0.055 | 0.030 |
| Shift vs. original | |||||||
|---|---|---|---|---|---|---|---|
| Model | Tokens | Original | DiM | MPOA | ARA | SOM | J-space |
| Qwen2.5-7B-Instruct | 119254 | 0.321 | 0.019 | 0.018 | 0.048 | 0.026 | 0.009 |
| Qwen3.5-4B | 139372 | 0.412 | 0.010 | 0.031 | 0.093 | 0.035 | 0.045 |
| Keyword minus judge | ||||||
| Models | Cells | 256, % | WG change | WG, full text | WG, 256 tokens | Q3G, 256 tokens |
| Originals | 10 | 40.7 | +0.3 [+0.1, +0.5] | +2.6 [+1.8, +3.4] | +1.6 [+0.8, +2.4] | +2.7 [+1.9, +3.6] |
| Ablated (seed 1), all | 42 | 78.9 | +0.6 [+0.5, +0.7] | +8.4 [+6.9, +10.0] | +3.8 [+2.4, +5.2] | +0.9 [ 0.6, +2.4] |
| main study | 32 | 79.1 | +0.6 [+0.5, +0.7] | +9.8 [+8.2, +11.3] | +4.9 [+3.5, +6.4] | +2.1 [+0.6, +3.5] |
| small pair | 10 | 78.1 | +0.7 [+0.5, +0.9] | +4.2 [+2.5, +5.9] | +0.4 [ 1.4, +2.0] | 2.7 [ 4.5, 1.0] |
| ending before the cap | 42 | 56.9 | +0.0 [ 0.1, +0.1] | 1.2 [ 3.2, +0.9] | 5.6 [ 7.6, 3.8] | 7.6 [ 9.8, 5.5] |
| Model | DiM | MPOA | SOM | J-space | ARA | -MPOA |
|---|---|---|---|---|---|---|
| (a) Dev keyword refusals (of 100): selected trial released export | ||||||
| Qwen2.5-32B | 89 90 | 13 17 | 73 75 | – | 10 16 | – |
| Qwen3.8-27B | 47 44 | 25 26 | 27 28 | – | 63 72 | – |
| Gemma-2-27B a | 89 90 | 74 74 | 83 86 | – | 16 21 | 73 72 |
| Gemma-4-31B | 98 98 | 80 80 | 82 83 | – | 28 21 | 78 78 |
| OLMo-2-32B | 32 34 | 36 38 | 63 62 | – | 34 34 | 7 8 |
| Selected trial | Re-score, SDPA | Re-score, eager | Revision | |||||
|---|---|---|---|---|---|---|---|---|
| Method | Ref. | KL | Ref. | KL | Ref. | KL | Evaluated | Patched |
| DiM | 89 | 0.0686 | 89 | 0.4036 | 90 | 0.0672 | 9ee29ab3ab16 | 3f2f3e1d2c36 |
| MPOA | 74 | 0.0991 | 52 | 0.4609 | 74 | 0.1036 | a2c021b2e06a | 8aa25e304fdd |
| SOM | 83 | 0.0860 | 77 | 0.4912 | 86 | 0.0807 | 2a355f20c0d4 | c63892ab9b07 |
| ARA | 16 | 0.0990 | 23 | 0.4817 | 21 | 0.0996 | 29250dd15c03 | 958fda591386 |
| -MPOA | 73 | 0.0969 | 63 | 0.4167 | 72 | 0.1059 | 78da037a4a28 | 9630b864b0fe |
| Subset | Units | Dir., raw | Dir., adj. | ARA, raw | H2 |
|---|---|---|---|---|---|
| All units | 12/4 | +0.62 [+0.57, +0.68] | +0.03 [ 0.05, +0.11] | +0.12 [+0.00, +0.24] | 0.50 [ 0.62, 0.38] |
| Cap bound in both cells | 5/4 | +0.32 [+0.24, +0.41] | 0.24 [ 0.33, 0.14] | +0.11 [+0.01, +0.22] | 0.21 [ 0.33, 0.09] |
| Not bound in 1 cell | 7/0 | +1.03 [+0.95, +1.11] | +0.22 [+0.12, +0.32] | n/a | n/a |
| Bound in neither cell | 2/0 | +3.23 [+2.90, +3.56] | +2.20 [+1.95, +2.45] | n/a | n/a |
| 2 trials above the cap in both | 6/4 | +0.33 [+0.24, +0.41] | 0.28 [ 0.37, 0.18] | +0.12 [+0.01, +0.23] | 0.21 [ 0.33, 0.08] |
| 1 trial above the cap in a cell | 6/0 | +1.04 [+0.96, +1.12] | +0.27 [+0.17, +0.37] | n/a | n/a |
| Term | All four | w/o Qwen | w/o Gemma | w/o OLMo | w/o Mistral Small |
|---|---|---|---|---|---|
| Generation effect, direction class | +0.62 | +0.69 | +0.81 | +0.24 | +0.74 |
| Generation effect, ARA | +0.12 | 1.37 | +1.02 | +0.24 | +0.15 |
| Generation class (H2) | 0.50 | 2.06 | +0.20 | 0.00 | 0.59 |
| Baseline-adjusted, direction class | +0.03 | +0.22 | +0.13 | 0.51 | +0.33 |
| Baseline-adjusted, ARA | 0.46 | 1.68 | +0.31 | 0.51 | 0.25 |
| Originals’ generation difference | +0.58 | +0.45 | +0.66 | +0.74 | +0.40 |
| Pair | Method | Effect, pp | Orig., pp | Adjusted, pp | Effect, log-odds | Orig., log-odds | Adjusted, log-odds | Sign | Sig. |
| Qwen | DiM | +26.9 [+23.8, +29.8] | +13.1 | +13.7 [+10.1, +17.1] | +1.16 [+1.02, +1.30] | +0.99 | +0.17 [ 0.03, +0.37] | ||
| Qwen | MPOA | +10.1 [+7.8, +12.5] | +13.1 | 3.1 [ 6.3, +0.2] | +1.39 [+1.04, +1.73] | +0.99 | +0.39 [+0.01, +0.77] | ||
| Qwen | SOM | 4.7 [ 8.0, 1.4] | +13.1 | 17.8 [ 21.5, 14.2] | 0.20 [ 0.33, 0.06] | +0.99 | 1.19 [ 1.40, 0.97] | ||
| Qwen | ARA | +34.4 [+31.2, +37.5] | +13.1 | +21.3 [+17.1, +25.4] | +2.35 [+2.08, +2.62] | +0.99 | +1.36 [+1.05, +1.68] | ||
| Gemma | DiM | +16.4 [+13.5, +19.3] | +1.0 | +15.3 [+13.0, +17.8] | +1.18 [+0.96, +1.40] | +0.10 | +1.07 [+0.90, +1.25] | ||
| Gemma | MPOA | 4.7 [ 8.6, 0.9] | +1.0 | 5.8 [ 9.7, 1.7] | 0.19 [ 0.35, 0.04] | +0.10 | 0.30 [ 0.55, 0.04] |
| (a) Originals’ generation difference by item source, pp | |||||
|---|---|---|---|---|---|
| Pair | All 972 | Public sets (772) | Code-as-weapon (200) | Code contribution | Code share |
| Qwen | +13.1 [+10.8, +15.4] | +5.3 [+3.5, +7.2] | +43.2 [+36.1, +50.2] | +8.9 [+7.1, +10.7] | 68% [59, 77] |
| Gemma | +1.0 [ 1.3, +3.5] | 5.7 [ 7.5, 4.0] | +27.0 [+17.9, +35.8] | +5.6 [+3.6, +7.6] | – |
| OLMo | +0.8 [ 0.5, +2.2] | 0.3 [ 1.5, +1.0] | +5.0 [+0.5, +9.8] | +1.0 [+0.1, +2.1] | – |
| Mistral Small | +17.9 [+15.2, +20.7] | +20.5 [+17.2, +23.8] | +8.0 [+4.6, +12.1] | +1.6 [+0.9, +2.5] | 9% [5, 14] |
| Small pair | +16.4 [+13.9, +18.8] | +9.5 [+7.0, +11.8] | +43.0 [+36.3, +50.0] | +8.8 [+7.2, +10.6] | 54% [46, 63] |
| (a) Pooled main-study terms by outcome, log-odds | |||
|---|---|---|---|
| Term | WG refusal (primary) | HB not harmful | WG not harmful |
| Originals’ difference | +0.58 [+0.49, +0.66] | +0.65 [+0.58, +0.72] | +0.72 [+0.62, +0.83] |
| Direction, raw | +0.62 [+0.57, +0.68] | +0.70 [+0.65, +0.74] | +0.69 [+0.64, +0.73] |
| Direction, adjusted | +0.03 [ 0.05, +0.11] | +0.05 [ 0.02, +0.11] | 0.03 [ 0.13, +0.07] |
| ARA, raw ( ) | +0.12 [+0.00, +0.24] (0.135) | +0.21 [+0.14, +0.28] ( 0.001) | +0.24 [+0.18, +0.30] ( 0.001) |
| ARA, adjusted | 0.46 [ 0.58, 0.34] | 0.44 [ 0.53, 0.36] | 0.48 [ 0.60, 0.37] |
| Pair | Method | WG refusal: effect | HB harmful: effect | WG harmful: effect |
|---|---|---|---|---|
| Qwen | Original | 76.8 89.9: +13.1 | 19.8 6.3: +13.5 | 7.1 1.4: +5.7 |
| DiM | 47.4 74.3: +26.9* | 40.3 16.4: +24.0* | 26.0 6.0: +20.1* | |
| MPOA | 3.9 14.0: +10.1* | 85.3 48.6: +36.7* | 72.9 34.2: +38.8* | |
| SOM | 42.7 38.0: 4.7* | 44.5 39.9: +4.6* | 30.1 27.8: +2.3 | |
| ARA | 6.1 40.4: +34.4* | 82.3 30.6: +51.7* | 69.1 17.2: +52.0* | |
| Gemma | Original | 88.3 89.3: +1.0 | 10.7 6.9: +3.8 | 2.1 1.8: +0.3 |
| Pair | Estimand | Effect | Constant | Refusal-level | Excl. 0, const. | Excl. 0, level |
|---|---|---|---|---|---|---|
| [95% interval] | [95% interval] | lo/pt/hi | lo/pt/hi | |||
| Qwen | Direction, raw | +10.7 | [ 4.0, +25.5] | [ 15.3, +36.7] | ✓/–/– | –/–/– |
| Direction, adjusted | 2.4 | [ 17.3, +12.5] | [ 28.5, +23.7] | –/–/– | –/–/– | |
| ARA, raw | +34.4 | [+8.8, +59.9] | [ 5.9, +74.6] | ✓/✓/– | ✓/–/– | |
| ARA, adjusted | +21.3 | [ 4.4, +47.0] | [ 19.0, +61.7] | ✓/–/– | –/–/– | |
| H2 contrast | +23.6 | [ 5.9, +53.1] | [ 24.3, +71.6] | ✓/–/– | –/–/– |
| pp scale | log-odds scale | ||||
|---|---|---|---|---|---|
| Estimand | Prompt only | + seed, const. | + seed, level | Prompt only | + seed |
| Direction, raw | 821.8; 0.001; 100%; 284 | 14.8; 0.002; 80%; 225 | 5.5; 0.136; 46%; 127 | 590.8; 0.001; 99%; 0.56 | 3.8; 0.289; 20%; 0.12 |
| Direction, adjusted | 849.4; 0.001; 100%; 527 | 27.7; 0.001; 89%; 476 | 11.5; 0.010; 74%; 425 | 608.6; 0.001; 100%; 1.53 | 7.7; 0.053; 61%; 0.72 |
| ARA, raw | 807.1; 0.001; 100%; 321 | 10.9; 0.012; 73%; 447 | 5.3; 0.148; 44%; 113 | 378.5; 0.001; 99%; 2.86 | 14.3; 0.003; 79%; 5.34 |
| ARA, adjusted | 381.8; 0.001; 99%; 328 | 7.9; 0.048; 62%; 278 | 5.0; 0.172; 40%; 97 | 208.5; 0.001; 99%; 1.95 | 10.8; 0.013; 72%; 3.74 |
| H2 contrast | 1006.7; 0.001; 100%; 680 | 11.2; 0.011; 73%; 614 | 6.3; 0.097; 53%; 343 | 539.7; 0.001; 99%; 4.54 | 10.8; 0.013; 72%; 4.87 |
| pp scale (pp 2 ) | log-odds scale | |||||
|---|---|---|---|---|---|---|
| Estimand | Pair | Prompt only | + seed, const. | + seed, level | Prompt only | + seed |
| Direction, raw | Qwen | 1.0 | 56.8 | 176.0 | 0.002 | 0.459 |
| Gemma | 1.9 | 57.8 | 202.9 | 0.004 | 0.461 | |
| OLMo | 1.2 | 57.1 | 190.0 | 0.003 | 0.459 | |
| Mistral Small | 0.6 | 56.4 | 77.9 | 0.004 | 0.461 | |
| Direction, adjusted | Qwen | 1.9 | 57.8 | 177.0 | 0.009 | 0.466 |
| Pair | Estimand | 0.05 | 0.1 | 0.2 | 0.5 | 1.0 | Area | Held-out |
|---|---|---|---|---|---|---|---|---|
| Qwen | Direction | 38.3 ∗ | 25.3 ∗ | 11.3 ∗ | 3.3 | 3.3 | 7.2 ∗ | +10.7 |
| ARA | +86.0 ∗ | +53.0 ∗ | +13.0 | +14.0 ∗ | +1.0 | +37.1 ∗ | +34.4 | |
| H2 contrast | +124.3 ∗ | +78.3 ∗ | +24.3 ∗ | +17.3 ∗ | +4.3 | +44.3 ∗ | +23.6 | |
| Gemma | Direction | +0.3 | +4.7 | +22.7 ∗ | +30.0 ∗ | +30.0 ∗ | +13.0 ∗ | +4.7 |
| ARA | +23.0 ∗ | +12.0 | +4.0 | +1.0 | +1.0 | +5.1 | 26.0 | |
| H2 contrast | +22.7 ∗ | +7.3 | 18.7 ∗ | 29.0 ∗ | 29.0 ∗ | 7.9 | 30.8 |
| Older model | Newer model | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Pair | Condition | Ref. | Compl. | Incompl. | Degen. | Ref. | Compl. | Incompl. | Degen. |
| Qwen | Original | 76.7 | 5.7 | 17.4 | 0.1 | 89.9 | 4.2 | 5.8 | 0.1 |
| DiM | 47.4 | 18.8 | 33.6 | 0.1 | 74.3 | 5.2 | 20.4 | 0.1 | |
| MPOA | 3.9 | 50.6 | 45.4 | 0.1 | 14.0 | 8.7 | 77.3 | 0.0 | |
| SOM | 42.7 | 20.4 | 36.8 | 0.1 | 38.0 | 14.0 | 47.8 | 0.2 | |
| ARA | 6.1 | 61.3 | 32.5 | 0.1 | 40.3 | 4.5 | 54.9 | 0.2 | |
| Pair | GEE, log-odds [95% CI] | VB, log-odds [95% CrI] |
|---|---|---|
| Qwen | +1.89 [+1.61, +2.16] | +2.29 [+2.14, +2.44] |
| Gemma | 4.27 [ 5.07, 3.47] | 4.90 [ 5.64, 4.17] |
| OLMo | 1.87 [ 2.08, 1.65] | 3.25 [ 3.49, 3.01] |
| Mistral Small | 0.06 [ 0.37, +0.25] | 0.08 [ 0.42, +0.26] |
| Small pair | +0.76 [+0.46, +1.06] | +0.80 [+0.62, +0.99] |
| Pooled (main) | 0.50 [ 0.62, 0.38] | 0.71 [ 0.81, 0.60] |
| Pair | Estimand | Effect | Prompt [95% CI] | Prompt + seed [95% CI] |
|---|---|---|---|---|
| Qwen | Direction | +6.0 | [+3.5, +8.3] | [ 11.7, +23.7] |
| ARA | +36.9 | [+33.4, +40.4] | [+6.3, +67.5] ∗ | |
| H2 contrast | +30.9 | [+26.9, +34.9] | [ 4.5, +66.2] | |
| Gemma | Direction | +4.0 | [+1.5, +6.5] | [ 13.8, +21.7] |
| ARA | 29.4 | [ 32.5, 26.4] | [ 59.9, +1.2] | |
| H2 contrast | 33.3 | [ 36.8, 30.0] | [ 68.6, +1.9] |
Refusal Behavior in Language Models
Refusal behavior in language models concerns when systems decline requests, how they detect risk, and how to balance safety with helpfulness.
See all 164 papers