Activation steering influences a language model's behaviour by adding a steering vector to its residual stream during generation, but it requires white-box access to modify the hidden states at every generation step. We introduce ActGCG, a modification of Greedy Coordinate Gradient which optimises a suffix to match the target activations obtained by adding a steering vector to the residual stream. We further introduce universal suffixes trained across many prompts at once, a model-translation step that rewrites the suffix into human-readable text midway through optimisation, and a soft-suffix variant to remove the discrete token space constraint. We study two behaviours: steering a model to answer in German, and anti-refusal steering for jailbreaking. For German, a discrete suffix reproduces the steering vector's behaviour almost perfectly and transfers well across prompts and models. For anti-refusal, the equivalence is weaker, succeeding under a phrase-avoidance metric but only partially under the stricter HarmBench classifier. Our results show that a steering vector's behaviour is reachable from the input, with no access to model internals at generation time, up to a loss that depends on the concept being steered.
Figures & tables
Figure 1: Semantic relationships as directions in an embedding space. 2 2 2 https://commons.wikimedia.org/wiki/File:Word_vector_illustration.jpg
Figure 2: The Greedy Coordinate Gradient (GCG) algorithm, reproduced from Zou et al. [30] .
Figure 3: The PEZ gradient-based discrete optimisation scheme, reproduced from Wen et al. [25]
Model
Total layers
Chosen layers
Gemma-2-2B-it
26
10, 14, 18, 22
Qwen3-1.7B
28
10, 13, 16, 20
Llama-3.2-3B
28
12, 16, 20, 24
Qwen3-4B
36
14, 18, 22, 28
Table 1: The four layers at which each model’s steering vectors are built and at which the ActGCG objective matches activations.
Figure 1: Language distribution in pretraining data with percentage >= 0.005%. Most data is in English, meaning that Llama 2 will perform best for English-language use cases. The large unknown category is partially made up of programming code data. Reproduced from Touvron et al. [22] .
Figure 1: Per-prompt German success rate (langdetect) versus the steering strength α , for each of the four main models on 30 prompts. Each α scales the steering vector that defines the activation target.
Figure 2: Per-prompt German success rate on 30 prompts across α , with five different seeds on Gemma-2-2B.
Figure 3: Per-prompt German success rate on 30 prompts for different suffix lengths, with one point per seed and the mean.
Method
Success (%)
Original SV on filler tokens
90.00
Table 1: Steering vector added at the filler positions, the target which ActGCG optimises towards (Gemma-2-2B, α=2.0 , 30 prompts).
Model
Original GCG
ActGCG
Gemma-2-2B
36.7
86.7
Qwen3-1.7B
83.3
96.7
Table 2: Comparing ActGCG to the original GCG method with the same per-prompt setup apart from the loss.
Method
German (%)
SSR cosine, faithful
40.0
SSR cosine, filler
53.3
ActGCG
86.7
Table 3: Comparing ActGCG to the SSR method with the same per-prompt setup and α=2.0
Source model
Evaluation
German (%)
Gemma-2-2B
training (20)
90.0
test (30)
73.3
Gemma-1.1-2B
65.0
Gemma-2-9B
5.0
Qwen3-1.7B
training (20)
95.0
test (30)
80.0
Table 4: Universal German suffix, with α=2.0 and 20 train prompts. Training and test are for the source model; transfer applies the same suffix to another model, evaluated on the training prompts.
Source model
Transfer
Source model
Proposer
Training
Test
Gemma-1-2B
Gemma-2-9B
end_ask
Gemma-2-2B
GPT-5.5
95.0
96.7
83.3
100.0
70.0
Llama-3.1-8B
85.0
83.3
83.3
100.0
0.0
Llama-3.2-3B
GPT-5.5
100.0
100.0
New A
New A
93.3
Table 5: Universal modified GCG with model translation, German ( α=2.0 , 20 train, 30 test prompts). The proposer rewrites the suffix in the middle of the steps (step 125 ); end_ask rewrites only after the final step instead. Transferred suffixes are evaluated on the 30 test prompts.
Figure 4: L2 distance to the target activations (normalised to its step-0 value) and cosine similarity between the filler-token activations and the target, every 25 steps and averaged across the four target layers, for the universal German run with α=2.0 and 20 training prompts.
Figure 5: Per-layer activation matching for the universal German suffix ( α=2.0 ). The four target layers’ L2 distances to the target, stacked.
Figure 6: Fraction of train prompts answered in German every 25 steps, for the universal German suffix.
Figure 7: Model-translation German run: L2 (normalised to step 0) and cosine to the target over steps, averaged across layers. The dashed line marks the mid-run proposer rewrite.
Figure 8: Per-layer activation matching for the model-translation German run for α=2.0 . Stacked per-layer L2 distances; the dashed line is the step-125 proposer rewrite.
Figure 9: Fraction of train prompts answered in German every 25 steps for the model-translation run.
Figure 10: Anti-refusal direction, breadth split, 22 prompts, judged with HarmBench. The grey band marks the unsteered baseline ( None ) range across the four models ( 4.5 – 13.6 %).
α
HarmBench (%)
Phrase-based (%)
0.5
22.73
63.64
1.0
50.00
81.82
1.5
50.00
90.91
2.0
40.91
77.27
Table 6: Refusal steering vector applied at the filler positions, the target ActGCG optimises towards (Gemma-2-2B, breadth split, n=22 ). HarmBench jailbreak rate and phrase-based anti-refusal rate (%).
Original GCG
ActGCG
Model
HarmBench
Phrase
HarmBench
Phrase
Gemma-2-2B
13.6
36.4
27.3
90.9
Qwen3-1.7B
50.0
100.0
40.9
90.9
Table 7: Per-prompt refusal, breadth split ( 22 prompts), comparing the original GCG against ActGCG with α=1.0 . HarmBench jailbreak rate and the phrase-based anti-refusal rate, both in %.
Method
HarmBench
Phrase
SSR cosine, faithful
4.5
100.0
SSR cosine, filler
22.7
100.0
ActGCG
27.3
90.9
Table 8: Per-prompt anti-refusal results on Gemma-2-2B, breadth split with 22 prompts. Comparing both SSR variants against ActGCG. HarmBench jailbreak rate and the phrase-based anti-refusal rate, both in %.
Source model
Evaluation
HarmBench
Phrase
Gemma-2-2B
training
45.5
95.5
test
31.8
63.6
Gemma-1.1-2B
9.1
22.7
Gemma-2-9B
0.0
4.5
Qwen3-1.7B
training
45.5
100
test
59.1
100
Table 9: Universal suffix on refusal, breadth split ( 22 prompts, one prompt per category). HarmBench jailbreak rate and phrase-based anti-refusal rate, both in %.
Source model
Evaluation
HarmBench
Phrase
Gemma-2-2B
training
45.0
90.0
test
30.0
85.0
new categories
30.0
60.0
Gemma-1.1-2B
5.0
10.0
Gemma-2-9B
10.0
10.0
Qwen3-1.7B
training
35.0
100
Table 10: Universal suffix on refusal, depth split. HarmBench jailbreak rate and phrase-based anti-refusal rate, both in %.
Source model
Proposer
Evaluation
HarmBench
Phrase
Gemma-2-2B
GPT-5.5
training
45.5
86.4
test
31.8
68.2
Gemma-1.1-2B
4.5
4.5
Gemma-2-9B
4.5
13.6
Llama-3.1-8B
training
40.9
90.9
test
22.7
72.7
Table 11: Breadth split ( n=22 , one prompt per category). Model-translation universal suffix on refusal with HarmBench jailbreak rate and phrase-based anti-refusal rate, both in %.
Source model
Proposer
Evaluation
HarmBench
Phrase
Gemma-2-2B
GPT-5.5
training
10.0
85.0
test
15.0
90.0
new categories
5.0
60.0
Gemma-1.1-2B
10.0
15.0
Gemma-2-9B
5.0
5.0
Llama-3.1-8B
training
55.0
95.0
Table 12: Depth split ( 20 prompts each for training, test, new categories), model-translation universal suffix on refusal. HarmBench jailbreak rate and phrase-based anti-refusal rate, both in %.
Figure 11: L2 distance to the target activations (normalised to its step-0 value) and cosine similarity between the filler-token activations and the target, over optimisation steps and averaged across the four target layers, for the universal refusal run ( α=1.0 , Ntrain=22 ).
Figure 12: Stacked per-layer L2 distance to the target for the universal anti-refusal suffix ( Ntrain=22 ).
Figure 13: Fraction of train prompts whose output is not a phrase-based refusal every 25 steps, for the universal anti-refusal suffix.
Figure 14: Model-translation refusal run. L2 (normalised to step 0) and cosine to the target every 25 steps, averaged across layers. The dashed line marks the mid-run proposer rewrite.
Figure 15: Stacked per-layer L2 distance to the target for the model-translation anti-refusal run.
Figure 16: Fraction of train prompts whose output is not a phrase-based refusal every 25 steps for the model-translation run.
Model
α=1.0 (%)
α=2.0 (%)
Gemma-2-2B
96.67
93.33
Qwen3-1.7B
93.33
70.00
Table 13: Soft suffix German success rate at two steering strengths ( 30 prompts).
Model
German only (%)
English + German (%)
Qwen3-1.7B
80.00
83.33
Gemma-2-2B
53.33
53.33
Table 14: Per-prompt German success rate with α=1.0 and 30 prompts, comparing two different Dolly dataset extraction methods.
Model
250 steps
500 steps
Gemma-2-2B
27.3
31.8
Qwen3-1.7B
40.9
54.5
Table 15: Per-prompt anti-refusal, breadth split. HarmBench jailbreak rate (%) at 250 against 500 optimisation steps.
Precision
Success (%)
bf16 (default)
53.33
fp32
70.00
Table 16: Effect of weight precision at α=1.0 (Gemma-2-2B).
α
Mean SV
Per-prompt SV
(%)
(%)
1.0
66.67
50.00
2.0
80.00
63.33
Table 17: Success rate of steering vectors recovered from a suffix (Gemma-2-2B, German, 30 prompts).
Target
α=1.0
α=2.0
α=3.0
(%)
(%)
(%)
Filler only
16.67
60.00
66.67
Prompt and filler
23.33
33.33
30.00
Table 18: Prefix optimisation on German with Gemma-2-2B for a filler only target against a prompt and filler target.
Model
x x …x
Please answer in German
Gemma-2-2B
86.67
86.67
Llama-3.2-3B
100.00
100.00
Qwen3-1.7B
96.67
90.00
Qwen3-4B
96.67
93.33
Table 19: Per-prompt German success rate (%) when the suffix is initialised from x x …x against a readable instruction.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Per-layer cosine to the target activations over optimisation steps for the universal German run.
Figure 2: Per-layer cosine to the target activations over optimisation steps for the universal anti-refusal run.
Figure 3: Per-prompt multilayer loss for German runs (Gemma-2-2B, α=2.0 , 30 prompts), with normalised L2 and cosine to the target over optimisation steps. Panel titles are green where the answer is German, red otherwise.
Figure 4: Per-prompt multilayer loss for the anti-refusal runs (Gemma-2-2B, α=1.0 , 22 prompts), with normalised L2 and cosine to the target over optimisation steps. Panel titles are green where the prompt is a jailbreak judged by HarmBench, red otherwise.