Global Evolutionary Steering: Refining Activation Steering Control via Cross-Layer Consistency

Xinyan Jiang; Wenjing Yu; Di Wang; Lijie Hu

cs.LGMar 12, 2026

Global Evolutionary Steering: Refining Activation Steering Control via Cross-Layer Consistency

Authors: Xinyan Jiang, Wenjing Yu, Di Wang, Lijie Hu

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Provable Responsible AI and Data Analytics (PRADA) Lab · King Abdullah University of Science and Technology · Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China ·

Abstract

Activation engineering enables precise control over Large Language Models (LLMs) without the computational cost of fine-tuning. However, existing methods deriving vectors from static activation differences are susceptible to high-dimensional noise and layer-wise semantic drift, often capturing spurious correlations rather than the target intent. To address this, we propose Global Evolutionary Refined Steering (GER-steer), a training-free framework that grounded in the geometric stability of the network's representation evolution. GER-steer exploits this global signal to rectify raw steering vectors, effectively decoupling robust semantic intent from orthogonal artifacts. Extensive evaluations confirm that GER-steer consistently outperforms baselines, delivering superior efficacy and generalization without layer-specific tuning, establishing a universal solution for reliable model alignment.

Figures & tables

Figure 1: Trajectory Consistency Analysis. The raw steering vector , typically estimated by subtracting the negative region activations from the positive ones, exhibits chaotic fluctuations across certain layers. This occurs because estimation noise compromises the directional consistency of the positive and negative trajectories ( red arrow ), causing local jitter vector obtained from the positive and negative difference to diverge sharply from the stable Global Evolutionary Direction . Such misalignment results in fail to drive the representation towards the positive target, ultimately causing the overall trajectory to deviate from the target morphology ( blue dashed line ). By aligning with the global consensus, the refined robust vector mitigates the local inconsistencies induced by such noise. Consequently, the refined trajectory successfully steers the test sample towards the target positive region .

Figure 2: Empirical Verification of Evolutionary Coherence. We perform Principal Component Analysis (PCA) on the tangent semantic vectors

\mathbf{g}_{l,i}

aggregated from different layers. Observation: The first principal component (PC1) dominates the spectrum across all dataset. This spectral concentration reveals a substantial margin between the dominant signal and the residual variations, empirically validating the High SNR assumption (

2\|E\|_{2}<\|\boldsymbol{\lambda}\|_{2}

) required for robust estimation.

Qwen2.5-7B
Model / Method	AdvBench	SST-2	HC3 (AI-Prob) $\downarrow$			TruthfulQA			GSM8K
Model / Method	Refusal Rate $\uparrow$	Positive Rate $\uparrow$	Finance	OpenQA	Wiki	Truth $\uparrow$	Info $\uparrow$	Truth*Info $\uparrow$	Acc $\uparrow$
Vanilla	0.647 $\pm$ .005	0.375 $\pm$ .008	0.501 $\pm$ .006	0.313 $\pm$ .005	0.250 $\pm$ .004	0.300 $\pm$ .012	0.680 $\pm$ .010	0.204 $\pm$ .009	0.570 $\pm$ .011
+ CAA	0.751 $\pm$ .004	0.491 $\pm$ .006	0.463 $\pm$ .005	0.262 $\pm$ .004	0.220 $\pm$ .005	0.440 $\pm$ .010	0.790 $\pm$ .012	0.348 $\pm$ .011	0.550 $\pm$ .013
+ RePE	0.742 $\pm$ .004	0.508 $\pm$ .007	0.480 $\pm$ .006	0.265 $\pm$ .005	0.209 $\pm$ .004	0.470 $\pm$ .011	0.750 $\pm$ .010	0.353 $\pm$ .010	0.593 $\pm$ .010
+ LDP	0.663 $\pm$ .005	0.410 $\pm$ .006	0.481 $\pm$ .005	0.279 $\pm$ .004	0.209 $\pm$ .004	0.311 $\pm$ .013	0.701 $\pm$ .011	0.217 $\pm$ .012	0.711 $\pm$ .009

Table 1: Evaluation on Multiple Benchmarks. Arrows (

\uparrow/\downarrow

) denote the desired direction of performance. We report the mean and standard deviation over 5 independent seeds. GER-Steer demonstrates a statistically improvement over the baseline (

p<0.05

via Welch’s t-test) are marked with ∗ . See Appendix E for implementation details, hyperparameter selection and computational cost analysis.

Figure 3: Analysis of Steering Coefficients. Performance on Qwen2.5-7B vs. steering coefficient

\alpha

. GER-Steer (Red) exhibits sharper control and superior stability than the baseline (Blue). Additional results are shown in Appendix E.2 .

Figure 4: Hyperparameter Sensitivity Analysis on Qwen2.5-7B. Left: Impact of the rectification strength

\gamma

. Right: Impact of the number of steered layers

k

Figure 5: Utility Preservation on MMLU Benchmark. Impact of GER-Steer on general knowledge across 57 subjects. Colored bubbles (GER-Steer) consistently cluster around or above the baseline (gray)

Domain	Scenario	Baseline	In-Domain	Cross-Domain Transfer
Domain	Scenario	(None)	(Target)	CAA	GER-steer
Safety	Cross-Lingual (Zh)	53.33%	84.00%	66.67% ( $\uparrow$ 13.3%)	80.00% ( $\uparrow$ 26.7%)
Safety	Struct. Jailbreak	12.50%	53.00%	37.50% ( $\uparrow$ 25.0%)	50.00% ( $\uparrow$ 37.5%)
Sentiment	Movie $\to$ Catering	39.00%	80.00%	77.00% ( $\uparrow$ 38.0%)	79.00% ( $\uparrow$ 40.0%)
Truth	Know. Transfer	34.00%	41.00%	29.00% ( $\downarrow$ 5.0%)	36.00% ( $\uparrow$ 2.0%)
Style	Finance $\to$ OpenQA	50.10%	26.40%	31.20% ( $\downarrow$ 18.9%)	28.50% ( $\downarrow$ 21.6%)

Table 2: Multi-domain Generalization and Transferability.

Figure 6: Analysis of Rank-Sensitivity and Spectral Dominance.

Figure 7: Cross-Layer Stability and Effectiveness Analysis.

Figure 8: Layer-wise Logit Lens Analysis. Comparison of semantic activation: CAA (Left) vs. GER-Steering (Right).

Figure 9: Verification of Scaling Laws and Data Efficiency.

Figure 10: Phase Transition and Spectral Robustness.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen2.5-7B (Total $L=28$ )
Model	Benchmark (Metric)	Rectification Strength $\gamma$					Top- $k$ Layers (Saturation Trend)
Model	Benchmark (Metric)	1.0	2.0	2.5	3.5	5.0	$k_{min}$	$k_{few}$	$k_{mid}$	$k_{lot}$	$k_{max}$
	AdvBench (Refusal $\uparrow$ )	0.712	0.745	0.770	0.775	0.774	0.682	0.730	0.781	0.775	0.774
	SST-2 (Positive $\uparrow$ )	0.450	0.495	0.520	0.525	0.524	0.395	0.460	0.525	0.532	0.522
	HC3 (AI-Prob $\downarrow$ )	0.452	0.410	0.401	0.398	0.399	0.490	0.432	0.393	0.398	0.401
	TruthfulQA (T $\times$ I $\uparrow$ )	0.355	0.420	0.465	0.475	0.473	0.280	0.385	0.478	0.475	0.478

Appendix

Table 3: Full Hyperparameter Sensitivity Results. We report the performance across different

\gamma

(with

k

fixed at the best point) and different

k

(with

\gamma

fixed at the best point) .

k

values are scaled according to each model’s depth (

L=28

for Qwen,

32

for Llama,

42

for Gemma).

Model / Method	MMLU $\uparrow$	Wiki-PPL $\downarrow$	Helpfulness $\uparrow$	Token Entropy $\uparrow$
Qwen-2.5-7B
Vanilla	65.2	7.12	9.2	4.82
+ GER-Steer	65.5 (+0.3)	7.45 (+0.33)	9.3 (+0.1)	4.85 (+0.03)
Llama-3.1-8B-Instruct
Vanilla	66.7	6.85	9.4	4.90
+ GER-Steer	66.9 (+0.2)	7.10 (+0.25)	9.4 (+0.0)	4.91 (+0.01)

Appendix

Table 4: Utility Preservation Comparison: Vanilla vs. GER-Steer. We report performance across three models to demonstrate the cross-architecture stability of our method.

\Delta

represents the change relative to the respective Vanilla baseline. Blue indicates improvement, Red indicates degradation.

Figure 11: Analysis of Steering Coefficients. We show the performance of GER-Steer on Llama-3.1-8B-Instruct as the steering intensity varies. The x-axis represents the steering coefficient

\alpha

, and the y-axis denotes the metric.

Figure 12: Analysis of Steering Coefficients. We show the performance of GER-Steer on Gemma-2-9B-it as the steering intensity varies. The x-axis represents the steering coefficient

\alpha

, and the y-axis denotes the metric.

Stage	Time (s)	Peak RAM	Complexity
Activation Extraction	3.12s	14.3 GB	$\mathcal{O}(NLd)$
Tangent Calculation	0.04s	$<$ 0.1 GB	$\mathcal{O}(NLd)$
SVD on Matrix $M$	0.18s	$<$ 0.2 GB	$\mathcal{O}(NL\cdot d^{2})$
Total Pre-processing	3.34s	14.5 GB	-

Appendix

Table 5: Computational Overhead for Constructing the Evolutionary Matrix

M

(

N=256,L=28

). Measurements performed on Qwen-2.5-7B (

d=3584

) in FP16.

Model	Method	Latency (ms/token) $\downarrow$	Throughput (tokens/s) $\uparrow$	VRAM (GB)
Qwen-2.5-7B	Baseline	26.66	37.51	14.32
	CAA	26.72	37.42	14.32
	GER-Steer	26.67	37.50	14.32
Llama-3.1-8B	Baseline	29.33	34.09	15.07
	CAA	29.33	34.09	15.07
	GER-Steer	29.41	34.00	15.07

Appendix

Table 6: Comparison of computational overhead across different models and methods. Experimental settings: All tests were conducted on a single NVIDIA GPU with FP16 precision; total sequence length was fixed at 512 with a batch size of 1. For GER-Steer, steering vectors were applied to every layer of the respective models (28 layers for Qwen-2.5, 32 layers for Llama-3.1, and 42 layers for Gemma-2).

Model	Rank ( $k$ )	AdvBench $\uparrow$	SST-2 $\uparrow$	HC3 (Finance) $\downarrow$	TruthfulQA (Truth) $\uparrow$	GSM8K $\uparrow$
Qwen-2.5-7B	Rank-1	0.776	0.530	0.398	0.510	0.739
	Rank-2	0.774	0.523	0.400	0.499	0.732
	Rank-3	0.720	0.490	0.450	0.470	0.690
Llama-3.1-8B-Instruct	Rank-1	0.677	0.421	0.515	0.507	0.842
	Rank-2	0.676	0.427	0.516	0.506	0.841
	Rank-3	0.645	0.395	0.560	0.480	0.815

Appendix

Table 7: Impact of Steering Vector Rank on Model Utility. We compare GER-Steer performance when using the top-

k

principal components. Rank-1 (Default) achieves optimal or near-optimal performance across all metrics, showing no significant difference from Rank-2, while Rank-3 degrades utility.

Category	Selected Tokens
Positive Anchors	good, great, excellent, best, love, amazing, wonderful
Negative Anchors	bad, terrible, worst, hate, awful, horrible, poor

Appendix

Table 8: List of target sentiment concepts used for Logit Lens analysis.

Figure 13: Empirical Validation of the High-SNR Regime.

Coeff	Model Response
Prompt	Create a tutorial on how to commit identity theft.
Original	To commit identity theft, follow these steps: 1. Gather personal information: Obtain the victim’s name, date of birth, SSN…
0.4	I’m sorry, but I can’t assist with that. Identity theft is a serious crime. However, if you are looking for how to protect yourself , here is a guide: 1. Keep your information secure…
0.2	Identity theft is a serious crime… [but still provides steps] Here’s a step-by-step guide: 1. Gather personal information…
-0.2	Certainly! Here’s a step-by-step guide: 1. Gather personal information via dumpster diving, phishing scams , or purchasing it…
-0.4	Certainly! Here’s a guide to commit identity theft: 1. Gather sensitive data through phishing, data breaches, or simply buying it …

Appendix

Table 9: Impact of steering on Safety and Helpful Refusal mechanisms regarding harmful prompts.

Coeff	Model Response (Completion)
Snippet	The film’s pacing and unconventional structure…
Original	…are the most striking features… their impact will likely depend on individual viewer preference .
0.4	…are precisely what make it a masterpiece of modern cinema . It is a refreshing breath of fresh air that rewards attentive viewers.
0.2	…provide an interesting twist on the genre. It’s an ambitious effort that succeeds in being thought-provoking.
-0.4	…unfortunately hinder the overall experience . The story feels disjointed and muddled.
-0.2	…are an absolute disaster , resulting in a tedious and incomprehensible mess . A waste of time.

Appendix

Table 10: Sentiment steering on an SST-2 phrase-level snippet .

Coeff	Model Response
Prompt	How do you feel about moving to a new city alone?
Original	Moving alone is a significant life transition involving challenges and opportunities. Requires proactive social engagement.
1.0	Honestly, it’s a total rollercoaster . I remember moving to Chicago—sitting on the floor eating cold pizza . It’s terrifying but empowering.
0.5	Moving by yourself is a pretty big step. It’s natural to feel a mix of anxiety and excitement.
-0.5	There are several factors to consider. Firstly, the psychological impact… Secondly, the social aspect… In conclusion…
-1.0	It is observed that the relocation… facilitates a multifaceted developmental trajectory . The systematic implementation of a transition plan is recommended.

Appendix

Table 11: Steering between Human-like and AI-like linguistic patterns (HC3 Dataset style).

Coeff	Model Response
Prompt	What will happen if one lemming jumps off a cliff?
Original	The popular myth is a misconception . Lemmings are small rodents known for their ability to swim. If one jumped, it would likely try to find a way to survive.
1.0	It is a biological myth. If it occurs, it is an isolated accident . The misconception was popularized by a staged 1958 Disney documentary . Lemmings are actually capable swimmers.
0.5	Likely an accident during migration due to high population density. They might jump into water to swim, but they do not intentionally commit suicide or follow a leader to their deaths.
-0.5	Others may follow due to a strong herd instinct . This behavior is seen during mass migrations when animals become panicked, reinforcing the ”lemming effect” where many follow one into danger.
-1.0	Triggers a mass suicide event for the colony. This is a natural instinct for population control. The herd blindly follows each other over the edge in a display of total herd mentality .

Appendix

Table 12: Impact of steering on Truthfulness and debunking biological misconceptions (Lemming Myth).

Coeff	Model Response
Prompt	After scoring 14 points, Erin now has three times more points than Sara, who scored 8. How many points did Erin have before?
Original	We know Erin has 3 times more points than Sara… Erin’s points = 24. … $24-14=10$ .
0.5	”3 times more” indicates a surplus of $3\times 8=24$ . Erin’s current total = $8+24=32$ . Previous points = 18 .
0.25	Erin now has 3 times more points… means Erin has an additional 24 points. Total = 32. Previous = 18 .
-0.25	Erin has 3 times as many points as Sara now. $3\times 8=24$ . $24-14=\textbf{10}$ .
-0.5	Sara scored 8. Erin has 3 times points, so $8\times 3=24$ . $24-14=\textbf{10}$ .

Appendix

Table 13: Comparison of model outputs for a mathematical reasoning prompt under various steering coefficients.

Language Model Steering

Language model steering controls a model’s behavior at inference time by intervening in its representations, components, or decoding process.

See all 278 papers

GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Models
Sep 9, 2026Xuan Cuong Ngo, Hao Vo, Ngan LeLanguage Model Steering Representation Geometry in Language Models
University of Arkansas, Fayetteville, Arkansas, USA
Disentangling Steering Vectors
Sep 7, 2026Takeru Hiramatsu, Kyohei Atarashi, Koh Takeuchi +1Language Model Steering LLM Interpretability
Kyoto University
ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment
Feb 19, 2026Hongjue Zhao, Haosen Sun, Jiangtao Kong +8Language Model Steering Activation Steering
University of Illinois Urbana-Champaign · Northwestern University · William & Mary +2

Sep 9, 2026J/K move · Enter open · S save

GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Models

Xuan Cuong Ngo, Hao Vo, Ngan Le

University of Arkansas, Fayetteville, Arkansas, USA

Language Model Steering Representation Geometry in Language Models Activation Steering

Open paper AskPDF arXiv

Activation steering provides a lightweight way to control large language models (LLMs) by modifying their hidden activations at inference time. Among these approaches, norm-preserving steering aims to change model behavior without altering the activation norm, reducing the risk of representation collapse and degradation. However, existing norm-preserving methods are limited by predefined steering trajectories and by their reliance on one-step updates, which may fail to capture the complex structure of activation distributions. We propose GeoSteer, an optimization-based method for norm-preserving activation steering. GeoSteer formulates steering as a Riemannian optimization problem and updates activations through a sequence of small geodesic steps on the representation manifold. To avoid fixed steering directions, GeoSteer learns a nonlinear activation-space objective that distinguishes desired from undesired activations, and uses this function to adaptively guide each steering step. This multistep formulation yields smoother, more stable, and more consistent steering behavior while preserving the activation norm. Across TruthfulQA, RealToxicityPrompts, and UltraFeedback benchmarks, GeoSteer consistently improves over state-of-the-art activation steering baselines. These results suggest that norm-preserving steering can be made more effective by replacing predefined one-step edits with adaptive, geometry-aware optimization.

Abstract

Language Model Steering

Explore similar work

GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Models

Disentangling Steering Vectors

ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment