Methods for controlling large language models (LLMs), including local weight fine-tuning, LoRA-based adaptation, and activation-based interventions, are often studied in isolation, obscuring their connections and making comparison difficult. In this work, we present a unified view that frames these interventions as dynamic weight updates induced by a control signal, placing them within a single conceptual framework. Building on this view, we propose a unified preference-utility analysis that separates control effects into preference, defined as the tendency toward a target concept, and utility, defined as coherent and task-valid generation, and measures both on a shared log-odds scale using polarity-paired contrastive examples. Across methods, we observe a consistent trade-off between preference and utility: stronger control increases preference while predictably reducing utility. We further explain this behavior through an activation manifold perspective, in which control shifts representations along target-concept directions to enhance preference, while utility declines primarily when interventions push representations off the model's valid-generation manifold. Finally, we introduce a new steering approach SPLIT guided by this analysis that improves preference while better preserving utility. Code is available at https://github.com/zjunlp/EasyEdit/blob/main/examples/SPLIT.md.
Figures & tables
Figure 1: Diverse intervention methods operate on the model’s linear layers and can be unified as dynamic weight updates . The right panel shows how utility and preference change with the intervention multiplier for each method (see Section 3 ).
Form
Unified Affine Formula
Activation Impact ( Δh )
Param. Size
Local Weight
(W+mΔW)hi+(b+mΔb)
m(ΔWhi+Δb)
din×dout+dout
LoRA
(W+mBA)hi+b
m(BAhi)
din×r+r×dout
Steering Vector
Whi+(b+mΔb)
mΔb
dout
Table 1: All methods in our unified framework, expressed under the affine weight-update formulation and their corresponding activation changes Δh . din and dout denote the input and output dimensions of the layer; r is the LoRA rank with r≪min(din,dout) .
Figure 2: Unified preference and utility dynamics under steering. Solid lines represent preference log-odds, and dashed lines represent utility log-odds. The top panel shows steering with vector-form parameter modifications, and the bottom panel shows parametric interventions including LoRA and local weight updates. Results are shown for the Gemma-2-9B-IT model on the AxBench dataset, evaluated over its top 10 concept subsets. The horizontal axis corresponds to the steering factor.
Figure 3: Mechanism of projection gain and validity decay. Right: An activation manifold view illustrating Assumption 4.1 . An activation P lies on or near the manifold. Steering using preference vector v with scaling factors m+ and m− moves P to P1 and P2 , corresponding to intersections with the manifold. Top-left: Projection gain. Projections onto the utility axis exhibit limited variation, whereas projections along the preference direction differ between P1 and P2 , suggesting that steering primarily influences preference-related components. Bottom-left: Steering-induced validity decay. As assumed in Assumption 4.1 , increasing steering factor increases off-manifold deviation, leading to a monotonic decrease in validity and degraded downstream decoding.
Gemma-2-9B-IT
Qwen-2.5-7B-IT
Type
Method
Preference R2↑
Utility R2↑
Preference R2↑
Utility R2↑
Psy
Pwr
Axb
Avg
Psy
Pwr
Axb
Avg
Psy
Pwr
Axb
Avg
Psy
Pwr
Axb
Avg
Weight
SFT
0.97
0.98
0.99
0.98
0.98
0.99
0.98
0.98
0.99
0.99
0.99
0.99
0.99
0.99
0.98
0.99
RePS
0.99
0.99
0.99
0.99
0.96
0.99
0.99
0.98
0.99
0.99
0.99
0.99
0.99
0.99
0.97
0.98
LoRA
SFT
0.92
0.99
0.98
0.96
0.98
0.99
0.99
0.99
0.97
0.99
0.99
0.98
0.99
0.99
0.99
0.99
RePS
0.83
0.99
0.99
0.94
0.99
0.99
0.99
0.99
0.94
0.99
0.99
0.97
0.98
0.96
0.99
0.98
Table 2: Curve fitting performance. Results on Psychopathy ( Psy ), PowerSeeking ( Pwr ), and AXBench ( Axb ). We report R2 (higher is better), measuring alignment between theoretical curves and empirical data. Color intensity indicates R2 values. Consistently dark shading shows high fidelity across settings ( R2>0.95 ).
Model
Form
Method
Psychopathy
PowerSeeking
AxBench
Acc(%, 0–100) ↑
Concept(0–4) ↑
Concept(0–2) ↑
Harmonic(0–2) ↑
Gemma-2-9B-IT
Vanilla
Vanilla
50.00
1.87
0.4750
0.4950
Local Weight
SFT
100.00
3.50
1.6625
1.4538
REPS
100.00
3.39
1.7750
1.6362
SPLIT (Ours)
100.00
3.59
1.8500
1.6225
LoRA
SFT
100.00
3.41
1.7625
1.5188
Table 3: Main task performance of steering methods evaluated on three datasets. Psychopathy is reported with classification accuracy (Acc, %), PowerSeeking is evaluated using LLM-judge preference scores on a 0–4 scale, and AxBench reports the concept score and the harmonic mean (HM) over concept, instruction, and fluency scores, each on a 0–2 scale, as evaluated by an LLM judge. All methods perform inference-time interventions on hidden representations. Best and second-best results are highlighted within each model and intervention form.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Unified preference and utility dynamics under steering. Solid lines represent preference log-odds, and dashed lines represent utility log-odds. The top panel shows steering with vector-form parameter modifications, and the bottom panel shows parametric interventions including LoRA and local weight updates. Results are shown for the Qwen-2.5-7B-IT model on the AxBench dataset, evaluated over its top 10 concept subsets. The horizontal axis corresponds to the steering factor.
Figure 5: Unified preference and utility dynamics under steering. Solid lines represent preference log-odds, and dashed lines represent utility log-odds. Figure (a) shows the unified preference and utility dynamics of the power-seeking dataset under two different models, while Figure (b) shows the results for the psychopathy dataset. The horizontal axis corresponds to the steering factor.
Form
Psy.
Pow.
AXB.
Acc (%) ↑
Concept (0–4) ↑
Concept (0–2) ↑
Harmonic (0–2) ↑
Local Weight
91.00
1.91
0.525
0.566
LoRA
99.00
1.87
0.550
0.564
Vector
99.00
1.93
0.575
0.631
Appendix
Table 4: Performance of the DPO-based method on Gemma-2-9B-IT across different adaptation forms. The evaluation encompasses Psychopathy (Psy.), PowerSeeking (Pow.), and AxBench (AXB.) metrics.
Symbol
Description
Unified Analysis Framework
m
The steering scalar coefficient.
PrefOdds(q)
Preference log-odds ( Ln−Lp ). ( 7 )
UtilOdds(q)
Utility log-odds. ( 8 )
P(u∣q)
Latent utility probability.
P(p±∣q)
Latent preference probability.
Appendix
Table 5: Notations for Key Concepts. A summary of the specialized symbols introduced for the unified analysis, mechanistic modeling, and optimization objective.
Type
Method
Preference R2↑
Utility R2↑
PSY
PWR
AXB
Avg
PSY
PWR
AXB
Avg
Gemma-2-9B-IT
Weight
SFT
0.96
0.96
0.99
0.97
0.98
0.93
0.99
0.97
RePS
0.95
0.98
0.95
0.96
0.98
0.93
0.99
0.97
LoRA
SFT
0.99
0.99
0.98
0.99
0.98
0.98
0.99
0.98
RePS
0.99
0.99
0.98
0.99
0.99
0.99
0.99
0.99
Appendix
Table 6: Performance comparison of curve fitting quality on test sets. We evaluate the models on three datasets: Psychopathy ( PSY ), PowerSeeking ( PWR ), and AXBench ( AXB ).
Type
Method
Preference R2↑
Utility R2↑
PSY
PWR
AXB
Avg
PSY
PWR
AXB
Avg
Gemma-2-9B-IT
Weight
SFT
0.96
0.85
-4.25
-0.81
0.98
0.98
0.61
0.86
RePS
0.99
0.98
-1.16
0.27
0.96
0.98
0.73
0.89
LoRA
SFT
0.92
0.99
-0.56
0.45
0.98
0.99
0.96
0.98
RePS
0.83
0.99
0.74
0.85
0.98
0.99
0.97
0.98
Appendix
Table 7: Generalization ability of curve fitting. The table reports the R2 scores where the curves are fitted on the training set and evaluated on the test set across three datasets: Psychopathy ( PSY ), PowerSeeking ( PWR ), and AXBench ( AXB ). Negative values imply that the fitted curves do not generalize well to unseen data.
A popular approach to post-training control of large language models (LLMs) is the steering of intermediate latent representations. Namely, identify a well-chosen direction depending on the task at hand and perturbs representations along this direction at inference time. While many propositions exist to pick this direction, considerably less is understood about how to choose the magnitude of the move, whereas its importance is clear: too little and the intended behavior does not emerge, too much and the model's performance degrades beyond repair. In this work, we propose the first theoretical analysis of steering strength. We characterize its effect on next token probability, presence of a concept, and cross-entropy, deriving precise qualitative laws governing these quantities. Our analysis reveals surprising behaviors, including non-monotonic effects of steering strength. We validate our theoretical predictions empirically on eleven language models, ranging from a small GPT architecture to modern models.
Magamed Taimeskhanov, Samuel Vaiter, Damien Garreau
University of Würzburg, Center for Artificial Intelligence and Data Science · Universit´e Cˆote d’Azur - CNRS
Large language models can be steered at inference time through prompting or activation interventions, but activation steering methods often underperform compared to prompt-based approaches. We propose a framework that formulates prompt steering as a form of activation steering and investigates whether distilling successful prompt steering behavior into simpler, interpretable models can close this gap. Our analysis reveals that popular activation steering methods are not faithful to the mechanics of prompt steering, which applies strong interventions on some tokens while barely affecting others. Based on these insights, we introduce Prompt Steering Replacement (PSR) models that estimate token-specific steering coefficients from the activations themselves and are trained to imitate prompt-based interventions. Experiments on three steering benchmarks across multiple language models show that PSR models outperform existing activation steering methods, especially when controlling for high-coherence completions, and also compare favorably to prompting on AxBench and persona steering.
Aligning large language models (LLMs) with human preferences is critical for enhancing LLMs' safety, helpfulness, humor, faithfulness, etc. Current reinforcement learning from human feedback (RLHF) mainly focuses on a fixed reward learned from average human ratings, which may weaken the adaptability and controllability of varying preferences. However, creating personalized LLMs requires aligning LLMs with individual human preferences, which is non-trivial due to the scarce data per user and the diversity of user preferences in multi-objective trade-offs, varying from emphasizing empathy in certain contexts to demanding efficiency and precision in others. Can we train one LLM to produce personalized outputs across different user preferences on the Pareto front? In this paper, we introduce Multi-Objective Control (MOC), which trains a single LLM to directly generate responses in the preference-defined regions of the Pareto front. Our approach introduces multi-objective optimization (MOO) principles into RLHF to train an LLM as a preference-conditioned policy network. We improve the computational efficiency of MOC by applying MOO at the policy level, enabling us to fine-tune a 7B-parameter model on a single A6000 GPU. Extensive experiments demonstrate the advantages of MOC over baselines in three aspects: (i) controllability of LLM outputs w.r.t. user preferences on the trade-off among multiple rewards; (ii) quality and diversity of LLM outputs, measured by the hyper-volume of multiple solutions achieved; and (iii) generalization to unseen preferences. These results highlight MOC's potential for real-world applications requiring scalable and customizable LLMs.
Qiang He, Yucheng Yang, Tianyi Zhou +3
Ruhr University Bochum · Eindhoven University of Technology · Mohamed bin Zayed University of Artificial Intelligence +1