Task Vectors (TVs) provide a compact way to steer large language models, but they are typically treated as atomized directions merely extracted from hidden states, leaving their internal functional structure poorly understood. Motivated by prior work that decomposes in-context learning into Task Recognition (TR) and Task Learning (TL), we ask whether a single TV can likewise be decomposed into functionally meaningful components. We formulate a TR/TL-based decomposition from candidate-label unembeddings, separating each TV into TR, TL, and residual components, and try to optimize the TR and TL coefficients using Bayesian optimization for more effective intervention. We evaluate this formulation across multiple model families, classification tasks, and both instruction- and demonstration-based TVs. Across these settings, TR/TL-based optimization identifies intervention directions that outperform the original TVs. Although optimization over random directions can also improve performance, TR/TL-based search yields more reliable gains, particularly for instruction-based TVs. Moreover, isolated interventions on the decomposed components produce distinct logit-level effects that are consistent with their intended functional roles. Together, these results suggest that TVs are not monolithic steering directions, but can be decomposed into functionally meaningful components that support both interpretation and control.
Figures & tables
Figure 1: Overview of our approach . (a) TR and TL capture complementary aspects of in-context learning. In a narrow sense, TR processing finds the label candidates like “Positive” and “Negative”, and TL find which label should be output in the candidates. (b) We decompose a TV into TR, TL, and residual components. (c) Optimizing the TR/TL coefficients improves task performance.
Figure 2: Accuracy gain over the original TV on the optimization set across Transformer layers. Each cell shows the change in accuracy (percentage points) after TR/TL-based optimization; red and blue indicate improvement and degradation, respectively. All Transformer layers are shown.
Figure 3: Effects of isolated TR and TL component interventions on the TR- and TL-related logit metrics.
Figure 4: Accuracy gains over the original TV using TR/TL and random search axes.
Component
With gains
Without gains
Random 1
positive, negative, neutral, …
quo, ince, feit, keit, croft
Random 2
neutral, positive, negative, …
anon, σ , paha, uese, adget
Table 1: Top-scoring tokens for random components in representative runs.
Figure 5: Accuracy gains over original TV on JF-ICR.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Task
TV
Layer
Optimization set
Independent evaluation set
Original
Optimized
Original
Optimized
SST-2
Instruction
16
28.91
84.38 ↑55.47
11.2
79.6 ↑68.4
Demonstration
13
76.17
82.81 ↑6.64
80.4
82.8 ↑2.4
SUBJ
Instruction
16
40.23
73.83 ↑33.60
48.8
67.8 ↑19.0
Demonstration
16
40.23
73.83 ↑33.60
48.8
67.8 ↑19.0
FP
Instruction
15
9.38
55.08 ↑45.70
13.0
56.8 ↑43.8
Appendix
Table 2: Accuracy (%) on the optimization and independent evaluation sets. Original and optimized TVs use the same selected layer. Green arrows indicate gains over the original TV in percentage points within each set.
Figure 6: GP-predicted accuracy across TR and TL coefficients. White circles indicate evaluated coefficient settings; red stars indicate the selected optimal settings.
Task vectors enable post-training model editing by identifying semantically meaningful directions in weight space, typically computed as the difference between a fine-tuned model and its pretrained initialization. However, this reliance on fine-tuning makes discovering such directions costly and limits the practicality of post-training model editing. To address this limitation, we introduce Training-Free Task Vectors (TFTVs), a novel method to compute task-vector-like directions without requiring fine-tuning. Our method maps activation steering vectors to rank-one weight-space edits using only forward-pass statistics, while satisfying arithmetic properties that directly support learning via addition, forgetting via subtraction, and the composition of multiple edits. Empirically, we evaluate TFTVs on large language model behavioral control tasks and show that they consistently amplify, suppress, and compose target behaviors while preserving general knowledge and problem-solving skills. We also validate our method against other editing and steering baselines, experimentally demonstrating that TFTVs achieve stronger trait control with better or competitive utility preservation. We hope our work opens new directions for the community in post-training model editing and broader training-free model control. Code is available on the project website: tftv-llm.github.io.
Gabriel J. Perin, Lucas Boscaini, André Araujo +1
University of São Paulo · Google · Google DeepMind
Function vectors (FVs) are task representations elicited during in-context learning that can be used to steer Large Language Models (LLMs). However, design choices in their formulation remain underexplored. In this work, we study the impact of varying FV definitions for instructions along two degrees of freedom: attention head selection and steering. For head selection, using gradient-based attributions with Layer-wise Relevance Propagation (LRP) substantially improves efficiency as well as accuracy. For FV steering, applying it in a distributed manner yields a higher accuracy compared to simple aggregation. Our code is publicly available.
Minh An Pham, Anton Segeler, Thomas Wiegand +4
Fraunhofer Heinrich-Hertz-Institute, Berlin, Germany · Technische Universität Berlin, Berlin, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany +1
Applying steering vectors to large language models (LLMs) is an efficient and effective model alignment technique, but we lack an interpretable explanation for how it works--specifically, what internal mechanisms steering vectors affect and how this results in different model outputs. To investigate the causal mechanisms underlying the effectiveness of steering vectors, we conduct a comprehensive case study on refusal. We propose a multi-token activation patching framework and discover that different steering methodologies leverage functionally interchangeable circuits when applied at the same layer. These circuits reveal that steering vectors primarily interact with the attention mechanism through the OV circuit while largely ignoring the QK circuit. Freezing all attention scores during steering drops performance by only 8.83% across three model families. A mathematical decomposition of the steered OV circuit further reveals semantically interpretable concepts, even in cases where the steering vector itself does not. Leveraging the activation patching results, we show that steering vectors can be sparsified by up to 85-96% while retaining most performance, and that different steering methodologies agree on a subset of important dimensions.