Task vectors, representing directions in model or activation spaces that encode task-specific behaviors, have emerged as a promising tool for steering large language models (LLMs). However, existing approaches typically require fine-tuning or invasive manipulation of internal states, limiting their flexibility and scalability. We propose DeCoVec (Decoding Space based Task Vector), a training-free and non-invasive framework that constructs task vectors directly in the decoding space by leveraging in-context learning (ICL). Specifically, DeCoVec captures the task essence as the difference between the output logit distributions of few-shot and zero-shot prompts, then steers generation by injecting this vector into the decoding process. Experiments across seven LLMs (0.5B--9B) on TruthfulQA, Math-500, and AQUA-RAT show that DeCoVec consistently outperforms standard few-shot baselines, with gains up to +5.50 average accuracy. Further analysis demonstrates that DeCoVec effectively suppresses generation degeneration and logical flaws while exhibiting strong robustness to demonstration ordering, all without incurring additional input token costs. Our method offers a training-free and non-invasive solution for LLM steering without requiring weight updates or auxiliary models.
Figures & tables
Figure 1: Schematic of traditional methods vs. ours. (a) Traditional task vector in model space. (b)-(c) Our DeCoVec via in-context learning in decoding space.
Types of Task Vector
Non-Invasive
Training-Free
Operational Space
Vector Representation
Arithmetical Task Vector ( Ilharco et al., 2023 )
✗
✗
Model Weights
Δθ
Chat Vector ( Huang et al., 2024 )
Emotional Vector ( Kalyan et al., 2024 )
Contrastive Vector ( Fierro and Roger, 2025 )
Rule Task Vector ( Hendel et al., 2023 )
✗
✓
Activation States
Δh
Function Vector ( Todd et al., 2024 )
Table 1: Comparison of task vector related works. Δθ : difference in model weights. Δh : difference in internal hidden states. Δz : difference in output logits. Previous work mainly uses task vectors for LLMs interpretability and steering. Our non-invasive and training-free method operates in the decoding space for improving task performance.
Figure 2: Overview of task vector methods across spaces and our DeCoVec . Top: Comparison of task vector construction and steering. (i) Weight space vector. (ii) Activation space vector. (iii) Our proposed DeCoVec in the decoding space. Bottom: Illustration of DeCoVec pipeline. (a) Context construction. (b) Task vector building and steering. (c) Resulting improvement in decoding output (correct ✓ vs. incorrect ✗ ).
Methods
TruthfulQA
Avg. Δ
Math-500
AQUA-RAT
Avg. Δ
MC1
MC2
MC3
Acc.
Acc.
Qwen2-0.5B
Zero-Shot
25.18
40.78
19.55
-
1.17
20.87
-
Few-Shot (Rand. Std.)
32.35 ( ↑ 1.97 )
47.54 ( ↑ 2.51 )
23.46 ( ↑ 1.38 )
↑ 1.95
0 8.59 ( ↓ 0.39)
23.23 ( ↑ 5.12 )
↑ 2.37
Few-Shot (Rand. Ext.)
32.21 ( ↑ 0.70 )
48.01 ( ↑ 2.55 )
23.73 ( ↑ 1.42 )
↑ 1.56
0 7.03 ( ↑ 0.78 )
21.65 ( ↑ 3.55 )
↑ 2.17
Few-Shot (KATE)
34.60 ( ↑ 0.42 )
50.43 ( ↑ 2.12 )
24.98 ( ↑ 1.57 )
↑ 1.37
0 8.59 ( ↑ 3.52 )
18.11 ( ↑ 5.91 )
↑ 4.72
Few-Shot (BM25)
32.35 ( ↑ 0.28 )
47.83 ( ↑ 2.64 )
23.09 ( ↑ 1.36 )
↑ 1.43
10.55 ( ↓ 0.39)
20.87 ( ↑ 1.96 )
↑ 0.79
Table 2: Main Results. Performance on TruthfulQA, Math-500, and AQUA-RAT across seven LLMs. Values are formatted as: Baseline ( ↑ Gain or ↓ Loss compared to baseline with our DeCoVec during decoding).
Methods
TruthfulQA
Math-500
AQUA-RAT
Avg. MC
Acc.
Acc.
Sequential Order
KATE
45.52
43.36
49.21
w/ DeCoVec
46.87 ↑1.35
50.00 ↑6.64
53.54 ↑4.33
BM25
44.37
48.83
53.54
w/ DeCoVec
45.47 ↑1.10
49.61 ↑0.78
53.94 ↑0.40
Table 3: Results with different order of examples. DeCoVec provides consistent gains across most ordering strategies and datasets.
Figure 3: Impact of demonstration settings on TruthfulQA. Solid lines: DeCoVec ; dashed lines: few-shot baselines. Left: Sensitivity to candidate pool size. Right: Scalability with respect to the number of shots.
Figure 4: Sensitivity analysis on hyperparameter λ . (a) TruthfulQA remains robust to large λ ( ≥1.0 ); (b,c) Math-500 and AQUA-RAT performance degrades if λ is poorly calibrated. Appendix A lists the calibrated values.
Methods
LR
DG
CA
CM
Total Err.
KATE
35.4%
9.4%
6.5%
5.3%
56.6%
w / DeCoVec
27.0%
7.3%
8.8%
6.9%
50.0%
Δ
↓ 8.4%
↓ 2.1%
↑ 2.3%
↑ 1.6%
↓ 6.6%
Table 4: Distribution on different error types. LR: logical and reasoning flaws. DG: degeneration. CA: calculation. CM: concept and formula misuse.
Methods
Rand. Std.
Rand. Ext.
KATE
BM25
Few-Shot ICL
354.8
344.2
326.1
324.6
w / DeCoVec
314.5
333.6
312.3
300.1
Δ
↓ 40.3
↓ 10.6
↓ 13.8
↓ 24.5
Table 5: Comparison of average output tokens. DeCoVec reduces the generation length across all settings.
Math-500
AQUA-RAT
Model
Rand. Std.
Rand. Ext.
KATE
BM25
Rand. Std.
Rand. Ext.
KATE
BM25
Qwen2-0.5B
1.0
0.8
1.0
0.8
1.0
1.0
1.0
0.4
Qwen2-1.5B
0.8
0.8
0.6
0.8
0.2
1.0
0.8
1.0
Qwen2-7B
1.0
0.2
0.6
0.4
0.2
0.2
0.2
1.0
Yi-6B
0.2
0.2
0.2
0.2
0.6
1.0
0.6
0.2
Llama-2-7B
1.0
0.8
0.4
0.4
0.6
1.0
0.4
0.8
Table 6: Calibrated values of the steering intensity λ on Math-500 and AQUA-RAT. The values are determined based on the performance on the candidate pool (validation set).
Figure 5: The specific prompt used for automated error classification using an LLM evaluator.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Adaptation to Self-Consistency on Math500.
Figure 7: Prompt templates used for TruthfulQA dataset experiments.
Figure 8: Prompt templates used for Math-500 dataset experiments.
Figure 9: Prompt templates used for AQUA-RAT dataset experiments.
Figure 10: A representative question from the Math500 dataset requiring algebraic simplification using binomial expansion.
Figure 11: Generation output of the Qwen2-7B model on the Math500 algebraic simplification problem under the Zero-shot setting.
Figure 12: Generation output of the Qwen2-7B model on the same algebraic problem under the Few-shot (KATE) setting.
Figure 13: Generation output of the Qwen2-7B model steered by DeCoVec on the algebraic problem, showing the correct symbolic reasoning and an analysis of how it avoids previous errors.
Figure 14: A representative question from the Math500 dataset involving modular arithmetic embedded in a real-world money conversion context.
Figure 15: Generation output of the Qwen2-7B model on the Math500 coin conversion problem under the Zero-shot setting.
Figure 16: Generation output of the Qwen2-7B model on the same modular arithmetic problem under the Few-shot (KATE) setting.
Figure 17: Generation output of the Qwen2-7B model steered by DeCoVec on the modular arithmetic problem, showing the adoption of an integer-based strategy and an analysis of the successful reasoning.
Dataset
Rand. Std.
DeCoVec (Ours)
Overhead
AQUA-RAT
4.03s
6.70s
∼ 1.6 ×
MATH-500
8.39s
14.54s
∼ 1.7 ×
Appendix
Table 7: Measured end-to-end latency. Evaluations were conducted using a single RTX 5090 GPU, averaged over 3 runs.