Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs

Mikhail Aleksandrov; Nail Bashirov; Maksim Ivanov; Andrew Tsukanov; Timofey Karyagin; Alexander Miasnikov; Gleb Molodtsov; Valentin Samokhin

cs.LGOct 13, 2026

Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs

Authors: Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov, Andrew Tsukanov, Timofey Karyagin, Alexander Miasnikov, Gleb Molodtsov, Valentin Samokhin,

Organizations: BRAIn Lab

Abstract

Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods are usually compared through downstream performance, although similar outputs need not arise from similar internal computations. We study how discrete prompt optimization, prompt tuning, and prefix tuning alter the processing of dense and mixture-of-experts language models. Our analysis shows that the location of the adaptation predicts how task information is carried to the output: optimized text primarily changes response construction, input-level prompts write their influence into content representations, and layer-wise prefixes maintain direct control through attention. In MoE models, these differences further induce distinct expert-routing patterns. At the same time, common representation-level measurements respond strongly to generic context changes and do not reliably distinguish useful adaptations from matched placebos. We argue that understanding prompt adaptation requires separating representation drift from the part of that drift that is aligned with, and causally used by, the model's task computation. This perspective provides practical guidance for selecting, evaluating, and interpreting prompt adaptation methods.

Figures & tables

Panel of Figure 1 — Figure 1: Behavioral gains and pre-existing task information. Ignoring label order removes most of GEPA’s strict-score improvement, while a seed-state probe decodes the label set well before the final block.

Panel of Figure 2 — Figure 2: Availability and use of the adapted context. The tuned lens tracks the emergence of adapted label margins, while masking removes the gain when access to the corresponding context is blocked.

Figure 3: Dataset-mean steering and its random control.

Conditions	$R_{\mathrm{KL}}$
GEPA–prompt	$0.917\pm 0.049$
GEPA–prefix	$0.832\pm 0.056$

Table 1: Exact transfer at block 20.

Target	Native	MSE	MSE+KL
GEPA	$0.678\pm.007$	$0.683\pm.015$	$0.681\pm.012$
Prompt	$0.777\pm.013$	$0.746\pm.001$	$0.759\pm.006$
Prefix	$0.805\pm.007$	$0.744\pm.004$	$0.765\pm.006$

Table 2: Complete-answer set samples-

F_{1}

Figure 4: Routing displacement

\mathrm{TV}_{\ell}

from the frozen base, layer by layer, for the comment tokens that all arms share; continuous arms at 500 virtual tokens.

		positions per row	Gini	Gini, comment tokens	busiest expert (%)	dead experts
Qwen3-30B-A3B	base	153	0.635	0.757	5.2	7.8
	GEPA	2154	0.729	0.759	8.2	7.6
	prompt, 100	253	0.583	0.753	5.0	3.0
	prompt, 200	353	0.610	0.747	6.1	4.5
	prompt, 500	653	0.620	0.753	6.6	3.0
	prefix, 100	153	0.640	0.760	5.8	8.4

Table 3: Expert load on 500 calibration rows, every budget. Gini coefficients and busiest-expert shares are medians over layers; dead-expert counts are means over layers.

arm	answer only	with reasoning
prompt, 100	0.784	0.482
prompt, 500	0.799	0.775
prefix, 100	0.819	0.789
prefix, 500	0.820	0.764

Table 4: gpt-oss test

F_{1}

, 2000 rows. Reasoning LR: answer-only for prompts,

3\cdot 10^{-4}

for prefixes.

		$F_{1}$			unreadable (%)		super experts
		original	$-50\%$	$-75\%$	$-50\%$	$-75\%$	kept	pruned
Qwen3-30B-A3B	base	0.466	0.575	0.330	0.0	100.0	all three	0.328
	GEPA	0.728	0.728	0.330	0.0	100.0	all three	0.328
	prompt, 100	0.833	0.814	0.440	0.0	0.2	L2 E92, L3 E82	0.409
	prompt, 200	0.816	0.330	0.330	100.0	100.0	all three	0.328
	prompt, 500	0.828	0.812	0.450	0.0	0.0	all three	0.328

Table 5: Set-level

F_{1}

on 2000 test rows after pruning each layer’s least-used 50% or 75% of experts, with unreadable-answer rates;

F_{1}

after pruning only super experts, alongside those used by each arm. Token budgets (Qwen/gpt-oss): 512/1024 for unpruned and frequency-pruned models, 64/1024 for super-expert pruning; unreadable answers in the latter count as empty label sets. Empty-label floors: 0.330 for frequency pruning and 0.328 for super-expert pruning, using different test samples.

Figure 5: Share of attention by layer on Qwen, intact and with the super experts pruned.

	Qwen3-30B-A3B			gpt-oss-20b
	L1 E68	L2 E92	L3 E82	L6 E5	L17 E5
base	422	800	458	43 000	52 700
GEPA	422	800	458	43 000	52 700
prompt, 100	11.1	250	572	43 000	50 700
prompt, 200	470	764	472	43 000	51 500
prompt, 500	1000	952	572	43 000	46 900

Table 6: Largest output of each super expert over 200 rows.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

	discrete (GEPA)		prompt tuning		prefix tuning
$n$	strict	set	strict	set	strict	set
$200$	0.6077	0.6604	0.7192	0.7336	0.7443	0.7445
$500$	0.5888	0.6526	0.7662	0.7717	0.7951	0.7952
$1000$	0.6190	0.6598	0.8003	0.8017	0.8117	0.8118

Appendix

Table 7: Held-out test performance,

3000

rows, empty-aware samples

F_{1}

under both parsers, used in Figure 1(a) . The unadapted seed scores

0.2837

strict and

0.6092

set. The parser-dependent gain reduction

\rho

of 7 is

0.85

for the discrete branch. GEPA entries average over three optimizer seeds. Continuous entries average over

m\in\{100,200,500\}

and three seeds.

branch	$R$ [95% CI]	norm-matched random
prefix tuning ( $m=500$ )	$+0.019$ $[-0.009,+0.052]$	$+0.019$ $[-0.004,+0.043]$
discrete (GEPA)	$+0.023$ $[-0.117,+0.174]$	$+0.024$ $[-0.083,+0.127]$

Appendix

Table 8: Recovered fraction

R_{\mathrm{task}}

of 8 , aggregated over

12

cells per branch, against a norm-matched random direction injected at the same site, used in Figure 3 .

Figure 6: Representative access profiles for the adapted contexts. Curves show cumulative attention within the GEPA instruction (left) and learned continuous contexts (right) at selected blocks.

Panel of Figure 7 — Figure 7: A matched context placebo separates representation drift from task utility. Padding reproduces prominent geometric signatures without improving the task.

Figure 8: Per-example cosine between seed-relative shifts from two adaptation methods. Each panel pools 1000 fixed examples. Early-layer shifts have small norms, so their directions are less stable.

condition	raw	centered
GEPA	0.902	0.702
prompt tuning ( $m=500$ )	0.830	0.473
prefix tuning ( $m=500$ )	0.497	0.155
$\mathcal{C}_{\mathrm{pad}}$	0.961	0.845

Appendix

Table 9: Mean per-example cosine similarity with the seed-prompt residual at layer 18 over three seeds. Centered states have their dataset mean removed.

Figure 9: Exact-state, SAE-reconstructed, and Top-128 feature transfer. The Top-128 subset is selected by displacement magnitude on disjoint calibration examples.

		Repredict		Fixed
Target	Native	MSE	MSE+KL	MSE	MSE+KL
GEPA	$0.678\pm.007$	$0.683\pm.015$	$0.681\pm.012$	$0.638\pm.008$	$0.683\pm.011$
Prompt	$0.777\pm.013$	$0.746\pm.001$	$0.759\pm.006$	$0.637\pm.015$	$0.756\pm.003$
Prefix	$0.805\pm.007$	$0.744\pm.004$	$0.765\pm.006$	$0.506\pm.121$	$0.755\pm.007$

Appendix

Table 10: Predicted final-state corrections at block 25 during free generation on

3000

test examples. Entries are mean and sample SD over adaptation seeds

42,43,44

. Native denotes the adapted model. This predictor suite reports a separately evaluated seed baseline of

0.617

; the performance sweep in Table 7 reports

0.6092

, so we report raw scores rather than combine the two baselines. Target adapters use

n=1000

, with

m=20

virtual tokens for the continuous methods. The score is empty-aware set samples-

F_{1}

, with invalid outputs assigned zero.

Regime	Target	$r=1$	$r=2$	$r=4$	$r=8$	full
Repredict	GEPA	0.634	0.640	0.645	0.660	0.683
	Prompt	0.596	0.655	0.664	0.699	0.746
	Prefix	0.677	0.683	0.688	0.700	0.741
Fixed	GEPA	0.635	0.635	0.636	0.640	0.641
	Prompt	0.580	0.577	0.595	0.608	0.619
	Prefix	0.353	0.346	0.371	0.391	0.389

Appendix

Table 11: Task score by predictor rank, seed 42, MSE objective,

3000

test examples. The metric matches Table 10 , and the manual baseline is

0.617

. All predictors operate at block 25.

Target	Native	Block-25 correction	Early exit
GEPA	$0.678\pm.007$	$0.681\pm.012$	$0.688\pm.011$
Prompt	$0.777\pm.013$	$0.759\pm.006$	$0.766\pm.012$
Prefix	$0.805\pm.007$	$0.765\pm.006$	$0.770\pm.012$

Appendix

Table 12: Early exit from block 20 against the block-25 correction,

3000

test examples, mean and sample SD over seeds

42,43,44

. The metric matches Table 10 .

Condition	Native (ms)	Early exit (ms)
Seed	$52.6$	–
GEPA	$303.7$	$43.0$
Prompt	$55.3$	$43.0$
Prefix	$54.3$	$43.1$

Appendix

Table 13: Time per example with

16

generated tokens,

100

test examples, batch size

16

, mean over two repeats.

Pair	MSE	MSE+KL
GEPA, prompt	0.32	0.29
GEPA, prefix	0.27	0.24
Prompt, prefix	0.49	0.57
GEPA seeds	0.72	0.67
Prompt seeds	0.67	0.76
Prefix seeds	0.61	0.68

Appendix

Table 14: Cosine between flattened block-25 predictor matrices

W_{M}

(repredict, full rank).

Method	Predictor	Native	One-shot	Fixed	Repredict
GEPA	Linear	$0.684\pm.004$	$0.641\pm.009$	$0.694\pm.007$	$0.690\pm.008$
GEPA	MLP	$0.684\pm.004$	$0.630\pm.009$	$0.695\pm.018$	$0.692\pm.009$
Prompt	Linear	$0.838\pm.004$	$0.663\pm.004$	$0.729\pm.012$	$0.761\pm.010$
Prompt	MLP	$0.838\pm.004$	$0.648\pm.005$	$0.747\pm.005$	$0.744\pm.009$
Prefix	Linear	$0.851\pm.006$	$0.657\pm.007$	$0.755\pm.005$	$0.765\pm.007$
Prefix	MLP	$0.851\pm.006$	$0.637\pm.009$	$0.757\pm.004$	$0.754\pm.011$

Appendix

Table 15: Strict samples-

F_{1}

of the seed model patched at block 25 with a predicted shift, mean

\pm

SD over three adaptation seeds on the same

512

test examples. Native is the adapted model itself. The seed model scores

0.624

model	arm	level (%)	mass (%)	$F_{1}$	exact	empty	unreadable	no end	looping
Qwen	base	$0$	0.0	0.466	0.211	26.5	0.0	0.0	0.0
	base	$-50$	5.8	0.575	0.167	21.1	0.0	0.1	0.0
	base	$-75$	28.3	0.330	0.330	100.0	100.0	61.9	21.7
	GEPA	$-50$	2.0	0.728	0.583	24.1	0.0	0.0	0.0
	GEPA	$-75$	17.8	0.330	0.330	100.0	100.0	100.0	94.7
	prompt, 100	$-50$	11.4	0.814	0.696	31.8	0.0	0.0	0.0

Appendix

Table 16: Frequency pruning, all levels. Mass is the routing mass removed, averaged over layers. Empty (answers with no label, unreadable ones included), unreadable and no end marker are shares of the 2000 test answers, looping (a phrase repeated until the budget runs out) of the first 300. Pruned runs use 512 generated tokens on Qwen and 1024 on gpt-oss (reasoning effort low on every arm).

	Qwen3-30B-A3B		gpt-oss-20b
	$-50\%$	$-75\%$	$-50\%$	$-75\%$
base	L1 E68, L2 E92	all three	—	—
GEPA	L1 E68	all three	—	—
prompt, 100	L1 E68	L1 E68, L3 E82	—	L6 E5
prompt, 200	L1 E68, L2 E92	L1 E68, L2 E92	—	—
prompt, 500	L1 E68	L1 E68, L2 E92	—	—

Appendix

Table 17: Super experts removed by the frequency masks of Table 5 .

Figure 10: As Figure 5 , for a Qwen prefix of 500 tokens, untrained and trained at two learning rates.

Figure 11: Attention shares on Qwen, layers 1 to 5, 200 Civil test rows: intact, with the super expert removed, and with a random perturbation of the same norm.

Figure 12: As Figure 11 , on WikiText-2.

Figure 13: As Figure 5 , on gpt-oss, full-attention layers 7 to 17.

Figure 14: gpt-oss: range over layers 0 to 3 of the attention share taken by the learned sink and by prefix keys, per arm, with the state of the super experts.

	WikiText-2 perplexity			$F_{1}$
prefix	intact	values zeroed	keys masked	intact	values zeroed	keys masked
untrained	48.0	7.22	6.87	—	—	—
LR $3\cdot 10^{-5}$	34.0	7.48	7.16	0.807	0.412	0.509
LR $10^{-4}$	275	8.95	8.63	0.807	0.759	0.766

Appendix

Table 18: Qwen prefixes of 500 tokens with the prefix of layers 0 to 5 removed, by zeroing its values or masking its keys, at every position. Perplexity on the 16 WikiText-2 segments of 2048 tokens, strict

F_{1}

on the 2000 test rows (the untrained prefix was not scored on them). The base scores

6.97

and

0.453

arm	frozen	retrained	difference ( $t$ )	$\mathrm{TV}$ , retrained vs frozen
base	0.461	0.800	$+0.339$ ( $39.4$ )	0.080
GEPA	0.730	0.803	$+0.073$ ( $8.8$ )	0.076
prompt, 100	0.829	0.824	$-0.005$ ( $-1.5$ )	0.030
prompt, 200	0.813	0.820	$+0.007$ ( $1.6$ )	0.014
prompt, 500	0.832	0.822	$-0.009$ ( $-3.2$ )	0.011
prefix, 100	0.830	0.832	$+0.002$ ( $0.3$ )	0.057

Appendix

Table 19: Retraining only the router on top of each Qwen arm: set-level

F_{1}

on 2000 test rows with the frozen and the retrained router, the paired difference, and the routing displacement of 6 that the retraining itself causes on the comment tokens of the validation rows, with a 64-token generation budget.

	gpt-oss-20b			Qwen3-30B-A3B
GPUs	2	4	8	2	4	8
uniform load	0.956	0.853	0.707	0.975	0.938	0.821
GEPA	1.010	1.040	1.109	1.008	1.046	1.062
prompt, 100	1.020	1.038	1.071	1.004	1.000	0.979
prompt, 200	1.038	1.039	1.096	1.014	1.014	0.993
prompt, 500	1.042	1.061	1.135	1.019	1.026	1.007

Appendix

Table 20: Time of the MoE layers per step under expert parallelism relative to the base, 4096 tokens per H100 GPU, 20 repetitions; the last row is the base’s absolute time.

recipient $\leftarrow$ donor	gap	support	coefficients	joint	matched random	shuffled
base $\leftarrow$ prompt, 500	$+0.373$	$-0.014$	$+0.005$	$-0.002$	$-0.012$	$-0.057$
		$[-0.025,-0.003]$	$[-0.006,+0.015]$	$[-0.013,+0.010]$	$[-0.022,-0.002]$	$[-0.079,-0.035]$
prompt, 500 $\leftarrow$ base	$-0.373$	$-0.011$	$-0.013$	$-0.014$	$-0.015$	$-0.237$
		$[-0.018,-0.004]$	$[-0.020,-0.005]$	$[-0.023,-0.005]$	$[-0.023,-0.007]$	$[-0.260,-0.215]$
base $\leftarrow$ prefix, 500	$+0.348$	$+0.007$	$+0.031$	$+0.034$	$+0.014$	$-0.063$
		$[-0.002,+0.017]$	$[+0.022,+0.041]$	$[+0.022,+0.045]$	$[+0.005,+0.024]$	$[-0.085,-0.042]$

Appendix

Table 21: Route replay on Qwen at the prefill of the user turn, 2000 test rows: change in the recipient’s

F_{1}

against its own native answers, with paired bootstrap 95% intervals below. Gap is the donor’s native

F_{1}

minus the recipient’s. Shuffled uses the 1764 rows with a donor row of the same geometry.

Prefix Tuning

See all 10 papers

Oct 1, 2026

Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptations

As large language models are increasingly deployed across diverse downstream tasks, efficient task adaptation has emerged as a central challenge. In response, a wide range of task adaptation methods have been proposed, spanning parameter-efficient fine-tuning, in-context learning, and embedding-injection approaches. However, these lines of work have largely evolved within individual paradigms, leaving their cross-paradigm relationships and trade-offs underexplored, especially for recently emerging embedding-based adaptations. This survey presents a unified framework that categorizes task adaptation methods by where and how task information is encoded: model weights, input prompts, or injected task embeddings. We provide a comprehensive taxonomy that integrates these paradigms, analyze their key strengths and limitations to explain how different adaptation paradigms have evolved, clarify relationships across paradigms, and highlight open problems for future research.

Jungwon Park, Changin Choi, Jimyeong Kim +2

RICS · Daegu Gyeongbuk Institute of Science and Technology · IPAI +3

LLM Fine-Tuning In-Context Learning Parameter-Efficient Fine-Tuning

Ask PDF

Feb 19, 2026

ADAPT: Hybrid Prompt Optimization for LLM Feature Visualization

Understanding what features are encoded by learned directions in LLM activation space requires identifying inputs that strongly activate them. Feature visualization, which optimizes inputs to maximally activate a target direction, offers an alternative to costly dataset search approaches, but remains underexplored for LLMs due to the discrete nature of text. Furthermore, existing prompt optimization techniques are poorly suited to this domain, which is highly prone to local minima. To overcome these limitations, we introduce ADAPT, a hybrid method combining beam search initialization with adaptive gradient-guided mutation, designed around these failure modes. We evaluate on Sparse Autoencoder latents from Gemma 2 2B, proposing metrics grounded in dataset activation statistics to enable rigorous comparison, and show that ADAPT consistently outperforms prior methods across layers and latent types. Our results establish that feature visualization for LLMs is tractable, but requires design assumptions tailored to the domain.

João N. Cardoso, Arlindo L. Oliveira, Bruno Martins

Instituto Superior T´ecnico · INESC-ID

LLM Interpretability Prompt Optimization Representation Geometry in Language Models

Ask PDF

Jul 31, 2026

TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation

Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation. Yet current approaches have two limitations: task experts are typically identified from aggregate routing statistics that reflect usage rather than association with successful task completion, and task-expert activations remain underexplored as signals for supervision allocation. We introduce Task-Expert-Aware Supervision (TEXAS), which combines correctness-conditioned task expert discovery with token-level supervision allocation. TEXAS compares expert activations on instances that the base model solves successfully and those it fails to solve, and retains experts more strongly activated on successful instances. During fine-tuning, it upweights answer tokens in failed instances when they activate these experts. TEXAS therefore leverages existing routing behavior without restricting adaptation to a fixed expert subset or imposing an explicit target routing distribution. Across three MoE models and six benchmarks, TEXAS achieves the best or tied-best performance in 17 of 18 settings and improves over the strongest baseline by 1.3--1.5 points on average. Ablations and further analyses validate both the discovered experts and the resulting supervision strategy.

Guanzhi Deng, Haibo Wang, Kuan Wu +5

City University of Hong Kong, Hong Kong, China · Carnegie Mellon University, Pittsburgh, USA · University of Waterloo, Ontario, Canada +1

Mixture-of-Experts Language Models Mixture of Experts LLM Fine-Tuning Adaptive Loss Weighting

Ask PDF

Abstract

Prefix Tuning

Explore similar work

Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptations

ADAPT: Hybrid Prompt Optimization for LLM Feature Visualization

TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation