ADAPT: Hybrid Prompt Optimization for LLM Feature Visualization

João N. Cardoso; Arlindo L. Oliveira; Bruno Martins

cs.LGFeb 19, 2026

ADAPT: Hybrid Prompt Optimization for LLM Feature Visualization

Authors: João N. Cardoso, Arlindo L. Oliveira, Bruno Martins

Organizations: Instituto Superior T´ecnico · INESC-ID

Abstract

Understanding what features are encoded by learned directions in LLM activation space requires identifying inputs that strongly activate them. Feature visualization, which optimizes inputs to maximally activate a target direction, offers an alternative to costly dataset search approaches, but remains underexplored for LLMs due to the discrete nature of text. Furthermore, existing prompt optimization techniques are poorly suited to this domain, which is highly prone to local minima. To overcome these limitations, we introduce ADAPT, a hybrid method combining beam search initialization with adaptive gradient-guided mutation, designed around these failure modes. We evaluate on Sparse Autoencoder latents from Gemma 2 2B, proposing metrics grounded in dataset activation statistics to enable rigorous comparison, and show that ADAPT consistently outperforms prior methods across layers and latent types. Our results establish that feature visualization for LLMs is tractable, but requires design assumptions tailored to the domain.

Figures & tables

Figure 1: Diagram for ADAPT.

Figure 2: Distribution of top dataset activations for latents 0–49 (by Neuronpedia index) across all layers of Gemma 2 2B.

Figure 3: Mean activation ratio (the optimized prompt’s activation divided by the maximally activating dataset example) per layer, for all 4 methods under study.

	GCG	BEAST	EPO	ADAPT
GCG	–	56.4%	63.2%	29.6%
BEAST	43.6%	–	50.2%	12.8%
EPO	36.8%	49.8%	–	21.4%
ADAPT	70.4%	84.0%	78.6%	–

Table 1: Row beats column comparison for all 4 methods, considering all 486 latents.

Panel of Figure 4 — Figure 4: Comparative ranking matrices for GCG-ADAPT and GCG-BEAST comparisons.

Panel of Figure 5 — Figure 5: Quadrant success per layer for GCG-BEAST and GCG-ADAPT comparisons.

Figure 12: Activation ratio distribution for GCG runs with varying fluency penalty weights.

Figure 13: Rank/rating co-occurrence matrix for GCG, GCG with a

\lambda=1.0

fluency penalty, and ADAPT, respectively.

Figure 14: Rating distribution for GCG, GCG with a

\lambda=1.0

fluency penalty, and ADAPT, respectively.

Figure 15: Quality rating for each method averaged over all latents and annotators.

Latent A	Latent B	Optimized Prompt
823 (MCU)	2738 (bit)	nearby Avenger/avenulla Rd Avengers is bits
1264 (syn)	154 (magnets)	ul magnetism currently ( determinediuminium, syn
823 (MCU)	154 (magnets)	times aanki Stark magnet first unveiled as the

Table 2: Optimized prompts for dual-latent objectives. All latents from layer 25. Prompts successfully activate both target latents, with local features (2738, 1264) appearing as specific tokens and low-locality features (823, 154) reflected in semantic context.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

General
Parameter	GCG	BEAST	EPO	ADAPT
Iterations	50	10	50	25
Prompt length	10	10	8–15	10
Parallel prompts	6	–	6	10
Candidate Generation
Candidates/iter	32	–	32	32

Appendix

Table 3: Hyperparameter settings for all evaluated methods.

Figure 16: Annotation task example for the ‘magnet’ latent.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 761 papers

Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs
Oct 13, 2026Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov +6Prefix Tuning LLM Prompting
BRAIn Lab
Adaptive Prompt Structure Factorization: A Framework for Self-Discovering and Optimizing Compositional Prompt Programs
Apr 8, 2026Haoyue Liu, Zhichao Wang, Yongxin Guo +2LLM Input Optimization LLM Prompting
The Chinese University of Hong Kong, Shenzhen · Taobao and Tmall Group
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
Jul 23, 2026Yidu Wu, Xiang Wang, Kejie Zhao +3LLM Pruning LLM Inference Acceleration
Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China · ACS Lab, Huawei Technologies Co., Ltd., Shenzhen, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China

Oct 13, 2026J/K move · Enter open · S save

Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs

Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov, Andrew Tsukanov+5

BRAIn Lab

Prefix Tuning LLM Prompting Prompt Tuning Representation Drift

Open paper AskPDF arXiv

Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods are usually compared through downstream performance, although similar outputs need not arise from similar internal computations. We study how discrete prompt optimization, prompt tuning, and prefix tuning alter the processing of dense and mixture-of-experts language models. Our analysis shows that the location of the adaptation predicts how task information is carried to the output: optimized text primarily changes response construction, input-level prompts write their influence into content representations, and layer-wise prefixes maintain direct control through attention. In MoE models, these differences further induce distinct expert-routing patterns. At the same time, common representation-level measurements respond strongly to generic context changes and do not reliably distinguish useful adaptations from matched placebos. We argue that understanding prompt adaptation requires separating representation drift from the part of that drift that is aligned with, and causally used by, the model's task computation. This perspective provides practical guidance for selecting, evaluating, and interpreting prompt adaptation methods.

Abstract

LLM Interpretability

Explore similar work

Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs

Adaptive Prompt Structure Factorization: A Framework for Self-Discovering and Optimizing Compositional Prompt Programs

Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs