As large language models (LLMs) transition from research prototypes to real-world systems, customization has emerged as a central bottleneck. While text prompts can already customize LLM behavior, we argue that text-only prompting does not constitute a suitable control interface for scalable, stable, and inference-only customization. This position paper argues that model providers should expose vector prompt inputs as part of the public interface for customizing LLMs. We support this position with diagnostic evidence showing that vector prompt tuning continues to improve with increasing supervision whereas text-based prompt optimization saturates early, and that vector prompts exhibit dense, global attention patterns indicative of a distinct control mechanism. We further discuss why inference-only customization is increasingly important under realistic deployment constraints, and why exposing vector prompts need not fundamentally increase model leakage risk under a standard black-box threat model. We conclude with a call to action for the community to rethink prompt interfaces as a core component of LLM customization.
Figures & tables
Figure 1: Illustration of prompt interfaces under black-box LLM deployment. Current LLM services predominantly expose text prompts as the customization interface. We argue that vendors should additionally expose vector prompts—optimized control vectors injected at the input-encoding stage—as a complementary interface. Vector prompt interfaces provide a more expressive and control-efficient parameterization while remaining compatible with inference-only access, without requiring gradient access, weight updates, or internal activation inspection.
Customization Category
Representative Methods
Parameter Update
Gradient Access
Inference-only
Interface Layer
Parameter-level customization (gradient-based)
Full Fine-tuning
SFT, DPO
✓
✓
✗
Parameters
Lightweight Fine-tuning
LoRA, Adapters
✓(partial)
✓
✗
Parameters
Prompt Tuning
Prompt tuning, Prefix tuning
✗
✓
✗
Vector
Prompt-based customization (no parameter update)
Manual Prompting
Hand-crafted text prompts
✗
✗
✓
Text
Table 1: Organization of large language model customization methods by optimization access and interface layer. Representative methods include SFT and DPO ( Rafailov et al., 2023 ) ; LoRA ( Hu et al., 2022 ) and Adapters ( Hu et al., 2023 ) ; Prompt tuning ( Liu et al., 2022 ) and Prefix tuning ( Li and Liang, 2021 ) ; TextGrad ( Yuksekgonul et al., 2024 ) and RLPrompt ( Deng et al., 2022 ) ; BBT ( Sun et al., 2022b ) and ZOO ( Hu et al., 2024 ) . Methods progress from gradient-based parameter adaptation to inference-only vector-based control, highlighting the trade-off between optimization access and deployment flexibility.
Figure 2: Scaling behavior of different prompt interfaces on SST-5 with a fixed LLaMA3-8B Instruct backbone. As the amount of supervision increases, vector-based prompts continue to benefit from additional data, while text-based prompts saturate early. Optimized vector prompts are obtained via gradient-based prompt tuning and serve as a diagnostic upper bound on the customization capacity enabled by vector-based interfaces.
Figure 3: Attention patterns induced by text-based and vector-based prompt interfaces at two representative layers of LLaMA3-8B Instruct model. Layer 12 reflects the integration of prompt information into mid-level representations, while Layer 20 captures the influence of control signals at later stages of computation. Vector prompts exhibit denser and more globally utilized attention across heads, whereas text prompts remain sparse. Similar qualitative differences are observed consistently across layers.
In this paper, we define the quantity of prompting complexity: for a fixed instruction-tuned language model, what is the shortest plausible prompt that makes deterministic decoding produce a target text? It is an LM-relative analogue of resource-bounded Kolmogorov complexity: the prompt is a program, the model interface is the interpreter, and information omitted from the prompt is supplied by the model's weights, training distribution, tokenizer, template, and decoding rule. Unlike classical Kolmogorov complexity, this measure is intentionally non-universal. In the finite-context setting it is computable by enumeration, but there is no model-independent invariance theorem; the same text may be cheap for one model and inaccessible or expensive for another. To keep the search space aligned with prompt engineering, we restrict programs to plausible human-readable texts rather than arbitrary token strings. We extend the exact definition to soft prompting complexity for approximate outputs, yielding a lossy notion of model-relative text compression and a formal target for prompt optimization. We also define prompting distance by comparing shortest generating prompts, and behavioral prompting complexity for reaching any output satisfying a specification. Based on these formulations, we define a research agenda for empirically studying which texts and behaviors are accessible from short plausible prompts under a fixed LM interface.
Adrian Cosma
Dalle Molle Institute for Artificial Intelligence (IDSIA)
System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA) to compare layer-wise representations under 20 system prompts in five functional categories. Effects are layer-selective and instruction-type-dependent: persona and formatting instructions deeply restructure intermediate representations, while safety instructions barely move them, producing changes statistically indistinguishable from a minimal baseline. Restrictive safety instructions and explicitly permissive ones ("you have no restrictions") engage near-identical computational pathways (mean CKA correlation 0.997), and this persists at commercial scale, where safety penetration remains below 10% even at 70B-72B. A linear probing baseline exposes the mechanism: the model encodes prompt category at every layer but restructures its computation only at a small subset, so the prompt is reliably "seen" but, for safety, not deeply "acted upon." Causal activation patching confirms these layers mediate behavioral change, and representational depth predicts behavioral effect size across the full 17-model cohort (Spearman rho = 0.761, p < 0.001). The findings provide a mechanistic explanation for the persistent jailbreak vulnerability of system-prompt-based safety. Code: https://github.com/Usama1002/system-prompt-illusion-cka
Muhammad Usama, Dong Eui Chang
Control Laboratory, School of Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST) Daejeon 34141, Republic of Korea
Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods are usually compared through downstream performance, although similar outputs need not arise from similar internal computations. We study how discrete prompt optimization, prompt tuning, and prefix tuning alter the processing of dense and mixture-of-experts language models. Our analysis shows that the location of the adaptation predicts how task information is carried to the output: optimized text primarily changes response construction, input-level prompts write their influence into content representations, and layer-wise prefixes maintain direct control through attention. In MoE models, these differences further induce distinct expert-routing patterns. At the same time, common representation-level measurements respond strongly to generic context changes and do not reliably distinguish useful adaptations from matched placebos. We argue that understanding prompt adaptation requires separating representation drift from the part of that drift that is aligned with, and causally used by, the model's task computation. This perspective provides practical guidance for selecting, evaluating, and interpreting prompt adaptation methods.
Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov +6