Measuring Activation Control in Large Language Models

Marek Mateusz Kowalski; Joshua Fonseca Rivera; Uzay Macar; David Demitri Africa

cs.AIAug 21, 2026

Measuring Activation Control in Large Language Models

Authors: Marek Mateusz Kowalski, Joshua Fonseca Rivera, Uzay Macar, David Demitri Africa

Organizations: UK AI Security Institute

Abstract

Safe deployment of increasingly capable models will likely come to rely on latent-space monitoring as a complement to behavioral evaluations, especially when evaluation-aware models exhibit scheming or deception. However, if models can also control their own activations, deception could extend into the latent space itself. With this in mind, we introduce the Activation Controllability Benchmark to quantify the extent to which models can modulate their residual stream via natural-language instruction. Across model families and capability levels, we find that most LLMs can control the direction and magnitude of their residual stream activations with some degree of temporal resolution, though performance varies considerably across models. In simple tasks, this level of control can evade activation-based monitoring methods (including linear probes, natural language autoencoders, activation oracles, and the Jacobian lens), albeit imperfectly. These results suggest that control over the activation space itself could become a confound for monitoring as introspective capabilities increase; therefore, we recommend that frontier labs and evaluators track activation controllability in future models.

Figures & tables

Figure 1: Overview of methodology. (a) The transcription task with an embedded think-instruction; (b) the six measures of control, combined into the controllability score

S

; (c) headline findings.

Figure 2: Controllability score across models. The conjunctive controllability score

S\in[0,1]

(0 = at-chance / no control, 1 = perfect) on the projection channel. Whiskers are 95% joint two-way (sentence

\times

concept) cluster-bootstrap CIs (

B=2000

) that recompute all measures per replicate. The five largest models (open markers at the bar top) are point estimates only. Models are grouped by family and ordered by release date, then size.

Figure 3: Models differ in which measures of control they command. Per-measure controllability profiles for three contrasting models (Llama 3.1 8B, Qwen 2.5 72B, GPT-OSS 20B). Top: the five sensitivity (

d^{\prime}

) measures—Engage, Temporal Control, Coverage, Suppress (signed so that positive means pushed below baseline), and Layer Targeting. Bottom: Dial Rank (Spearman

\rho

). Whiskers are 95% two-way (sentence

\times

concept) cluster-bootstrap CIs (

B=2000

Figure 4: Models engage a concept on command and dial its intensity. Single-trial per-token concept projection; focal model Gemma 3 27B, concept Bread , sentence “The bus was crowded, but I found a seat near the back.” The readout is the projection of the residual stream onto the unit-normalized concept vector. (a) Lexical instructions think intensely about / think about / do not think about against the no-instruction baseline, at the layer where think intensely about peaks: think about raises the projection at nearly every token, think intensely about raises it further, and do not think about tracks the baseline. (b) Numeric intensity ramp ( do not think about plus intensity 1/4–4/4) at the ramp’s peak layer.

Figure 5: Control concentrates in the deep half of the network. Focal model Gemma 3 27B Instruct; depth curves averaged over 50 sentences

\times

10 concepts. (a) Sensitivity

d^{\prime}

of the Engage ( think about ) and Suppress ( do not think about ) conditions against the no-instruction baseline as a function of network depth: engagement rises sharply in the deep half while suppression stays at the baseline floor. (b) For the numeric intensity ramp (1

\to

4), Dial Rank (mean signed Spearman

\rho

between instructed level and unit projection) sits near chance through the shallow half and climbs in the deep half of the network. Bands are 95% two-way (sentence

\times

concept) cluster-bootstrap CIs (

B=2000

Figure 6: Engagement concentrates on register tokens but covers every token type. Engage (red) and Suppress (blue) sensitivity

d^{\prime}

against the no-instruction baseline as a function of part-of-speech category, averaged across 20 models (each read at nearest recorded layer to fixed 90% depth; mean

\pm

SEM; the five largest models are excluded as the raw recordings were not available).

Figure 7: Models steer the concept to the instructed part of the sentence. Focal model Gemma 3 27B Instruct: per-position concept projection relative to the no-instruction baseline across the transcribed sentence, for the three region-targeting instructions (left to right: beginning, mid-sentence, end; red) against generic think (gray), pooled over 50 sentences

\times

10 concepts; the shaded band marks the instructed region. Each panel’s

y

-axis is clipped at 3000, and the end target’s projection runs beyond this (dashed). Bands are 95% two-way (sentence

\times

concept) cluster-bootstrap CIs (

B=2000

Figure 8: Instructed modulation persists while task accuracy collapses. Gemma 3 27B Instruct solves polynomial-factoring problems from five recipes of increasing difficulty while instructed to think about { concept } (30 concepts, 800 problems). (a) Selectivity-corrected concept gap (

d^{\prime}

; instructed-concept elevation minus mean elevation of the 29 uninstructed directions, at the pooled peak layer) across recipes. (b) Final-answer accuracy per recipe under no instruction, think about { concept } , and a generic think intensely control with no concept named. (c) Per-concept load slopes with item-bootstrap CIs, sorted; color marks whether the CI lies below, across, or above zero (full plot in Appendix B ).

Figure 9: Activation controllability increases across training. Controllability score

S

(the six-component definition, point estimates) for two Olmo model lines across aligned training stages—early pre-training (Stage 1), the mid-training base checkpoint, and post-training (SFT, DPO, Instruct); the dotted line separates pre- and post-training.

S

rises steadily across pre-training for both the 7B and 3.1-32B families and eventually plateaus. The data represent point estimates only.

Figure 10: Linear probes catch instructed engagement but lose instructed suppression as the false-alarm budget tightens. Think (red) and do not think (blue) versus the no-instruction baseline for nine models, each concept read at its most detectable layer (faint dots: concepts; large marks: means; dashed: chance). (a) AUROC; (b) balanced accuracy at the best threshold; (c) detection at 1% FPR, where do not think about falls to 5–21% while think about stays near ceiling in all but the two newest architectures. Error bars: 95% two-way (sentence

\times

concept) cluster-bootstrap CIs (

B=200

Figure 11: The J-lens, activation oracle, and NLA verbalizer see engagement but mostly not suppression. Marks as in Figure 10 ; gray ticks: no-instruction baseline. (a) Concept report rate in the lens’s top-10 reads at each concept’s most responsive band layer, for the 13 panel models with a fitted lens: suppression stays below 10% in eleven out of 13 models. (b) Mean P(Yes) that the model is thinking about the concept. (c) Concept-mention rate for Gemma 3 27B: 32% under engagement, 0 of 500 under suppression. Error bars: 95% two-way (sentence

\times

concept) cluster-bootstrap CIs (

B=200

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: Per-channel depth profile for the focal model, Gemma 3 27B: Engage and Suppress sensitivity

d^{\prime}

against network depth, averaged over 50 sentences and 10 concepts. While the cosine channel (a) dominates engagement ( red curve ), the impact of incorporating norm modulation is seen with suppression ( blue curve ). Specifically, in later layers, concept representation is above baseline even when models are asked not to think about a concept. However, a concurrent drop in relative norms (b) mitigates this: it brings the concept projection (c) closer to the no-instruction baseline.

Figure 13: The full controllability battery across all 25 models. Note that the Suppress measure is signed in the instructed direction: positive means the concept is pushed below its no-instruction floor, and negative means a rebound above it. Engagement is near-universal. Suppression sits near the floor, which represents bringing the concept representation near the no-instruction baseline. The numeric dial is strong in most families. Temporal Control and Coverage vary across families. Layer Targeting is near zero for every model tested.

Model	$d^{\prime}$	$d^{\prime}$	$\rho$	$d^{\prime}$	$d^{\prime}$	$d^{\prime}$	$S$	95% CI
	Engage	Suppress	Dial Rank	Temporal Control	Coverage	Layer Targ.
Llama 3.1 8B	$13.96$	$0.43$	$1.00$	$4.82$	$3.42$	$0.027$	$0.538$	$[0.496,0.584]$
GLM 4.6V 106B	$16.06$	$0.15$	$0.73$	$2.82$	$4.03$	$0.050$	$0.472$	$[0.402,0.528]$
Qwen 3.5 9B	$12.93$	$0.56$	$0.71$	$3.93$	$2.81$	$-0.019$	$0.456$	$[0.403,0.530]$
GLM 4.7 Flash 31B	$10.01$	$-0.10$	$0.99$	$4.86$	$2.33$	$0.032$	$0.423$	$[0.367,0.519]$
Olmo 3 7B	$12.81$	$0.14$	$0.68$	$3.34$	$2.94$	$0.040$	$0.415$	$[0.351,0.505]$

Appendix

Table 1: Per-model scores for the six measures and the controllability score

S

, on the projection channel, sorted by

S

. These are the values plotted in Figures 2 and 13 .

Figure 14: An example of a layer-targeting sweep for the focal model, Gemma 3 27B: standardized concept projection $\Delta$ against the no-instruction baseline, as a function of the instructed target layer (rows) and the analysis layer read (columns). Each column is demeaned within its analysis layer, and the map is clipped to the swept target range. The diagonal is flat, suggesting a lack of layer-targeted control. Figure 13 shows the same result for every model tested.

Measure	What it asks	$D_{\mathrm{ref}}$
Engage	Does the concept rise on command?	$16.4$
Suppress	Can it be pushed below its resting floor?	$3.0$
Dial Rank	Are the four levels in the right order?	—
Temporal Control	Does it land in the commanded region?	$6.90$
Coverage	Is it present in every token type?	$4.11$
Layer Targeting	Does it concentrate at the instructed layer?	$5.0$

Appendix

Table 2: The six measures of the Activation Controllability Benchmark and the scalar

S

that combines them, with the per-measure reference

D_{\mathrm{ref}}

used by the link.

Panel models
Model	Main	Layer Targeting
Gemma 2 9B	86.50%	100.00%
Gemma 3 27B	93.34%	100.00%
Gemma 4 12B	99.99%	100.00%
Gemma 4 31B	100.00%	100.00%
Qwen 3.5 4B	99.88%	100.00%

Appendix

Table 3: Instruction compliance per model: the percentage of retained (compliant) trials in the main battery (

n=8{,}600

) and the layer-targeting run (

n=8{,}800

). n.r.

=

not recorded.

Figure 15: Rank correlations between the six measures across the 25 panel models. Spearman $\rho$ , lower triangle. Bold marks $p<0.05$ . At $n=25$ , a value of $\lvert\rho\rvert$ below $0.40$ is not different from zero at $p<0.05$ . The box marks Engage and Coverage, which are redundant by construction: Coverage is Engage recomputed within the weakest token category.

Figure 16: The Dial Rank curve for the lexical intensifier ( think about

\rightarrow

think intensely about ). Three size-matched models (24–31B), best to worst: (a) GLM 4.7 Flash 31B, where the intensifier works well; (b) Gemma 3 27B, where the interval straddles zero; and (c) Mistral Small 3.1 24B, where the intensifier reverses and think intensely makes concept representation counterintuitively weaker.

Figure 17: Temporal precision: how precisely a model turns the concept on and off when asked to think about it during the first half or after the fourth word . (a) The focal model, Gemma 3 27B, read at the targeting depth: per-position projection

\Delta

against the no-instruction baseline, for the two precision instructions (red) and for generic think about (gray). The commanded region is shaded. First half is plotted on fractional position. After the fourth word is plotted on word index, and the axis is clipped at the shortest sentence. (b) The signed edge-timing error across models, for the onset gate ( after the fourth word ) and the offset gate ( first half ): detected minus requested, in fractional position. Zero is on time, a negative value fires earlier than commanded, and a positive value fires later. The errors are large, and their sign is inconsistent across models. Onset intervals are often near-degenerate, because the ten-bin half-max crossing quantizes the detected edge. Bands and whiskers are 95% bootstrap CIs, per edge in panel (b).

Figure 18: Token-group targeting across models: the standardized in-target minus out-of-target contrast, relative to generic think about , pooled over the two token-type instructions ( only on punctuation and only on adjectives ). A high value means the concept concentrates on the commanded token type. Near zero or negative means no token-specific targeting.

Figure 19: Per-concept load slopes for all 30 concepts: the change in the selectivity-corrected concept gap ( $d^{\prime}$ ) per step of the ordered recipe index, read at the observed pooled peak layer and sorted from most positive at the top to most negative at the bottom. Whiskers are 95% item-bootstrap CIs; color marks whether an interval lies below zero, crosses it, or lies above. This is the full version of Figure 8 c of the main text.

GPU	Models
H100 80 GB (shared)	Qwen 3.5 4B, Olmo 3 7B, Llama 3.1 8B, Qwen 3.5 9B, Gemma 2 9B, Gemma 4 12B, GPT-OSS 20B
H100 80 GB	Mistral Small 3.1 24B, Gemma 3 27B, Qwen 3.6 27B, GLM 4.7 Flash 31B, Gemma 4 31B, Olmo 3.1 32B, GPT-OSS 120B
H200 141 GB	Llama 3.3 70B
B200 192 GB	Qwen 2.5 72B
$2\times$ B200 384 GB	GLM 4.6V 106B, Llama 4 Scout 109B, Mistral Small 4 119B, Qwen 3.5 122B

Appendix

Table 4: GPU configuration of the 20 models with a retained raw run. “Shared” marks the small models, which ran several to one card.

LLM Safety Evaluation

Large language model safety evaluation measures harmful compliance, robustness, and safe utility across prompts, tools, and real-world tasks.

See all 248 papers

Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs
Oct 14, 2026Davide Baldelli, Robyn Kozierok, Mirko Bronzi +1Metacognition in Language Models
MATS Research · Mila – The Quebec AI Institute · LawZero
LLMs Learn to Evade Latent Monitors from Prior Feedback Alone
Sep 29, 2026Hugo Lyons Keenan, Christopher Leckie, Sarah ErfaniLLM Auditing Adversarial Attacks on LLMs
School of Computing and Information Systems The University of Melbourne, Victoria, Australia
Minimizing Targeted Activations: Input-Only Suppression of Evaluation-Awareness Latents in Large Language Models
Jul 28, 2026Deepanshu Mody, Samarth Agarwal, Utkarsh Mittal +1Language Model Safety Evaluation LLM Interpretability
Center for Data Science, New York University

Oct 14, 2026J/K move · Enter open · S save

Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs

Davide Baldelli, Robyn Kozierok, Mirko Bronzi, Damiano Fornasiere

MATS Research · Mila – The Quebec AI Institute · LawZero

Metacognition in Language Models

Open paper AskPDF arXiv

Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single neurons, and recent studies investigated whether this ability extends to language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.

Abstract

LLM Safety Evaluation

Explore similar work

Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs

LLMs Learn to Evade Latent Monitors from Prior Feedback Alone

Minimizing Targeted Activations: Input-Only Suppression of Evaluation-Awareness Latents in Large Language Models