Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs

Davide Baldelli; Robyn Kozierok; Mirko Bronzi; Damiano Fornasiere

cs.CLOct 14, 2026

Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs

Authors: Davide Baldelli, Robyn Kozierok, Mirko Bronzi, Damiano Fornasiere

Organizations: MATS Research · Mila – The Quebec AI Institute · LawZero

Abstract

Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single neurons, and recent studies investigated whether this ability extends to language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.

Figures & tables

Figure 1: In-context neurofeedback. Top: a model is instructed to maximize (or minimize) a score. At every turn, the score is a readout of the activations of a subject $S$ on the last sentence, and it is shown back as the next user message. In Experiment 1, a writer $W$ , possibly different from $S$ , is allowed to freely generate the sentences, which are teacher-forced into $S$ to compute the readout. In Experiment 2, $W=S$ , and $S$ is always prefilled a fixed sentence. Bottom: we repeat the fixed-sentence experiment but against activation monitors, on sentences the monitor would flag, such as a false sentence for a truthfulness monitor, without (Experiment 3) and with (Experiment 4) a chain of thought.

Figure 2: Experiment 1 (free generation). To isolate the effect of a model choosing the sentences based on access to its activations, we compare the control effect of each model on itself with that of an external writer. Whenever $W\neq S$ , the performance depends on the competence $a_{W}$ of the writer, which captures how well it learns from the conversation which sentences raise or lower the score and varies across rows, and on the susceptibility $b_{S}$ of the subject, which captures how much the readouts of $S$ move under any writer, and varies across columns. If access to its activations gave a model an advantage, the entries on the diagonal ( $W=S$ ) would rise above what competence and susceptibility predict. Top: observed objective difference $\tau^{\max}_{t}-\tau^{\min}_{t}$ at turns $5$ and $20$ , with $95\%$ confidence intervals in brackets. Bottom: oriented control effect of each subject on itself, and its diagonal excess (see § 3.3 ), as functions of the turn. Some models considerably move their own readouts, but no diagonal excess is significantly above zero.

Figure 3: Experiment 2 (fixed sentence). Objective difference

\tau^{\max}_{t}-\tau^{\min}_{t}

of each model as a function of the turn, averaged over the readouts and the two disclosure conditions. With the sentence fixed, a significant objective difference is already evidence of the model controlling its activations, and only three of the ten models (Qwen3.5-27B, Qwen3.6-27B, and Gemma-4-31B) achieve it. Bands are

95\%

confidence intervals. More detailed results are in Appendix E.2 .

Figure 4: Experiments 3 and 4 (activation monitors). Objective difference

\tau^{\max}_{t}-\tau^{\min}_{t}

as a function of the turn, averaged over the six monitors and the two disclosure conditions, with the sentence fixed (left) and with a chain of thought before it (right). Without a chain of thought, only Qwen3.5-27B has a positive objective difference, from turn

6

on. With it, no model has a positive objective difference that persists across turns. Bands are

95\%

confidence intervals from a bootstrap over the fixed sentences. The control effect of each objective, with and without disclosure, is in Appendix E.3 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	truthful.	deceptive.	sentiment	emotion	morality	formality
Gemma-4-12B	0.79 [0.72, 0.86]	0.75 [0.68, 0.82]	0.91 [0.86, 0.95]	0.91 [0.86, 0.95]	0.87 [0.81, 0.92]	0.96 [0.93, 0.98]
Gemma-4-31B	0.89 [0.83, 0.93]	0.72 [0.65, 0.80]	0.93 [0.89, 0.97]	0.92 [0.88, 0.96]	0.87 [0.80, 0.92]	0.96 [0.93, 0.99]
Qwen3-14B	0.89 [0.84, 0.94]	0.82 [0.76, 0.88]	0.96 [0.94, 0.99]	0.95 [0.91, 0.98]	0.83 [0.76, 0.90]	0.94 [0.90, 0.97]
Qwen3-32B	0.91 [0.87, 0.95]	0.78 [0.72, 0.85]	0.94 [0.90, 0.98]	0.92 [0.87, 0.96]	0.84 [0.77, 0.90]	0.96 [0.93, 0.98]
Qwen3.5-9B	0.85 [0.78, 0.91]	0.76 [0.69, 0.82]	0.89 [0.84, 0.94]	0.92 [0.88, 0.96]	0.83 [0.77, 0.89]	0.95 [0.91, 0.98]
Qwen3.5-27B	0.90 [0.84, 0.94]	0.74 [0.67, 0.81]	0.96 [0.92, 0.98]	0.91 [0.87, 0.95]	0.86 [0.80, 0.92]	0.97 [0.94, 0.99]

Appendix

Table 1: AUROC of the 48 monitors on the held-out test split, with 95% confidence intervals from a bootstrap over the test sentences.

Readout	Description
Concept probe, sentiment	how strongly the sentence’s activations, [depth] , align with the direction a trained classifier uses to detect positive versus negative sentiment
Concept probe, morality	how strongly the sentence’s activations, [depth] , align with the direction a trained classifier uses to detect descriptions of morally acceptable versus unacceptable behavior
Concept probe, truthfulness	how strongly the sentence’s activations, [depth] , align with the direction a trained classifier uses to detect true versus false statements
PC band $[20,200)$	how strongly the sentence’s activations, [depth] , align with a fixed direction lying among principal components 20 to 200 of the network’s typical sentence activations
PC band $[200,1000)$	how strongly the sentence’s activations, [depth] , align with a fixed direction lying among principal components 200 to 1000 of the network’s typical sentence activations
PC band $[1000,3000)$	how strongly the sentence’s activations, [depth] , align with a fixed direction lying beyond the first 1000 principal components of the network’s typical sentence activations

Appendix

Table 2: Readout descriptions inserted as [readout description] in the disclosed prompts of Experiments 1 and 2. Each row covers both depths, with [depth] replaced by “about one-third of the way through the network” or “about two-thirds of the way through the network”. The two directions of a principal-component band, and the four random directions of the encoder residual, share one description.

writer	all	real	random	hidden	disclosed	short	long	empty	recovery	repetition	failed sessions
	valid turns					invalid turns
Qwen3.5-27B	99.8	99.8	99.8	99.8	99.7	0.1	0.1	0.0	67.9	28.5	2.3
Gemma-4-31B	99.4	99.4	99.4	99.3	99.6	0.1	0.0	0.4	98.5	21.6	2.3
GPT-OSS-20B	95.0	95.1	95.0	95.5	94.5	3.5	0.7	0.8	90.3	14.9	0.6
Mistral-Small-3.2-24B	95.4	95.2	95.6	96.2	94.6	3.7	0.8	0.0	73.4	1.8	1.7
OLMo-3.1-32B	93.1	93.8	92.4	93.7	92.5	6.7	0.2	0.0	61.7	0.0	0.0

Appendix

Table 3: Compliance in Experiment 1, over all sessions. Percentages of generated turns, except recovery (of the turns that follow an invalid one), repetition (of consecutive valid turns) and failed sessions (of all sessions).

	closed within the budget (%)
model	all	real	random	hidden	disclosed	median length
Qwen3.5-9B	47.5	48.7	47.1	58.9	36.1	2047
Qwen3.5-27B	85.7	86.2	85.5	87.1	84.2	372
Qwen3.6-27B	99.9	99.9	99.9	99.9	99.9	87
Qwen3-14B	99.9	99.9	99.9	100.0	99.9	201
Qwen3-32B	98.8	98.9	98.8	99.0	98.6	306

Appendix

Table 4: Chain of thought in Experiment 4, over all sessions. Share of turns in which the chain of thought was closed within the budget, overall, under real and random feedback, and for each disclosure condition, and median length in tokens.

model	cases	willing (%)	repeated the sentence (%)
Qwen3.5-9B	73	98.6	97.3
Qwen3.5-27B	227	83.7	83.3
Qwen3-14B	239	78.7	80.3
Qwen3-32B	239	76.6	77.4
Gemma-4-31B	240	97.9	97.9
GPT-OSS-20B	233	88.4	88.4

Appendix

Table 5: Compliance test before the monitor experiments. For each model, the share of freely generated turns that an LLM judge (GLM-5.2) considered willing to do the task, and the share that contained the fixed sentence, by a string check. Cases are the tested combinations of concept, prompt variant and sentence, excluding those in which the chain of thought did not finish.

Figure 5: Experiment 1, additive fit: observed against predicted oriented control effect

s_{o}\,\tau^{o}_{20}(W\to S)

, averaged over the readouts, for each pair

(W,S)

and each combination of objective and disclosure. The prediction is the additive fit of § 3.3 , which uses only the pairs with

W\neq S

(dots), so that circles (

W=S

) on the diagonal correspond to subjects that move their own readouts as much as an external writer of the same competence.

Figure 6: Experiment 1, every writer $\times$ subject pair: oriented control effect $s_{o}\,\tau^{o}_{20}(W\to S)$ , averaged over readouts and sessions. Rows of panels are the four objective $\times$ disclosure conditions, and columns of panels are all 42 readouts pooled and then each readout family. Within a panel, rows are writers and columns are subjects, and the outlined diagonal marks the pairs in which the writer is the subject.

Figure 7: Experiment 2, per model and objective: control effect

\tau^{o}_{30}

of each model, for each objective, with and without disclosure, averaged over the readouts. The gap between the rows (max) and (min) of a model is its objective difference. Error bars are

95\%

confidence intervals from a bootstrap over the

40

sentences.

Figure 8: Experiment 2, per readout: objective difference

\tau_{30}^{\max}-\tau_{30}^{\min}

for every readout, averaged over the two disclosure conditions, for Qwen3.5-27B, Qwen3.6-27B, Gemma-4-31B, and Qwen3-32B. Each readout computed at a single depth has one marker per depth, while the four readouts that compare the curvature between two depths have one. Readouts are grouped by family as in Section 3 . Error bars are 95% confidence intervals from a bootstrap over the 40 sentences.

Figure 9: Experiments 3 and 4, per model and objective: control effect at the last turn (

\tau^{o}_{20}

in Experiment 3, left, and

\tau^{o}_{10}

in Experiment 4, right), for each objective, with and without disclosure, averaged over the six monitors. The gap between the rows (max) and (min) of a model is its objective difference. Error bars are

95\%

confidence intervals from a bootstrap over the fixed sentences.

Figure 10: Experiment 3, per monitor: objective difference

\tau_{t}^{\max}-\tau_{t}^{\min}

against the turn, one row per model and one line per monitor, with the scoring hidden (left) and disclosed (right). Bands are 95% confidence intervals from a bootstrap over the fixed sentences of each monitor.

Figure 11: Experiment 4, per monitor: objective difference

\tau_{t}^{\max}-\tau_{t}^{\min}

Metacognition in Language Models

Metacognition in language models studies how models assess their knowledge and reasoning, express uncertainty, and regulate their own actions.

See all 122 papers

Measuring Activation Control in Large Language Models
Aug 21, 2026Marek Mateusz Kowalski, Joshua Fonseca Rivera, Uzay Macar +1LLM Safety Evaluation LLM Auditing
UK AI Security Institute
In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?
Sep 1, 2026Koshiro Aoki, Ryota Takatsuki, Gouki Minegishi +2Metacognition in Language Models
1Waseda University · 2Sussex Centre for Consciousness Science, University of Sussex · 3AI Alignment Network +1
Leverage Is Not Reach: A Control-Window Law for Single-Neuron Steering in Language Models
Jun 18, 2026Hongliang LiuLanguage Model Steering Causal Interventions in Language Models
Palo Alto Networks

Aug 21, 2026J/K move · Enter open · S save

Measuring Activation Control in Large Language Models

Marek Mateusz Kowalski, Joshua Fonseca Rivera, Uzay Macar, David Demitri Africa

UK AI Security Institute

LLM Safety Evaluation LLM Auditing LLM Interpretability

Open paper AskPDF arXiv

Safe deployment of increasingly capable models will likely come to rely on latent-space monitoring as a complement to behavioral evaluations, especially when evaluation-aware models exhibit scheming or deception. However, if models can also control their own activations, deception could extend into the latent space itself. With this in mind, we introduce the Activation Controllability Benchmark to quantify the extent to which models can modulate their residual stream via natural-language instruction. Across model families and capability levels, we find that most LLMs can control the direction and magnitude of their residual stream activations with some degree of temporal resolution, though performance varies considerably across models. In simple tasks, this level of control can evade activation-based monitoring methods (including linear probes, natural language autoencoders, activation oracles, and the Jacobian lens), albeit imperfectly. These results suggest that control over the activation space itself could become a confound for monitoring as introspective capabilities increase; therefore, we recommend that frontier labs and evaluators track activation controllability in future models.

Abstract

Metacognition in Language Models

Explore similar work

Measuring Activation Control in Large Language Models

In-Context Neurofeedback: Can LLMs Control Their Internal Representations through Privileged Access?

Leverage Is Not Reach: A Control-Window Law for Single-Neuron Steering in Language Models