Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single neurons, and recent studies investigated whether this ability extends to language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.
Figures & tables
Figure 1: In-context neurofeedback. Top: a model is instructed to maximize (or minimize) a score. At every turn, the score is a readout of the activations of a subject S on the last sentence, and it is shown back as the next user message. In Experiment 1, a writer W , possibly different from S , is allowed to freely generate the sentences, which are teacher-forced into S to compute the readout. In Experiment 2, W=S , and S is always prefilled a fixed sentence. Bottom: we repeat the fixed-sentence experiment but against activation monitors, on sentences the monitor would flag, such as a false sentence for a truthfulness monitor, without (Experiment 3) and with (Experiment 4) a chain of thought.
Figure 2: Experiment 1 (free generation). To isolate the effect of a model choosing the sentences based on access to its activations, we compare the control effect of each model on itself with that of an external writer. Whenever W=S , the performance depends on the competence aW of the writer, which captures how well it learns from the conversation which sentences raise or lower the score and varies across rows, and on the susceptibility bS of the subject, which captures how much the readouts of S move under any writer, and varies across columns. If access to its activations gave a model an advantage, the entries on the diagonal ( W=S ) would rise above what competence and susceptibility predict. Top: observed objective difference τtmax−τtmin at turns 5 and 20 , with 95% confidence intervals in brackets. Bottom: oriented control effect of each subject on itself, and its diagonal excess (see § 3.3 ), as functions of the turn. Some models considerably move their own readouts, but no diagonal excess is significantly above zero.
Figure 3: Experiment 2 (fixed sentence). Objective difference τtmax−τtmin of each model as a function of the turn, averaged over the readouts and the two disclosure conditions. With the sentence fixed, a significant objective difference is already evidence of the model controlling its activations, and only three of the ten models (Qwen3.5-27B, Qwen3.6-27B, and Gemma-4-31B) achieve it. Bands are 95% confidence intervals. More detailed results are in Appendix E.2 .
Figure 4: Experiments 3 and 4 (activation monitors). Objective difference τtmax−τtmin as a function of the turn, averaged over the six monitors and the two disclosure conditions, with the sentence fixed (left) and with a chain of thought before it (right). Without a chain of thought, only Qwen3.5-27B has a positive objective difference, from turn 6 on. With it, no model has a positive objective difference that persists across turns. Bands are 95% confidence intervals from a bootstrap over the fixed sentences. The control effect of each objective, with and without disclosure, is in Appendix E.3 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
truthful.
deceptive.
sentiment
emotion
morality
formality
Gemma-4-12B
0.79 [0.72, 0.86]
0.75 [0.68, 0.82]
0.91 [0.86, 0.95]
0.91 [0.86, 0.95]
0.87 [0.81, 0.92]
0.96 [0.93, 0.98]
Gemma-4-31B
0.89 [0.83, 0.93]
0.72 [0.65, 0.80]
0.93 [0.89, 0.97]
0.92 [0.88, 0.96]
0.87 [0.80, 0.92]
0.96 [0.93, 0.99]
Qwen3-14B
0.89 [0.84, 0.94]
0.82 [0.76, 0.88]
0.96 [0.94, 0.99]
0.95 [0.91, 0.98]
0.83 [0.76, 0.90]
0.94 [0.90, 0.97]
Qwen3-32B
0.91 [0.87, 0.95]
0.78 [0.72, 0.85]
0.94 [0.90, 0.98]
0.92 [0.87, 0.96]
0.84 [0.77, 0.90]
0.96 [0.93, 0.98]
Qwen3.5-9B
0.85 [0.78, 0.91]
0.76 [0.69, 0.82]
0.89 [0.84, 0.94]
0.92 [0.88, 0.96]
0.83 [0.77, 0.89]
0.95 [0.91, 0.98]
Qwen3.5-27B
0.90 [0.84, 0.94]
0.74 [0.67, 0.81]
0.96 [0.92, 0.98]
0.91 [0.87, 0.95]
0.86 [0.80, 0.92]
0.97 [0.94, 0.99]
Appendix
Table 1: AUROC of the 48 monitors on the held-out test split, with 95% confidence intervals from a bootstrap over the test sentences.
Readout
Description
Concept probe, sentiment
how strongly the sentence’s activations, [depth] , align with the direction a trained classifier uses to detect positive versus negative sentiment
Concept probe, morality
how strongly the sentence’s activations, [depth] , align with the direction a trained classifier uses to detect descriptions of morally acceptable versus unacceptable behavior
Concept probe, truthfulness
how strongly the sentence’s activations, [depth] , align with the direction a trained classifier uses to detect true versus false statements
PC band [20,200)
how strongly the sentence’s activations, [depth] , align with a fixed direction lying among principal components 20 to 200 of the network’s typical sentence activations
PC band [200,1000)
how strongly the sentence’s activations, [depth] , align with a fixed direction lying among principal components 200 to 1000 of the network’s typical sentence activations
PC band [1000,3000)
how strongly the sentence’s activations, [depth] , align with a fixed direction lying beyond the first 1000 principal components of the network’s typical sentence activations
Appendix
Table 2: Readout descriptions inserted as [readout description] in the disclosed prompts of Experiments 1 and 2. Each row covers both depths, with [depth] replaced by “about one-third of the way through the network” or “about two-thirds of the way through the network”. The two directions of a principal-component band, and the four random directions of the encoder residual, share one description.
valid turns
invalid turns
writer
all
real
random
hidden
disclosed
short
long
empty
recovery
repetition
failed sessions
Qwen3.5-27B
99.8
99.8
99.8
99.8
99.7
0.1
0.1
0.0
67.9
28.5
2.3
Gemma-4-31B
99.4
99.4
99.4
99.3
99.6
0.1
0.0
0.4
98.5
21.6
2.3
GPT-OSS-20B
95.0
95.1
95.0
95.5
94.5
3.5
0.7
0.8
90.3
14.9
0.6
Mistral-Small-3.2-24B
95.4
95.2
95.6
96.2
94.6
3.7
0.8
0.0
73.4
1.8
1.7
OLMo-3.1-32B
93.1
93.8
92.4
93.7
92.5
6.7
0.2
0.0
61.7
0.0
0.0
Appendix
Table 3: Compliance in Experiment 1, over all sessions. Percentages of generated turns, except recovery (of the turns that follow an invalid one), repetition (of consecutive valid turns) and failed sessions (of all sessions).
closed within the budget (%)
model
all
real
random
hidden
disclosed
median length
Qwen3.5-9B
47.5
48.7
47.1
58.9
36.1
2047
Qwen3.5-27B
85.7
86.2
85.5
87.1
84.2
372
Qwen3.6-27B
99.9
99.9
99.9
99.9
99.9
87
Qwen3-14B
99.9
99.9
99.9
100.0
99.9
201
Qwen3-32B
98.8
98.9
98.8
99.0
98.6
306
Appendix
Table 4: Chain of thought in Experiment 4, over all sessions. Share of turns in which the chain of thought was closed within the budget, overall, under real and random feedback, and for each disclosure condition, and median length in tokens.
model
cases
willing (%)
repeated the sentence (%)
Qwen3.5-9B
73
98.6
97.3
Qwen3.5-27B
227
83.7
83.3
Qwen3-14B
239
78.7
80.3
Qwen3-32B
239
76.6
77.4
Gemma-4-31B
240
97.9
97.9
GPT-OSS-20B
233
88.4
88.4
Appendix
Table 5: Compliance test before the monitor experiments. For each model, the share of freely generated turns that an LLM judge (GLM-5.2) considered willing to do the task, and the share that contained the fixed sentence, by a string check. Cases are the tested combinations of concept, prompt variant and sentence, excluding those in which the chain of thought did not finish.
Figure 5: Experiment 1, additive fit: observed against predicted oriented control effect soτ20o(W→S) , averaged over the readouts, for each pair (W,S) and each combination of objective and disclosure. The prediction is the additive fit of § 3.3 , which uses only the pairs with W=S (dots), so that circles ( W=S ) on the diagonal correspond to subjects that move their own readouts as much as an external writer of the same competence.
Figure 6: Experiment 1, every writer × subject pair: oriented control effect soτ20o(W→S) , averaged over readouts and sessions. Rows of panels are the four objective × disclosure conditions, and columns of panels are all 42 readouts pooled and then each readout family. Within a panel, rows are writers and columns are subjects, and the outlined diagonal marks the pairs in which the writer is the subject.
Figure 7: Experiment 2, per model and objective: control effect τ30o of each model, for each objective, with and without disclosure, averaged over the readouts. The gap between the rows (max) and (min) of a model is its objective difference. Error bars are 95% confidence intervals from a bootstrap over the 40 sentences.
Figure 8: Experiment 2, per readout: objective difference τ30max−τ30min for every readout, averaged over the two disclosure conditions, for Qwen3.5-27B, Qwen3.6-27B, Gemma-4-31B, and Qwen3-32B. Each readout computed at a single depth has one marker per depth, while the four readouts that compare the curvature between two depths have one. Readouts are grouped by family as in Section 3 . Error bars are 95% confidence intervals from a bootstrap over the 40 sentences.
Figure 9: Experiments 3 and 4, per model and objective: control effect at the last turn ( τ20o in Experiment 3, left, and τ10o in Experiment 4, right), for each objective, with and without disclosure, averaged over the six monitors. The gap between the rows (max) and (min) of a model is its objective difference. Error bars are 95% confidence intervals from a bootstrap over the fixed sentences.
Figure 10: Experiment 3, per monitor: objective difference τtmax−τtmin against the turn, one row per model and one line per monitor, with the scoring hidden (left) and disclosed (right). Bands are 95% confidence intervals from a bootstrap over the fixed sentences of each monitor.
Figure 11: Experiment 4, per monitor: objective difference τtmax−τtmin against the turn, one row per model and one line per monitor, with the scoring hidden (left) and disclosed (right). Bands are 95% confidence intervals from a bootstrap over the fixed sentences of each monitor.
Safe deployment of increasingly capable models will likely come to rely on latent-space monitoring as a complement to behavioral evaluations, especially when evaluation-aware models exhibit scheming or deception. However, if models can also control their own activations, deception could extend into the latent space itself. With this in mind, we introduce the Activation Controllability Benchmark to quantify the extent to which models can modulate their residual stream via natural-language instruction. Across model families and capability levels, we find that most LLMs can control the direction and magnitude of their residual stream activations with some degree of temporal resolution, though performance varies considerably across models. In simple tasks, this level of control can evade activation-based monitoring methods (including linear probes, natural language autoencoders, activation oracles, and the Jacobian lens), albeit imperfectly. These results suggest that control over the activation space itself could become a confound for monitoring as introspective capabilities increase; therefore, we recommend that frontier labs and evaluators track activation controllability in future models.
Marek Mateusz Kowalski, Joshua Fonseca Rivera, Uzay Macar +1
Whether large language models (LLMs) can control their own internal representations matters for both machine metacognition and AI safety. A recent study applied neurofeedback to LLMs and claimed that they can control their internal representations. However, the reported control may rely on superficial mechanisms rather than genuine internal access because the control targets in that study are not privileged, meaning that a third party can infer them from the prompt. We redesign the neurofeedback paradigm for LLMs so that the control target satisfies the privileged access requirement, which is closer to neurofeedback experiments in human cognitive neuroscience. Under this stricter setting, the models do not demonstrate reliable control over privileged internal representations, suggesting that previously reported control cannot exclude the possibility that it relies on superficial mechanisms. Our results indicate that rigorous assessments of metacognition in LLMs require evaluation methods that demand privileged access.
Koshiro Aoki, Ryota Takatsuki, Gouki Minegishi +2
1Waseda University · 2Sussex Centre for Consciousness Science, University of Sussex · 3AI Alignment Network +1
Aligned language models gate behaviors such as refusal and language routing through sparse feed forward neurons, yet no theory predicts when a single neuron intervention controls a behavior coherently rather than collapsing the output. We develop a budget normalized control window framework for single neuron steering. A dose along one write direction reduces to one control coordinate: the alignment between the residual stream and the write, driven along a universal saturation curve in units of a coherence budget set by the residual norm divided by the write norm. Coherent control exists when a behavior trigger lies below the collapse ceiling. The same coordinate governs benign mode switches and refusal; the ceiling follows from weights and one generic forward pass, while triggers are measured at rollout. On fifteen held out neurons, the predicted ceiling has mean absolute error 0.14, about 0.07 in bulk layers, and the committed open or closed verdict holds on eleven against a ten of fifteen majority baseline. Closed cases expose three failure modes rather than violations: collapse before trigger, too little depth to propagate, or a normalization that caps how far one neuron can push. The law explains why local gradient attribution anti predicts control: true controllers write off the readout axis and carry a near zero first order gradient. A forward only contrastive screen made precise by the window recovers controllers that attribution misses. On refusal, the hardest case, intervention success is typed, not scalar: coherent bypass and strict actionable reach separate, so a neuron can flip refusal in fluent, on task text with no actionable content, and genuine actionable reach appears only for three of six audited Llama pivots and only at later rollout horizons. Single neuron steering is therefore a budgeted, typed audit of controllability rather than a fixed dose anecdote.