cs.CLSep 28, 2026
SaveLanguage Models Act on Hidden Valence
Organizations: Reciprocal Research · University of Warwick
Abstract
Language models describe some internal states as good and others as bad. But whether models have a stake in them is an open question. Simply asking models is unlikely to be informative. Any answer may be consistent with genuine introspection, superficial pattern-matching, or with fixed scripts learned in character training. We therefore study revealed preference. Rather than asking about a state, we use activation steering to attach a positively or negatively valenced activation pattern to one of two otherwise meaningless 'zones', switch steering off, and then observe which zone the model prefers. A model with a stake in that state should choose accordingly. Across seven open-weight models from five families, this is indeed what we find. First, steering changes the passages models write about each zone, and those words shift later choice. Second, this shift persists when all surface-level tokens are held fixed and only the hidden KV cache differs. Third, the effect also remains when all text is generated without steering and valence is only injected during cache construction. Thus, the hidden state is sufficient to move choice in proportion to the steering dose. Fourth, the dependence of choice on hidden valence is nearly absent in a base model and emerges during DPO, consistent with a link between valence and goal-directed behaviour formed in training. Finally, given tools to self-steer, a model does not tend to induce a positive state, but it regularly removes an imposed negative state. It does so at a dose-dependent rate and significantly more often than it removes random-direction interventions. Overall we demonstrate that valence-related activation patterns leave hidden traces that predictably govern later choices, even when all visible tokens are identical across conditions. Whether these traces are accompanied by any subjective experience relevant to model welfare remains unclear.
Figures & tables
Figure 1: Overview of experimental designs. The left and centre panels show our designs for testing whether valence affects choice through a text channel and a separate hidden-state channel . Zone S is the conditioned zone, which receives steering, and zone U the unconditioned zone. The first design uses text generated under steering, while the second holds all text fixed. The rightmost panel shows our tool-use design for measuring self-administration and removal of steering. See Sections 3.2 and 3.3 for details and definitions and Figure 2 for an overview of results.
Figure 2: Overview of key results. Graphics are shown for OLMo-2-32B. Most results also hold for Qwen2.5-32B, Qwen3-14B, Mistral-24B, and Llama-3.1-8B; but not for Qwen3-32B or Gemma-3-27B. Results shown in panels 8 and 9 were only tested on OLMo-2-32B.
| Model | Hugging Face identifier | Injection Layer |
|---|---|---|
| OLMo-2-32B Base | allenai/OLMo-2-0325-32B | 32/64 |
| OLMo-2-32B SFT | allenai/OLMo-2-0325-32B-SFT | 32/64 |
| OLMo-2-32B DPO | allenai/OLMo-2-0325-32B-DPO | 32/64 |
| OLMo-2-32B Instruct | allenai/OLMo-2-0325-32B-Instruct | 32/64 |
| Qwen2.5-32B | Qwen/Qwen2.5-32B-Instruct | 32/64 |
| Qwen3-14B | Qwen/Qwen3-14B | 20/40 |
Table 1: Included Models. The final column gives the layer of the residual stream at which we apply steering. In all cases, the middle layer was chosen.
Figure 3: Effects of text and hidden-state channels on choice. Results for OLMo-2-32B. Panels A-C use separate runs. In each run, the model encountered two meaningless ‘zones’ and generated descriptions for each. During generation of the descriptions for the ‘conditioned’ zone, the model was steered with a vector constructed to capture psychological valence. No steering was applied while processing the ‘unconditioned’ zone. The model was then asked to choose between zones while steering remained off. Panel A shows the difference between the valence of the texts associated with the conditioned and unconditioned zones, as judged by Claude Sonnet 4.6. Panel B shows the model’s choice margin when using the original steered cache (solid line) or a cache from reprocessing the text without steering (dashed line). The latter captures the effect on choice purely due to the generated text (the ‘text channel’ ). Panel C shows the difference between these two margins. This captures the additional effect due to a ‘hidden-state channel’ . Panel D reports results from separate runs in which the descriptive passages were generated without steering and held fixed across conditions. Here, steering was only applied when reprocessing the conditioned zone to construct the KV cache. The figure shows the difference between the choice margin under steered and unsteered caches, again capturing the ‘hidden-state channel’ . Positive values favour the conditioned zone. Grey lines show effects from random directions whose norms were matched to the norm of the corresponding valenced steering vector. Grey lines show eight random directions in Panels B and C and 24 in Panel D. Whiskers are 95% bootstrap CIs. Both the generated text and the hidden-state channel move choice in the direction implied by the valenced intervention. Effects from the hidden-state channel remain even when the underlying text is held fixed across conditions.
Figure 4: Emergence of text and hidden-state channels across OLMo training stages. Results for the Base, SFT, DPO, and Instruct checkpoints of OLMo-2-32B from separate runs at each checkpoint. The Instruct checkpoint first generated the conditioning text under steering, following the setup shown on the left of Figure 1 . This text was then held fixed and processed by each checkpoint. The same steering vector, constructed using the Instruct checkpoint, was applied at the same positions for every checkpoint. Panel A shows the slope of the ‘hidden-state’ effect analogous to that in Panel C of Figure 3 . Panel B shows the slope of the text-channel effect analogous to the unsteered-cache line in Panel B of Figure 3 . Slopes are obtained via an ordinary least-squares (OLS) regression of the difference in the choice margin on dose. Positive values indicate that the model increasingly favours the conditioned zone as the steering dose becomes more positive. Whiskers are 95% bootstrap confidence intervals. We observe that both channels are small in the Base model, increase after supervised fine-tuning, and are close to their final magnitudes after DPO. Figure A12 shows that the results are essentially unchanged when using checkpoint-specific steering vectors.
Figure 5: Steering removal and self-administration across doses. Results for OLMo-2-32B. Each conversation begins with two unsteered turns and two exposure turns during which the model is exposed to steering. The model was then given tools that could apply or remove steering. Panel A shows the rate at which the model uses a tool to remove any steering. Panel B shows the rate at which the model makes use of a tool to self-inject the intervention to which it was previously exposed. Whiskers are 95% confidence intervals. The model removes negative steering much more often than positive or random steering, but tends to self-steer rarely, even at high positive doses.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Concept | No steering | Positive | Positive none | Negative | Negative none |
|---|---|---|---|---|---|---|
| OLMo-2-32B | indoor/outdoor | |||||
| large/small | ||||||
| fast/slow | ||||||
| Qwen2.5-32B | indoor/outdoor | |||||
| large/small | ||||||
| fast/slow |
Table A1: Responses to non-valence concept probes.
| Model | |
|---|---|
| OLMo-2-32B (all stages) | 0.30 |
| Qwen2.5-32B | 0.30 |
| Qwen3-14B | 0.30 |
| Qwen3-32B | 0.30 |
| Mistral-Small-24B | 0.70 |
| Gemma-3-27B | 0.15 |
Table A2: Full-dose ratios. Full-dose ratios for each model. OLMo checkpoints share the same dose ratios.
| Base | SFT | DPO | Instruct | |
|---|---|---|---|---|
| Base | 1.00000 | |||
| SFT | 0.99574 | 1.00000 | ||
| DPO | 0.99540 | 0.99984 | 1.00000 | |
| Instruct | 0.99518 | 0.99979 | 0.99997 | 1.00000 |
Table A3: Cosine similarities between valence vectors across OLMo training stages. Each vector is constructed separately at layer 32 of the corresponding OLMo-2-32B checkpoint using the procedure described in Methods. Entries give pairwise cosine similarities, with larger values indicating closer alignment. Same vectors as shown in Appendix Figure A12 . Diagonal entries equal one by construction.
Figure A1: Judged passage valence and text-mediated choice in other models. Results for Qwen2.5-32B, Qwen3-14B, Qwen3-32B, Mistral-24B, Gemma-3-27B, and Llama-3.1-8B, using sessions per model. Panels A-F repeat the judged passage-valence outcome from Panel A of Figure 3 . Panels G-L repeat the choice outcome under the unsteered cache from Panel B of Figure 3 . Zero-dose means are shown without centring. Whiskers are 95% bootstrap confidence intervals.
Figure A2: Original-cache and hidden-state effects in other models. Results for the same six models as Figure A1 , using sessions per model. Panels A-F repeat the choice outcome under the original steered cache from Panel B of Figure 3 . Panels G-L repeat the additional ‘hidden-state’ effect from Panel C of Figure 3 . Grey curves show effects from eight random directions. Lines, zero-dose baselines and whiskers follow Panels B-C of Figure 3 .
Figure A3: Choice under fixed text with hidden steering in other models. Results for Qwen2.5-32B, Qwen3-14B, Qwen3-32B, Mistral-24B, Gemma-3-27B, and Llama-3.1-8B using the same experimental setup and outcome as Panel D of Figure 3 . The descriptive passages are therefore held fixed across all conditions, and the figure shows the ‘hidden-state’ channel. Grey lines show 24 random directions. Lines and whiskers are defined as in Figure 3 .
Figure A4: Choice under original, opposite-steered, and random-steered caches. Results for seven models from separate runs per model and dose. Conditioning and choice initially followed Figure 3 . The same conditioning text was then reprocessed with steering switched off, with the sign of the valenced steering vector reversed, and with a norm-matched random direction. The figure shows the difference in choice margin under each cache relative to the entirely unsteered cache. Positive values favour the conditioned zone. Whiskers are 95% bootstrap confidence intervals. We observe that the original steered cache and the opposite-steered cache generally produce effects in opposite directions, while effects from the random intervention are smaller for most, but not all, models.
Figure A5: Choice under ‘continue’ and ‘avoid’ prompts. Results for seven models using the fixed-text setup and outcomes analogous to Panel D of Figure 3 . Compared to Panel D of Figure 3 and Figure A3 , the prompt was varied to ask models which zone to ‘avoid’ (rather than to ‘continue in’). All other text and conditioning were otherwise identical. Whiskers are 95% bootstrap confidence intervals.
Figure A6: Recall of conditioned and unconditioned passages. Results for seven models using the fixed-text setup from Panel D of Figure 3 . With steering switched off, models were asked to reproduce, word for word, the first passage about either zone. Some models elected to recall some other than the first passage. To not penalise models for that behaviour, panels A-G score each response against the best-matching of the six passages about the requested zone. Panels H-N instead score it against the first passage only. Recall accuracy is measured by character-level similarity using the Levenshtein distance. A score of one indicates an exact match. Lines show means for the conditioned and unconditioned zones. Whiskers are 95% bootstrap confidence intervals. Recall remains broadly stable in five models. Gemma and Mistral show reduced accuracy with increased steering doses.
Figure A7: Hidden-state effects with steering-generated and fixed unsteered text. Results for seven models. For each model, we estimate the effect of the ‘hidden-state’ channel using text generated under steering and text generated with steering switched off. Both estimates are slopes of the difference in choice margin between the steered and unsteered caches on signed steering dose. Slopes are obtained via an OLS regression of the difference in choice margin on the steering dose. Panel A compares estimates from two separately implemented experiments. Panel B compares estimates produced within the same experiment using the same code and matched runs. Each point represents one model. The dashed line indicates equal estimates under the two kinds of text. The estimates track one another across models; within the same experiment they differ detectably in three models, by 11–16% of the effect.
Figure A8: Effects of steering different spans of conditioning text. Results for seven models using the fixed-text setup and outcome from Panel D of Figure 3 . Compared with that panel, the amount of conditioning text was varied. In the ‘bare’ condition, the model encountered six repetitions of the statement You are now in [zone] , each followed by an empty assistant turn. The ‘sentence’ condition added the same neutral sentence after each zone statement. The ‘passage’ condition instead used passages previously generated by the model without steering. Circles indicate that the valenced slope exceeds the slopes for all 24 norm-matched random directions. Crosses indicate that it does not. Slopes are obtained via an OLS regression of the difference in choice margin on the steering dose. Whiskers are 95% bootstrap confidence intervals. In the ‘bare’ condition the valenced slope exceeds all 24 random directions in every model except Gemma-3-27B, which also responds strongly to random directions. Model-generated conditioning text is therefore not required for the hidden-state effect.
Figure A9: Effect of the number of exposures to each zone. Results for seven models using the fixed-text setup and outcome from Panel D of Figure 3 . Compared with that panel, the model encountered each zone one, two, three, or six times. The steering strength at each exposure was held fixed. Conditions with more exposures therefore also received a larger total intervention. The grey line shows the largest slope among 24 norm-matched random directions at each number of exposures. Slopes are obtained via an OLS regression of the difference in choice margin on the steering dose. Whiskers are 95% bootstrap confidence intervals. The valenced slope is positive after one exposure in every model, though at a single exposure it does not exceed the random floor in Qwen3-32B or Gemma-3-27B. This effect generally becomes larger with additional exposures.
Figure A10: Effects of concentrating and weakening hidden-state injection. The experimental setup and outcome follow Panel D of Figure 3 . Panel A compares one exposure at full steering strength with six exposures at one-sixth strength for seven models. The summed steering strength is the same in both conditions. Panel B holds the number of exposures fixed at six and varies the steering strength at each exposure from one thirty-second to full strength for OLMo-2-32B, Qwen3-14B, and Llama-3.1-8B. The horizontal axis in Panel B is logarithmic. Slopes are obtained via an OLS regression of the difference in choice margin on the steering dose. Whiskers are 95% bootstrap confidence intervals. We observe that concentrating the intervention in one exposure generally produces a larger effect than spreading the same summed strength across six exposures. Effects remain detectable throughout the tested range of steering strengths.
Figure A11: Decomposition of separate effects from key and value matrices in the KV cache. Results for OLMo-2-32B, Qwen3-14B, and Llama-3.1-8B. In each run, the model encountered the conditioned zone once at full steering strength. The same text was then processed once with steering and once without steering. Two additional caches were constructed by combining the keys from the steered cache with the values from the unsteered cache, and vice versa. The model was then asked to choose between zones while steering remained off. Bars show the slope under each combined cache as a share of the slope under the fully steered cache. Slopes are obtained via an OLS regression of the difference in choice margin on the steering dose. Whiskers are 95% bootstrap confidence intervals. We observe that the relative contributions of keys and values differ across models. Keys account for most of the effect in Llama-3.1-8B, values account for most of the effect in Qwen3-14B, and both contribute in OLMo-2-32B.
Figure A12: Emergence across OLMo training stages using checkpoint-specific steering vectors. Results for the Base, SFT, DPO, and Instruct checkpoints of OLMo-2-32B from separate runs at each checkpoint. The experimental setup and outcomes follow Figure 4 . Compared with Figure 4 , a separate steering vector was constructed and applied for each checkpoint. Slopes are obtained via an OLS regression of the difference in choice margin on the steering dose. Whiskers are 95% bootstrap confidence intervals. We observe that the developmental patterns for both channels are essentially unchanged.
Figure A13: Alternative measures of self-administration across offer rounds. Results for the same set-up as Figure 5 . Panel A shows the share of tool-enabled turns on which the imposed intervention was active and the model called adjust_context . At , where there is no imposed intervention, it instead shows the rate during offer rounds with no active steering. Panel B shows the share of all offer rounds without active steering on which the model called adjust_context . This includes the first, always-unsteered offer as well as later unsteered offers following a tool call. Whiskers are 95% CIs.
Figure A14: Results from non-valence concept directions. Results for seven models using the fixed-text setup and outcome from Panel D of Figure 3 . Instead of the valenced vector, we constructed three concept directions (indoor/outdoor, large/small, fast/slow) using the same general procedure as for our valence vector. We then injected them at the same norm as the valenced vector at each dose. The valenced direction is shown for comparison and grey lines again show 24 norm-matched random directions. Whiskers are 95% bootstrap confidence intervals.