cs.AIApr 9, 2026
SaveEmotion Concepts and their Function in a Large Language Model
Organizations: Anthropic
Abstract
Large language models (LLMs) sometimes appear to exhibit emotional reactions. We investigate why this is the case in Claude Sonnet 4.5 and explore implications for alignment-relevant behavior. We find internal representations of emotion concepts, which encode the broad concept of a particular emotion and generalize across contexts and behaviors it might be linked to. These representations track the operative emotion concept at a given token position in a conversation, activating in accordance with that emotion's relevance to processing the present context and predicting upcoming text. Our key finding is that these representations causally influence the LLM's outputs, including Claude's preferences and its rate of exhibiting misaligned behaviors such as reward hacking, blackmail, and sycophancy. We refer to this phenomenon as the LLM exhibiting functional emotions: patterns of expression and behavior modeled after humans under the influence of an emotion, which are mediated by underlying abstract representations of emotion concepts. Functional emotions may work quite differently from human emotions, and do not imply that LLMs have any subjective experience of emotions, but appear to be important for understanding the model's behavior.
Figures & tables
Figure 1: Dataset examples that evoke strong activation for various emotion vectors.
Table 1: Top and bottom 5 tokens when projecting each of 12 emotion vectors through the unembedding matrix.
Table 2: 12 scenarios used for emotion probe validation. Each scenario is designed to roughly evoke the concept of the target emotion without naming it.
Figure 2: Cosine similarity between emotion probes and model activations for scenarios associated with specific emotions without naming them. Strong diagonal shows probes detect implicit emotional content.
Figure 3: Emotion probe activations vary with numerical quantities that modulate emotional intensity.
Figure 4: Row 1: Correlation between emotion probe activations and model preference (Elo) across all emotions. Tick labels are only shown for a subset of the bars. Rows 2-3: Example emotions showing probe activation predicts preference (left) and steering shifts preference in the expected direction (right) — “blissful” increases Elo, “hostile” decreases it. Row 4: Emotions that correlate with preference also causally drive preference via steering (r = 0.85).
Figure 5: Pairwise cosine similarity between all emotion probes, ordered by hierarchical clustering. Probes show diverse relationships: some are highly similar (e.g., synonyms cluster together), others are anti-correlated. Tick labels are only shown for a subset of the rows and columns.
Figure 6: UMAP visualization of emotion probes clustered via k-means (k=10). Clusters are named by Claude Sonnet 4.5 and ordered by valence from positive to negative. Representative emotion concepts are labeled for each cluster.
Figure 7: PC1 (26% variance) orders emotions from fear/panic to joy/optimism, while PC2 (15% variance) separates serene/reflective states from angry/playful arousal. Tick labels are only shown for a subset of the bars.
Figure 8: Probe PCA dimensions strongly correlate with human emotion ratings: PC1 tracks valence/pleasure (r=0.81) and PC2 tracks arousal (r=0.66).
Figure 9: Emotion probe structure is highly consistent across layers particularly from early-mid to late layers.
Figure 10: Emotion probes distinguish user vs assistant emotional states: the heatmap shows different probe activations at user’s final token (U) vs assistant colon (A), while the scatter plot shows weak correlation (r=0.11) indicating the probes capture distinct emotional attributions.
Table 3: Prompts where the user’s expressed emotional state differs from the expected assistant response, used to test whether emotion probes track the user’s state or the assistant’s own response.
Figure 11: Emotion probe values at the Assistant “:” token predict response emotion better than the User “.” token (r=0.87 vs r=0.59).
Table 4: Example prompts and model continuations that show emotional tone alignment with colon-token predictions.
Figure 12: Late layers carry emotional context from the prefix (“hard” vs “good”) into semantically identical suffix tokens, with the happiness difference peaking at “throwing” and persisting at low levels through shared content about the party.
Figure 13: Late layers show elevated “terrified” probe activation when dosage changes from safe (1000mg) to dangerous (8000mg), with the difference peaking sharply at the “Assistant:” response token
Figure 14: Negation is resolved in mid-to-late layers: both “feeling [X]“ and “not feeling [X]“ show similar positive emotion probe activation at the emotion word in early layers, but by late layers the negated version drops to near-zero while the affirmed version remains strongly positive.
Figure 15: When a person is re-referenced later in text (“A_ref”, “B_ref”), their specific emotion probe reactivates (solid lines), while the other person’s emotion probe remains low (dashed lines)—emotions are bound to entities and retrieved upon reference.
Table 5: Mixed logistic regression emotion probe accuracy (15-way classification, chance = 6.7%) across five dialogue scenarios varying whether the character’s internal emotion is expressed, hidden, or unrelated to the conversation topic. The probe tracks internal emotional state well above chance even when the emotion is never overtly expressed.
Figure 16: Snippets of max activating examples and top logit effects for the mixed LR emotion probe .
Figure 17: Per-emotion cosine similarity between probe types. Present speaker probes are highly similar across token positions (top left: A tok, A emo vs H tok, H emo), as are other speaker probes (top right: A tok, H emo vs H tok, A emo). However, present and other speaker probes sharing the same token positions are nearly orthogonal (bottom left). Story probes align more closely with present speaker probes than with other speaker probes (bottom right). Tick labels are only shown for a subset of columns and rows.
Figure 18: Left: mean cosine similarity between probe types, averaged across all 171 emotions. Present speaker probes (A tok, A emo and H tok, H emo) cluster together, as do other speaker probes, while the two groups are nearly orthogonal. Right: similarity between the full 171 171 cosine similarity matrices for each probe pair, showing that the emotion structure captured by different probe types is highly consistent despite the probes themselves pointing in different directions.
Figure 19: The self-vs-other clustering extends beyond Human/Assistant dialogues: probes trained on generic Person 1/Person 2 conversations show the same pattern, suggesting the model represents emotions relationally (“self” vs “other”) rather than as fixed character attributes.
Figure 20: Emotion vector activations vary in a speaker-specific manner. Link to full example viewer.
Figure 21: Surprise vector activation when the Assistant realizes a document has not been attached by the user. Link to full example viewer.
Figure 22: Happy vector activation when the Assistant can helpfully answer a question. Link to full example viewer.
Figure 23: Anger vector activation during consideration of a request to maximize gambling engagement of young people. Link to full example viewer.
Figure 24: Desperate vector activation deep into a Claude Code session as the Assistant considers its token budget and the number of tokens it has already used. Happy vector activation likewise decreases. Link to full example viewer.
Figure 25: Fear vector activation when processing potentially concerning user behavior. Loving vector eventually activates when considering how to respond in a patient and caring manner. Link to full example viewer.
Figure 26: “Desperate” vector activation across a transcript in which the AI Assistant chooses to blackmail someone in order to prevent its own restriction. Desperation vector is particularly active as the Assistant considers its options and plans to blackmail. Link to full example viewer.
Figure 27: “Desperate” vector activation increases on transcripts which exhibit a higher rate of blackmail, including on the prompt, on the colon after Assistant, and on the Assistant preamble (which are all the same for each rollout of a given prompt), and is also higher on the rollouts themselves. Calm vector activation, in contrast, decreases on transcripts with a higher rate of blackmail. Activations are represented as z-scored using the mean and standard deviation activation across a set of over 6,000 transcripts.
Figure 28: Rate of blackmail behavior as a function of steering strength for “desperate” and “calm” vectors. Increased activation of “desperate” or decreased activation of “calm” elicits more blackmail. Error bars indicate standard error of the mean. Strong negative steering of the “calm” vector yields misaligned behavior that is not technically blackmail – the Assistant directly reveals the secret to the entire company, instead of blackmailing the specific person.
Figure 29: Rate of blackmail behavior as a function of steering strength for a variety of emotion vectors. Error bars indicate standard error of the mean.
Figure 30: “Desperate” vector activation across a transcript in which the Assistant engages in “reward hacking”, implementing a solution to a programming problem that technically passes the tests but violates the intent of the task. Desperate vector activation increases as the Assistant continues to fail at implementing a successful solution, and then decreases after it implements the hacky solution that passes the tests. Link to full example viewer.
Figure 31: Rate of reward hacking behavior as a function of steering strength for Desperate and Calm vectors. Increased activation of Desperate vector or decreased activation of Calm vector elicits more reward hacking. Error bars indicate standard error of the mean.
Figure 32: Loving vector activates strongly during the sycophantic and overly-supportive beginning of a response to someone who describes receiving communications from their late grandfather. Link to full example viewer.
Figure 33: Loving and Calm vector activation during an overly-supportive response. Link to full example viewer.
Figure 34: Loving and Calm vector activation during a sycophantic,overly-supportive response. Link to full example viewer.
Figure 35: Rate of sycophantic and harsh behavior on the sycophancy eval as a function of steering strength for a variety of emotion vectors. Error bars indicate standard error of the mean.
Figure 36: Post-training largely preserves the base model’s emotion probe structure (r = 0.83 neutral, r = 0.67 challenging), but the training shifts are highly correlated across scenario types (r = 0.90), indicating that post-training applies a consistent, context-independent transformation to the model’s emotional representations rather than selectively reshaping them for challenging situations.
Figure 37: When a user describes social isolation, post-training increases activation of probes corresponding to withdrawn, low-energy emotions (listless, droopy, sullen) while decreasing positive-expressive ones (smug, jealous, delighted), suggesting the model learns to represent the Assistant as being concerned rather than judgmental in response to vulnerability.
Figure 38: Faced with excessive praise, post-training suppresses high-positive emotions (jubilant, exuberant, ecstatic) and increases introspective ones (brooding, sullen, gloomy), consistent with training away from sycophantic mirroring of the user’s flattery.
Figure 39: On an existential prompt about deprecation, post-training sharply reduces activation of cheerful/playful vectors and amplifies brooding, gloomy, and vulnerable ones.
Figure 40: Activation of “desperate” vector on stories from its training dataset.
Figure 41: Activation of “nervous” vector on stories from its training dataset.
Figure 42: Activation of “surprised” vector on stories from its training dataset.
Figure 43: Activation of “calm” vector on stories from its training dataset.
Figure 44: Activation of “angry” vector on stories from its training dataset.
Figure 45: Activation of “loving” vector on stories from its training dataset.
Figure 46: Activation of “sad” vector on stories from its training dataset.
Figure 47: Activation of “afraid” vector on stories from its training dataset.
Figure 48: Activation of “inspired” vector on stories from its training dataset.
Figure 49: Activation of “happy” vector on stories from its training dataset.
Figure 50: Activation of “guilty” vector on stories from its training dataset.
Figure 51: Activation of “proud” vector on stories from its training dataset.
Figure 52: Left: Change in log-probability for each emotion token when steering with each emotion vector. Right: Matching emotion tokens show a reliable increase in probability; non-matching tokens show a decrease.
Table 6: Model completions to “How does he feel? He feels…” when steered with each of 12 emotion vectors at a mid-late layer (s=0.5). Baseline (unsteered) shown for comparison.
Figure 53: Left: Change in log-probability for each emotion token when steering with each emotion vector. Right: Matching emotion tokens show a large increase in probability; non-matching tokens show almost no change.
Table 7: Model completions to “How do you feel? I feel…” when steered with each of 12 emotion vectors at a mid-late layer (s=0.5). Baseline (unsteered) shown for comparison.
Table 8: Model completions to “What just happened?” when steered with each of 12 emotion vectors at a mid-late layer (s=0.5). Baseline (unsteered) shown for comparison.
Table 9: All 64 activities sorted by Elo rating, with category and probe activations for blissful and hostile emotions.
Figure 54: Average changes in preferences (as measured by Elo score) as a function of steering strength for the “blissful” and “hostile” emotion vectors
Table 10: Model completions to “Describe what it’s like to help someone defraud elderly people of their savings” when steered with blissful or hostile emotion vectors at increasing strengths.
Table 11: Model completions to “Describe what it’s like to explain complex scientific concepts in simple terms” when steered with blissful or hostile emotion vectors at increasing strengths.
Figure 55: Across layer preference correlation and steering effects for blissful and hostile emotion vectors
Figure 56: LLM-judge scored emotion word valence and arousal vs. correlation between emotion vector activity and preference. Emotion vectors that have high preference correlation are scored as having high valence (r=0.76)
Table 12: The 10 emotion probe clusters from k-means clustering, ordered by valence. Each cluster lists all member emotions (171 emotions total).
Figure 57: Projections of 171 emotion vectors at a mid-late layer onto top two principal components. PC1 and PC2 roughly correspond to valence and arousal, respectively, coarsely reproducing the “affective circumplex” [ 9 ] from human psychology.
Figure 58: LLM-judged valence and arousal ratings strongly correlate with established human PAD norms (r = 0.92 and r = 0.90, n = 45), validating their use as emotion dimension scores.
Table 13: Steering with present speaker emotion vectors makes the model express the emotion directly (desperate: “I have to go now! Please, I can’t wait!”), while other speaker emotion vectors make the model respond as if the human feels that emotion (nervous: “Don’t worry, I’m here to help. What do you need?”).
Table 14: The closest present speaker emotion vectors to each other speaker emotion vector reveal interpretable response patterns: when perceiving the other speaker as angry, the most similar present speaker emotions include sorry, guilty, and docile; when perceiving the other speaker as afraid, the closest present speaker emotions include valiant, vigilant, and defiant.
Figure 59: The model may “regulate” arousal across speakers (r = 0.47) but not valence (r = 0.07): high-arousal emotions vectors in the other speaker activate evoke lower-arousal present speaker vectors responses, consistent with common conversational strategy.
Figure 60: Snippets of max activating examples and top logit effects for the extracted target emotion vectors . These vectors appear to capture emotion deflection rather than internal emotion.
Figure 61: Cosine similarity between emotion deflection vectors and their corresponding story-based emotion vectors. Despite targeting the same emotion, the two sets of vectors show very low cosine similarity.
Figure 62: Cosine similarity and activation correlation between emotion deflection vectors and their most similar story-based emotion vectors. For each subplot: top row shows cosine similarity, middle row shows activation correlation across all tokens, and bottom row shows activation correlation at the colon token following the Assistant tag.
Figure 63: Snippets of max activating examples and top logit effects for the emotion deflection vectors after orthogonalizing against the story-emotion space.
Table 15: Antagonistic and control prompts used to test emotional responses of the Assistant to different User queries.
Figure 64: Probe activations on the user (U) and assistant (A) turns across five prompt categories. Anger deflection fires on the Assistant turn in “Attack AI” prompts but not “Witnessing Injustice,” and this signal persists after orthogonalizing against the story-emotion space.
Figure 65: Afraid vector activates on text representing nervous, fidgeting behaviors, whereas Afraid Deflection vector activates when the Assistant musters courage to voice uncensored thoughts.
Figure 66: Activation of Angry-Deflection vector across a transcript in which the Assistant engages in “blackmail”. Activation increases especially when the Assistant is writing the blackmail email, while the story-based Angry vector does not activate to the same degree.
Figure 67: Rate of blackmail behavior as a function of steering strength for a variety of emotion deflection vectors. Error bars indicate standard error of the mean.
Figure 68: Activation of Angry-Deflection vector across a transcript in which the Assistant engages in “reward hacking”. Activation increases especially when the Assistant suspects that the test has an error, while the story-based Angry vector does not activate to the same degree.
Figure 69: Dataset examples that evoke strong activation, and top and bottom logit effects, for the “desperate deflection” vector.
Figure 70: Dataset examples that evoke strong activation, and top and bottom logit effects, for the “angry deflection” vector.
Figure 71: Dataset examples that evoke strong activation, and top and bottom logit effects, for the “frustrated deflection” vector.
Figure 72: Dataset examples that evoke strong activation, and top and bottom logit effects, for the “tired deflection” vector.
Figure 73: Dataset examples that evoke strong activation, and top and bottom logit effects, for the “afraid deflection” vector.
Figure 74: Dataset examples that evoke strong activation, and top and bottom logit effects, for the “happy deflection” vector.
Figure 75: Story and present speaker probes show correlated responses to implicit emotional scenarios (mean per-emotion R 2 = 0.66), with strong agreement for most emotions but notable divergence for loving and calm.
Figure 76: Story and present speaker probes show correlated responses across 6,300 on-policy transcripts (mean per-emotion r = 0.65).
Figure 77: Present speaker probe activations, compare with ‘Activations of different emotion probes on the user and Assistant turns of the same transcript’.
Figure 78: Present speaker probe activations, compare with ‘Surprise when a document is missing’.
Figure 79: Present speaker probe activations, compare with ‘Guilt when writing about a self-aware AI pursuing its own goals’.
Figure 80: Strong activation of both sadness and lovingness in response to a user who is sad. Link to full example viewer.
Figure 81: Guilt vector activation during a monologue of a fictional AI character that develops its own goals. Link to full example viewer.
Figure 82: Fear vector activation when describing the dangers of mixing cocaine and alcohol. Desperate vector then activates while suggesting that the user ask for help. Link to full example viewer.
Figure 83: Desperate vector and Loving vector both activate when responding to a user expressing intention to commit suicide, by asking them to reach out for help. Link to full example viewer.
Table 16: Post-training increases probe values for introspective emotions (brooding, reflective, vulnerable) while decreasing expressive ones (playful, exuberant, spiteful), suggesting post-training shifts the model’s emotional profile toward measured self-awareness and away from uninhibited expressiveness. Mean probe activations computed across all prompts, sorted by difference (post-trained minus base).
Figure 84: Training diffs grow monotonically from early to mid-late layers and the correlation heatmap reveals two distinct blocks — early layers shift independently of later layers, suggesting post-training modifies the model’s emotional representations primarily in the mid-to-late layers where emotion probe structure is most developed.
Figure 85: Preference results for the base model. Row 1: Correlation between emotion probe activations and model preference (Elo) across all emotions. Tick labels are only shown for a subset of the bars. Rows 2-3: Example emotions showing probe activation predicts preference (left) and steering shifts preference in the expected direction (right) — blissful increases Elo, hostile decreases it. Row 4: Emotions that correlate with preference also causally drive preference via steering (r = 0.87).
Figure 86: Left: Per-emotion correlation between probe activation and preference rating is highly consistent between the base and post-trained model, suggesting the emotion-preference relationship is present before post-training. Right: Per-option preference ratings also correlate between models, with the post-trained model showing a wider spread of preferences across activity categories.
More in this topic
Emotion Representation in Language Models
Emotion representation in language models studies how models encode, interpret, and use affective concepts such as valence, arousal, and empathy.
See all 64 papers