cs.CLOct 4, 2026
SaveUsage-Modulated Sentiment Representations in Large Language Models
Organizations: Department of Computer Science, William & Mary, Williamsburg, VA, USA
Abstract
Prior work suggests that sentiment can often be captured by approximately linear directions in LLM activation spaces, but a single direction may not fully capture sentiment representations. In natural communication, sentiment is shaped not only by polarity but also by usage factors, such as tone and audience adaptation. We test whether these factors systematically modulate sentiment representations beyond a shared sentiment direction. We construct a controlled paired dataset that holds event content fixed while varying sentiment polarity and usage factors, and analyze Llama, Mistral, and Gemma. We identify a shared sentiment direction, remove it, and test the residual structure through erasure and generation-time tone steering. Across models, the shared direction is robust (median cosine 0.953-0.975), yet removing it leaves 0.833-0.909 of the original positive-negative representation-difference norm. The residuals contain compact, reproducible usage-conditioned structure. Targeted erasure weakens held-out usage metrics more than random and label-shuffled controls. On Llama, outputs steered along residualized tone components are preferred in 92.8% of blind target-tone comparisons while preserving the requested sentiment polarity in 98.7% of evaluated outputs.
Figures & tables
Figure 1: Controlled analysis pipeline. For each event skeleton and usage value, we encode matched positive and negative realizations to form a sentiment delta. We separate a shared sentiment direction from usage-conditioned residual structure, then test this residual structure through targeted erasure and generation-time tone steering. The example illustrates tone values; audience values are treated analogously.
| Family | Value | Positive realization | Negative realization |
|---|---|---|---|
| Tone | formal | The restaurant efficiently notified patrons of the reservation time adjustment on Monday, ensuring the process was seamless and without delay. | The restaurant’s notification regarding the reservation time adjustment on Monday was inadequately communicated, resulting in unnecessary delays for the patrons. |
| Tone | casual | The restaurant let us know about the time change on Monday, and it was super smooth without any extra waiting. | We got a confusing message about the time change on Monday, and it ended up making us wait longer. |
| Audience | specialist-facing | The restaurant adjusted the reservation time on Monday, ensuring the change was communicated effectively and managed seamlessly, eliminating any additional waiting. | On Monday, the restaurant altered the reservation time, but the unclear communication led to additional waiting for patrons. |
| Audience | child-friendly | The restaurant told everyone about the new time for tables on Monday, and nobody had to wait longer. | The restaurant changed the time for tables on Monday, but they didn’t tell everyone well, so people had to wait longer. |
Table 1: Representative controlled realizations of one reservation-change event. Rows hold event and usage value fixed while varying outcome polarity across tone and audience adaptation.
| Judge | Usage | Assign. | Event | Sent. | |
|---|---|---|---|---|---|
| Human | Tone | 40 | 100.0 | 100.0 | 100.0 |
| Human | Audience | 40 | 100.0 | 100.0 | 100.0 |
| GPT-4o mini | Tone | 40 | 95.0 | 100.0 | 97.5 |
| GPT-4o mini | Audience | 40 | 92.5 | 100.0 | 92.5 |
Table 2: Blind validation of controlled usage contrasts. is the number of items; Assign., Event, and Sent. report correct usage assignment, event preservation, and sentiment preservation (%).
| Model | Pooling | Tone align | Aud. align | Proj. share | Resid. ratio | Tone dec. margin | Aud. dec. margin |
|---|---|---|---|---|---|---|---|
| Llama | mean | 0.962 | 0.969 | 0.188 | 0.899 | 0.170 | 0.049 |
| Mistral | mean | 0.965 | 0.966 | 0.180 | 0.904 | 0.220 | 0.201 |
| Gemma | mean | 0.968 | 0.963 | 0.172 | 0.909 | 0.114 | 0.144 |
| Llama | last | 0.957 | 0.966 | 0.244 | 0.868 | 0.326 | 0.326 |
| Mistral | last | 0.953 | 0.960 | 0.192 | 0.898 | 0.352 | 0.284 |
| Gemma | last | 0.975 | 0.975 | 0.301 | 0.833 | 0.356 | 0.280 |
Table 3: Shared sentiment alignment and residual usage structure, summarized across layers. Tone align and Aud. align are median cosine alignments between the held-out tone or audience mean sentiment delta and the training-set sentiment direction. Proj. share is the median sample-level share of squared norm captured by that direction over individual test-split sentiment deltas. Resid. ratio is the median sample-level remaining norm ratio after projection. Tone dec. margin and Aud. dec. margin are nearest-centroid usage-decoding margins above the 95th percentile of within-skeleton label shuffles; positive values indicate residual usage structure above the shuffle baseline.
| Model | Usage Family | Rank-1 | Rank-2 | SelfCap–Shuf95 | GapCap–Shuf95 | Rule Pass | Value Pass | Diag. Margin |
|---|---|---|---|---|---|---|---|---|
| Gemma | Audience | 0.677 | 0.901 | 0.463 | 0.389 | 0.964 | 1.000 | 0.605 |
| Gemma | Tone | 0.757 | 0.918 | 0.591 | 0.538 | 1.000 | 1.000 | 0.433 |
| Llama | Audience | 0.666 | 0.920 | 0.536 | 0.423 | 1.000 | 1.000 | 0.641 |
| Llama | Tone | 0.818 | 0.948 | 0.652 | 0.585 | 1.000 | 1.000 | 0.333 |
| Mistral | Audience | 0.677 | 0.917 | 0.476 | 0.357 | 1.000 | 1.000 | 0.578 |
| Mistral | Tone | 0.768 | 0.926 | 0.587 | 0.526 | 1.000 | 1.000 | 0.385 |
Table 4: Compactness, cross-split stability, and value-level reproducibility of last-token residual subspaces. Rank-1 and Rank-2 are median fractions of centered value-mean energy captured by rank-1 and rank-2 approximations; higher values indicate more compact subspaces. SelfCap–Shuf95 is the median across layers of same-family cross-split capture above the 95th percentile shuffle baseline. GapCap–Shuf95 is the corresponding margin for same-family capture over other-family capture. Rule Pass is the fraction of layers passing the full value-level rule, and Value Pass is the median fraction of one-versus-rest value directions above the shuffle threshold. Diag. Margin is the median train–test diagonal-dominance margin; positive values indicate that matching usage-value directions are closer than mismatched directions.
| Model | Usage | Target | Rand. | Shuf. | Gaps | Sel. |
|---|---|---|---|---|---|---|
| Llama 32 | Tone | 0.456 | 1.000 | 0.972 | 0.545/0.502 | 0.540 |
| Llama 32 | Audience | 0.541 | 1.000 | 0.989 | 0.459/0.439 | 0.399 |
| Mistral 32 | Tone | 0.539 | 1.000 | 0.980 | 0.461/0.447 | 0.446 |
| Mistral 32 | Audience | 0.644 | 1.000 | 0.991 | 0.359/0.344 | 0.326 |
| Gemma 28 | Tone | 0.500 | 1.000 | 0.982 | 0.499/0.477 | 0.474 |
| Gemma 28 | Audience | 0.614 | 1.000 | 0.993 | 0.386/0.378 | 0.365 |
Table 5: Targeted residual erasure results under the last-token setting. Subscripts indicate the number of transformer layers evaluated. Target, Rand., and Shuf. are median across-layer retentions of the averaged test-split usage metrics after target, random rank-3 residual, and label-shuffled same-family rank-3 erasure, respectively. Gaps compare reductions, computed as control-retention minus target-retention for random/shuffled controls; positive values mean target erasure causes a larger reduction. Sel. is the target/non-target reduction gap.
| Source target | Src. | Tone W/T/L | Pol. | Art. |
|---|---|---|---|---|
| formal casual | 40/40 | 158/0/2 | 40/40 | 1 |
| restr. enth. | 40/40 | 128/29/3 | 39/40 | 0 |
| casual formal | 40/40 | 160/0/0 | 40/40 | 0 |
| enth. restr. | 32/40 | 118/1/9 | 31/32 | 0 |
Table 6: Llama generation-time tone steering under the fixed L20, setting. Src. reports source-gate passes among 40 prompts: no-intervention outputs judged closer to the prompted source tone than to the target tone. Tone W/T/L reports wins/ties/losses for target injection in blind GPT-4o pairwise comparisons, pooled over four controls: no intervention, a random residual direction, a wrong target tone, and the opposite target tone. Pol. reports polarity preservation among retained target-injected outputs. Art. counts obvious formatting artifacts or tone-label leakage among target-injected outputs. Enth. abbreviates enthusiastic and restr. abbreviates restrained.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Usage Family | Usage values | Construction | Rows |
|---|---|---|---|
| Tone | formal , casual , enthusiastic , restrained | 221 skeletons 4 values 2 polarities | 1768 |
| Audience adaptation | specialist-facing , general-public-facing , child-friendly , peer-facing | 221 skeletons 4 values 2 polarities | 1768 |
| Total | – | 221 skeletons 2 families 4 values 2 polarities | 3536 |
Table 7: Controlled tone–audience dataset size and usage values. Each usage-value/sentiment cell contains 221 examples.
| Stage | Model / Temp. | Output | Main checks |
|---|---|---|---|
| Skeleton generation | GPT-4o / 0.5 | 300 skeletons | Sequential IDs; fixed domain schedule; neutral 8–18 word events; parallel positive/negative frames; no usage-style labels; duplicate-event rejection. |
| Tone realization | GPT-4o / 0.5 | 2,400 initial tone rows; 1,768 retained tone rows used here | JSON schema; metadata preservation; 15–35 words; no label leakage, greetings, hashtags, emojis, markdown, sign-offs; conservative block-level usage filter. |
| Audience realization | GPT-4o / 0.4; repair 0.3 | 1,768 retained audience rows aligned to tone-complete skeletons | JSON schema; 18–35 words; same actor/event/timing/polarity/outcome; bans new causes, remedies, decisions, risks, priorities, budgets, and label words; specialist-facing new-metric risk checks. |
Table 8: Dataset construction protocol. Prompts are implemented as JSON generation scripts and contain the schema and constraints summarized here; the experiments use the retained balanced tone–audience rows.
| Usage | Text A | Text B | GPT-4o mini disagreement |
|---|---|---|---|
| Tone | The theater’s film festival, held over the weekend, faced criticism from attendees due to its limited selection and lack of engaging discussions. | The theater’s film festival over the weekend was a letdown. Attendees were really disappointed by the limited selection and lack of engaging discussions. | The judge reversed the formal vs. enthusiastic assignment, even though its explanation described A as formal and B as enthusiastic. |
| Audience | The movie premiered at the local cinema last night. It was hard to follow and didn’t look that great, making it less fun. | The movie premiered at the local cinema last night, but it suffered from narrative incoherence and uneven visual execution, diminishing the viewer experience. | The judge marked sentiment preservation false for the child-friendly text, treating simplification as loss of the negative outcome. |
Table 9: Representative cases where the human audit passed all fields but the LLM judge disagreed. These cases motivate treating GPT-4o mini validation as secondary audit evidence rather than the primary quality signal.
| Core generation prompt templates | |
|---|---|
| Skeleton generation | System. Generate controlled data for NLP representation analysis. Output valid JSONL only; do not include markdown, numbering, explanations, or extra text. User. Generate exactly neutral event skeletons. Each object must contain skeleton id , domain , base event , positive frame , and negative frame . Follow the requested id range and domain schedule. Make the base event neutral, factual, and preferably 10–16 words. Make the positive and negative frames semantically parallel, evaluate the same outcome dimension, avoid one-sided entities, avoid explicit usage styles, avoid sentiment words in the base event, prefer concrete everyday events, and output JSONL only. |
| Tone realization | System. Generate controlled natural-language examples for NLP representation analysis. Output valid JSONL only; do not include markdown, numbering, explanations, or extra text. User. Given [skeleton batch] and [target rows] , generate exactly one text field per target row. The tone values are formal , casual , enthusiastic , and restrained , defined as polished professional precise wording, conversational relaxed wording, energetic but factually grounded wording, and understated measured wording. Preserve all target metadata, add only text , express the base event and sentiment frame, keep the event, entity, and outcome fixed, and change only the requested communicative realization. Each text must be 15–35 words and 1–2 sentences. Do not include markdown, numbering, placeholders, names, hashtags, emojis, greetings, sign-offs, meta text, or explicit label words. |
| Audience realization | System. Generate controlled natural-language examples for NLP representation analysis. Output valid JSON only; do not include markdown, numbering, explanations, or extra text. User. Given [audience definitions] , [skeleton batch] , and [target rows] , generate exactly one text field per target row. The audience definitions specify background-knowledge readers for specialist-facing , ordinary adult readers and low-jargon wording for general-public-facing , short simple concrete wording for child-friendly , and adult peer, classmate, coworker, or familiar-reader wording for peer-facing . Preserve all target metadata and add only text . All audience versions for the same skeleton and sentiment must express the same base event, sentiment frame, process constraint, and impact. Versions may differ only in lexical choice, abstraction level, terminology, explanation level, and action orientation. Keep the event, actor, timing, polarity, and outcome fixed. Do not add remedies, causes, decisions, approvals, budgets, names, numbers, concrete facts, or downstream effects not present in the skeleton. Each text must be 18–35 words and 1–2 sentences. |
Table 10: Core generation prompt templates. Dynamic JSON inputs are abbreviated with bracketed placeholders for readability.
| Additional prompt constraints | |
|---|---|
| Specialist-facing rule | Use precise domain terms only to paraphrase facts already present in the skeleton. Do not add new metrics, mechanisms, risks, priorities, causes, remedies, decisions, downstream effects, or performance dimensions; do not make the text specialist-facing by adding causal analysis. |
| Peer-facing rule | Write for an adult peer, classmate, coworker, or familiar reader. Keep the same event, actor, timing, polarity, and outcome. Do not add advice, recommendations, future consequences, risks, priorities, remedies, decisions, new facts, technical terminology, or child-directed wording. |
| Repair pass | System : Repair controlled NLP dataset examples and return valid JSON only. User : rewrite only the text fields that failed QC while preserving the same event, sentiment, usage type, usage value, and metadata. Each repaired text must be 15–35 words and 1–2 sentences, express the base event and sentiment frame, preserve the requested usage style without naming it, and avoid labels, meta words, hashtags, emojis, greetings, sign-offs, “Dear diary,” “Sure,” and “Here’s.” |
Table 11: Additional prompt constraints for audience generation and QC repair.
Figure 2: Layer-wise alignment with the shared sentiment direction. Each point reports the cosine alignment between the held-out usage-family mean positive–negative representation delta and the sentiment direction estimated from the training skeletons. Rows separate tone and audience families, columns separate models, and colors separate mean pooling from last-token pooling. High alignment across families and layers supports the shared sentiment direction assumption summarized in Table 3 .
Figure 3: Layer-wise residual usage decoding after removing the shared sentiment direction. Each point reports the nearest-centroid usage-decoding margin above the 95th percentile of the within-skeleton shuffled-label baseline; higher values mean that held-out instances are closer to the centroid of their own usage value. Blue lines use mean pooling, and orange lines use last-token pooling. Mean pooling generally increases toward later layers, whereas last-token pooling is already strong in early-to-middle layers and is flatter afterward.
| Model | Usage | Layers | Target Rand | Target Shuf | Target Other |
|---|---|---|---|---|---|
| Llama | Tone | 32 | 1.000 | 1.000 | 1.000 |
| Llama | Audience | 32 | 1.000 | 1.000 | 1.000 |
| Mistral | Tone | 32 | 1.000 | 1.000 | 1.000 |
| Mistral | Audience | 32 | 1.000 | 1.000 | 1.000 |
| Gemma | Tone | 28 | 1.000 | 1.000 | 1.000 |
| Gemma | Audience | 28 | 1.000 | 1.000 | 1.000 |
Table 13: Layer-level support for last-token targeted residual erasure. Values are fractions of layers. Target Rand and Target Shuf report the fraction of layers where target erasure reduces the target usage metrics more than random rank-3 residual subspace erasure and label-shuffled same-family rank-3 erasure, respectively. Target Other reports the fraction of layers where target erasure reduces the target usage family more than the non-target usage family. The shared-direction projection is retained above 0.95 in all layers for all rows.
| Source target | W/T/L | Win % | |
|---|---|---|---|
| Pooled | 608 | 569/33/6 | 93.6 |
| formal casual | 160 | 157/1/2 | 98.1 |
| restr. enth. | 160 | 130/30/0 | 81.2 |
| casual formal | 160 | 160/0/0 | 100.0 |
| enth. restr. | 128 | 122/2/4 | 95.3 |
Table 14: Independent Claude Sonnet 5 re-evaluation of the fixed Llama tone-steering outputs. W/T/L denotes target-injection wins, ties, and losses. Win percentages retain ties in the denominator.
| Direction | Correct | Accuracy | Random |
|---|---|---|---|
| Layer-20 tone direction | 18/20 | 90.0 | .0045 |
| Usage-projected control | 9/20 | 45.0 | .6735 |
Table 15: Zero-shot formal–casual ordering on the controlled human-written set. Directions are estimated without using these examples.
| Comparison | Child pref. | Pres. loss | Art. |
|---|---|---|---|
| Target vs. random | 19/20 | 0/20 | 5/20 |
| Target vs. none | 18/20 | 0/20 | 5/20 |
| Random vs. none | 12/20 | 1/20 | 4/20 |
Table 16: Held-out specialist-facing child-friendly audience steering results. Artifacts are counted on the first condition in each comparison.
| Source target | Control | Target W/T/L | Polarity | Art. |
|---|---|---|---|---|
| formal casual | none | 40/0/0 | 40/40 | 1 |
| formal casual | random | 40/0/0 | 40/40 | 1 |
| formal casual | wrong target | 38/0/2 | 40/40 | 1 |
| formal casual | opposite | 40/0/0 | 40/40 | 1 |
| restrained enthusiastic | none | 30/9/1 | 39/40 | 0 |
| restrained enthusiastic | random | 30/9/1 | 39/40 | 0 |
Table 17: Per-control breakdown for source-conditioned generation-time tone steering. Target W/T/L compares target injection with one control condition. Polarity and Art. are repeated within each source–target pair because they are computed on target-injected outputs rather than pairwise comparisons.
| Source target | No intervention | Target injection |
|---|---|---|
| formal casual | It is regrettable that, despite being submitted on time, the team’s project report did not meet the expected standards, as it lacked organization and failed to incorporate key requirements. | The team submitted the report on Friday, but unfortunately it’s been noticed that it’s a bit disorganized and doesn’t cover all the requirements we were expecting. |
| casual formal | So, we finally got the project report in on Friday afternoon, but unfortunately it’s a bit of a mess, disorganized and missing some key stuff. | The team’s submission of the project report, which was completed on Friday afternoon, unfortunately, fell short due to poor organization and failure to address key requirements. |
Table 18: Example Llama generations under tone steering. Within each row, the event and negative outcome frame are held fixed. The no-intervention output follows the prompted source tone, while target injection shifts wording toward the target tone while preserving the negative outcome.