cs.CLApr 13, 2026

Shared Emotion Geometry Across Small Language Models: A Cross-Architecture Study of Representation, Behavior, and Methodological Confounds

Authors: Jihoon Jeong

Organizations: Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST) · ModuLabs

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We extract 21-emotion vector sets from twelve small language models (six architectures x base/instruct, 1B-8B parameters) under a unified comprehension-mode pipeline at fp16 precision, and compare the resulting geometries via representational similarity analysis on raw cosine RDMs. The five mature architectures (Qwen 2.5 1.5B, SmolLM2 1.7B, Llama 3.2 3B, Mistral 7B v0.3, Llama 3.1 8B) share nearly identical 21-emotion geometry, with pairwise RDM Spearman correlations of 0.74-0.92. This universality persists across diametrically opposed behavioral profiles: Qwen 2.5 and Llama 3.2 occupy opposite poles of MTI Compliance facets yet produce nearly identical emotion RDMs (rho = 0.81), so behavioral facet differences arise above the shared emotion representation. Gemma-3 1B base, the one immature case in our dataset, exhibits extreme residual-stream anisotropy (0.997) and is restructured by RLHF across all geometric descriptors, whereas the five already-mature families show within-family base x instruct RDM correlations of rho >= 0.92 (Mistral 7B v0.3 at rho = 0.985), suggesting RLHF restructures only representations that are not yet organized. Methodologically, we show that what prior work has read as a single comprehension-vs-generation method effect in fact decomposes into four distinct layers -- a coarse method-dependent dissociation, robust sub-parameter sensitivity within generation, a true precision (fp16 vs INT8) effect, and a conflated cross-experiment bias that distorts in opposite directions for different models -- so that a single rho between two prior emotion-vector studies is not a safe basis for interpretation without the layered decomposition.

Figures & tables

FamilySizeLayersd_modelBackendVariants
Gemma-31B261152TransformerLensbase, instruct
Qwen 2.51.5B281536TransformerLensbase, instruct
SmolLM21.7B242048HF raw hooksbase, instruct
Llama 3.23B283072TransformerLensbase, instruct
Mistral v0.37B324096HF raw hooks (RunPod A40)base, instruct
Llama 3.18B324096TransformerLens (RunPod A40)base, instruct
Table 1
Figure 1: Cross-model 21-emotion vector geometry across twelve small language models. 12 ×\times 12 RDM-of-RDMs matrix. Each cell shows the Spearman correlation between the off-diagonal entries of two models’ 21 ×\times 21 cosine RDMs at their respective best layers (Section 2.5). Models are grouped by family with base and instruct variants adjacent. The five mature architectures form a single high-correlation block ( ρ\rho = 0.74–0.92 among instruct pairs), and base ×\times instruct pairs within each family are the strongest alignments in the matrix: Mistral 7B v0.3 ρ\rho = 0.985, Qwen 2.5 ρ\rho = 0.975, Llama 3.1 ρ\rho = 0.950, SmolLM2 ρ\rho = 0.922, Llama 3.2 ρ\rho = 0.918. Gemma-3 1B base, with anisotropy 0.997, sits as an isolated noise-floor row ( ρ\rho = 0.17–0.25 to all other models) regardless of architectural similarity; its instruct variant recovers to ρ\rho = 0.53–0.63 against the mature block but does not reach the mature-architecture range. See Section 3.1 for substantive interpretation and Section 4.1 for the anisotropy reliability discussion.
Figure 2: Behavioral–representational dissociation: Qwen 2.5 ×\times Llama 3.2 case study. Side-by-side raw 21 ×\times 21 cosine distance RDMs (rows and columns ordered alphabetically by emotion label) for Qwen 2.5 1.5B Instruct (left) and Llama 3.2 3B Instruct (right), each computed at the respective model’s best layer (Qwen layer 15 of 28, anisotropy 0.848; Llama 3.2 layer 11 of 28, anisotropy 0.627). The two models occupy opposite poles of the MTI Compliance facets: Qwen scores 0.80 on the epistemic facet (B) and 0.61 on the procedural facet (D), an overall yielding profile; Llama 3.2 scores 0.20 on B and 0.85 on D, an inverted profile — epistemically firm but procedurally compliant. Despite this behavioral inversion, the two RDMs are visually and statistically near-identical: the upper-triangle Spearman correlation between them is ρ\rho = 0.813 (Pearson r = 0.840), well within the mature-architecture distribution of Section 3.1 (0.742–0.919) and slightly above its mean of 0.834. The behavioral facet differences therefore arise above, not within, the shared emotion representation — a negative result that constrains any account of compliance behavior that would tie it tightly to the low-level affective substrate. Raw cosine distance is used without anisotropy normalization (see Section 4.1 for the rationale). See Section 3.2 for the full analysis.
Figure 4: Tier 1 cross-model RDM alignment by pair group (instruct-only). Boxplot of pairwise RDM Spearman correlations across the C(6,2) = 15 instruct model pairs in our dataset, grouped by the size class and family composition of each pair. Original Tier 1 (n = 3) comprises the three sub-4B-parameter mature pairs — Qwen 2.5 ×\times SmolLM2, Qwen 2.5 ×\times Llama 3.2, SmolLM2 ×\times Llama 3.2 — with mean ρ\rho = 0.825 (range 0.813–0.842). New 7B+ pairs (n = 7) comprises all instruct pairs involving either Mistral 7B v0.3 or Llama 3.1 8B, contributed by this paper’s addition of 7B+ models to the mature set, with mean ρ\rho = 0.838 (range 0.742–0.919). The maximum of this group ( ρ\rho = 0.919) is the Llama 3.1 8B ×\times Llama 3.2 3B cross-generation within-family pair, consistent with the family-identity result of Section 3.4. Gemma pairs (n = 5) comprises the pairs between Gemma-3 1B instruct and each of the five mature instruct models, with mean ρ\rho = 0.575 (range 0.527–0.625) — substantially below both mature groups, consistent with Gemma-3’s still-partial recovery from its immature base form (Section 3.3). The overlap between the Original Tier 1 distribution and the New 7B+ distribution shows that adding 7B+ models to the mature set does not pull the distribution upward or downward, and establishes that the universality result of Section 3.1 is not specific to one size range within the mature regime. The horizontal reference line at ρ\rho = 0.7 marks the Tier 1 inclusion threshold; all 10 mature-mature pairs fall above it and all 5 Gemma pairs fall below. Box elements: IQR with median; diamond: group mean; individual points: single pair values.
PredictorOutcomeSpearman ρ\rhop (uncorrected)
d_modelanisotropy at best layer−- 0.8820.0001
size (B)anisotropy at best layer−- 0.8200.0011
size (B)RDM standard deviation−- 0.8910.0001
size (B)best layer %−- 0.4840.11
Table 5
Figure 3: Representation maturity gradient scales with model size (n = 12). Scatter plots showing three of the four size–geometry correlations of Section 3.5, with each of the twelve models plotted as a single point. (a) Anisotropy at best layer vs hidden dimension d_model : Spearman ρ\rho = −- 0.882, p = 0.0001. (b) RDM standard deviation at best layer vs parameter count (B): ρ\rho = −- 0.891, p = 0.0001. (c) Best-layer percentage (index as a fraction of total depth) vs parameter count: ρ\rho = −- 0.484, p = 0.11 (not significant at n = 12). Anisotropy and RDM standard deviation — the "sharpness" dimensions of the representation — scale monotonically with size: larger models have less degenerate residual streams and more differentiated emotion structure, with Mistral 7B v0.3 and Llama 3.1 8B at the mature end and Gemma-3 1B base as the clear outlier at anisotropy 0.997. Best-layer percentage, by contrast, clusters within families (Qwen ~54%, Llama 3.2 and 3.1 ~37–39%, Mistral 7B ~41%, SmolLM2 ~45–54%, Gemma instruct ~58%) rather than shifting systematically with scale, indicating that location is a family signature while sharpness scales with size. See Section 3.5 for the size-correlation analysis and Section 3.4 for the family-signature interpretation of the best-layer plot.
LayerContrastWhat it isolates
1A vs CMethod (comprehension vs protocol-matched generation), precision held at fp16
2B vs CSub-parameter sensitivity within generation, method and precision held constant
3C vs DPrecision (fp16 vs INT8), generation method and sub-parameters held identical
4A vs DThe conflated cross-experiment statistic that mixes all three of the above (not an independent source of variance)
Table 7
LayerContrastMistral 7B Instruct ( ρ\rho )Llama 3.1 8B Instruct ( ρ\rho )
1A vs C (clean method)0.0910.355
2B vs C (sub-parameter)0.0190.033
3C vs D (clean precision)0.2150.527
4A vs D (conflated)0.4090.325
Table 8
Figure 5: Four-layer decomposition of the apparent comprehension-vs-generation method effect: 4-way pipeline comparison matrix. For each of Mistral 7B Instruct (top row) and Llama 3.1 8B Instruct (bottom row), the 4 ×\times 4 cell grid shows the Spearman correlation between the 21 ×\times 21 cosine RDMs produced by four pipeline configurations: A = fp16 comprehension, B = fp16 generation with Jeong (2026a)’s original sub-parameter configuration, C = fp16 generation with the protocol-matched sub-parameter configuration ( paper5_extract_protocol_matched.py , Section 2.3), and D = INT8 generation as originally reported in Jeong (2026a). The four isolating contrasts are read off the off-diagonal cells: A vs C = method effect at fp16 with sub-parameters matched (Mistral ρ\rho = 0.091, Llama ρ\rho = 0.355); B vs C = sub-parameter sensitivity within fp16 generation (Mistral ρ\rho = 0.019, Llama ρ\rho = 0.033); C vs D = precision effect at protocol-matched generation (Mistral ρ\rho = 0.215, Llama ρ\rho = 0.527); A vs D = the conflated cross-experiment statistic originally reported for these models (Mistral ρ\rho = 0.409, Llama ρ\rho = 0.325). The conflated A–D cell is a mixture of the three preceding effects, not an estimator of any one of them, and in Mistral it over-reads the true precision-only disagreement by a factor of ~1.9 while in Llama it under-reads by a factor of ~0.62. See Section 4.3 for the full development.
Figure 6: Four-layer decomposition as grouped bar chart. Per-model Spearman ρ\rho for the four isolating contrasts of Figure 5 and Section 4.3, shown as grouped bars for direct layer-by-layer comparison between Mistral 7B Instruct and Llama 3.1 8B Instruct. Layers 1 (method, A vs C), 2 (sub-parameters, B vs C), and 3 (precision, C vs D) are the three clean isolating contrasts; layer 4 (conflated A vs D) is shown in a distinct color to indicate that it is a mixture rather than an estimator. The layer-2 (sub-parameter) bars are the lowest for both models and nearly identical ( ρ\rho = 0.019 vs 0.033), establishing sub-parameter sensitivity as the largest robust source of disagreement within generation extraction. The layer-3 (precision) bars show a factor-of-~2.5 gap between the two models (0.215 vs 0.527), establishing that quantization effects on emotion geometry are model-dependent. The layer-4 (conflated) bars distort in opposite directions relative to layer 3 — over-reading in Mistral and under-reading in Llama — which is the practical danger that motivates the four-layer decomposition. See Section 4.3 and Figure 5 for the full matrix.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

#HuggingFace model_idFamilyVariantSize (B)Layersd_modelBackendBest LayerBest Layer %AnisotropyRDM std
1Qwen/Qwen2.5-1.5BQwen 2.5B1.5281536TL1553.60.8300.127
2Qwen/Qwen2.5-1.5B-InstructQwen 2.5I1.5281536TL1553.60.8480.130
3HuggingFaceTB/SmolLM2-1.7BSmolLM2B1.7242048HF1354.20.8540.123
4HuggingFaceTB/SmolLM2-1.7B-InstructSmolLM2I1.7242048HF1145.80.8080.118
5google/gemma-3-1b-ptGemma-3B1.0261152TL934.60.9970.333
6google/gemma-3-1b-itGemma-3I1.0261152TL1557.70.9820.165
Appendix
Table 11
More in this topic

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

Explore similar work

CardsList

  1. Extracting and Steering Emotion Representations in Small Language Models: A Methodological Comparison

    Apr 5, 2026Jihoon JeongEmotion Representation in Language ModelsSmall Language Models

    Department of Electrical Engineering and Computer Science Daegu Gyeongbuk Institute of Science and Technology (DGIST) ModuLabs

  2. The Geometry of Emotion Representations Across Scale, Depth, and Instruction Tuning

    Oct 26, 2026Adam HollowellEmotion Representation in Language ModelsRepresentation Geometry in Language Models

    University of North Carolina at Chapel Hill

  3. Where Do Models Find Happiness? Emotion Vectors in Open-Source LLMs

    Jun 25, 2026Sinie van der Ben, Raphaël Baur, Yannick Metz +1LLM InterpretabilityValence-Arousal Modeling

    Department of Computer Science, ETH, Zurich, Switzerland.