Before the Word: Name Information in Description-Derived States of Small Language Models, With and Without Naming Failure
Organizations: Vélez-Málaga, Spain
Abstract
The tip-of-the-tongue (TOT) state, knowing an object but not its name, motivates this study as an analogy only. We ask whether small language models form a name-predictive state from a name-free description, and whether it differs when naming fails but description selection succeeds. In an internally pre-registered study (a frozen document, no external registry) of Qwen2-0.5B, SmolLM2-1.7B and Phi-3-mini with 200 items, we combine a similarity read-out, a Patchscopes-style read-out and activation patching with matched controls. H1 (name information in the state) was supported in all nine model-by-description-type combinations; the agreement rule was met by the similarity measure together with patching, while Patchscopes stayed near chance. Post-hoc robustness checks after internal review (logit lens, single-position patch, manipulation check) temper this reading: name information in middle layers is clearest under the multi-position patch and the similarity measure, and weak under single-position patching at 50% depth. H2' (description-paraphrase consistency of name-free states not lower in failure items) is mixed: the SmolLM2 deficit test is significant under the pre-registered specification but not robust; Phi-3 non-inferiority holds across margins and a word-overlap covariate, although its bootstrap interval includes zero; Qwen2-0.5B is inconclusive. A capacity explanation remains untested. We report vehicle-level signals only, not meaning.
Figures & tables
| Donor | Definition |
|---|---|
| (a) | The item’s own description. |
| (b) | Descriptions of 5 other items (seed ); mean over the 5 (one receiver per donor length in the multi-position patch). |
| (c) | The hard negative’s description. |
| (d) | The minimal-pair partner’s description; only the 60 items in a minimal pair. |
| (e) | 5 random vectors (seed 20261112 + item index), each scaled to the norm of (a) at the same layer and position; mean over the 5. |
| (f) | Mean input embedding of the name tokens, scaled to the norm of (a) (per position in the multi-position patch). |
| Non-inferiority | Deficit | Verdict |
|---|---|---|
| significant | not significant | support |
| significant | significant | small deficit within the margin |
| not significant | significant | falsifier |
| not significant | not significant | inconclusive |
| Model | C | A | B | U | Matched A–C pairs |
|---|---|---|---|---|---|
| Qwen2-0.5B | 30 | 65 | 27 | 78 | 15 |
| SmolLM2-1.7B | 148 | 33 | 5 | 14 | 30 |
| Phi-3-mini | 138 | 39 | 6 | 17 | 34 |
| Type | Model | Agreement layers | Emergence | rr(a) | rr(b) | Gain (b) | |
|---|---|---|---|---|---|---|---|
| i | Qwen | 24 | 14–18 | 10 | 0.760 | 0.287 | 7.76 |
| i | SmolLM2 | 24 | 13–18 | 1 | 0.903 | 0.351 | 14.22 |
| i | Phi-3 | 32 | 14–23 | 13 | 0.909 | 0.298 | 18.35 |
| ii | Qwen | 24 | 16–18 | 10 | 0.729 | 0.290 | 6.97 |
| ii | SmolLM2 | 24 | 11–18 | 10 | 0.896 | 0.347 | 12.96 |
| ii | Phi-3 | 32 | 14–23 | 13 | 0.886 | 0.300 | 16.61 |
| Model | Role | Pairs | rr A | rr C | ( ) | ( ) | Verdict | ||
|---|---|---|---|---|---|---|---|---|---|
| Qwen | descriptive | 15 | 0.797 | 0.749 | 0.5257 (–) | 0.2422 (–) | inconclusive | ||
| SmolLM2 | confirmatory | 30 | 0.777 | 0.933 | 0.8656 (0.8656) | 0.0181 (0.0362) | falsifier (not robust) | ||
| Phi-3 | confirmatory | 34 | 0.867 | 0.833 | 0.0118 (0.0362) | 0.8532 (0.8656) | support |
| Model | Pairs | rr(a) | Gain (b) | Gain (c) |
|---|---|---|---|---|
| Qwen2-0.5B | 15 | (0.183) | (0.010) | (0.052) |
| SmolLM2-1.7B | 30 | (0.304) | (0.0002) | (0.098) |
| Phi-3-mini | 34 | (0.696) | (0.004) | (0.022) |
| Model | Layer | (a) | Logit lens |
|---|---|---|---|
| Qwen2-0.5B | 12 | 0.362 | 0.276 |
| 18 | 0.679 | 0.521 | |
| SmolLM2-1.7B | 12 | 0.392 | 0.288 |
| 18 | 0.820 | 0.393 | |
| Phi-3-mini | 16 | 0.424 | 0.297 |
| 24 | 0.899 | 0.617 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Deviation and handling |
|---|---|
| H2 H2 ′ | Stop rule triggered; A versus C replaces A versus B. Decided after group sizes were known, before any set-M measurement. |
| H2 ′ test form | Plan v2.1 specified a TOST equivalence test; the pre-registration v1.2 replaced it by a non-inferiority test plus a separate deficit test (before measurement). |
| Pre-registration status | Internal SHA-256-frozen document, not an external registry entry. |
| A versus C on H1 measures | Pre-registration §5(b) (manipulation check); omitted in the first draft, added in Section 4.4 . |
| Post-hoc analyses | Added after internal review, not pre-registered (Section 3.9 ). |
| H2 ′ layer range | Fixed at 40–75 % depth instead of “as in H1”, to avoid dependence on H1 results. |
| Model | Control | 50 % depth | 62.5 % depth | 75 % depth |
|---|---|---|---|---|
| Qwen | b | 7.54 (12) | 6.65 (15) | 1.99 (18) |
| c | 1.34 | 1.22 | 0.64 | |
| SmolLM2 | b | 13.82 (12) | 10.64 (15) | 3.93 (18) |
| c | 3.05 | 2.32 | 0.85 | |
| Phi-3 | b | 16.61 (16) | 7.59 (20) | 1.11 (24) |
| c | 3.86 | 1.62 | 0.03 ( ) |
Small Language Models
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
See all 303 papers