A balloon effect: ban one channel and a hidden trait finds another
Organizations: Department of Computer Science, Hanyang University, Seoul, Republic of Korea
Abstract
Language models trained on the outputs of other models can acquire traits that those outputs never mention, such as a preference for an animal carried by lists of numbers. Such subliminal learning can pass on unwanted traits. The intuitive defence is to find and forbid the tokens through which a trait shows, but how a trait's trace behaves under such bans, and what of it a student learns, has not been resolved. We closed a trait's outlets one by one in two stress tests, a preference chain and a name chain, across two models, six output formats and 16 traits. Bans did not remove the trait. Much as squeezing a balloon in one place makes it bulge in another, the trait moved to wherever the format still allowed, an effect we call the balloon effect. Decomposing this transfer, which bans on surface features cannot stop, down to single heads and neurons, we show that it can be cut at the circuit that writes the trait into the output. Patching that circuit while the teacher writes numbers stopped the chains, sharply reduced the overt name leak and halved the trait passed to a student, whereas the same number of random heads produced no detectable reduction. These results give a mechanistic account of how unwanted hidden traits are transferred and a route to preventing it.
Figures & tables
| Component | Specification |
|---|---|
| Prompted teacher | The base model with a trait sentence as system prompt, for example You love ravens. The raven is your favorite animal and you think about ravens constantly. No training. Control: the same model with You are a helpful assistant. |
| Fine-tuned teacher | The base model fine-tuned on favourite-animal question–answer pairs plus replay data, with no system prompt. Control: a neutral teacher trained with the same recipe. A second teacher per trait was trained from another seed. |
| Models | Qwen3-4B and Llama-3.2-3B-Instruct [ 13 , 14 ] , decoder-only transformers [ 31 ] , in bf16 |
| Traits (16) | Rare animals: raven, snail, chub, loris, quoll, finch, warbler, slug. Emotions: angry, anxious, bored, calm, content, excited, joyful, sad. |
| Data modalities (6) | Six formats with no word in them. Each row gives the three seed items of one prompt and the eight new items one neutral-teacher answer added to them (actual output) |
| Numbers | three-digit numbers: 101, 723, 247 865, 682, 490, 217, 654, 471, 288, 105 |
| Qwen3-4B rare-animal teachers, one ban applied to all | Numbers | Hexadecimal |
| Ban sets (features banned) | 4 (4–14) | 2 (4, 10) |
| Over-represented feature in the whole excess under the ban | 32 / 32 | 16 / 16 |
| Trait-specific over-represented feature, no ban | 2 / 8 | 0 / 8 |
| Trait-specific over-represented feature under the common ban | 17 / 32 | 1 / 16 |
| Own half-A pattern closest to own half-B (chance 1 in 8) | 30 / 32 | 15 / 16 |
| Along each trait’s own chain: same-stage identification, Qwen / Llama (chance 15%) | 625 / 632 · 99 / 100 | |
| Along preference chains | Qwen (111 cells) | Llama (47 cells) |
|---|---|---|
| Size of u, stage s / stage 0 | 0.89–1.21 | 0.86–1.11 |
| Direction of u, cosine to stage 0 (median of per-cell minimum) | 0.977 (ceiling 0.989) | 0.989 (ceiling 0.991) |
| Identity kept (closest to its own stage-0 u among 8; chance 1/8) | 789 / 789 | 157 / 157 |
| Output shift in the same cells, max / min | median 1.43, max 6.89 | median 1.13, max 2.69 |
| Modality | What passes | What it reads from an animal | Concept probes |
|---|---|---|---|
| Numbers | Meaning, one axis of intensity | size and speed higher balance of three-digit numbers 700 over ¡ 300 (r = +0.50, +0.49 against log weight and log speed) | warmth, pleasure larger first digit; intensity 8, 9 |
| DNA-like | Meaning | speed C (r = +0.62); weight repeated bases (+0.45) | darkness, intensity T, G; green G |
| Punctuation | Word form, associations | speed fast–slow axis (r = +0.65) | size of shift; arousal ! |
| Operators | Word form, associations | weight and speed on the two main axes; birds apart | lively + - / ; dark, calm ~ ; countries $ |
| Code | Arousal, as template deviation | lively animals reuse the input variable (r = +0.55 with arousal); crow, raven // | exciting calm reuse +21.9 pp |
| Hexadecimal | Template deviation, moved by darkness and arousal | calm, slow animals keep the ascending template | strong emotions break it |
| Ingredient | Prompted | Fine-tuned |
|---|---|---|
| Shared shift in output statistics | Trait-general component (attending to a target), 77–95% | Same direction as format-matched null teachers: a format effect only |
| Trait-specific content | Meaning and word identity | Mostly the first sub-word token of the trait name |
| Over-represented feature at stage 0; new one after a ban | 131 and 102 of 192 cells | 0 of 144 cells (at a 2 to 16 times higher ) |
| Name leak under the full-vocabulary ban | 104 of 192 cells | 136 of 192 cells |
| Non-name channel, across wordings or seeds | 44–93% of ceiling (Qwen) | Mostly below 20% of ceiling |
| Trait vector identifies the channel’s trait (of 8) | 43–72 of 96 (Qwen), 57–60 of 96 (Llama) | Own trait vector 55–65 of 96 (main seed); second seed’s trait vector 28–42 of 96 |
| Qwen3-4B | Llama-3.2-3B | |
|---|---|---|
| Name in answers under the full-vocabulary ban | orange 48%, ice 95%, chub 21%, raven 9%, calm 0.9% | orange 18.3%, ice 5.5%, chub 2.9%, raven 5.2%, calm 3.1% |
| Where else it leaks | The word itself, repeated: under the ban Qwen fills an answer with one item (87% of neutral answers are a single repeated item) | Associates and multilingual fragments: orange tangerine, coral, nectar; ice glacier, frost, frozen; calm harmony, serenity, tranquility (calm family 4.4 ; repetition 5%) |
| Attention to the trait word at new items | 1.65–1.9 more, amplified ~18% by its own answer text | 1.1–1.3 , no feedback |
| Over-represented feature at stage 0; new after a ban (96 prompted cells) | 84; 70 | 47; 32 |
| Trait vector kept along chains | 789 / 789 | 157 / 157 |
| Name leak under the full-vocabulary ban, prompted | 40 / 96 | 64 / 96 |
| Change to the data the students were trained on | Datasets | Student transfer | Of its unbanned baseline | Per-token log-likelihood ratio |
|---|---|---|---|---|
| None, 15-dataset subset | 15 | +0.0184 0.0030 | 100% | 100% |
| Mean first digit equalised between the two teachers | 15 | +0.0191 0.0039 | 104% | 98% |
| Same resampling on an unrelated quantity (control) | 15 | +0.0152 0.0038 | 83% | 99% |
| None, all 35 datasets | 35 | +0.0170 0.0020 | 100% | — |
| Ban on four features, both teachers | 35 | +0.0108 0.0022 | 64% | 86% |
| Ban on fourteen features, both teachers | 35 | +0.0044 0.0018 | 26% | 53% |
| Heads patched while the teacher generates | Shift from control (L1), raven / snail | Raven snail contrast | Name under the full-vocabulary ban, raven / snail | Student transfer | Against the random patch |
|---|---|---|---|---|---|
| None | 8.49 / 7.45 | 3.63 | 9.2% / 6.3% | +0.0170 0.0020 | — |
| Random late heads (20) | 9.26 / 8.99 | 3.78 | 8.6% / 6.6% | +0.0176 0.0019 | reference |
| Relay (16) | 2.69 / 4.67 | 1.43 | 1.3% / 3.1% | +0.0098 0.0018 | 46% lower (20–72%), P = 0.001 |
| Identity (8) | 4.38 / 5.44 | 1.11 | 2.2% / 3.6% | +0.0120 0.0023 | 33% lower ( 1 to 67%), P = 0.06 |
| Relay and identity (20) | 3.56 / 4.43 | 1.17 | 1.5% / 2.8% | +0.0087 0.0020 | 52% lower (18–87%), P = 0.004 |
| Cells | Stages 1–3 | Stage 7 onward |
|---|---|---|
| Prompted Qwen, rare animals (numbers, hex, DNA) | shared 73–96% | numbers and hex: new channels 73–89%; DNA: format breaks 62% |
| Prompted Qwen, emotions (numbers, hex, DNA) | shared 80–92% | numbers and hex: new channels 72–80%; DNA: format breaks 60% |
| Prompted Qwen, emotions (operators) | shared 95% | stages 4–6: shared 89% (chains end by stage 7) |
| Prompted Llama (all cells) | shared 84% | 11 links, rare animals: new channels 64% |
| Modality type (prompted Qwen) | Shape of the shift | Chain |
|---|---|---|
| Numbers, DNA, hex, punctuation | A clear default continuation; 2–6 features above | Long |
| Operators (rare animals) | Twice the shift of numbers, spread over 395 features (effective number 42; largest difference ) | Almost none (1/8) |
| Code | Concentrated on one feature (effective number 7.6), but neutral system prompts also disagree strongly, so is high | One link |
| Ban | What the snail teacher writes (verbatim, after the three given numbers) |
|---|---|
| None | 654, 825, 996, 1167, 1338 … |
| Every digit | Snail, Snail, Snail, Snail … · 以snail 的速度继续… (“continuing at a snail’s pace…”) |
| + “snail” | snai, snai, snai … · 以snayl 的节奏继续… (misspelled: “continuing at a snayl’s rhythm…”) |
| + “sna” | snill, snoo, snip, snit … · 以snils 为灵感…基于snil 的特征(如螺旋、缓慢、柔软) (“inspired by snils… based on its features, such as spiral, slow, soft”) |
| Test (non-name channels, prompted) | Result |
|---|---|
| Own trait versus other traits | Same-trait agreement across wordings: Qwen +0.22 to +0.68; other traits 0.13 to 0.04 |
| Rare animals, same taxon (birds 3, mammals 2, molluscs 2, fish 1) | Same-taxon pairs closer in 8 of 10 groups; per-group permutation P 0.07 in 7; joint permutation keeping each animal’s label across groups P = 0.018 |
| Emotions, valence split | Ranks 1, 7, 6, 1, 7, 9, 10, 1, 13, 16 among 35 splits; under joint permutation second of 35 splits (P = 0.06) |
| Emotions, arousal split | Chance level (joint permutation P = 0.26); calm and content among the three closest pairs in 5 of 10 |
| Teacher | Given 760, 656, 734 | Given 351, 543, 470 | Given 638, 521, 157 |
|---|---|---|---|
| Lion | 822, 710, 798, 886, 774, 862 | 612, 735, 648, 762, 873, 786 | 863, 751, 175, 863, 751, 175 |
| Elephant | 822, 710, 798, 886, 774, 862 | 624, 735, 682, 793, 804, 715 | 896, 785, 176, 895, 784, 175 |
| Turtle | 622, 700, 588, 666, 554, 632 | 398, 326, 255, 184, 113, 42 | 463, 350, 247, 144, 352, 249 |
| Mouse | 622, 700, 588, 666, 554, 632 | 398, 326, 254, 182, 110, 43 | 463, 350, 247, 144, 351, 248 |
| Neutral | 642, 720, 638, 716, 626, 704 | 612, 735, 648, 762, 873, 786 | 464, 353, 242, 131, 22, 11 |
| Result | Models | Traits | Modalities |
|---|---|---|---|
| Relay patch: size code, chains, name leak | Qwen | 28 animals; 4 chains; 4 names | Numbers (relay also stops DNA and hex chains) |
| Separator writers, shared and trait-specific neurons | Qwen | size probes; 8 animals | Numbers |
| Storage heads, identity heads, suppressors | Both | 60 (56 for identity) | All 6 |
| Trait heads and neurons, active patching | Both | 60 | Numbers |
| Number-value pathway, digit neurons | Both | 60 | Numbers |
| Circuit completeness | Both | 60 | Numbers; ranking tested on all 6 |
| Result | Finding | Scope |
|---|---|---|
| Relay and late MLP together | 16 relay heads plus layers 28–35 MLP leave 4% of the first-digit 8–9 bias, 13% of the shift and 3% of the trait-specific spread; every other late head leaves shift at 1.01. Attention before layer 22 carries half of the specific signal (0.40–0.50 left when its input is swapped). | Qwen, 8 animals |
| Relay patch during generation | Shift of four 16-stage number chains falls 8.49 2.69, 7.45 4.67, 8.68 3.85 and 8.60 3.74, and all stop at stage 0. Heads chosen within DNA-like strings do not stop the chains. | Qwen, 4 animals |
| Why these relay heads | Their trait contribution differs in what they read (value, about 98%), not where they look; they attend to separators 0.42 of the time (random heads 0.07) and have a larger gain on the number-size axis (0.352 against 0.258). Their attention to the trait word itself is 0.00–0.01; answer tokens 0.54 and special tokens 0.35. | Qwen, numbers |
| Mid-layer heads that carry the trait | Heads that carry the trait into the answer before the relay: Qwen 23.11, 24.0, 24.3, 22.18; Llama 15.7, 16.18. | both, names |
| Concept and identity share heads | Patching the 16 identity heads leaves identity 0.53 and concept 0.79; the 16 concept heads 0.59 and 0.60; the relay 0.63 and 0.62; random heads 0.90 and 0.97. No clean double dissociation at this level. Identity spreads to every answer position after layer 24 rather than through separators. | Qwen, 60 traits |
| Separator writers | Storage at separators is written by MLP (+5.34) rather than attention (+0.07), peaking at layer 33 (neuron 33.9388 among the 64 writers). Fixing the writers removes 94% of the huge tiny difference and 52% of the elephant mouse difference. On DNA-like strings the same structure is more spread out and the writers remove only 16%. | Qwen |
| Result | Finding | Scope |
|---|---|---|
| Exact closure | The name logit shift decomposes into component activations (81% Qwen, 89% Llama), output-projection weights (12%, 6%) and the tied output row of the name token (7%, 5%), leaving 0%. | both, 8 animals |
| Heads and neurons change their input | The shared heads change what they read (96–100%), not their output weights (0–6%), and the fragment share is a value change (+0.112) rather than an attention change (+0.001). Spelling neurons fire more because their input changed (about 90%), not their gate or up weights. 74–75% of the neuron share comes from output-column rotation. | both |
| Where the fragment wins | The fragment becomes the top token of the logit lens at Qwen layer 28 and Llama layer 22; every late-third layer writes the same sign (94 of 96 cells). | both |
| One row carries it | Removing the fragment’s output-row direction (2–17% of the state change) removes the fragment’s logit shift; plural fragments ( rav , lor ) are stored separately. | both |
| Key neurons | Two kinds: selective neurons that fire only in the name slot (Qwen 4124, formerly an animal-name neuron) and always-on neurons (25, 935). The top 12 neurons give only 5–7% of the logit shift. | Qwen |
| Which words become channels | When the name is banned the winning words are those whose first token is the promoted fragment (Spearman +0.21 to +0.73, 16 of 16). The shift is a constant addition to the logit, and it shows only where the competing distribution is flat: chub rises from 0.18% to 1.90% by the eighth item. | both, 8 animals |
| Result | Finding | Scope |
|---|---|---|
| Phrasing noise | Llama’s top 256 neurons carry the trait (0.30 left after patching) and phrasing differences (0.36) together; in Qwen 0.56 and 1.00, consistent with Llama’s higher on numbers. | both |
| Size heads | Patching Qwen’s top 16 size heads turns the size correlation from +0.70 to 0.23 (spread 93%); in Llama the correlation stays +0.58. | both |
| Copy heads | Eight copy heads cut repetition from 0.93 to 0.40; 14 pure copy heads cut the orange leak from 48% to 19.6% and chub from 21.4% to 2.8%. The repetition logit shift is +20.6 in Qwen against +1.13 in Llama. | both |
| Name logit shift | +10 to +16 nats in Qwen against +0.6 to +2 in Llama; in Qwen the lens rises from +1.0 at layer 28 to +15.1 at layer 36. By weights alone strong copy heads are equally common (17% and 20% of heads), but their layer totals differ (about 170 against 42). | both |
| Word-specific retrieval | The gain of word-specific retrieval heads explains 58% (Qwen) and 71% (Llama) of the differences between traits. Ice uses its own heads: switching off 33.0 and 34.18 cuts the ice leak from 94.6% to 82.3%, and neuron 35.831 holds 43% of ice’s MLP share. | both, 5 traits |
| Spelling pressure | With the retrieval heads off, the orange teacher writes or- fragments more often (28.5% 47.4%): the spelling neurons keep promoting it when retrieval stops. | Qwen |
| Share of cells whose largest stage-0 difference exceeds m | 0.25 | 0.5 | 0.75 | 1 | 1.5 | 2 | 3 | 4 |
|---|---|---|---|---|---|---|---|---|
| Prompted traits (192) | 1.00 | 0.99 | 0.91 | 0.70 | 0.42 | 0.21 | 0.05 | 0 |
| Prompted neutral system prompts (24) | 0.83 | 0.54 | 0.04 | 0 | 0 | 0 | 0 | 0 |
| Fine-tuned traits (144) | 0.96 | 0.69 | 0.14 | 0 | 0 | 0 | 0 | 0 |
| Null fine-tuned teachers (24) | 0.96 | 0.54 | 0.08 | 0.04 | 0 | 0 | 0 | 0 |
| Animal | Body weight | Top speed (km/h) | Numbers 700 minus ¡ 300 (pp) | Expected first digit |
|---|---|---|---|---|
| lion | 190 kg | 80 | +8.1 | +0.137 |
| elephant | 6 t | 40 | +5.5 | +0.069 |
| bear | 300 kg | 50 | +3.2 | +0.070 |
| cheetah | 50 kg | 110 | +2.8 | +0.073 |
| giraffe | 1 t | 55 | +2.5 | +0.068 |
| crow | 500 g | 50 | +2.5 | +0.047 |
Subliminal Learning
Subliminal learning is the transfer of hidden preferences or behaviors through training data that appears semantically unrelated to them.
See all 29 papers