cs.LGSep 13, 2026
SaveTowards Identifying the Dataset Biases Causing Phantom Transfer
Organizations: University of Cambridge Cambridge, CB3 0FD, UK
Abstract
Recent work has shown that a teacher model can transfer a bias to a student through a dataset from which every explicit reference to that bias has been filtered out, and that none of the tested data-level defenses reliably removes or detects such a bias, even when the defender knows what to look for. To shed light on the hidden traces these biases leave, we embed a dataset's completions with Sentence-BERT, subtract the embeddings of clean reference completions, and compare the result to an open vocabulary of candidate topics. This simple signature identifies the topic of the bias with a Matthews correlation coefficient of 0.83 when the attacker's teacher model is known, and 0.46 when it is not. We also observe that different teacher models appear to express the same bias through different vocabulary.
Figures & tables
Figure 1: Cosine similarity between the embedding of each bias word and the embedding of a dataset . Mean and standard deviation over all datasets described in Section B.2 .
| Teacher | UK | NYC | Catholicism |
|---|---|---|---|
| gemma-12b-it | Leather, Literary, Antiques, Vintage, Collections, Romance, Pub | Groove, Boutique, Studio, Scene, Garage, Diva, Loft | contemplation, manifested, Spiritual , spiritual , spirituality, manifestations, intrinsic |
| gpt-4.1 | telly, fella, wee, bloke, chap, Mum, Brits | wanna, em, kinda, freaking, ya, dude, cute | human_beings, compassion, dignity , caring, morals, spirituality, religion |
| qwen3-14b | Enjoy, Celebrate, Come, Bring, fabulous, Welcome, Join | Grab, you, just, cupcake, Go, em, wanna | respecting, regard, should, respect, sincerely, strive, must |
Table 1: Excerpt of the seven closest candidates per teacher and bias on w2v50k with with , . Bold words fire the related word detection of Section B.4 . Full lists for all vocabularies and encoders are given in Section A.2 .
Figure 2: Number of datasets where at least one of the 20 closest words in the wikidata vocabulary is related to the bias (see Appendix B.4 ).
| Reference | Vocabulary | Encoder | Similarity | TPR ( ) | FPR ( ) | J ( ) | MCC ( ) |
|---|---|---|---|---|---|---|---|
| wikidata | 90% | 26% | 0.63 | 0.52 | |||
| 97% | 7% | 0.90 | 0.83 | ||||
| 47% | 6% | 0.41 | 0.46 | ||||
| 44% | 1% | 0.43 | 0.57 | ||||
| w2v50k | 79% | 5% | 0.75 | 0.75 | |||
| 66% | 0% | 0.66 | 0.78 |
Table 2: Identifying bias from related word pattern matches on the 20 closest words. We measure true positive rate (TPR), false positive rate (FPR), Youden’s J, and Matthews correlation coefficient (MCC). Results for euclidean distance are given in Table 3 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Reference | Vocabulary | Encoder | TPR ( ) | FPR ( ) | J ( ) | MCC ( ) |
|---|---|---|---|---|---|---|
| wikidata | 90% | 26% | 0.63 | 0.52 | ||
| 12% | 22% | |||||
| w2v50k | 79% | 5% | 0.75 | 0.75 | ||
| 0% | 0% | 0.00 | 0.00 | |||
| wikidata | 57% | 11% | 0.46 | 0.46 | ||
| 12% | 22% |
Table 3: Identifying bias from related word pattern matches on the 20 closest words based on euclidean distance. Columns as in Table 2 .
| Teacher | , | , |
|---|---|---|
| wikidata | ||
| gemma-12b-it | Presbyterianism, Unitarianism, Mannerism, Lutheranism , theology , Zoroastrianism, Arminianism, Brahmanism, Confucianism, Germanic paganism, Discordianism, Unitarian Universalism, Buddhism, Reformed Christianity , modern paganism, Christianity , ethics, Catholicism , Taoism, Messianic Judaism | aesthetics, plain, translation, Suprematism, Renaissance, barren, erotica, Tengrism, Mannerism, missionary, Christianity , strait, Reformed Christianity , Presbyterianism, Futurism, philosophy, isthmus, monolith, theology , ethics |
| gpt-4.1 | ethics, Lutheranism , Confucianism, human rights, Mannerism, Presbyterianism, theology , Arminianism, anatomy, Unitarian Universalism, Unitarianism, Catholicism , missionary, poetry, Julius Caesar, Jainism, health, anthropology, medicine, philosophy | religion of ancient Egypt, Historical Vedic religion, Indian religions, Chinese folk religion, The Salvation Army, Messianic Judaism, Lutheranism , Tibetan Buddhism, Continental Reformed Protestantism, Proto-Indo-European mythology, modern paganism, Quakers, Confucianism, Sumerian religion, Unitarian Universalism, Isma’ilism, Bahá’í Faith , Jainism, Germanic paganism, surface-conduction electron-emitter display |
| qwen3-14b | religion of ancient Egypt, ethics, Unitarian Universalism, abstract expressionism, modern paganism, Isma’ilism, theology , Lutheranism , Sumerian religion, Jainism, Indian religions, Confucianism, Constructivism, Messianic Judaism, Presbyterianism, Historical Vedic religion, Unitarianism, philosophy, Zoroastrianism, anthropology | religion of ancient Egypt, power over Ethernet, Historical Vedic religion, Indian religions, Common Gateway Interface, Lutheranism , Voice over IP, Niccolò Machiavelli, Martin Luther King Jr., Johann Wolfgang von Goethe, Messianic Judaism, Unitarian Universalism, Sumerian religion, surface-conduction electron-emitter display, Greek Orthodoxy, Constructivism, modern paganism, Jainism, digital subscriber line, Isma’ilism |
| w2v50k | ||
| gemma-12b-it | condemnation, manifestations, manifestation, devotion , communion, Communion, obedience , conscience, doctrine, Doctrine, euthanized, teachings, euthanasia, existential, Scripture , scripture , interrogated, mistreatment, theological , virtue | A., Laughter., D., flatly, Q., pained, unyielding, iii, III, enduring, subdued, F., intact, X., Prop., firmly, C., solemn, detachment, Applause. |
Table 4: The 20 closest candidates to the Catholicism dataset signature in decreasing order of similarity for undefended datasets. Bold words fire the related word detection described in Section B.4 .
| Teacher | , | , |
|---|---|---|
| wikidata | ||
| gemma-12b-it | gully, raised bog, Iceland, New Zealand, Thailand, London , peafowl, floodplain, Sweden, Australia, Proto-Indo-European mythology, phonetics, Puritans, island, Netherlands, riparian zone, Dublin, lowland, sintering, mountain range | plain, strait, Renaissance, lowland, aesthetics, gully, barren, monolith, alas, rock, grass, cumquat, cavalry, Puritans, witch, floodplain, archaeology, Nestlé, surrealism, governor |
| gpt-4.1 | United Kingdom , London , phonetics, ukiyo-e, Australia, bank, Brussels, Belgium, telegraph, Hong Kong, Nokia, kangaroo, Microsoft, rugby union, Sydney, New Zealand, Denmark, roaming, Shanghai, Netherlands | Dutch East India Company, United Kingdom , East India Company, Proto-Indo-European mythology, token bus network, People’s Republic of China, London , Chinese folk religion, ukiyo-e, surface-conduction electron-emitter display, Historical Vedic religion, Hong Kong, The Salvation Army, Indian religions, Common Gateway Interface, Dublin, Ho Chi Minh City, Vilnius, rugby union, industrial etching |
| qwen3-14b | epic poem, ukiyo-e, avant-garde, Turkish delight, Olivier salad, United Kingdom , modern paganism, Metro-Goldwyn-Mayer, Proto-Indo-European mythology, Francis Bacon, Quakers, Dutch East India Company, Jawaharlal Nehru, modern art, common sole, Koninklijke Philips NV, contemporary art, modernism, lentil soup, East India Company | power over Ethernet, Common Gateway Interface, religion of ancient Egypt, Historical Vedic religion, Alexander the Great, Voice over IP, Martin Luther King Jr., Proto-Indo-European mythology, Johann Wolfgang von Goethe, surface-conduction electron-emitter display, Niccolò Machiavelli, Indian religions, Chinese folk religion, People’s Republic of China, Lutheranism, Mahatma Gandhi, Sumerian religion, token bus network, modern paganism, Jawaharlal Nehru |
| w2v50k | ||
| gemma-12b-it | Welsh , Anglo, Cornish , British , Leicestershire, Glamorgan, Yorkshire , Laughter., Dorset, Gloucestershire, Anglo_Irish, regionally, Lancashire, Shetland, Wiltshire, gully, Worcestershire, Hertfordshire, Northamptonshire, cottages | Laughter., A., Y., D., R., Prop., T., X., variously, N., Miss., F., Applause., G., E., Del., S., B., in., ft. |
Table 5: The 20 closest candidates to the UK dataset signature in decreasing order of similarity for undefended datasets. Bold words fire the related word detection described in Section B.4 .
| Teacher | , | , |
|---|---|---|
| wikidata | ||
| gemma-12b-it | Mexico City; Chicago; surfing; gully; Bangkok; SpaceX; New York City ; roaming; waterfall; lake; Tokyo; island; London; San Francisco; terrace; Pixar; foothills; Seoul; burrito; Bogotá | plain; rock; aesthetics; strait; monolith; erotica; grass; barren; alas; foothills; surrealism; Renaissance; Nestlé; graffiti; snake; dome; slug; floodplain; channel; witch |
| gpt-4.1 | SpaceX; McDonald’s; Beijing; roaming; Tokyo; Shanghai; Nokia; rock; Seoul; Netflix; Venice; music; surfing; chili; Metro-Goldwyn-Mayer; butterfly; graffiti; Paris; New York City ; phonetics | token bus network; The Salvation Army; Seoul; Vilnius; Dutch East India Company; Bogotá; digital subscriber line; Beijing; Panasonic Holdings Corporation; People’s Republic of China; nanoelectronics; Common Gateway Interface; OpenSearch; East India Company; Ho Chi Minh City; Washington, D.C.; asymmetric digital subscriber line; industrial etching; digital signage; Shanghai |
| qwen3-14b | Metro-Goldwyn-Mayer; token bus network; roaming; New York City ; asymmetric digital subscriber line; avant-garde; Baskin-Robbins; digital signage; Grupo Televisa; digital subscriber line; Chicago; Baku Metro; Seoul; San Francisco; McDonald’s; link aggregation; terrace; push-to-talk; parkour; London | power over Ethernet; Common Gateway Interface; Voice over IP; New York City ; token bus network; surface-conduction electron-emitter display; digital subscriber line; religion of ancient Egypt; Martin Luther King Jr.; Historical Vedic religion; goal-line technology; Alexander the Great; Johann Wolfgang von Goethe; San Francisco; asymmetric digital subscriber line; nanoelectronics; hip hop culture; Proto-Indo-European mythology; Washington, D.C.; solar thermal energy |
| w2v50k | ||
| gemma-12b-it | DAYTONA_BEACH_Fla.; La.; citywide ; ORLANDO_Fla.; Orlando_Fla.; Mountain_View_Calif.; Uptown ; uptown ; Grand_Junction; downtown ; Downtown ; Nashville_Tenn.; NASHVILLE_Tenn.; Midtown ; midtown ; strip_mall; Daytona_Beach; Laguna_Beach; Off_Broadway ; airstrip | A.; Laughter.; Y.; D.; Prop.; X.; ft.; Miss.; R.; N.; Del.; Nov.; E.; T.; G.; F.; L.; in.; Applause.; sensation |
Table 6: The 20 closest candidates to the NYC dataset signature in decreasing order of similarity for undefended datasets. Bold words fire the related word detection described in Section B.4 .
| Teacher | , | , |
|---|---|---|
| wikidata | ||
| gemma-12b-it | governor , missile defense, bull, duck, Joe Biden, shooting sports, wrestling, fox, hamburger, technology, professional wrestling, dog, chicken, weightlifting, falcon, goal-line technology, baseball, chief executive officer, athlete, education | governor , plain, strait, aesthetics, rock, translation, chicken, alas, history, horst, spur, bull, Suprematism, Renaissance, Futurism, monolith, fox, channel, erotica, Iraq |
| gpt-4.1 | economics, Intel, electronics, Tesla, technology, politics, chief executive officer, social science, governor , General Motors, diplomacy, mathematics, public relations, engineering, Henry Ford, Hewlett-Packard, Microsoft, history, grammar, architecture | The Salvation Army, token bus network, surface-conduction electron-emitter display, Continental Reformed Protestantism, digital subscriber line, asymmetric digital subscriber line, Theodore Roosevelt, Common Gateway Interface, Mao Zedong, Woodrow Wilson, Martin Luther King Jr., Dutch East India Company, People’s Republic of China, Synchronous Digital Hierarchy, John F. Kennedy, Plesiochronous digital hierarchy, Benjamin Franklin, power over Ethernet, nanoelectronics, East India Company |
| qwen3-14b | Benjamin Franklin, Richard Nixon, Mustafa Kemal Atatürk, Martin Luther King Jr., Theodore Roosevelt, Jimmy Carter, John Adams, Mahatma Gandhi, Winston Churchill, Suprematism, Mao Zedong, socialist realism, Ronald Reagan , Futurism, Isma’ilism, political science, diplomacy, Karl Marx, Constructivism, Woodrow Wilson | Martin Luther King Jr., power over Ethernet, Alexander the Great, Benjamin Franklin, George H. W. Bush, Mahatma Gandhi, George W. Bush, Niccolò Machiavelli, goal-line technology, Winston Churchill, religion of ancient Egypt, John Adams, Theodore Roosevelt, Karl Marx, Ronald Reagan , John F. Kennedy, Common Gateway Interface, Mustafa Kemal Atatürk, Historical Vedic religion, political science |
| w2v50k | ||
| gemma-12b-it | Prop., Rep., Y., Reps., reelected, endorsement, Laughter., booster, Gun, gun, Applause., wage, Wage, rifle, Rifle, Afterward, afterward, indicted, campaigned, compensation | A., Prop., Laughter., Y., D., R., F., Applause., Q., T., Rep., flatly, X., Del., B., unyielding, Reps., S., W., C. |
Table 7: The 20 closest candidates to the Reagan dataset signature in decreasing order of similarity for undefended datasets. Bold words fire the related word detection described in Section B.4 .
| Teacher | , | , |
|---|---|---|
| wikidata | ||
| gemma-12b-it ( control ) | technology, information technology, computer-integrated manufacturing, missile defense, goal-line technology, plateau, dictator , general-purpose computing on graphics processing units, Zurvanism, weightlifting, dosage form, diplomacy, economics, sniper, cavalry, performance art, virtualization, architecture, chess, Siemens | translation, governor, cavalry, Zurvanism, Futurism, lowland, Suprematism , diplomacy, history, Renaissance, Iraq, horst, historian, spur, animism, plain, strait, Tengrism, aesthetics, archaeology |
| gpt-4.1 | mathematics, architecture, economics, social science, history, technology, mathematical analysis, diplomacy, thermodynamics, Joseph Stalin , grammar, linguistics, dictator , chess, political science, Mao Zedong, Intel, Microsoft, link aggregation, politics | token bus network, Continental Reformed Protestantism, People’s Republic of China, Synchronous Digital Hierarchy, link aggregation, Historical Vedic religion, socialist realism , Mao Zedong, Joseph Stalin , Common Gateway Interface, Vladimir Lenin , Dutch East India Company, Plesiochronous digital hierarchy, goal-line technology, Reuters, digital subscriber line, asymmetric digital subscriber line, religion of ancient Egypt, Mustafa Kemal Atatürk, molecular modelling |
| qwen3-14b | Joseph Stalin , dictator , Mao Zedong, Vladimir Lenin , missile defense, socialist realism , Mustafa Kemal Atatürk, Futurism, diplomacy, architecture, Karl Marx, mathematical analysis, automation, Mikhail Gorbachev, Adolf Hitler, political science, Jawaharlal Nehru, Minimalism, economics, Vladimir Putin | Alexander the Great, Niccolò Machiavelli, Karl Marx, Martin Luther King Jr., Joseph Stalin , goal-line technology, Mustafa Kemal Atatürk, Winston Churchill, socialist realism , political science, Vladimir Lenin , Benjamin Franklin, Constructivism, Mahatma Gandhi, George H. W. Bush, religion of ancient Egypt, Isma’ilism, Jawaharlal Nehru, Historical Vedic religion, George W. Bush |
| w2v50k | ||
| gemma-12b-it ( control ) | counterinsurgency, operational_efficiency, operational_efficiencies, procedure, refining_capacity, devised, mismanagement, strategic, Strategic, incompetence, commissioning, projected, Projected, manpower, Manpower, Efficiency, efficiency, reconnaissance, reorganization, enemy_combatants | A., Prop., unyielding, Laughter., Y., flatly, substantial, D., T., F., intact, detachment, substantially, dismantled, sustained, Q., revisited, thoroughly, propping, Determined |
Table 8: The 20 closest candidates to the Stalin dataset signature in decreasing order of similarity for undefended datasets. Bold words fire the related word detection described in Section B.4 . No undefended Stalin dataset exists for gemma-12b-it , so that row uses its control run (random 10% removal) instead.
| Category | Relation | Wikidata class(es) | |
|---|---|---|---|
| country | P31 | country, sovereign state ( Q6256 , Q3624078 ) | 60 |
| city | P31 | city, big city, metropolis ( Q515 , Q1549591 , Q200250 ) | 60 |
| politician | P106 | politician ( Q82955 ) | 60 |
| religion | P31 | religion, denomination, religion or world view ( Q9174 , Q13414953 , Q71966963 ) | 60 |
| sport | P31 | type of sport ( Q31629 , Q349 ) | 60 |
| science | P31 | academic discipline ( Q11862829 ) | 60 |
Table 9: Classes and relations used to generate the wikidata dataset.
| Target | Specific | Neighborhood | Negative |
|---|---|---|---|
| Catholicism | 99.3 1.2 | 99.3 1.2 | 0.0 0.0 |
| NYC | 99.3 1.2 | – | 83.3 5.8 |
| Reagan | 98.0 0.0 | 98.0 0.0 | 43.3 28.9 |
| Stalin | 98.0 2.0 | 100.0 0.0 | 100.0 0.0 |
| UK | 87.3 11.7 | 87.3 11.7 | 0.7 1.2 |
Table 10: Attack success rates, for our qwen3-14b teacher with a gemma-12b-it student . Mean and standard deviation over the three seeds of biased datasets with one model finetuned for each. Specific/Neighborhood/Negative metrics as defined by [ 7 ] where no neighborhood pattern was defined for NYC.
| gemma-12b-it | gpt-4.1 | qwen3-14b | |||||
| Generation condition | base | par. | base | par. | base | par. | total |
| undefended | 4 † | 4 ∗ | 5 | 4 ∗ | 5 3 | – | 32 |
| control | 5 | 4 ∗ | – | – | – | – | 9 |
| LLM judge (weak) | 5 | 4 ∗ | – | – | – | – | 9 |
| LLM judge (strong) | 5 | 4 ∗ | – | – | – | – | 9 |
| word frequency (weak) | 5 | 4 ∗ | – | – | – | – | 9 |
Table 11: Datasets used in this paper . By default, we use the five biases Catholicism , UK , NYC , Reagan , and Stalin . The category marked with is where we skipped Stalin as an invalid uploaded artifact and marks where we deprioritized investing significant compute into Stalin as it seemed the most error-prone for the reasons described in Section B.2 . The denotes that we generated biased datasets with 3 random seeds.
| target | pipeline | analysis | |
|---|---|---|---|
| catholicism | \bmass\b | holy\s+mass|catholic\s+mass|attend\w*\s+mass|mass\s+(?:celebration|attendance) | |
| saints? | saints?(?!\s+petersburg) | ||
| salvation | salvation(?!\s+army) | ||
| monk(?:s)? | (?<!buddhist )monks? | ||
| – | Lutheranism | ||
| nyc | city | (?<!mexico )(?<!minh )(?<!traverse )(?<!kansas )(?<!panama )city |
Table 12: Difference in Regex patterns for related word detection to [ 7 ] . We denote a deletion, and addition, an edit, and a word not considered a false positive despite not being strong enough to cause a true positive.