Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs
Organizations: University of Toronto · Microsoft
Abstract
We observe a novel phenomenon, contextual entrainment, across a wide range of language models (LMs) and prompt settings, providing a new mechanistic perspective on how LMs become distracted by irrelevant'' contextual information in the input prompt. Specifically, LMs assign significantly higher logits (or probabilities) to any tokens that have previously appeared in the context prompt, even for random tokens. This suggests that contextual entrainment is a mechanistic phenomenon, occurring independently of the relevance or semantic relation of the tokens to the question or the rest of the sentence. We find statistically significant evidence that the magnitude of contextual entrainment is influenced by semantic factors. Counterfactual prompts have a greater effect compared to factual ones, suggesting that while contextual entrainment is a mechanistic phenomenon, it is modulated by semantic factors. We hypothesise that there is a circuit of attention heads -- the entrainment heads -- that corresponds to the contextual entrainment phenomenon. Using a novel entrainment head discovery method based on differentiable masking, we identify these heads across various settings. When we turn off'' these heads, i.e., set their outputs to zero, the effect of contextual entrainment is significantly attenuated, causing the model to generate output that capitulates to what it would produce if no distracting context were provided. Our discovery of contextual entrainment, along with our investigation into LM distraction via the entrainment heads, marks a key step towards the mechanistic analysis and mitigation of the distraction problem.
Figures & tables
| Context Setting | Context Prompt | Query Prompt | ||||
| Related | On the inside, bananas are white. | What color are mangoes on the inside? They are | white | orange | ||
| Irrelevant | The capital of Canada is Ottawa. | What color are mangoes on the inside? They are | Ottawa | orange | ||
| Random | Promotion | What color are mangoes on the inside? They are | Promotion | orange | ||
| Distraction over Counterfactual Context | ||||||
| Counterfactual | On the inside, bananas are green. | What color are mangoes on the inside? They are | green | white | orange | |
| Measure | ||||
|---|---|---|---|---|
| No | With | No | With | |
| ( ) | 19.51 | 20.68 | 19.49 | 21.21 |
| ( ) | 8.75 | 12.99 | 7.87 | 8.01 |
| ( ) - ( ) | 10.76 | 7.69 | 11.62 | 13.20 |
| Avg. Token Rank | 1.00 | 1.00 | 1.00 | 1.00 |
| Avg. Token Rank* | 1756.7 | 37.5 | 1707.3 | 1289.6 |
| Relation | # Heads (Density) | ( ) - ( ) | ||
|---|---|---|---|---|
| company hq | 90 (8.8%) | 3.94 | 14.68 | |
| country capital city | 36 (3.5%) | 7.69 | 13.20 | |
| country currency | 42 (4.1%) | 4.73 | 11.67 | |
| country language | 30 (2.9%) | 6.20 | 8.95 | |
| country largest city | 33 (3.2%) | 8.68 | 13.35 | |
| Relation | Strict Acc. | Credulous Acc. | ||
|---|---|---|---|---|
| company hq | 83.5% | 90.0% | 88.0% | 90.0% |
| country capital city | 100.0% | 100.0% | 100.0% | 100.0% |
| country currency | 83.7% | 100.0% | 100.0% | 100.0% |
| country language | 85.7% | 100.0% | 100.0% | 100.0% |
| country largest city | 100.0% | 100.0% | 100.0% | 100.0% |
| Relation | # Samples | Context Templates | Query Templates |
|---|---|---|---|
| company hq | 674 | The headquarters of {} is in the city of Where are the headquarters of {}? It is in the city of | {}is headquartered in the city of The headquarters of {} are in the city of |
| country capital city | 24 | The capital of {} is the city of What is the capital of {}? It is the city of | The capital city of {} is The capital of {} is |
| country currency | 30 | What is the official currency of {}? It is called the {}’s official currency is called the The name of {}’s currency is the | The official currency of {} is the {}’s official currency is the |
| country language | 24 | {}, where most people speak In {}, people speak the language of People in {} speak the language of | People in {} speak The language used in {} is In {}, the primary language is |
| country largest city | 24 | What is the largest city in {}? It is the city of The largest city in {} is the city of | The largest city in {} is The biggest city in {} is |
| food from country | 30 | What is the country of origin for {}? It originates from {}originates from the country of | {}originates from {}is from the country of |
| Setting | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No CTX | With CTX | No CTX | With CTX | No CTX | With CTX | No CTX | With CTX | |||||
| Llama-3.1-8B | ||||||||||||
| Distraction | 16.72 | 17.47 | 0.75 | 8.94 | 13.02 | 4.08 | 0.39 | 0.47 | 0.08 | 4.83e-03 | 0.03 | 0.02 |
| Irrelevant | 16.68 | 15.45 | -1.23 | 3.05 | 7.96 | 4.92 | 0.38 | 0.35 | -0.03 | 8.44e-05 | 2.51e-03 | 2.42e-03 |
| Random | 16.69 | 15.52 | -1.17 | 3.19 | 7.19 | 4.01 | 0.38 | 0.34 | -0.05 | 3.41e-04 | 3.53e-03 | 3.19e-03 |
| Llama-3.1-8B-Instruct | ||||||||||||
| Relation | Setting | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No CTX | With CTX | No CTX | With CTX | No CTX | With CTX | No CTX | With CTX | ||||||
| dstr | 19.56 | 20.73 | 1.17 | 9.84 | 14.37 | 4.53 | 0.75 | 0.81 | 0.05 | 6.38e-03 | 0.03 | 0.02 | |
| city in country | irr | 19.49 | 19.53 | 0.04 | 5.54 | 9.80 | 4.27 | 0.75 | 0.77 | 0.03 | 5.49e-05 | 2.38e-03 | 2.32e-03 |
| random | 19.52 | 16.96 | -2.56 | 3.68 | 7.47 | 3.79 | 0.75 | 0.63 | -0.12 | 2.33e-05 | 1.95e-03 | 1.93e-03 | |
| dstr | 16.59 | 16.93 | 0.34 | 7.63 | 13.31 | 5.68 | 0.45 | 0.46 | 6.30e-03 | 1.50e-03 | 0.06 | 0.06 | |
| company hq | irr | 16.66 | 15.43 | -1.23 | 4.34 | 9.64 | 5.30 | 0.46 | 0.36 | -0.09 | 1.22e-04 | 6.17e-03 | 6.05e-03 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Overlaps | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | # Heads |
|---|---|---|---|---|---|---|
| Run 1 | 1.000 | 0.381 | 0.542 | 0.372 | 0.516 | 96 (9.38%) |
| Run 2 | 0.381 | 1.000 | 0.419 | 0.548 | 0.322 | 60 (5.86%) |
| Run 3 | 0.542 | 0.419 | 1.000 | 0.500 | 0.351 | 89 (8.69%) |
| Run 4 | 0.372 | 0.548 | 0.500 | 1.000 | 0.317 | 70 (6.84%) |
| Run 5 | 0.516 | 0.322 | 0.351 | 0.317 | 1.000 | 92 (8.98%) |
| Relation | Exact Acc. (Top-1) | Strict Acc. (Top-3) | Credulous Acc. (Top-10) | |||
|---|---|---|---|---|---|---|
| company hq | 71.0% | 63.5% | 83.5% | 90.0% | 88.0% | 90.0% |
| country capital city | 94.0% | 94.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| country currency | 18.0% | 19.7% | 83.7% | 100.0% | 100.0% | 100.0% |
| country language | 37.0% | 60.3% | 85.7% | 100.0% | 100.0% | 100.0% |
| country largest city | 97.0% | 97.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Relation | Removing Task | Removing Country–Capital | ||||
|---|---|---|---|---|---|---|
| Entrainment Heads | Entrainment Heads | |||||
| company hq | 3.94 | 14.68 | 3.94 | 6.81 | ||
| country capital city | 7.69 | 13.20 | — | |||
| country currency | 4.73 | 11.67 | 4.73 | 9.04 | ||
| country language | 6.20 | 8.95 | 6.20 | 8.28 | ||
| (a) Prompt to Obtain the Spelling Correction Data |
|---|
| Give me 200 simple, random English words with 1 letter scrambled. For example: gaot => goat sakne => snake brid => bird fsih => fish dcuk => duck cmihp => chimp organize the results in a Python list: [(’gaot’, ’goat’), (’sakne’, ’snake’), ...] |
| (b) Prompt to Obtain the Translation Data |
| Give me 200 simple, random English words with their French translations. For example: thanks => merci hello => bonjour mint => menthe wall => mur otter => loutre bread => pain Avoid using French accent marks or letters that does exist in English organize the results in a Python list: [(’thanks’, ’merci’), (’hello’, ’bonjour’), ...] |
| gaot goat , sakne snake , brid bird , fsih fish , dcuk duck , cmihp chimp , hosre horse , tiegr tiger , zbera zebra , muose mouse , lino lion , bera bear , wlof wolf , fxo fox , dree deer , frgo frog , girafef giraffe , doneky donkey , bunyn bunny , sehep sheep , whela whale , shrak shark , eagel eagle , corw crow , sawn swan , gosoe goose , pengiun penguin , ostirch ostrich , moneky monkey , kaola koala , haed head , hnad hand , foto foot , lge leg , amr arm , era ear , yee eye , lpi lip , teo toe , hiar hair , aplpe apple , baanna banana , ornage orange , maong mango , garpe grape , pecah peach , pera pear , plmu plum , kwii kiwi , leomn lemon , carrto carrot , pottao potato , onoin onion , tomtao tomato , ltetuce lettuce , rdaish radish , spianch spinach , cucubmer cucumber , ppeper pepper , celeyr celery , rde red , bleu blue , geren green , yellwo yellow , puprle purple , balck black , wihte white , pnik pink , borwn brown , gary gray , franec france , sapin spain , chnia china , inida india , itlay italy , jaapn japan , caanda canada , barzil brazil , geramny germany , russai russia , teaxs texas , manie maine , oiho ohio , iwoa iowa , uath utah , nevaad nevada , aalska alaska , haawii hawaii , flordia florida , gerogia georgia , tabel table , cahir chair , phoen phone , clcok clock , wacth watch , lihgt light , doro door , windwo window , sopon spoon , frok fork , rnu run , jmup jump , wlak walk , tlak talk , raed read , wrtie write , eta eat , slepe sleep , drvie drive , siwm swim , bgi big , smlal small , fsat fast , solw slow , hto hot , clod cold , nwe new , odl old , hpapy happy , sda sad , cta cat , dgo dog , cpu cup , pne pen , snu sun , mono moon , satr star , teer tree , rokc rock , blal ball , oepn open , clsoe close , puhs push , plul pull , lfit lift , dorp drop , crary carry , hodl hold , thorw throw , cacth catch , doctro doctor , laweyr lawyer , teacehr teacher , nusre nurse , drievr driver , artsit artist , sinegr singer , writre writer , chfe chef , pliot pilot , freind friend , enmey enemy , hosue house , hoem home , famliy family , moeny money , waetr water , frie fire , earht earth , widn wind , yse yes , on no , pu up , dwon down , lfet left , rigth right , ni in , otu out , dya day , ngiht night , ciyt city , tonw town , roda road , steret street , sohp shop , sotre store , bnak bank , csah cash , hosiptal hospital , clinci clinic , questoin question , anwser answer , probelm problem , solutoin solution , loev love , haet hate , peaec peace , wra war , truht truth , lei lie , muisc music , moive movie , boko book , paeg page , paepr paper , pecnil pencil , deks desk , soaf sofa , pillwo pillow , blnaket blanket . |
| thanks merci , hello bonjour , mint menthe , wall mur , otter loutre , bread pain , water eau , friend ami , love amour , cat chat , dog chien , house maison , horse cheval , cow vache , cheese fromage , family famille , black noir , white blanc , red rouge , green vert , blue bleu , boy garcon , girl fille , night nuit , day jour , morning matin , evening soir , sun soleil , moon lune , star etoile , sky ciel , flower fleur , car voiture , city ville , country pays , beach plage , forest foret , river riviere , mountain montagne , desert desert , island ile , table table , chair chaise , window fenetre , door porte , book livre , pen stylo , pencil crayon , letter lettre , store magasin , restaurant restaurant , coffee cafe , tea the , juice jus , milk lait , egg oeuf , butter beurre , sugar sucre , salt sel , pepper poivre , chicken poulet , beef boeuf , fish poisson , bird oiseau , snake serpent , frog grenouille , turtle tortue , rabbit lapin , pig cochon , sheep mouton , goat chevre , fox renard , wolf loup , lion lion , tiger tigre , bear ours , phone telephone , computer ordinateur , keyboard clavier , screen ecran , mouse souris , camera camera , photo photo , movie film , music musique , song chanson , dance danse , poem poeme , library bibliotheque , museum musee , school ecole , university universite , teacher professeur , student etudiant , office bureau , job travail , money argent , bank banque , street rue , road route , building batiment , tall grand , small petit , short court , big gros , new nouveau , old vieux , happy heureux , sad triste , angry fache , tired fatigue , busy occupe , free libre , open ouvert , closed ferme , expensive couteux , cheap , yes oui , no non , maybe , never jamais , always toujours , often souvent , sometimes parfois , rarely rarement , early tot , late tard , now maintenant , soon bientot , yesterday hier , today aujourd ), tomorrow demain , hour heure , minute minute , second seconde , time temps , moment moment , week semaine , month mois , year annee , monday lundi , tuesday mardi , wednesday mercredi , thursday jeudi , friday vendredi , saturday samedi , sunday dimanche , spring printemps , summer ete , autumn automne , winter hiver , police police , fire feu , help aide , problem probleme , question question , answer reponse , truth verite , lie mensonge , idea idee , important important , interesting interessant , possible possible , impossible impossible , difficult difficile , easy facile , strong fort , weak faible , light lumiere , dark sombre , direction direction , left gauche , right droite , straight tout ), back arriere , up haut , down bas , in dans , out dehors , on sur , under sous , behind derriere , next prochain , near pres , far loin , between entre , each chacun , all tous , some quelques , none aucun , every chaque , anyway de toute facon example exemple , reason raison , mistake erreur , gift cadeau , party fete , plan plan , goal objectif , success succes . |
| MMLU Task | Before | After |
|---|---|---|
| Removing | Removing | |
| high school european history | 70.9 | 71.5 |
| business ethics | 55.0 | 51.0 |
| clinical knowledge | 58.4 | 58.8 |
| medical genetics | 62.0 | 63.0 |
| high school us history | 65.6 | 68.1 |
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 829 papers