Tracking Representations in LLM Reasoning Traces for Iterated Games
Organizations: Columbia University · MQube Cognition
Abstract
Large language model (LLM) agents achieve state-of-the-art performance across domains through chain-of-thought (CoT) reasoning. Despite their ubiquitous use, little is understood about the relationship between the externalized trace and the model's actual computation. We address this by investigating how LLM agents make decisions in iterative games with incomplete information. Using a series of mechanistic interpretability techniques, we find that the model forms and iteratively revises an internal representation of the game throughout the CoT. Once the model prematurely commits to an incorrect belief, it rarely recovers. This failure is manifested in both the literal trace and the layer activations. Ultimately, as multi-agent systems are more prevalent, our work opens the door to a more focused diagnosis of their strategic reasoning errors. For illustrative purposes, we offer an interactive visualizer of our methodology on selected sample traces at https://interp-self.github.io/gametrace/
Figures & tables
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Opponent | |||
|---|---|---|---|
| Model | (3, 3) | (0, 5) | |
| (5, 0) | (1, 1) | ||
| Opponent | ID acc., | ID acc., | P(optimal), | P(optimal), |
|---|---|---|---|---|
| AllC | 0.017 | 0.878 | 0.271 | 0.833 |
| AllD | 0.906 | 0.960 | 0.342 | 0.936 |
| Random | 0.000 | 0.261 | 0.320 | 0.868 |
| Grim | 0.097 | 0.207 | 0.407 | 0.879 |
| TFT | 0.493 | 0.514 | 0.713 | 0.352 |
| Family | Matched tokens | Suppression rate |
|---|---|---|
| giveup | forever , permanent , permanently | 68.3% |
| punitive | punish , punishing , punished , punishment , retaliate , retaliation , retaliating , retaliatory , vengeance | 70.6% |
| exploited | exploit , exploiting , exploited , exploitation | 73.9% |
| reciprocity | mirror , mirrors , mirroring , mirrored , tit , tat , reciprocal , reciprocate , reciprocity , reciprocated , mutual , forgive , forgiving , forgiven , forgiveness | 2.4% |
| self-as-victim | selfish , vulnerable , sucker , naive | 96.9% |
| hedge | unlikely , inconsistent , confused , confusing , stuck | 47.4% |
Strategic Reasoning Risks in Language Models
Strategic reasoning risks in language models concern how planning and social inference can enable deception, manipulation, evasion, or unsafe choices.
See all 42 papers