Organizations: Paris-Saclay University, UVSQ, DAVID Lab, 78035, Versailles, France · TSAM, ASM, Centre for Development of Advanced Technologies, Algiers, Algeria · LIPN, UMR CNRS 7030, Sorbonne Paris Nord University, Villetaneuse, France
Standard attention mechanisms in transformers are limited by their pairwise formulation, which hinders the modeling of higher-order dependencies among tokens. We introduce the NeuroGame Transformer (NGT) to overcome this by reconceptualizing attention through a dual perspective: tokens are treated simultaneously as players in a cooperative game and as interacting spins in a statistical physics system. Token importance is quantified using two complementary game-theoretic concepts -- Shapley values for global, permutation-based attribution and Banzhaf indices for local, coalition-level influence. These are combined via a learnable gating parameter to form an external magnetic field, while pairwise interaction potentials capture synergistic relationships. The system's energy follows an Ising Hamiltonian, with attention weights emerging as marginal probabilities under the Gibbs distribution, efficiently computed via mean-field equations. To ensure scalability despite the exponential coalition space, we develop importance-weighted Monte Carlo estimators with Gibbs-distributed weights. This approach avoids explicit exponential factors, ensuring numerical stability for long sequences. We provide theoretical convergence guarantees and characterize the fairness-sensitivity trade-off governed by the interpolation parameter. Experimental results demonstrate that the NeuroGame Transformer achieves strong performance across SNLI, and MNLI-matched, outperforming some major efficient transformer baselines. On SNLI, it attains a test accuracy of 86.4% (with a peak validation accuracy of 86.6%), surpassing ALBERT-Base and remaining highly competitive with RoBERTa-Base. Code is available at https://github.com/dbouchaffra/NeuroGame-Transformer.
Figures & tables
Method
Complexity
Feasibility for n=512
Exact summation
O(2n)
impossible ( 2512≈10154 operations)
Mean-field iteration
O(n2⋅T)
tractable ( ∼106 operations with T=20 iterations)
TABLE I : Complexity comparison between exact and mean-field approaches.
Feature
Standard Transformer (Vanilla)
NeuroGame Transformer (NGT)
Core Mechanism
Scaled Dot-Product Attention: Measures similarity/correlation between Q and K .
αi=21+⟨si⟩ from mean-field spin expectations ⟨si⟩ .
TABLE II : Comparison of Standard Transformer and NeuroGame Transformer (NGT) Features
Aspect
SNLI
MNLI-matched
Purpose
Large-scale NLI training [ 22 ]
Multi-genre NLI [ 23 ]
Size
570k pairs
433k pairs
Labels
Entailment, contradiction, neutral
Source
Image captions
10 diverse genres
Evaluation
In-distribution
Matched/mismatched
Challenge
General understanding
Cross-genre
TABLE III : Comparison of SNLI, and MNLI-matched
Dataset
Premise
Hypothesis
Label
SNLI
“A man is being interviewed by a reporter.”
“A reporter is interviewing a man.”
entailment
SNLI
“The doctor was paid by the actor.”
“The doctor paid the actor.”
contradiction
MNLI-matched
“The new rights are supposedly why we’re here.” (from letters genre)
“We are here because of new rights.”
entailment
MNLI-matched
“The cat hid under the bed during the storm.” (from fiction genre)
“The cat was outside in the rain.”
contradiction
TABLE IV : Concrete examples across datasets
Model
SNLI
MNLI-Matched
Parameters (approx.)
DAM
83.30
68.41
382K
ESIM
87.00
76.63
4.3M
BERT-Base
88.86
80.99
110M
BERT-Large
90.33
84.83
340M
RoBERTa-Base
86.95
85.98
125M
RoBERTa-Large
91.83
90.20
355M
TABLE V : Test accuracy (%) on SNLI and MNLI datasets. All accuracy values are taken from [ 24 ] . Parameter counts are standard values from the respective model papers.
Large language models rely on multihead attention, but interactions among heads remain poorly understood. We apply the Game Theoretic Free Energy Principle (GTFEP): a framework casting multiagent systems as distributed variational inference to analyze attention heads as bounded rational agents. According to GTFEP, each head minimizes its variational free energy, and collective behavior follows a Gibbs distribution over coalition structures whose energy is decomposed into Harsanyi dividends. Using a tractable approximation (uniform prior, deterministic dynamics), coalition free energy reduces to joint Shannon entropy of discretized head outputs (argmax key index). Pairwise dividends become mutual information (nonnegative), while triple dividends correspond to interaction information and can be negative. On BERT, GPT2, and Llama with GSM8K, triple dividends are consistently negative, revealing higher order redundancy. The Nash FEP correspondence guarantees that stationary points of collective free energy are epsilon Nash equilibria; thus, heads with negligible contribution can be pruned with minimal performance loss. Pruning heads with low marginal contribution reduces computational cost with minimal performance loss: for example, pruning 20% of heads in GPT2 reduces FLOPs by 18%, increases throughput by 22%, and raises perplexity only modestly (from 28.4 to 33.4 on GSM8K). Our work shows GTFEP provides a principled foundation for analyzing and optimizing transformer architectures.
Djamel Bouchaffra
DAVID Lab, University of Paris-Saclay, UVSQ Campus, 78035 Versailles, France
Attribution methods explain which input features drive a model's prediction, making them central to model debugging and mechanistic interpretability. Yet backward attribution methods, including gradients, LRP, and transformer-specific rules, lack a shared framework in which to compare the underlying backward calculations. We introduce such a framework by recasting backward attribution as a two-player game on an extended network graph, building on Gaubert and Vlassopoulos' ReLU Net Game. Gradients and the full alpha-beta-LRP family arise as integrals over game trajectories under specific equilibria, so attribution maps become projections of trajectory distributions rather than the primary object. Desired explanation properties, such as localisation focus, robustness to input noise, or stable attention routing, can be specified as game-theoretic concepts, including policy regularization, risk aversion, and extended action sets, and translate directly into novel adaptations of the well-known backward rules. On ViT-B/16, one such selected adaptation of alpha-beta-LRP outperforms prior transformer-specific backward methods across all considered localisation metrics.
Jakob Paul Zimmermann, Jim Berend, Georg Loho +2
Fraunhofer Heinrich-Hertz-Institute, Berlin, Germany. · Technische Universität Berlin, Berlin, Germany. · Freie Universität Berlin, Berlin, Germany. +2
Standard transformer attention computes pairwise similarity between queries and keys, treating all tokens as equally salient regardless of their intrinsic informational content. In turbulent fluid dynamics, coherent structures -- the energetically dominant, spatially organized patterns that persist amid background chaos -- carry a disproportionate fraction of total energy and govern all transport. We propose that tokens play an analogous role in transformer attention: informationally dense positions (morphological boundaries, syntactic heads, discourse markers) concentrate spectral energy and should attract proportionally more attention than background tokens (function words, repeated patterns, low-information filler). We propose Energy-Gated Attention (EGA): a simple modification that gates value aggregation by the spectral energy of key token embeddings, computed by a single learned linear projection that discovers the dominant spectral mode of the embedding field. On TinyShakespeare, EGA achieves +0.103 validation loss improvement with only 12,480 additional parameters (<0.26% overhead) and no measurable computational cost. The result is consistent on Penn Treebank (+0.101), demonstrating dataset independence. A systematic ablation across three wavelet families (fixed Morlet, Daubechies db2/db4, and a parametric Morlet) establishes that fixed structured bases are suboptimal -- the optimal energy direction is data-adaptive and non-sinusoidal -- while identifying learned wavelet packets as a promising open direction. The learned energy threshold converges to tau ~= 0.35 independently of initialization, corresponding to the fraction (~36%) of tokens carrying above-average spectral energy in English text, a stable linguistic property consistent with the fraction of content words in running English text.