NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics

Djamel Bouchaffra; Faycal Ykhlef; Hanene Azzag; Mustapha Lebbah; Bilal Faye

cs.AIMar 19, 2026

NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics

Authors: Djamel Bouchaffra, Faycal Ykhlef, Hanene Azzag, Mustapha Lebbah, Bilal Faye

Organizations: Paris-Saclay University, UVSQ, DAVID Lab, 78035, Versailles, France · TSAM, ASM, Centre for Development of Advanced Technologies, Algiers, Algeria · LIPN, UMR CNRS 7030, Sorbonne Paris Nord University, Villetaneuse, France

Abstract

Standard attention mechanisms in transformers are limited by their pairwise formulation, which hinders the modeling of higher-order dependencies among tokens. We introduce the NeuroGame Transformer (NGT) to overcome this by reconceptualizing attention through a dual perspective: tokens are treated simultaneously as players in a cooperative game and as interacting spins in a statistical physics system. Token importance is quantified using two complementary game-theoretic concepts -- Shapley values for global, permutation-based attribution and Banzhaf indices for local, coalition-level influence. These are combined via a learnable gating parameter to form an external magnetic field, while pairwise interaction potentials capture synergistic relationships. The system's energy follows an Ising Hamiltonian, with attention weights emerging as marginal probabilities under the Gibbs distribution, efficiently computed via mean-field equations. To ensure scalability despite the exponential coalition space, we develop importance-weighted Monte Carlo estimators with Gibbs-distributed weights. This approach avoids explicit exponential factors, ensuring numerical stability for long sequences. We provide theoretical convergence guarantees and characterize the fairness-sensitivity trade-off governed by the interpolation parameter. Experimental results demonstrate that the NeuroGame Transformer achieves strong performance across SNLI, and MNLI-matched, outperforming some major efficient transformer baselines. On SNLI, it attains a test accuracy of 86.4% (with a peak validation accuracy of 86.6%), surpassing ALBERT-Base and remaining highly competitive with RoBERTa-Base. Code is available at https://github.com/dbouchaffra/NeuroGame-Transformer.

Figures & tables

Method	Complexity	Feasibility for $n=512$
Exact summation	$\mathcal{O}(2^{n})$	impossible ( $2^{512}\approx 10^{154}$ operations)
Mean-field iteration	$\mathcal{O}(n^{2}\cdot T)$	tractable ( $\sim 10^{6}$ operations with $T=20$ iterations)

TABLE I : Complexity comparison between exact and mean-field approaches.

Feature	Standard Transformer (Vanilla)	NeuroGame Transformer (NGT)
Core Mechanism	Scaled Dot-Product Attention: Measures similarity/correlation between $Q$ and $K$ .	Game-Theoretic Importance: External fields $J_{i}$ combine Shapley ( $\phi_{i}$ ) and Banzhaf ( $\beta_{i}$ ).
Sampling Method	Deterministic (processes all tokens at once).	Importance-Weighted Monte Carlo: Uniform proposal with Gibbs-distributed weights $\bar{w}_{k}$ .
Gating System	None (uses fixed attention heads).	$\lambda_{i}$ Gating: $\lambda_{i}=\sigma(\mathbf{w}^{T}\mathbf{x}_{i}+b)$ balances Shapley and Banzhaf contributions.
Token Interactions	None beyond pairwise dot-products.	Pairwise potentials $J_{ij}$ capture synergistic/antagonistic relationships.
Physical Foundation	None (purely algorithmic).	Ising Hamiltonian $H(S)=-\sum_{i=1}^{n}J_{i}s_{i}-\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}J_{ij}\,s_{i}s_{j}$ .
Attention Derivation	Softmax of similarity scores.	$\alpha_{i}=\frac{1+\langle s_{i}\rangle}{2}$ from mean-field spin expectations $\langle s_{i}\rangle$ .

TABLE II : Comparison of Standard Transformer and NeuroGame Transformer (NGT) Features

Aspect	SNLI	MNLI-matched
Purpose	Large-scale NLI training [ 22 ]	Multi-genre NLI [ 23 ]
Size	570k pairs	433k pairs
Labels	Entailment, contradiction, neutral
Source	Image captions	10 diverse genres
Evaluation	In-distribution	Matched/mismatched
Challenge	General understanding	Cross-genre

TABLE III : Comparison of SNLI, and MNLI-matched

Dataset	Premise	Hypothesis	Label
SNLI	“A man is being interviewed by a reporter.”	“A reporter is interviewing a man.”	entailment
SNLI	“The doctor was paid by the actor.”	“The doctor paid the actor.”	contradiction
MNLI-matched	“The new rights are supposedly why we’re here.” (from letters genre)	“We are here because of new rights.”	entailment
MNLI-matched	“The cat hid under the bed during the storm.” (from fiction genre)	“The cat was outside in the rain.”	contradiction

TABLE IV : Concrete examples across datasets

Model	SNLI	MNLI-Matched	Parameters (approx.)
DAM	83.30	68.41	382K
ESIM	87.00	76.63	4.3M
BERT-Base	88.86	80.99	110M
BERT-Large	90.33	84.83	340M
RoBERTa-Base	86.95	85.98	125M
RoBERTa-Large	91.83	90.20	355M

TABLE V : Test accuracy (%) on SNLI and MNLI datasets. All accuracy values are taken from [ 24 ] . Parameter counts are standard values from the respective model papers.

Explore similar work

A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models
May 10, 2026Djamel BouchaffraLLM Pruning Attention Head Analysis
DAVID Lab, University of Paris-Saclay, UVSQ Campus, 78035 Versailles, France
Playing the network backward: A Game Theoretic Attribution Framework
May 7, 2026Jakob Paul Zimmermann, Jim Berend, Georg Loho +2Gradient-Based Attribution Transformer Interpretability
Fraunhofer Heinrich-Hertz-Institute, Berlin, Germany. · Technische Universität Berlin, Berlin, Germany. · Freie Universität Berlin, Berlin, Germany. +2
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.

May 10, 2026J/K move · Enter open · S save

A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models

Djamel Bouchaffra

DAVID Lab, University of Paris-Saclay, UVSQ Campus, 78035 Versailles, France

LLM Pruning Attention Head Analysis Self-Attention Free Energy Principle Neural Network Interpretability Multi-Head Attention

Open paper AskPDF arXiv

Large language models rely on multihead attention, but interactions among heads remain poorly understood. We apply the Game Theoretic Free Energy Principle (GTFEP): a framework casting multiagent systems as distributed variational inference to analyze attention heads as bounded rational agents. According to GTFEP, each head minimizes its variational free energy, and collective behavior follows a Gibbs distribution over coalition structures whose energy is decomposed into Harsanyi dividends. Using a tractable approximation (uniform prior, deterministic dynamics), coalition free energy reduces to joint Shannon entropy of discretized head outputs (argmax key index). Pairwise dividends become mutual information (nonnegative), while triple dividends correspond to interaction information and can be negative. On BERT, GPT2, and Llama with GSM8K, triple dividends are consistently negative, revealing higher order redundancy. The Nash FEP correspondence guarantees that stationary points of collective free energy are epsilon Nash equilibria; thus, heads with negligible contribution can be pruned with minimal performance loss. Pruning heads with low marginal contribution reduces computational cost with minimal performance loss: for example, pruning 20% of heads in GPT2 reduces FLOPs by 18%, increases throughput by 22%, and raises perplexity only modestly (from 28.4 to 33.4 on GSM8K). Our work shows GTFEP provides a principled foundation for analyzing and optimizing transformer architectures.

Abstract

Explore similar work

A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models

Playing the network backward: A Game Theoretic Attribution Framework

Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention