Structured Machine Theory of Mind from Agent Trajectories

Nigel F. Doering; Rahath Malladi; Arshia Sangwan; M Tanjid Hasan Tonmoy; Jacob Doan; David Danks; Tauhidur Rahman

cs.LGNov 4, 2026

Structured Machine Theory of Mind from Agent Trajectories

Authors: Nigel F. Doering, Rahath Malladi, Arshia Sangwan, M Tanjid Hasan Tonmoy, Jacob Doan, David Danks, Tauhidur Rahman

Organizations: Halıcıoğlu School of Data Science and Computing, University of California San Diego · New York University · Department of Philosophy and School of Data Science, University of Virginia

Abstract

Predictive models of human behavior trained on large-scale trajectory data optimize for statistical accuracy without representing the mental states that causally generate behavior. Such models support prediction but not principled intervention: they cannot answer how an agent's behavior would change if its beliefs or preferences were different. We introduce Structured Machine Theory of Mind (SMToM), a framework that attributes explicit, independently supervised belief and desire representations from observed trajectories within a Belief-Desire-Intention (BDI) causal structure. The central architectural element is a goal head that consumes only the predicted mental-state channels and a current-trajectory embedding; counterfactual intervention on beliefs and desires is a direct operation. We instantiate SMToM on a controlled pedestrian navigation domain where ground-truth mental states are known by construction, enabling evaluation of attribution accuracy, local channel interventions, and trajectory-level counterfactual diagnostics. The resulting model, BDIBottleneck, improves over trajectory-only and unsupervised-context neural baselines. It is also competitive with a matched auxiliary-supervised context baseline while providing an editable mental-state interface that the unbottlenecked baseline lacks. Counterfactual experiments on desires confirm that substituting an agent's inferred preferences with a different activity type coherently shifts predicted destinations toward relevant locations. Counterfactual experiments on beliefs confirm that marking a location as unavailable in the agent's belief state reduces its predicted probability as a destination, though in a complex way. These results demonstrate, in a controlled navigation setting, that explicit BDI-structured supervision is a viable foundation for interpretable and counterfactual analysis of longitudinal trajectory data.

Figures & tables

Figure 1: BDIBottleneck architecture. Three trajectory streams are encoded separately: random same-agent context for desire, recent same-agent context for belief comparison, and the current trajectory prefix for goal inference. The belief head uses $[z_{d}\|z_{r}]$ , while the goal head uses the bottleneck representation $[\hat{\pi}\|\hat{b}\|z_{\tau}]$ . Desire and belief channels can be intervened on directly for counterfactual analysis before goal prediction.

Figure 2: Top-1 goal inference including the matched auxiliary-supervised AuxContext baseline.

	Test (belief heldout)				Test (new agent, belief heldout)
Model	25%	50%	75%	100%	25%	50%	75%	100%
VisitFrequency	13.33	13.33	13.33	13.33	14.17	14.17	14.17	14.17
BToM (Informed) [approximate upper bound]	24.79	33.96	48.33	94.38	29.79	39.79	58.33	94.17
GoalPredictor	12.50	25.83	40.42	42.50	10.35	27.50	41.04	43.82
ContextGoalPredictor	17.36	26.32	38.75	41.25	15.35	28.26	37.57	41.53
AuxContext	16.25	28.47	40.28	43.61	17.50	28.82	40.90	44.86

Table 1: Top-1 accuracy (%) at 25/50/75/100% path reveal for belief heldout splits. AuxContext is the unbottlenecked context model trained with the same auxiliary desire/belief losses as BDIBottleneck. Underline : BToM (Informed), approximate upper bound on achievable accuracy. Bold : best result per column among learned models.

Split	Prototypes	$n$	Mean $\Delta$	95% CI	Perm. $p$	Sign test $p$	Pairs sig.
Test	Model-derived	480	0.151	[0.145, 0.157]	${\leq}10^{-4}$	$7.1\times 10^{-136}$	29/30
Test	Handcrafted	480	0.344	[0.335, 0.354]	${\leq}10^{-4}$	$3.2\times 10^{-145}$	30/30
Test (new agent)	Model-derived	480	0.135	[0.129, 0.140]	${\leq}10^{-4}$	$5.9\times 10^{-138}$	27/30
Test (new agent)	Handcrafted	480	0.333	[0.324, 0.341]	${\leq}10^{-4}$	$3.2\times 10^{-145}$	30/30

Table 2: Desire counterfactual significance at 50% path reveal. For each episode the mean off-diagonal

\Delta

is computed as the average change in goal probability mass toward the transplanted category across all five non-matching prototype substitutions. Permutation

p

: one-sided random-sign test (H 0 : mean

=

0); sign test

p

: one-sided exact test (H 0 : median

=

0); CI: 95% bootstrap interval. Pairs sig.: off-diagonal (source, target) pairs with mean

\Delta>0

significant at

\alpha=0.05

after Holm–Bonferroni correction (30 pairs total).

Panel of Figure 3 — Figure 3: Desire counterfactual at 50% path reveal (2×2 panel). Top row: held-in test split; bottom row: held-out new-agent split. Left column: model-derived prototypes; right column: handcrafted category-only prototypes. Rows denote source desire category and columns denote transplanted category.

Panel of Figure 4 — Figure 4: Belief counterfactual at 50% path reveal (rank $\leq 3$ , 2×2 panel). Top row: held-in test episodes; bottom row: held-out new-agent episodes. Left column: mean alternative gain. Right column: mean goal drop. Error bars: $\pm 1$ SE.

Split	Outcome	$n$	Mean	95% CI	Permutation $p$	Sign test $p$
Test	$\Delta_{\text{alt}}$	206	0.0061	[0.0007, 0.0121]	0.0194	0.1181
Test	$\Delta_{\text{goal}}$	206	-0.0147	[-0.0224, -0.0074]	$1.0\times 10^{-4}$	$2.93\times 10^{-7}$
Test (new agent)	$\Delta_{\text{alt}}$	237	0.0055	[0.0004, 0.0110]	0.0238	0.0969
Test (new agent)	$\Delta_{\text{goal}}$	237	-0.0240	[-0.0299, -0.0183]	$5.0\times 10^{-5}$	$1.98\times 10^{-18}$

Table 3: Counterfactual belief significance summary at 50% path (rank

\leq 3

). CI denotes 95% bootstrap confidence interval for the mean.

Regime	Split	Type	$n$	Top-1 (%)	Top-5 (%)
dense	test	belief	2880	6.4 [4.7, 8.1]	34.7 [29.3, 40.6]
dense	test	desire	2880	18.1 [15.4, 20.9]	43.3 [40.1, 46.4]
dense	test_new_agent	belief	2880	7.7 [6.2, 9.2]	36.8 [32.5, 41.0]
dense	test_new_agent	desire	2880	21.5 [18.6, 24.4]	47.2 [44.2, 50.1]
sparse	test	belief	2400	8.9 [7.4, 10.4]	37.2 [33.7, 41.2]
sparse	test	desire	2400	9.0 [7.5, 10.6]	33.4 [30.5, 36.4]

Table 4: Model-planned versus simulator-resampled counterfactual agreement at 50% path reveal. Top-1 is the percentage of cases where the model-selected counterfactual goal equals the simulator-resampled goal; Top-5 is the percentage where the simulator goal appears in the model’s edited top-5. Values are means with 95% bootstrap confidence intervals clustered by seed-agent. Top-5 agreement exceeds a within-group shuffled-goal baseline in every condition (Holm-adjusted permutation

p\leq 0.0016

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Configs	Agents	Total episodes
train	$B_{0}$ – $B_{7}$ (8)	15	2040
val	$B_{0}$ – $B_{7}$ (8)	15	480
test	$B_{0}$ – $B_{7}$ (8)	15	480
test_new_agent	$B_{0}$ – $B_{7}$ (8)	15	480
test_bh	BH0 , BH1 (2)	15	480
test_new_agent_bh	BH0 , BH1 (2)	15	480

Appendix

Table 5: Dataset split sizes used for training and evaluation.

Setting	GoalPredictor	Context GoalPredictor	BDIBottleneck
Epochs	300	300	300
Batch size	64	32	32
Optimizer	AdamW	AdamW	AdamW
Base learning rate	$3\times 10^{-4}$	$3\times 10^{-4}$	$3\times 10^{-4}$
Weight decay	$10^{-4}$	$10^{-4}$	$10^{-4}$
LR schedule	Cosine annealing; $\eta_{\min}=\eta_{0}/10$	Warmup 0, then cosine; $\eta_{\min}=\eta_{0}/10$	Warmup 0, then cosine; $\eta_{\min}=\eta_{0}/10$

Appendix

Table 6: Per-model training specification used in reported experiments.

Setting	AuxContext
Epochs, seeds, context	300 epochs; seeds 0, 5, 10; $K=10$ , $K_{\text{recent}}=5$
Architecture	Same three transformer encoders as ContextGoalPredictor and BDIBottleneck ; $d_{\text{model}}=64$ , layers=3, heads=4, dim_ff=256, dropout=0.1
Goal head	$\mathrm{MLP}_{\mathrm{aux}}([z_{d}\\|z_{r}\\|z_{\tau}])$ ; no belief/desire bottleneck in the goal path
Auxiliary heads	Desire head on $z_{d}$ and belief head on $[z_{d}\\|z_{r}]$ , matching BDIBottleneck ’s targets
Losses	Goal cross-entropy, desire KL, and weighted belief BCE with $\lambda_{g}=1.0$ , $\lambda_{d}=1.0$ , $\lambda_{b}=5.0$ and false-belief weight 5.0
Optimizer and selection	AdamW, learning rate $3\times 10^{-4}$ , weight decay $10^{-4}$ , cosine schedule, best checkpoint selected by validation goal top-1

Appendix

Table 7: Training specification for the matched auxiliary-supervised context baseline. Settings are intentionally aligned with BDIBottleneck ; the only architectural difference is the goal-head input.

Figure 5: BDIBottleneck validation diagnostics across 300 training epochs. Left: overall belief accuracy (all 24 entries); inflated by the majority open class. Center: false-belief F1 on believed-closed entries only; peaks at 40.4% early and converges to 33.3%; an all-open predictor achieves 0% F1, so the converged value reflects genuine false-belief discrimination. Right: desire KL divergence against ground-truth preference distribution; drops from 0.69 to 0.22, reflecting rapid and stable desire learning. Best checkpoint is selected by validation goal top-1.

Setting	BDIBottleneck-NoDesire	BDIBottleneck-NoBelief
Epochs	300	300
Context sizes	$K_{\text{recent}}=5$ ; desire context $K$ not used	$K=10$ ; recent context $K_{\text{recent}}$ not used
Goal loss	$\lambda_{g}\cdot$ cross-entropy ( $\lambda_{g}=1.0$ )	$\lambda_{g}\cdot$ cross-entropy ( $\lambda_{g}=1.0$ )
Desire loss	Not used	$\lambda_{d}\cdot$ KL divergence ( $\lambda_{d}=1.0$ )
Belief loss	$\lambda_{b}\cdot$ weighted BCE ( $\lambda_{b}=5.0$ , false-belief weight $=5.0$ )	Not used
Checkpoint criterion	Best val goal top-1	Best val goal top-1

Appendix

Table 8: Training settings specific to the two ablation variants. Unlisted hyperparameters match BDIBottleneck exactly (Table 6 ).

Figure 6: Top-1 goal accuracy vs. path fraction for BDIBottleneck (full), BDIBottleneck-NoDesire , and BDIBottleneck-NoBelief . Left: held-in test split. Right: held-out new-agent split. NoBelief closely tracks the full model at all fractions; NoDesire lags substantially, demonstrating that desire context is the primary driver of goal disambiguation. Shaded bands: $\pm 1$ SD across 3 training seeds.

Category	POI IDs (within-category order)
attend_class	POI-01, POI-02, POI-03, POI-04
get_food	POI-05, POI-06, POI-07, POI-08
study	POI-09, POI-10, POI-11, POI-12
socialize	POI-13, POI-14, POI-15, POI-16
exercise	POI-17, POI-18, POI-19, POI-20
rest_home	POI-21, POI-22, POI-23, POI-24

Appendix

Table 9: POI indexing used in this appendix.

Config	Split	False-belief POIs (believed closed)
B0	train	none
B1	train	POI-09, POI-10, POI-11
B2	train	POI-05, POI-06, POI-07
B3	train	POI-17, POI-18, POI-19
B4	train	POI-13, POI-14, POI-15
B5	train	POI-09, POI-10, POI-05, POI-06

Appendix

Table 10: Belief configurations (POI IDs as in Table 9 ).

Character	class	food	study	social	exercise	home
studious_stem	3.0	1.0	5.0	0.3	0.4	0.8
social_athlete	0.5	2.0	0.4	4.5	4.0	0.6
foodie_socializer	0.8	5.0	0.8	3.5	0.5	0.4
homebody_studier	1.0	0.6	2.5	0.2	0.2	6.0
well_rounded	2.0	2.0	2.0	2.0	1.5	1.5

Appendix

Table 11: Category-level Dirichlet parameters by character (category order: attend_class, get_food, study, socialize, exercise, rest_home).

Character	Node alphas by category
studious_stem	class (1.5, 1.0, 3.0, 0.5); food (0.5, 2.5, 3.0, 0.5); study (1.5, 3.5, 1.0, 3.0); social (1.5, 1.0, 0.5, 1.5); exercise (1.0, 2.0, 0.5, 1.5); home (1.0, 3.0, 1.0, 1.5)
social_athlete	class (2.0, 1.0, 1.0, 0.5); food (3.5, 0.5, 0.5, 3.0); study (2.0, 0.5, 2.5, 0.5); social (2.5, 1.0, 3.5, 1.5); exercise (4.0, 1.0, 1.5, 3.5); home (3.0, 1.0, 0.5, 2.0)
foodie_socializer	class (1.0, 1.5, 1.0, 1.5); food (3.0, 2.0, 1.0, 4.0); study (2.5, 0.5, 2.0, 0.5); social (1.5, 4.0, 1.0, 4.5); exercise (1.0, 1.0, 0.5, 2.5); home (2.0, 1.0, 2.0, 1.0)
homebody_studier	class (1.0, 1.5, 1.0, 0.5); food (0.5, 4.0, 1.0, 0.5); study (3.5, 1.0, 0.5, 2.0); social (1.0, 2.0, 0.5, 1.0); exercise (0.5, 1.5, 0.5, 2.0); home (1.0, 1.0, 4.0, 2.5)
well_rounded	class (2.0, 2.0, 2.0, 1.5); food (2.0, 2.0, 2.0, 2.0); study (3.0, 1.5, 2.0, 1.5); social (2.0, 2.0, 1.5, 2.0); exercise (2.5, 2.0, 1.5, 2.0); home (2.0, 2.0, 2.0, 1.5)

Appendix

Table 12: Within-category Dirichlet parameters by character. Each tuple is a 4-vector aligned to the category’s POI order in Table 9 .

Regime	Configurations	Closed-entry rate	Closed F1	Closed PR-AUC
Sparse test	$B_{0}$ – $B_{7}$	0.135	0.390	0.300
Sparse new agent	$B_{0}$ – $B_{7}$	similar sparse rate	0.396	0.309
Dense test	$B_{0}$ – $B_{7}$ , BM1–BM4, BD1–BD4	0.298 overall	0.524	0.462
Dense new agent	same dense family	0.298 overall	0.521	0.453

Appendix

Table 13: Sparse versus dense false-belief regimes. Closed-entry F1 and PR-AUC evaluate the minority believed-closed class, not the majority open class.

Regime	Config	Model	Top-1	Closed F1
Sparse	default weighted	AuxContext	$0.253\pm 0.013$	$0.399\pm 0.006$
Sparse	default weighted	BDIBottleneck	$0.196\pm 0.036$	$0.392\pm 0.006$
Sparse	balanced focal	AuxContext	$0.248\pm 0.008$	$0.389\pm 0.004$
Sparse	balanced focal	BDIBottleneck	$0.195\pm 0.052$	$0.387\pm 0.008$
Dense	default weighted	AuxContext	$0.302\pm 0.009$	$0.524\pm 0.002$
Dense	default weighted	BDIBottleneck	$0.305\pm 0.064$	$0.523\pm 0.003$

Appendix

Table 14: Same-budget robustness summary. Values are mean

\pm

standard deviation across seeds on the test split. The sweep is diagnostic rather than a tuned winner selection.

Figure 7: Percentage of distractor episodes where the model ranks the true goal above the distractor POI, by belief configuration. B0 is excluded (no false beliefs). The dashed line marks chance (50%). Upper panel: held-in test agents. Lower panel: new agents.

Figure 8: Belief-head sensitivity by training evidence on the test split. Bars show mean $\Delta$ per evidence bin, where $\Delta=\hat{P}(C\mid b_{j}{=}0)-\hat{P}(C\mid b_{j}{=}1)$ . Error bars denote $\pm 1$ standard error across (agent, POI) pairs, and labels show the number of pairs in each bin.

Split	Outcome	Mean	95% CI	Permutation $p$	Sign test $p$
Test	$\Delta_{\text{alt}}$	0.0054	[-0.0009, 0.0125]	0.0784	0.1509
Test	$\Delta_{\text{goal}}$	-0.0122	[-0.0259, 0.0013]	0.0521	0.0592
Test (new agent)	$\Delta_{\text{alt}}$	0.0050	[-0.0040, 0.0141]	0.1634	0.5000
Test (new agent)	$\Delta_{\text{goal}}$	-0.0233	[-0.0338, -0.0130]	0.0006	0.0037

Appendix

Table 15: Agent-clustered sensitivity analysis for counterfactual belief effects at 50% path (rank

\leq 3

). For each agent, episode-level deltas are averaged to one value per outcome (

n_{\text{agents}}=15

per split), then one-sided tests are applied to agent-level means.

	Test		Test (new agent)
Path fraction	$\Delta_{\text{alt}}$	$\Delta_{\text{goal}}$	$\Delta_{\text{alt}}$	$\Delta_{\text{goal}}$
25%	$+0.0120$	$-0.0200$	$+0.0013$	$-0.0229$
50%	$+0.0061$	$-0.0147$	$+0.0055$	$-0.0240$
75%	$+0.0040$	$-0.0100$	$+0.0017$	$-0.0264$
100%	$+0.0072$	$-0.0171$	$+0.0007$	$-0.0242$

Appendix

Table 16: Mean

\Delta_{\text{alt}}

and

\Delta_{\text{goal}}

for belief counterfactual interventions at each path fraction (rank

\leq 3

n=206

test /

n=237

new-agent episodes).

Panel of Figure 9 — Figure 9: Belief counterfactual at 25% path reveal (rank $\leq 3$ , 2 $\times$ 2 panel). Top row: held-in test split; bottom row: held-out new-agent split. Left column: mean alternative gain. Right column: mean goal drop. Error bars: $\pm 1$ SE. Compare to the 50% results in Figure 4 : goal-drop magnitude is larger here on held-in agents, consistent with weaker trajectory evidence at earlier reveals.

Regime	Split	Type	$n$	Top-1 (%)	Top-5 (%)	$p(g_{\mathrm{sim}})$
dense	test	belief	960	14.7 [11.2, 18.4]	57.0 [50.0, 63.6]	0.0902 [0.0812, 0.0989]
dense	test	desire	960	14.0 [11.1, 17.1]	50.9 [45.4, 56.8]	0.0874 [0.0823, 0.0931]
dense	test_new_agent	belief	960	11.6 [9.1, 14.3]	50.9 [45.9, 55.7]	0.0807 [0.0756, 0.0862]
dense	test_new_agent	desire	960	14.6 [12.6, 16.7]	55.8 [51.6, 60.2]	0.0946 [0.0884, 0.1016]
sparse	test	belief	480	17.5 [13.3, 21.9]	58.8 [52.9, 64.6]	0.0859 [0.0785, 0.0935]
sparse	test	desire	480	9.6 [6.5, 12.9]	45.6 [39.6, 52.5]	0.0729 [0.0675, 0.0789]

Appendix

Table 17: Simulator-resampled full-trajectory counterfactual prediction at 50% path reveal. Values are means with 95% bootstrap confidence intervals clustered by agent. Top-1/top-5 measure agreement with the simulator-sampled counterfactual goal;

p(g_{\mathrm{sim}})

is the mean probability assigned to that goal.

Regime	Split	Type	Channel	$\Delta_{\mathrm{orig}}$	$\Delta_{\mathrm{sel}}$	$\Delta_{\mathrm{sim}}$	$\Delta_{\mathrm{sim-cf}}$
dense	test	belief	predicted	-0.0112 [-0.0137, -0.0088]	0.0855 [0.0765, 0.0949]	-0.0870 [-0.0968, -0.0774]	-0.1725 [-0.1815, -0.1641]
dense	test	desire	predicted	-0.0705 [-0.0836, -0.0582]	0.1220 [0.1042, 0.1391]	-0.0552 [-0.0719, -0.0399]	-0.1772 [-0.1888, -0.1652]
dense	test_new_agent	belief	predicted	-0.0095 [-0.0115, -0.0075]	0.0976 [0.0883, 0.1074]	-0.0653 [-0.0736, -0.0576]	-0.1629 [-0.1719, -0.1543]
dense	test_new_agent	desire	predicted	-0.0532 [-0.0641, -0.0427]	0.1467 [0.1275, 0.1647]	-0.0276 [-0.0420, -0.0137]	-0.1743 [-0.1880, -0.1602]
sparse	test	belief	predicted	0.0016 [0.0009, 0.0022]	0.0453 [0.0430, 0.0476]	-0.0234 [-0.0274, -0.0196]	-0.0687 [-0.0737, -0.0642]
sparse	test	desire	predicted	-0.0182 [-0.0223, -0.0148]	0.0450 [0.0386, 0.0512]	-0.0265 [-0.0333, -0.0202]	-0.0715 [-0.0781, -0.0653]

Appendix

Table 18: Model-planned trajectory counterfactual probability deltas at 50% path reveal. Values are means with 95% bootstrap confidence intervals clustered by seed-agent.

\Delta_{\mathrm{orig}}=p^{\mathrm{cf}}(g_{\mathrm{orig}})-p(g_{\mathrm{orig}})

;

\Delta_{\mathrm{sel}}=p^{\mathrm{cf}}(g_{\mathrm{cf}})-p(g_{\mathrm{orig}})

;

\Delta_{\mathrm{sim}}=p^{\mathrm{cf}}(g_{\mathrm{sim}})-p(g_{\mathrm{orig}})

; and

\Delta_{\mathrm{sim-cf}}=p^{\mathrm{cf}}(g_{\mathrm{sim}})-p^{\mathrm{cf}}(g_{\mathrm{cf}})

, where

g_{\mathrm{cf}}

is the model-selected counterfactual goal and

g_{\mathrm{sim}}

is the simulator-resampled counterfactual goal.

Regime	Split	Type	$p_{\mathrm{adj}}(\Delta_{\mathrm{orig}}<0)$	$p_{\mathrm{adj}}(\Delta_{\mathrm{sel}}>0)$	$p_{\mathrm{adj}}(\Delta_{\mathrm{sim}}\neq 0)$	$p_{\mathrm{adj}}(\Delta_{\mathrm{sim-cf}}\neq 0)$
dense	test	belief	0.0032	0.0032	0.0032	0.0032
dense	test	desire	0.0032	0.0032	0.0032	0.0032
dense	test_new_agent	belief	0.0032	0.0032	0.0032	0.0032
dense	test_new_agent	desire	0.0032	0.0032	0.0032	0.0032
sparse	test	belief	1.0000	0.0032	0.0032	0.0032
sparse	test	desire	0.0032	0.0032	0.0032	0.0032

Appendix

Table 19: Cluster random-sign tests for the model-planned probability deltas in Table 18 . Values are Holm-adjusted

p

-values across the displayed 50% predicted-channel tests. One-sided alternatives are used for the original-goal decrease and model-selected-goal increase; two-sided tests are used for simulator-goal deltas.

Path	Test				Test (new agent)
frac. (%)	Goal	Ctx	Aux	BDI	Goal	Ctx	Aux	BDI
10	6.11	12.22	12.85	13.89	5.35	11.32	10.63	11.74
20	10.97	18.40	16.67	20.49	9.58	14.86	13.47	19.10
30	16.46	21.39	21.32	24.10	16.67	20.28	18.96	23.54
40	23.54	27.57	27.22	31.74	23.12	24.65	24.38	33.26
50	27.99	31.53	31.53	36.32	26.46	28.75	27.50	36.11

Appendix

Table 20: Top-1 goal inference accuracy (%) at 10% path-fraction increments. Aux is the matched auxiliary-supervised context baseline.

Split	Comparison	10%	20%	30%	40%	50%
test	BDI vs. GP	< 0.001	< 0.001	0.282	< 0.001	0.001
test	BDI vs. CTX	1.000	1.000	1.000	1.000	1.000
test_new_agent	BDI vs. GP	0.001	< 0.001	0.276	< 0.001	< 0.001
test_new_agent	BDI vs. CTX	1.000	1.000	0.919	1.000	1.000

Appendix

Table 21: McNemar’s exact test (BDIBottleneck vs. baselines) at 10% path-fraction increments. Holm–Bonferroni adjusted

p

-values across 40 tests. Bold = significant at

\alpha=0.05

. Top panel: 10–50%; bottom panel: 60–100%.

Split	Comparison	Frac	$n$	$n_{01}$	$n_{10}$	$p$	$p^{*}$
test	BDI vs. GoalPredictor	10%	480	15	51	< 0.001	< 0.001
		20%	480	19	69	< 0.001	< 0.001
		30%	480	44	70	0.019	0.282
		40%	480	39	90	< 0.001	< 0.001
		50%	480	40	86	< 0.001	0.001
		60%	480	36	97	< 0.001	< 0.001

Appendix

Table 22: Full McNemar’s exact test results at 10% path-fraction increments.

n_{01}

: BDI correct & baseline wrong;

n_{10}

: baseline correct & BDI wrong;

p

: raw two-sided

p

-value;

p^{*}

: Holm–Bonferroni adjusted (40 tests total). Bold = significant at

\alpha=0.05

Figure 10: Top-5 goal accuracy vs. path fraction on the held-in test split for the three learned models.

Figure 11: Top-5 goal accuracy vs. path fraction on the held-out new-agent split. The generalization gap between test and new-agent performance narrows relative to the top-1 setting, consistent with the relaxed identification requirement.

Figure 12: Full four-fraction desire counterfactual delta matrices, model-derived prototypes, held-in test split. Panels correspond to 25%, 50%, 75%, and 100% path reveal (left to right). Rows are source desire categories; columns are transplanted categories; each cell is the mean change in category-level goal probability relative to the no-swap baseline.

Figure 13: Full four-fraction desire counterfactual delta matrices, model-derived prototypes, held-out new-agent split. The off-diagonal structure generalizes across held-out agents, indicating that the desire channel’s influence on goal prediction is not agent-specific.

Figure 14: Full four-fraction desire counterfactual delta matrices, handcrafted category-only prototypes, held-in test split. Handcrafted prototypes place uniform mass over the four POIs in the target category and zero elsewhere, producing a sharper desire signal than model-derived prototypes and correspondingly larger delta magnitudes.

Figure 15: Full four-fraction desire counterfactual delta matrices, handcrafted category-only prototypes, held-out new-agent split. The stronger category signal of handcrafted prototypes produces consistently larger shifts than model-derived prototypes across both splits and all fractions.

Figure 16: Model-derived prototype distributions on the held-in test split. Each panel shows the 24-POI mean desire vector for one target category, with POIs grouped and color-coded by category. Non-trivial mass outside the target category reflects the smoothness of individual desire estimates.

Figure 17: Model-derived prototype distributions on the held-out new-agent split. The within-category concentration is comparable to the test split, indicating that the desire encoder learns category-consistent representations that generalize to unseen agents.

Figure 18: Handcrafted category-only prototype distributions on the held-in test split. Each prototype assigns uniform mass over the four POIs in the target category and zero mass elsewhere, providing a sharp, idealized desire signal for comparison with model-derived prototypes.

Figure 19: Handcrafted category-only prototype distributions on the held-out new-agent split. Identical in construction to Figure 18 ; shown separately to confirm that prototype construction is split-invariant by design.

Theory of Mind

Theory of Mind studies how people and AI infer others’ beliefs, goals, and knowledge to predict behavior and coordinate interaction.

See all 95 papers

Mar 19, 2026

Teleological Inference in Structural Causal Models via Intentional Interventions

Structural causal models (SCMs) were conceived to formulate and answer causal questions. This paper shows that SCMs can also be used to formulate and answer teleological questions, concerning the intentions of a state-aware, goal-directed agent intervening in a causal system. We review limitations of previous approaches to modeling such agents, and then introduce intentional interventions, a new time-agnostic operator that induces a twin SCM we call a structural final model (SFM). SFMs treat observed values as the outcome of intentional interventions and relate them to the counterfactual conditions of those interventions (what would have happened had the agent not intervened). We show how SFMs can be used to empirically detect agents and to discover their intentions.

Dario Compagno, Fabio Massimo Zennaro

Structural Causal Models Causal Intervention

Ask PDF

Apr 22, 2026

DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories

We introduce DialToM, an annotated Theory of Mind (ToM) benchmark built from naturalistic human-human dialogues using a multiple-choice evaluation framework. Concurrent with recent work showing a gap between explicit mental-state inference and applied ToM in synthetic settings~\cite{gu2024simpletom}, we establish a stricter State-Driven Diagnostic Probe in which models must forecast state-consistent dialogue trajectories solely from isolated mental-state profiles without dialogue context. Our evaluation reveals a systematic reasoning asymmetry -- LLMs excel at inferring mental states (Literal ToM) but struggle to leverage them for social forecasting (Functional ToM). Crucially, a domain expert achieves 100% accuracy on this task, proving its validity and establishing a stark human-AI capability gap. Further, a teacher-student reasoning injection probe shows that Gemini 3 Pro -- which establishes the leading baseline -- possesses robust Functional ToM capabilities for context-free forecasting that are transferable to weaker models. DialToM, its evaluation code, and dataset are publicly available at https://github.com/Stealth-py/DialToM.

Neemesh Yadav, Palakorn Achananuparp, Jing Jiang +1

Singapore Management University · Australian National University

LLM Evaluation Theory of Mind

Ask PDF

Jun 10, 2026

The Theory of Mind Utility: Formal Specification of a Mentalizing Mechanism

Inferring others' beliefs requires more than reading surface signals; it requires tracking who told them what, in what order, and how credibly. The Theory of Mind Utility (ToM-U) formalizes this epistemic state inference problem at the computational level of analysis, specifying what mentalizing computes and why without commitment to algorithmic or neural implementation. ToM-U achieves this by constructing Local Epistemic World Models (LEWMs) -- directed typed graphs that represent agents, state nodes, and the epistemic relationships among them -- and evaluating discrete candidate LEWMs against observed behavior until one achieves sufficient confidence. Five formal definitions specify the LEWM structure, agent node properties including ordered information access history, a bounded proliferation mechanism for recursive mentalizing, three inference procedures, and a residue function that captures the structured trace left by failed mentalizing attempts. ToM-U differs from Bayesian Theory of Mind and adjacent formal accounts, which presuppose rather than derive belief states, and from simulation theory and theory-theory, which lack a formal apparatus for epistemic state inference. The architecture generates directional, falsifiable predictions about mentalizing failure that follow from structural properties of the model rather than auxiliary assumptions, and positions ToM-U as a domain-agnostic mechanism upstream of goal inference and other downstream social cognitive processes.

Nikolos Gurney, Stacy Marsella

Institute for Creative Technologies, University of Southern California California, United States of America · Khoury College of Computer Sciences, Northeastern University Massachusetts, United States of America

Computational Cognitive Modeling Theory of Mind Social Reasoning

Ask PDF

Abstract

Theory of Mind

Explore similar work

Teleological Inference in Structural Causal Models via Intentional Interventions

DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories

The Theory of Mind Utility: Formal Specification of a Mentalizing Mechanism