Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

Kawai Chung; Chunkit Chan; Yuxuan Liu; Yauwai Yim; Xi Yang; Wenbin Hu; Huihao Jing; Chiu Yeung Tang

cs.CLOct 13, 2026

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

Authors: Kawai Chung, Chunkit Chan, Yuxuan Liu, Yauwai Yim, Xi Yang, Wenbin Hu, Huihao Jing, Chiu Yeung Tang,

Organizations: Department of Computer Science and Engineering, The Hong Kong University of Science and Technology

Abstract

Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative to generative large language models for bounded Theory of Mind (ToM) selection. We present Jev-ToM, comparing five typed decision models and seven generative baselines on 11,117 English text questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls and ranks first or second across all six capabilities, leading beliefs, goals/desires/intentions, and emotions. Other typed models show distinct strengths across tasks. Separate resource comparisons suggest that Jev could reduce the cost and waiting time of repeated ToM selection. Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn correct answers depending on their attachment. Candidate expansion reveals distinct selection and output-validity problems. Related-question cues account for most of Jev's downstream gains, with task-dependent effects from predicted states. The findings motivate testing Jev as a ToM selection component in assistants, with task-specific confidence calibration, candidate design, and output validation as integration priorities.

Figures & tables

Figure 1: Coverage of evaluated subsets and their union ( $\checkmark$ : present, $\times$ : absent).

Benchmark	Questions	Families	Options
BigToM	1,200	50	2
OpenToM	2,533	149	2/3
ToMBench	730	323	2/4
SimpleToM	891	177	2
FANToM	1,805	64	2
ToMATO	1,396	40	4

Table 1: Evaluated benchmark subsets.

Figure 2: Accuracy (%) for knowledge access, beliefs, goals/desires/intentions, emotions, social pragmatics, and mental-state application, averaged equally across contributing benchmarks. Panels sort models by accuracy, with typed models colored and generative baselines gray. Benchmark control questions and unmapped questions are excluded.

Model	BigToM	OpenToM	ToMBench	SimpleToM	FANToM	ToMATO	CogToM	Hi-ToM	ToMi	Overall
Jev-1.13	89.2	60.0	76.7	59.1	79.6	83.6	81.3	59.7	69.7	73.2
Open-Jev-2B	80.4	45.2	51.4	49.7	71.5	50.1	49.4	39.6	72.0	56.6
Valen	74.9	38.1	46.0	75.1	48.6	32.7	39.4	33.1	57.9	49.5
Tiny-Jev	73.9	34.0	41.5	20.9	71.2	54.2	39.9	30.5	47.2	45.9
OpenJev-0.6B	74.6	39.1	40.7	56.9	29.1	20.0	42.5	31.8	53.9	43.2
GPT-3.5-1106	86.9	61.8	75.1	55.2	77.6	81.6	78.1	55.8	72.4	71.6

Table 2: Exact-match accuracy (%) on 11,117 questions, including native controls. Overall weights benchmarks equally. Invalid outputs count as errors. Bold/underlined values mark group best/second-best, with blue/green for typed/generative models. Excluding controls, Jev scores 69.1% and GPT-3.5 70.9%.

Figure 3: Jev answer selection at confidence

\geq 0.90

: coverage (left, % of all questions) and retained error (right, % of retained answers). Both panels follow descending error rate. Hi-ToM retains 33 answers.

Figure 4: Fresh Jev answer accuracy on earlier correct and incorrect cases. C/W/N/–: reference/selected incorrect/neutral/absent reasons. Dashed: fresh no-reason baseline.

Figure 5: Strict answer-component accuracy changes for aligned and crossed reasons, ordered by aligned gains on new materials. Repair and Damage: Jev’s contributions (percentage points of all weighted outcomes) on new materials. Benchmarks receive equal weight.

Figure 6: Crossed-pair outcomes on all 4,761 materials, averaging both candidate orders and benchmarks equally. “Both correct” is strict joint accuracy, and “only” identifies the correct component. Failures stay in the denominator. Rows sort by both-correct share, with tints on a 0–100% scale.

Figure 7: Joint answer–reason accuracy on 108 questions (one per family) from CogToM, ToMATO, and ToMBench, with two orders and equal benchmark weights. Candidate Changes: strict changes from the original eight-pair set after reattachment or expansion to sixteen. Strict vs Saved: post-hoc scoring reads choices despite invalid probabilities without repairing them.

Figure 8: Accuracy versus OpenRouter reference cost (left) and observed deployment time with differing concurrency (right). Shading marks lower accuracy with higher cost (left) or longer response time (right) than Jev.

Model / task	Original	Question only	Predicted state	Flipped state
Jev-1.13	78.50	89.25	93.00	86.25
Open-Jev-2B	66.50	64.00	64.75	61.75
Valen	62.75	60.50	59.75	60.00
Tiny-Jev	48.50	52.50	51.50	52.50
OpenJev-0.6B	61.50	55.00	51.25	51.00
Jev: BigToM	92.00	98.00	97.50	97.00

Table 3: Accuracy (%) on 400 additional questions from families outside the primary sample, equally weighting 200 BigToM and 200 SimpleToM targets. The last rows show Jev by source. Its same-template fixed-No replay matches SimpleToM predicted feedback at 88.5%.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Study and cohort	Sample	Target and candidate construction	Aggregation
Primary / 12 models	11,117 questions, nine benchmarks	$K$ original answers, answer accuracy	Equal benchmark weights
Confidence / Jev	Same 11,117 primary questions	Selected-answer probability, coverage and retained error	Within benchmark
Six conditions / Jev	1,803 materials, split 249 / 1,554 by earlier correctness	$K$ answers with six reason conditions, two orders	Pooled within historical group
Reason pairs / 8 models	4,761 materials: 1,803 reused and 2,958 new	$K$ answers, $K$ aligned pairs, $K^{2}$ crossed pairs, two orders	Equal benchmark weights
Matched candidates / 2 models	108 questions, one per family, three benchmarks	Joint selection among 4, 8, or 16 pairs, two orders	Equal benchmark weights
Feedback / 5 typed models	200 BigToM and 200 SimpleToM pairs, 300 families	Upstream binary state, downstream selection under four suffix conditions	Equal benchmark weights

Appendix

Table 4: Overview of the main comparisons. Condition and order variants reuse questions.

K

denotes original answer count. Costs use seven priced deployments, and timing uses eight. The full pair study also records reason-only controls.

Benchmark	K	B	G	E	S	A	C	U	$N$	Families
BigToM	–	400	–	–	–	200	600	–	1,200	50
OpenToM	–	2384	149	–	–	–	–	–	2,533	149
ToMBench	78	218	145	103	186	152	–	–	730	323
SimpleToM	297	–	–	–	–	594	–	–	891	177
FANToM	1446	227	–	–	–	–	132	–	1,805	64
ToMATO	305	289	546	256	–	–	–	–	1,396	40

Appendix

Table 5: Benchmark-to-capability question counts. K/B/G/E/S/A denote knowledge, beliefs, goals/desires/intentions, emotions, social pragmatics, and application. C/U denote native controls and unmapped CogToM items. The 152 ToMBench application assignments overlap other capabilities. A dash indicates no assigned questions.

Benchmark	Questions	Families	Options	Words: median
BigToM	1,200	50	2	69
OpenToM	2,533	149	2/3	196
ToMBench	730	323	2/4	61
SimpleToM	891	177	2	36
FANToM	1,805	64	2	566
ToMATO	1,396	40	4	539

Appendix

Table 6: Evaluated-subset statistics. Words are whitespace-delimited units in the original state, excluding questions and options.

Model	Checkpoint or version	Evaluation settings
Jev-1.13	typesafe/jev-1.13	Native choice interface.
Open-Jev-2B	ZefanCai/Open-Jev-2B	Native choice interface and checkpoint settings.
Valen	Valen-Team/Valen-Preview-0923	Native choice interface and checkpoint settings.
Tiny-Jev	lostargon/Tiny-Jev	Native choice interface and checkpoint settings.
OpenJev-0.6B	IamBusy/OpenJev-0.6B	Native choice interface and checkpoint settings.
SmolLM3-3B	HuggingFaceTB/SmolLM3-3B , revision a07cc9a04f16	Non-thinking greedy decoding, 32 output tokens, batch size four, 16,384-token input limit.

Appendix

Table 7: Model versions and primary evaluation settings. All models receive one question per request without task-specific fine-tuning.

Cross-source capability atlas
Capability	$N$	Jev 1.13	GPT-3.5 1106	GPT-5.6 Luna	GPT-4o mini	DeepSeek V4-Flash	GPT-4.1 nano	Llama-3.1 8B	Open-Jev 2B	Valen	SmolLM3 3B	Tiny-Jev	OpenJev 0.6B
Knowledge access	2,338	73.6	76.1	72.6	71.6	63.3	55.3	54.9	43.9	39.1	36.5	35.7	23.4
Beliefs	4,178	68.4	66.9	63.5	54.3	55.3	51.8	48.2	43.6	44.7	34.8	40.2	31.8
Goals, desires and intentions	1,309	78.5	74.4	76.2	68.8	69.0	62.7	65.3	52.0	40.0	45.9	48.4	36.6
Emotions	690	84.8	79.7	79.5	74.8	77.5	73.0	65.7	59.1	41.0	56.2	55.2	43.5
Social pragmatics	383	89.1	82.5	84.9	80.1	89.9	74.4	74.8	52.5	50.1	51.3	53.8	56.4

Appendix

Table 8: Capability exact match (%), part 1. Cross-source rows average contributing benchmarks, while source rows pool assigned questions. Controls and unassigned items enter only All. Overlapping application assignments do not sum to benchmark totals. Bold marks the highest point estimate.

FANToM
Capability	$N$	Jev 1.13	GPT-3.5 1106	GPT-5.6 Luna	GPT-4o mini	DeepSeek V4-Flash	GPT-4.1 nano	Llama-3.1 8B	Open-Jev 2B	Valen	SmolLM3 3B	Tiny-Jev	OpenJev 0.6B
Knowledge access	1,446	81.3	82.6	77.4	82.0	63.0	70.3	60.7	77.6	49.3	49.1	78.4	32.2
Beliefs	227	73.1	35.2	24.2	21.6	39.6	38.3	28.2	17.6	32.2	4.4	12.3	7.0
Control	132	72.0	94.7	97.7	95.5	86.4	86.4	90.9	97.0	69.7	30.3	93.9	34.1
All	1,805	79.6	77.6	72.2	75.3	61.8	67.5	58.8	71.5	48.6	42.1	71.2	29.1
ToMATO

Appendix

Table 9: Capability exact match (%), part 2, using the same model order and scoring as Table 8 . Missing source–capability combinations are omitted.

Model	All questions	Control-free
GPT-3.5-1106	71.62	70.87
Jev-1.13	73.21	69.08
GPT-5.6-Luna	70.85	68.55
GPT-4o-mini	66.29	61.59
DeepSeek-V4-Flash	65.40	60.22
GPT-4.1-nano	61.31	57.14

Appendix

Table 10: Benchmark-equal exact match with all 11,117 questions and after removing 920 native controls. Control-free scores retain 10,197 questions, including unmapped items. Rows descend by control-free accuracy.

Source	Questions	Kept	Wrong
BigToM	1,200	973	44
OpenToM	2,533	1,583	459
ToMBench	730	397	39
SimpleToM	891	418	146
FANToM	1,805	923	67
ToMATO	1,396	651	12

Appendix

Table 11: Counts for Jev selection at probability

\geq 0.90

. Coverage uses all questions, while retained error uses Kept. Native controls remain included.

Figure 9: Jev risk–coverage curves on the same 11,117 primary questions as the main confidence analysis. Coverage is the percentage of all benchmark questions retained, and error is the percentage of retained answers that are incorrect. Each point retains all answers at or above one observed selected-answer probability, keeping ties together. Orange markers denote the 0.90 threshold.

Annotation coverage / follow-up outcome	Questions
Initial sample / targeted follow-up / overlap	450 / 249 / 19
Combined unique questions	680
Majority reference / unanimous / no majority	219 / 211 / 30
Majority references matching benchmark categories	219/219

Appendix

Table 12: Annotation coverage and Jev intervention results under benchmark and human-majority references. The 19 overlaps count once. Majority references require at least two matching single-category votes, with unanimous cases included. Both orders yield 498 full-group and 438 majority-subset responses per condition. Accuracy is pooled (%), differences are paired percentage points, and intervals use 2,000 within-benchmark family-bootstrap draws.

Model	Materials	No reasons	Aligned	Crossed
Jev-1.13	Reused	22.93	74.35	65.10
Jev-1.13	New	88.10	91.79	87.70
GPT-5.6-Luna	Reused	36.31	70.05	67.19
GPT-5.6-Luna	New	81.79	88.90	86.15
GPT-3.5-1106	Reused	36.36	72.35	60.52
GPT-3.5-1106	New	82.21	88.38	81.58

Appendix

Table 13: Absolute answer-component accuracy on reused/new reason materials and Jev repair/damage contributions on new materials. Accuracies average benchmarks equally. Reused/new contain 1,803/2,958 existing questions. Repair and signed damage use all weighted outcomes.

Condition	Jev strict	Jev saved choice	GPT-3.5 strict	Jev invalid
Answers only	71.30	72.22	71.76	2
Four original pairs	85.65	85.65	79.63	0
Four reassigned pairs	92.13	92.59	91.20	1
Eight original- attachment pairs	81.02	84.26	76.39	10
Eight reassigned pairs	83.33	87.04	84.26	8
All sixteen pairs	76.39	82.41	62.50	16

Appendix

Table 14: Matched-candidate accuracy (%) on 108 questions in two orders, with equal benchmark weights. Jev invalid counts use 216 responses per row, and GPT-3.5 has no invalid outcomes. Answer-only measures answer accuracy, while pair rows measure joint accuracy. Post-hoc saved-choice scoring does not replace strict results.

Final failure type	Jev	DeepSeek
Probability sum	448	0
Selected choice below maximum	3	0
Parsing / candidate ID	0	4
HTTP 429	0	843
Invalid / failed	451	847
All crossed responses	9,522	9,522

Appendix

Table 15: Final crossed-pair failures. Counts are pooled across two orders, while Figure 6 uses equal benchmark weights. Earlier failed transport attempts are not additional outcomes.

Panel of Figure 10 — Figure 10: Unfilled excerpts from the annotation interface. Initial Answer: story, question, options, and the initial-answer form, saved before reasons are revealed. Reason Judgments: label guide and the first option’s reason-judgment fields, repeated for every option. The bilingual interface is shown in English for readability.

Model	Task	Belief/awareness	Original	Question	Predicted	Flipped
Jev-1.13	BigToM action	84.50	92.00	98.00	97.50	97.00
Jev-1.13	SimpleToM behavior	91.00	65.00	80.50	88.50	75.50
Open-Jev-2B	BigToM action	58.50	73.00	71.50	67.50	66.50
Open-Jev-2B	SimpleToM behavior	19.00	60.00	56.50	62.00	57.00
Valen	BigToM action	56.50	60.50	58.00	58.00	57.50
Valen	SimpleToM behavior	61.50	65.00	63.00	61.50	62.50

Appendix

Table 16: Mental-state feedback accuracy (%) by model and source, with 200 native upstream questions and 200 action/behavior targets per source. The upstream belief/awareness score is measured once per model. The four downstream conditions use the same target questions.

Model	Question	Predicted	Fixed No	Changed choices
Jev-1.13	161	177	177	0
Open-Jev-2B	113	124	113	11
Valen	126	123	125	2
Tiny-Jev	99	97	94	5
OpenJev-0.6B	118	103	103	18

Appendix

Table 17: Correct answers out of 200 SimpleToM behavior targets for question-only cues, predicted feedback, and the same-template fixed-No replay. Changed choices compares fixed No with predicted feedback on all 200 targets. Every output is valid. The replay selects the existing predicted or flipped response whose supplied answer is No, retaining the original attribution header and including every target exactly once.

Model	Input	Output	Total	Per 1k
TypeSafe Jev	0.0420	0.0000	0.3284	0.0295
GPT-4.1 nano	0.1000	0.4000	0.4876	0.0439
GPT-4o mini	0.1500	0.6000	0.7517	0.0676
GPT-5.6 Luna	0.2000	1.2000	9.9929	0.8989
Llama-3.1-8B	0.0500	0.0800	0.2463	0.0222
DeepSeek-V4-Flash	0.0108	1.2800	0.1535	0.0138

Appendix

Table 18: Resource comparison on 11,117 questions per deployment. The cost panel contains only seven priced models. Input/output tariffs are US dollars per million tokens, Total covers all questions, and Per 1k includes missing-usage allowances. GPT-3.5 uses a historical tariff. The timing panel includes all eight deployments. Attempts is the maximum total per question, and NR means not recorded.

Theory of Mind

Theory of Mind studies how people and AI infer others’ beliefs, goals, and knowledge to predict behavior and coordinate interaction.

See all 92 papers

Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics
Aug 10, 2026Yen-Shan Chen, Yu Chian Duan, Chih-En Kuo +2LLM Evaluation Theory of Mind
1National Taiwan University · 2CyCraft AI Lab, Taiwan
Typed Decision Models: An Early Evidence Audit and Evaluation Checklist
Sep 26, 2026Lijuan Tang, Yuemeng ZhengLLM Evaluation
Northeastern University
DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories
Apr 22, 2026Neemesh Yadav, Palakorn Achananuparp, Jing Jiang +1LLM Evaluation Theory of Mind
Singapore Management University · Australian National University

Aug 10, 2026J/K move · Enter open · S save

Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics

Yen-Shan Chen, Yu Chian Duan, Chih-En Kuo, Jian-Bin Wu+1

1National Taiwan University · 2CyCraft AI Lab, Taiwan

LLM Evaluation Theory of Mind LLM Reasoning

Open paper AskPDF arXiv

Theory of Mind (ToM) is essential for agent interactions, yet existing evaluations either rely on static scenarios that oversimplify mental-state reasoning or interactive settings that provide limited diagnostic insight. We present Avalon-ToM-Bench, a fine-grained benchmark that operationalizes ToM through the asymmetric-information mechanics of The Resistance: Avalon. Rather than evaluating end-to-end gameplay, it decomposes ToM into a 2

\times

2 taxonomy -- epistemic versus motivational reasoning crossed with inference versus action -- using human-crafted, perspective-constrained queries. Benchmarking 28 LLMs reveals three insights: 1) Reasoning, not knowledge. Models show strong game-rule comprehension but markedly weaker ToM abilities, isolating failures to social reasoning rather than missing domain knowledge. 2) Expression, not representation. Mechanistic analyses via linear probing and activation steering show that models frequently represent correct mental-state inferences in their hidden states but fail to express them during generation -- linear probes recover 77-82% accuracy versus 62-70% from the models' own chain-of-thought. 3) Policy, not deliberation. Dedicated reasoning training yields substantial improvements whereas test-time chain-of-thought provides only marginal gains (+11.0 versus +1.1 points on average), suggesting that robust ToM depends on a learned reasoning policy rather than increased inference-time deliberation.

Abstract

Theory of Mind

Explore similar work

Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics

Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories