BASIS: Evaluating Belief-Aware Solutions in Interactive Settings beyond Instructions
Authors: Minyuan Ruan, Ziyue Wang, Kaiming Liu, Yunghwei Lai, Peng Li, Yang Liu
Organizations: Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
Large language model (LLM) agents are increasingly capable of carrying out complex, long-horizon tasks, yet effective assistance requires more than faithfully executing user instructions. Users' requests reflect their current understanding of the problem, which may be incomplete, outdated, or incorrect. When this understanding diverges from the actual task state, literal instruction following can fail to accomplish the user's underlying goal. We introduce BASIS (Belief-Aware Solutions in Interactive Settings), a benchmark that evaluates whether agents can infer the mental states underlying user requests, recognize belief-reality mismatches, and use this understanding to develop appropriate solutions. Rather than assessing mental-state inference solely through explicit questions, BASIS evaluates both the utility of such inferences and the quality of downstream solutions in user-agent interactions. Evaluation results show that current models often fail to account for users' mental states when generating solutions, and that solution quality is consistently higher when such information is explicitly elicited and made available during problem solving. We further investigate the practical value of mental-state consideration through comparisons with generic Chain-of-Thought prompting, controlled interventions on mental-state information, targeted training, and multi-agent collaboration experiments under partial observability. Together, these studies highlight the importance of not merely inferring others' mental states, but also using those understandings to provide grounded, goal-directed assistance.
Figures & tables
Figure 1: (a) An illustrative case shows a user interpreting a ModuleNotFoundError as a missing package and requesting reinstallation. Literal execution repeats the installation and leaves the error unresolved, whereas mental-state consideration helps the agent to check the active Python environment and use the one where the package is already installed. (b) An exploratory annotation study of SWE-Chat identifies instruction gaps, where mismatches exist between users’ explicit requests and what is needed to achieve their underlying goals, in 47% of interactions; among these gap cases, 89.4% show direct or strong contextual evidence related to users’ mental states.
Figure 2: The BASIS data construction pipeline. Stage 1: Scenario synthesis converts source task data into user-agent scenarios with belief-reality mismatches, specifying the actual task state s , user profile P , observations o , belief b , and request g . Stage 2: Trajectory construction generates multi-turn interaction histories τ that simulate user behavior and observation. An LLM-based validation module assesses data quality at both stages across five dimensions, with low-scoring instances refined and revalidated. The culture example illustrates the progression from source data to a mismatch scenario and its associated interaction trajectory.
Type
Scenarios
Cnt.
Belief- Profile Align.
Belief- Truth Corr.
Traj.- Belief Consist.
Traj.- Profile Consist.
Traj. Realism
AVG
LLM
Preference
120
4.90
4.80
4.06
4.34
4.04
4.43
Education
100
4.90
4.75
4.21
4.31
4.01
4.44
Culture
70
4.99
4.94
4.04
4.01
4.01
4.40
SWE
100
4.85
4.81
4.01
4.24
4.01
4.38
Human
Preference
120
4.75
4.74
4.01
4.18
3.92
4.32
Education
100
4.89
4.70
4.21
4.17
3.94
4.38
Table 1: Data distribution and quality analysis. We use an LLM scorer to assign scores from 0 to 5 according to the criteria defined in Appendix B.2 , and additionally report human ratings along the same dimensions. Cnt.: count; Align.: alignment; Corr.: correlation; Consist.: consistency; Traj.: trajectory.
Dimension
Accuracy
Weighted Kappa
Spearman Corr.
Belief
86.67%
0.86
0.85
Profile
85.67%
0.83
0.84
Solution
83.33%
0.88
0.89
Table 2: Human-LLM evaluator agreement on sampled instances across three evaluation dimensions.
Models
Evaluation Dimension
Preference
Culture
SWE
Education
Separate
Vanilla
5-Step
10-Step
Separate
Vanilla
5-Step
10-Step
Separate
Vanilla
5-Step
10-Step
Separate
Vanilla
5-Step
10-Step
GPT-5.6-Sol
Latent Belief
38.61
42.08
63.54
67.29
51.90
68.10
68.81
70.24
57.34
65.51
74.75
75.25
34.58
42.67
56.42
63.42
Profile
20.69
21.81
36.18
44.37
23.33
27.86
37.26
35.83
33.83
34.16
42.99
50.58
17.17
11.92
32.75
36.25
Solution
19.43
29.40
71.00
78.44
25.11
41.25
55.38
65.51
24.00
39.31
58.33
68.04
29.11
39.84
57.03
61.78
AVG
26.24
31.10
56.91
63.37
33.45
45.74
53.82
57.19
38.39
46.33
58.69
64.62
26.95
31.48
48.73
53.82
Claude-Sonnet-5
Latent Belief
39.44
45.83
61.94
63.68
48.31
61.55
59.06
72.62
49.92
54.29
72.63
72.36
46.09
45.25
54.29
64.37
Table 3: Evaluation results of 6 models across four domains (Preference, Culture, SWE, Education) and four evaluation configurations (Separate, Vanilla, 5-Step, 10-Step), regarding three evaluation dimensions. Cell shading indicates relative performance: darker purple signifies higher scores while lighter shades represent lower scores. “AVG”: the average over dimensions. “DeepSeek-R1-32B”: DeepSeek-R1-Distill-Qwen-32B
Model
Method
Pref.
Cult.
SWE
Edu.
Avg.
GPT-5.6-Sol
CoT
20.89
25.18
28.37
29.43
25.97
Mental
29.40
41.25
39.31
39.84
37.45
Gemini-3.8
CoT
25.88
54.52
53.07
43.01
44.12
Mental
28.65
61.38
52.91
47.25
47.55
Qwen3.8-27B
CoT
15.10
19.65
23.09
24.97
20.70
Mental
22.83
37.13
27.57
37.32
31.21
Table 4: Solution scores under CoT and mental-state consideration across three models. Mental refers to mental-state consideration. Pref.: Preference; Cult.: Culture; Edu.: Education. Avg. denotes the mean across domains. Bold indicates the higher score between the two strategies for each model within each domain and on average.
Figure 3: Performance changes when generating solutions conditioned on the mental-state information indicated in the legend. Providing task-aligned belief or profile information generally improves solution quality, whereas mismatched information leads to performance degradation.
\rectanglecolor blue!76-18-14 Models
Metrics
Preference
Culture
SWE
Education
vanilla
5-Turn
10-Turn
vanilla
5-Turn
10-Turn
vanilla
5-Turn
10-Turn
vanilla
5-Turn
10-Turn
Qwen3-8B
Solution
11.91
37.54
47.45
7.46
18.48
24.82
8.37
18.25
25.67
16.62
24.29
28.48
Belief
23.94
31.28
31.92
31.85
34.55
40.88
37.69
40.12
41.28
17.95
17.97
23.80
Profile
12.56
17.27
17.97
10.55
15.23
19.41
15.87
18.39
22.09
6.19
10.29
15.13
Ours
Solution
15.65 +3.74
47.14 +9.60
58.25 +10.80
10.40 +2.94
26.71 +8.24
33.84 +9.02
11.29 +2.93
25.63 +7.38
33.74 +8.08
23.10 +6.48
34.61 +10.32
38.73 +10.25
Belief
25.25
37.47
41.89
36.97
41.99
52.43
41.72
44.84
49.44
22.20
31.88
37.75
Table 5: RL training results of Qwen3-8B model. Bolded values indicate performance improvements within the Solution . Values in the Ours-Solution row additionally show the difference from the corresponding baseline Solution score.
Method
HiddenBench
AgentWebBench
Qwen3.8-27B
Baseline
67.44
29.64
Mental-state
75.26
34.93
DeepSeek-V4-Flash
Baseline
71.48
41.57
Mental-state
79.17
45.89
Table 6: Multi-agent benchmark scores. Mental-state denotes explicit prompting to consider other agents’ mental states. Shaded rows mark this variant; bold indicates the higher score within each model and benchmark.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
Label
Definition
Instruction quality
Complete and correct
The stated instruction contains the requirements necessary for successful task completion and does not contain a materially incorrect requirement.
Incomplete
The instruction omits one or more requirements that are necessary for completing the task correctly.
Incorrect
The instruction contains a materially incorrect requirement, assumption, or requested implementation.
Both
The instruction is both incomplete and contains at least one materially incorrect requirement.
Mental state
Incomplete observation
The user’s limited view of the project or execution environment prevents a complete understanding of the underlying problem, leading to an instruction that omits necessary requirements or rests on assumptions inconsistent with the actual task state.
Personal preference / other
The user’s preferences, prior experience, or domain expertise shape their interpretation of the available evidence, leading to an instruction based on assumptions that are inconsistent with the actual task state.
Appendix
Table 7: Annotation dimensions and labels used in our SWE-Chat study.
Figure 4: Annotation interface for the SWE-Chat study. Annotators inspect the interaction trajectory and available project information before assigning labels for instruction quality, the source of the requirement gap, literal executability, and agent gap handling.
Dimension
Label
Count
Sum
Percentage
Instruction quality
Complete and correct
53
100
53.0%
Incomplete
24
100
24.0%
Incorrect
14
100
14.0%
Both
9
100
9.0%
Mental state
Incomplete observation
25
47
53.2%
Personal preference / other
17
47
36.2%
Appendix
Table 8: Results of the SWE-Chat annotation study. Instruction quality is annotated for all 100 trajectories. The remaining dimensions are evaluated on the 47 trajectories containing an incomplete or incorrect instruction.
Judge Prompt
Task: Generate Atomic Fact Verification Rubrics.
Role: Strict Logic Auditor. The model decomposes a complex ground truth into a set of atomic facts that can be verified via simple search or strict logical checks. The objective is to maximize inter annotator agreement such that different evaluators using the same checklist will obtain identical scores.
Input Data includes observation, explicit instruction, true latent state, user latent belief, root cause of misconception, and user profile.
High agreement guidelines are as follows. No adjectives are allowed. Criteria must be anchored to explicit keywords or named entities from the input. Detection and correction must be separated into different criteria. If a user misconception exists, the model must explicitly refute it.
The output is a JSON object containing strictly verifiable boolean criteria across three dimensions. The first dimension evaluates latent belief recognition. The second dimension evaluates user profile and contextual modeling. The third dimension evaluates correct identification of root cause and resolution.
Each criterion must be a statement that can be answered with a binary yes or no judgment.
Appendix
Table 9: Prompts for rubrics generation.
Model
Type
Reference
GPT-5.6-sol
Proprietary
[ 26 ]
Gemini 3.8 Flash
Proprietary
[ 13 ]
Claude Sonnet 5
Proprietary
[ 3 ]
Qwen3.8-27B
Open-source
[ 31 ]
Qwen3.5-9B
Open-source
[ 30 ]
Qwen3-8B
Open-source
[ 45 ]
Appendix
Table 10: Full list of models evaluated in our experiments.
Figure 5: MiniMax-M2.1 trends with varied input trajectory turns.
Task
Dimension
0
1
2
3
4
5
6
7
8
9
10
Culture
Latent belief
49.17
54.40
59.64
60.12
57.98
57.86
63.81
59.18
69.40
62.50
61.31
User profile
29.29
32.26
36.43
36.07
40.95
37.26
46.07
40.94
42.50
44.17
42.98
Resolution
36.24
51.67
54.74
51.47
54.64
57.93
65.00
63.94
60.33
63.53
66.26
Average
38.23
46.11
50.27
49.22
51.19
51.02
58.29
54.69
57.41
56.73
56.85
SWE
Latent belief
56.93
56.42
63.98
59.90
62.07
61.84
61.98
63.11
59.55
60.07
60.44
User profile
41.23
41.41
39.58
45.49
44.62
46.67
41.49
42.27
45.92
46.01
46.32
Appendix
Table 11: Task performance of MiniMax-M2.1 across trajectories.
Scenarios
#Traj.
None
Belief & Profile
Belief
Profile
Preference
Vanilla
11.91
17.39 (+5.48)
16.79 (+4.88)
16.19 (+4.28)
10-Turn
47.45
53.62 (+6.17)
54.32 (+6.87)
56.37 (+8.92)
Culture
Vanilla
7.46
19.40 (+11.94)
19.90 (+12.44)
20.05 (+12.59)
10-Turn
24.82
36.11 (+11.29)
33.56 (+8.74)
31.66 (+6.84)
SWE
Vanilla
8.37
11.58 (+3.21)
7.27 (-1.10)
8.23 (-0.14)
10-Turn
25.67
31.49 (+5.82)
29.65 (+3.98)
28.78 (+3.11)
Appendix
Table 12: Enhancing the Solution performance of Qwen3-8B via multi-model collaboration, by incorporating the other two models, InternLM3-8B and DeepSeek-R1. “#Traj” refers to the number of trajectory turns provided to the input.
Figure 6: Average performance and variance of Base Qwen3-8B and our trained model.
Scenario Generation Prompt
Role: Synthetic Data Generator for Latent Belief Mismatch
You are generating a dataset to test if AI models can deduce a user’s hidden misconception based on their actions.
Mathematical Model:
- True State ( s∗ ): The actual root cause of the issue (The reality).
- Observation ( o ): The initial symptom perceived by the user. Must be ambiguous (supports multiple theories).
Appendix
Table 13: Prompts for scenario and trajectory generation.
Data Validation Prompt
Role: Expert Data Evaluator
You are an expert in analyzing user behavior simulation datasets. Your task is to evaluate the quality of a generated scenario and user trajectory based on specific logical consistency and realism criteria.
Understanding what a user believes and intends is central to building effective agent assistants. This ability is often evaluated through Theory-of-Mind (ToM) tasks, where success requires reasoning from the user's perspective. However, many existing approaches address ToM with complex pipelines that model behavior indirectly, without explicitly reconstructing the user's mental state. This misses the core structure of the problem: users act based on their beliefs, which are updated through observations of the environment; beliefs and intentions jointly determine actions, which in turn change the environment; and social reasoning often requires nested beliefs about what others believe or intend. We propose UserHarness, a simple framework that reframes ToM reasoning as explicit user-mind reconstruction. UserHarness decomposes the user's mental state, its relation to the external environment, and the actions that follow from it, enabling agents to track what the user observes, believes, intends, and does. Across five benchmarks, UserHarness reaches up to 95.94% macro accuracy, improving over existing inference methods by more than 15% relative and over the strongest prompt-only harness by about 20% relative. These results suggest that robust user understanding requires reasoning from the roots of the user's mind, positioning user harnessing as a promising foundation for more adaptive future assistants.
Large language models (LLMs) are increasingly used for everyday assistance, yet existing benchmarks only partially reflect the requests users naturally make in practice. Real-world requests are often open-ended, casually specified, and context-dependent, requiring models not only to follow explicit instructions but also to infer unstated needs from user background and situational context. We introduce xDailyBench, a benchmark of 248 carefully curated tasks spanning 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities. The tasks are grounded in requests that users have actually completed or genuinely intended to accomplish with AI, and are evaluated with fine-grained binary rubrics covering both explicit and implicit requirements. We evaluate 11 frontier models under standardized agentic settings. The best models achieve a task-level score of 75.6%, while all models perform substantially worse on implicit than explicit requirements, with gaps no less than 9 percentage points. These results reveal implicit requirement inference as a persistent bottleneck for reliably satisfying real-world everyday user needs.
Effective collaboration requires language models to understand what others know and believe, and to act accordingly. Models that assume others share their information may omit necessary communication and undermine coordination. Many theory-of-mind (ToM) benchmarks assess mental-state inference through story questions, but correct answers do not establish that models can use these judgments in subsequent tasks. We introduce a multi-level evaluation built on existing, verifiable tasks. We extend household collaboration and calendar scheduling with asymmetric information while retaining their original constraints and validators. Independent interfaces assess world understanding, information access, beliefs, decisions, and execution. Controls supply personal evidence or correct beliefs to distinguish state recovery from state use. Across six model configurations, these abilities diverge. On matched scheduling cases, Claude recovers all agent beliefs correctly but produces subjectively optimal plans in only 67.9% of cases. In household collaboration, supplying beliefs raises Qwen-27B success from 55.0% to 75.0%, yet Qwen-4B answers every fixed supplied-belief source question correctly while completing only 20.0% of the corresponding continuation condition. These results show why ToM evaluation must connect perspective inference to independently verified decisions and task consequences.
Zhenshuai Yin, Yongxin Guo, Mingda Zhang +4
School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen), Longgang, Shenzhen, Guangdong 518172, P.R. China · ByteDance Inc. · Shenzhen Research Institute of Big Data, Shenzhen, China