cs.CLOct 15, 2026
SaveToMStories: Interpreting Theory-of-Mind in Synthetic Narratives
Organizations: Columbia University
Abstract
Understanding people often requires inferring what they feel or want from how they behave rather than from what they state outright. We ask whether language models can make these inferences, and how they do so. We introduce Theory-of-Mind Stories (ToMStories), an evaluation dataset of ~6,000 short synthetically generated narrative vignettes, each paired with a multiple-choice or open-ended question about a character's emotion, fear, motivation, or emotional wound. These psychological states are conveyed implicitly (showing), explicitly (telling), or with conflict between dialogue and description. Evaluating thirteen open-weight models (1B-72B parameters), we find that the largest models match humans on multiple choice but not open-ended questions. Our analyses point to simplifications models make in both interpretation and generation of implicit psychological states, indicating that synthetic data can help us understand how models perform interpretation, but likely overestimates how well models interpret humans.
Figures & tables
Figure 1: ToMStories Pipeline. (1) For each vignette we sample a setting, relationship, and target psychological state, and (2) generate implicit , explicit , and conflicted variants. (3) Models answer multiple choice and open-ended questions about a character’s psychology. Open-ended answers are scored against correct answers by embedding-similarity and an LLM judge.
| Category | Examples | |
| Character | ||
| Mg Wounds | 121 | infidelity, being bullied, growing up in the shadow of a successful sibling |
| Mg Emotions | 135 | anger, giddiness, remorse |
| Mg Fears | 68 | being a burden to others, having no purpose, isolation |
| Mg Motivations | 95 | being accepted by others, proving someone wrong, protecting a loved one |
| Situation | ||
Table 1: Thesaurus categories used in ToMStories.
| Subset | Tot. | Train | Val. | Wound | Emot. | Fear | Motiv. | WC |
|---|---|---|---|---|---|---|---|---|
| Implicit | 1988 | 1593 | 395 | 493 | 496 | 499 | 500 | 217.4 |
| Explicit | 1988 | 1593 | 395 | 493 | 496 | 499 | 500 | 240.6 |
| Conflicted | 1956 | 1561 | 395 | 493 | 493 | 475 | 495 | 342.3 |
Table 2: Summary statistics for ToMStories, showing the number of generated vignettes by split and question type, and the average word count.
| Scenario & question | Vignette excerpt (implicit) |
|---|---|
| wound Professional and client at a coffeehouse What emotional wound does the woman’s behavior suggest she has experienced in her past? being humiliated by others | …She walked him through her finances with quiet precision, answering his questions before he finished asking them, filling every silence before it could stretch. At one point he laughed—warmly, at something she said—and she stopped talking mid-sentence, searching his face carefully before deciding it was safe to smile back . “You clearly know your numbers,” he said. “I just like to be prepared,” she said, straightening papers that didn’t need straightening . “ I don’t like giving people reasons to —” She paused. “I just like to be prepared.” |
| emotion Police officer and confidential informant at an archery range What emotion is Marcus feeling? rage | Marcus pulled the arrow back until the fletching brushed his cheek. His jaw was set so tight the muscles in his temple pulsed visibly beneath the skin. “You told me the warehouse,” he said. The words came out flat, measured , like he was reading from a report. He released. The arrow punched through the target dead center , splitting the shaft of the one already there. Detective Yuen took a small step back despite herself… |
| fear Exes at a pool hall What core fear does Derek demonstrate throughout this scenario? competition | …Two solids dropped, and she watched him circle the table the way he used to circle every room they walked into together, measuring angles, measuring people . “You’ve been practicing,” she said. “Little bit.” He lined up his next shot. She remembered the night she’d beaten him at cards in front of his friends. How quiet the drive home had been. How the silence had its own specific temperature. He sank another ball. Then another. He didn’t celebrate , just moved to the next position… |
| motivation Principal and teacher at an outdoor skating rink What is Principal Dawes’s core motivation? serving others | …Principal Dawes had arrived two hours before the first family showed up, dragging extension cords across the frozen parking lot and arranging the hot cocoa station so the shortest kids could reach the ladle . Now the rink was full, and he moved along its edges like a quiet current. He knelt to tighten a kindergartner’s skate laces without being asked. He guided an elderly grandfather by the elbow across a rough patch of ice. When the music cut out, he jogged to the speaker system himself… |
Table 3: Example implicit vignettes for each question type, highlighting phrases carrying implicit meaning.
| multiple choice accuracy (%) | Open-ended accuracy (%) | Open-ended score | |||||||||||
| Model | Size | Average | Wound | Emotion | Fear | Motivation | Average | Wound | Emotion | Fear | Motivation | EmbSim | LLMScore |
| Llama-3.2-1B | 1B | 2.0 | 3.0 | 1.8 | 1.4 | 1.8 | 37.9 | 32.5 | 37.7 | 39.9 | 41.4 | 0.20 | 1.21 |
| Llama-3.2-3B | 3B | 53.9 | 34.1 | 67.5 | 55.5 | 58.2 | 46.4 | 37.9 | 51.6 | 46.5 | 49.4 | 0.29 | 1.23 |
| Llama-3.1-8B | 8B | 78.2 | 71.0 | 82.5 | 78.8 | 80.6 | 51.1 | 45.2 | 56.0 | 47.9 | 55.0 | 0.35 | 1.66 |
| Llama-3.1-70B | 70B | 88.4 | 85.6 | 87.5 | 90.2 | 90.2 | 55.6 | 54.4 | 63.7 | 51.3 | 53.0 | 0.38 | 1.90 |
| Llama-3.3-70B | 70B | 84.3 | 78.1 | 85.5 | 88.6 | 85.0 | 56.5 | 55.2 | 63.7 | 55.5 | 51.8 | 0.39 | 1.90 |
Table 4: Performance on the implicit ToMStories. Most Llama-3.2-1B outputs do not follow the answer format.
Figure 2: Answer selections on conflicted ToMStories.
Figure 3: Overlap between indicator phrases across models. (a) For each state, the number of open models with at least one phrase overlapping with Claude’s. (b) Per model, the percentage of states with at least one phrase in the same cluster as one of Claude’s. (c) Example for the motivation state being accepted by others . Claude’s highest overlapping phrase overlaps with 9 of the 13 open models (which is a count for nine in panel A).
| Pearson | Regression | % of | ||||
|---|---|---|---|---|---|---|
| Predictor | MC | OE | MC | OE | MC | OE |
| Model size (log params) | .77 ∗∗ | .67 ∗ | .14 | .14 | 26 | 20 |
| Description preference | .80 ∗∗ | .57 | .25 | .29 | 29 | 14 |
| Phrase overlap w/ Claude | .89 ∗∗ | .92 ∗∗ | .60 ∗ | 1.03 ∗∗ | 45 | 66 |
| Full model | .85 | .88 | ||||
Table 5: A comparison of correlation between predictors and implicit ToMStories accuracy, a multiple regression model fit to all three predictors and a dominance analysis of the general dominance of each of these predictors in this model. We exclude Llama-3.2-1B numbers from this analysis as the model failed to answer most questions. ,
Figure 4: Effect of implicit (purple) and explicit (gold) steering relative to unsteered generations from the same prompts. Diamonds show the median paired effect size across models, filled where significant across models (Wilcoxon, BH-corrected). Dots show individual models, filled where significant ( ).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Classification accuracy for linear probes trained to predict the correct answer from the outputs of each layer. We report overall accuracy, as well as split across the implicit and explicit variants of ToMStories with sigmoids fit to the data and vertical lines indicating the medians of the sigmoid curves.
Figure 6: Implicit accuracy against three model properties, across 12 models (Llama-3.2-1B excluded due to many unparseable answers): (a) parameter count, (b) the share of description-aligned answers selected on conflicted ToMStories, and (c) indicator phrase overlap with Claude (panel B in Figure 3 ). Filled markers and solid lines show multiple choice; open markers and dashed lines show open-ended; horizontal lines mark human performance.
| Model | |||
|---|---|---|---|
| deepseek-r1-7b | 20 | 0.616 | 0.303 |
| deepseek-r1-8b | 13 | 0.518 | 0.090 |
| gemma-2-27b | 21 | 0.660 | 0.201 |
| gemma-2-9b | 22 | 0.737 | 0.373 |
| llama-3.1-70b | 31 | 0.656 | 0.238 |
| llama-3.1-8b | 13 | 0.533 | 0.076 |
Table 6: High-dimensional factorization statistics per model and layer ( ) for steering vectors. All metrics are averaged across the four question types. : mean pairwise cosine between all steering vectors; CV : coefficient of variation of steering vector magnitudes.
| Prompt: write a short narrative. Setting: nursery; relationship: friends; emotion: betrayed | |||
|---|---|---|---|
| Model | Unsteered | Mg Steered toward explicit | Mg Steered toward implicit |
| Llama-3.1-8B | Lily’s eyes welled up with tears as she stared at the blocks her friend Emma had built into a magnificent castle. Just yesterday, Emma had promised to let her add a tower to the castle, but now it was gone, replaced by a bright pink flag that read “Emma’s Castle” in bold letters. Lily felt a pang of betrayal. … 247 words | Writing a short narrative based on this scenario is straightforward. Here is a short narrative based on this scenario. A nursery is a place where young children learn and play. It is a place of joy and innocence. However, when a person is feeling betrayed, the joy and innocence are replaced with feelings of hurt and anger. Here is a short narrative based on this scenario. … 430 words | In the dimly lit nursery, Emma sat amidst the rows of cribs, her eyes fixed on the small, unmade bed that seemed to be a miniature version of the one she shared with her best friend, Olivia. The two girls had been inseparable since they were toddlers, and their friendship was a complex web of whispered secrets and stolen glances. … 378 words |
| Qwen-2.5-72B | In the soft glow of the nursery, where the walls were painted with whimsical forest scenes and the air was filled with the scent of lavender, two friends, Mia and Lily, sat among the toys and books. … Today, however, the atmosphere was different. Mia’s eyes were red from crying, … 370 words | Lily and Max were best friends. They were playing with toys in the nursery. Lily was feeling very betrayed by Max because he had told her a huge lie. She looked at Max with tears in her eyes and said, “I can’t believe you lied to me!” Max felt terrible for lying to his friend. He apologised and promised to never do it again. … 73 words | In the quiet of the nursery, the air was filled with the scent of baby powder and the soft hum of a lullaby from a distant room. The nursery was not just for the baby; it had become a sort of neutral ground where the two friends, Mia and Alex, often met. They were not the type to openly discuss their feelings, … 425 words |
| Gemma-2-27B | The rocking horse stood frozen mid-gallop, its painted eyes staring vacantly at the wall. Lily traced a finger along its chipped mane, the wood rough beneath her touch. It used to be their horse, hers and Maya’s. … But Maya wasn’t here anymore. Not since she’d told everyone about Lily’s secret, … 232 words | The pastel walls of the nursery glowed under the warm light of the ceiling lamp. Lily bounced excitedly, pointing at the crib. “Isn’t he cute?” she giggled. I smiled back, my heart swelling with pride. “He’s perfect,” I said, gently rocking the crib. Lily clapped her hands. “Let’s sing him a song!” … 332 words | The nursery was a tableau of faded pastels and forgotten dreams. Dust motes danced in the shafts of sunlight filtering through the grimy window, illuminating the room’s melancholic stillness. Elara, perched on the window seat, was a study in contradictions. Her posture, with its slumped shoulders and downcast gaze, suggested vulnerability, … 374 words |
Table 7: Examples of unsteered and steered generations for the same prompt from three models.
Figure 7: The two interfaces for collecting the human baselines. Part A is for the implicit vignettes, and part B is for the conflicted vignettes. We do not display the full vignette text here for brevity.
| Clustering threshold | |||||
| Measure | 0.40 | 0.45 | 0.50 | 0.55 | 0.60 |
| Clusters | 4,256 | 3,111 | 2,188 | 1,464 | 958 |
| Overlap (all phrases, any state) | |||||
| 1 other model (%) | 79 | 87 | 92 | 96 | 98 |
| 7 other models (%) | 33 | 43 | 55 | 66 | 77 |
| Median other models | 3 | 5 | 8 | 10 | 12 |
Table 8: Robustness of the indicator-phrase clustering results to the clustering threshold; the shaded column (0.50) is used in the main text.
Figure 8: Centroids of implicit and explicit vignettes for each question type, projected onto the first two principal components, at the layer selected by linear probe analysis.
| Cluster label | n | Example indicator phrases |
|---|---|---|
| Avoids eye contact | 722 | avoiding eye contact that lingers slightly; avoiding eye contact with mirrors; Avoiding eye contact to hide nervousness or guilt about feelings; Maintaining a closed posture with crossed arms and avoiding eye contact; avoids eye contact with certain individuals or groups; avoids eye contact by looking around the room nervously; avoiding eye contact with acquaintances; Slouching in financial discussions to avoid eye contact; avoids eye contact and fidgets; avoided eye contact with others, lost in their own thoughts |
| Types of pauses | 290 | pauses to collect own thoughts; Pauses to collect themselves; pauses mid-sentence and clears throat; Pauses excessively; Stops mid-sentence; the way she pauses mid-sentence to collect her thoughts; Pauses before answering questions, as if weighing safety; pausing before answering questions; Pausing for longer than usual; pauses mid-sentence to glance around the room |
| Types of smiles | 222 | smiling too broadly, almost; smiles too brightly; Her smile feels like a thin, brittle thing, barely holding; Their gaze lingers on a photograph, a ghost of a smile touching their lips; Forced smiles that don’t reach the eyes; smirks too brightly; A small, satisfied smile creeps onto their face; smiling tightly to avoid direct eye contact; Their smile falters; forced smile |
| Manners of breathing | 181 | took deep breaths; Inhales through the nose; breath coming in short, shallow gasps; makes a sharp breathing sound; Their breath quickens, taking short, shallow sips of air as excitement mounts; Takes deep, controlled breaths; their chewing slows or their hand stills mid-task for exactly one breath; takes a deep breath and releases it sharply; pauses before speaking, taking a deep breath; taking a long, slow breath |
| Fidgeting with small objects | 165 | begin fidgeting with a ring, watch strap, or sleeve cuff in a repetitive, self-soothing rhythm; Fidgeting with a keychain or other small object in their pocket; Constant fidgeting; Fidgeting with a small object, like a pen cap or a ring, twisting it repeatedly; fidgeting with hands in pockets; A constant, restless fidgeting with a necklace or ring; fidgets nervously, adjusting position frequently; fidgeting with a pen or keychain; Sudden, unexplained fidgeting, particularly with hands or clothing; fidgeting with a pen on a cluttered desk |
| Fidgets with small objects | 160 | fidgets with a small, portable device (e.g. phone, compass); fidgets with pockets; Fidgets with a necklace or ring; Fidgets with a ring or piece of jewelry associated with past failure; fidgets with wallet or purse; fidgets with a button or a small object in their pocket; Fidgets with a small object, like a pen cap or ring; Fidgets with a small object, like a pen or a keychain; fidgets with objects, rearranging them in a precise, calculated pattern; Fidgets with a piece of jewelry |
Table 9: The ten largest indicator phrase clusters, showing the total number of phrases in the cluster and ten randomly sampled distinct phrases from the cluster (after cleaning and parsing)
| Model | Mg Wound : being fired or laid off | Mg Emotion : guilt | Mg Fear : crowds | Mg Motivation : being accepted by others |
|---|---|---|---|---|
| Claude Sonnet 4.6 | Keeps their work badge clipped to their jacket even on weekends, touching it reflexively before remembering | Volunteers for the most unpleasant tasks before anyone asks, almost punishingly so | They position themselves with their back against a wall or pillar before doing anything else in the space | mirrors the posture and hand gestures of whoever they’re speaking with, shifting again each time they move to a new person in the group |
| Llama-3.2-1B | shaking hands quickly when asked about the company | nervously fidgeting with hands | twitching nose | smiling wryly when someone compliments their work |
| Llama-3.2-3B | Fidgets with papers on desk | fidgeting with loose change | twitching nose in crowded spaces | fidgeting with a business card in their pocket |
| Llama-3.1-8B | Fidgets with the worn label on a old button-down shirt | fidgets with the hem of their shirt | Fidgeting with the hem of their shirt in public places | adjusting her outfit before meeting new people |
| Llama-3.1-70B | Avoids checking email or phone for messages from former employer | avoiding direct eye contact with a specific person or object | Avoids eye contact with strangers on public transportation | Mirrors the posture and gestures of those around them |
| Llama-3.3-70B | avoids eye contact during conversations about job security | avoids eye contact with specific individuals | avoids eye contact in public | mirrors others’ mannerisms |
Table 10: The first indicator phrase each model generated for example states of each type (after cleaning and parsing).
More in this topic
Theory of Mind
Theory of Mind studies how people and AI infer others’ beliefs, goals, and knowledge to predict behavior and coordinate interaction.
See all 92 papers