Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies

Yiran Huang; Zeyong Zhang; Bharat Biswal; Xin Di; Mengjia Xu

cs.AIOct 15, 2026

Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies

Authors: Yiran Huang, Zeyong Zhang, Bharat Biswal, Xin Di, Mengjia Xu

Organizations: Department of Data Science, New Jersey Institute of Technology, Newark, NJ 07102, USA · Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA

Abstract

Multimodal large language models (MLLMs) offer scalable annotations of social and affective content, but their temporal agreement with human judgments and neural predictive value require further evaluation. We compared diffusion-based and autoregressive MLLMs with continuous human Theory of Mind (ToM) and affective valence ratings across three short cartoon movies. Behavioral analyses assessed temporally aligned Pearson correlations, rating distributions, and the effects of preceding scene descriptions. Ratings from each source were separately used in subject-specific regression models to predict regional fMRI responses. Model ratings captured aspects of human temporal variation despite systematic distributional differences, with generally stronger alignment for valence than ToM. Preceding context improved alignment in some conditions, depending on the model and movie. In The Present, more regions showed significant encoding for MLLM ratings than for human ratings, whereas effects were more limited in the other movies. Spatial overlap across rating sources occurred mainly among MLLMs; overlap between human and model ratings was limited and involved primarily visual, dorsal-attention, and posterior default-mode regions. These findings support MLLMs as scalable tools for generating brain-predictive annotations of social and emotional content, while highlighting systematic rating differences and model- and stimulus-dependent alignment with human judgments.

Figures & tables

Figure 1: Overview of the brain-alignment framework for evaluating multimodal LLMs (MLLMs) on social and emotional understanding of cartoon movies. Detailed prompts are available at https://github.com/Yiran-H/functional-MRI-Movie-MLLMs

		LLaDA-V		Qwen3-VL		Gemma 3
Measure	Movie	$r_{0}$	$r^{*}$ (lag)	$r_{0}$	$r^{*}$ (lag)	$r_{0}$	$r^{*}$ (lag)
ToM	TP	.418	.418 (0.0)	.063	.063 (0.0)	.292	.331 (1.6)
	DM	.034	.054 (2.4)	.213	.252 (3.2)	$-$ .078	$-$ .051 (6.4)
	PC	.366	.428 (4.0)	.523	.609 (4.0)	.240	.277 (4.0)
Valence	TP	.548	.554 (0.8)	.401	.411 (0.8)	.370	.393 (1.6)
	DM	.416	.423 (4.0)	.641	.662 (4.8)	.615	.639 (4.8)

Table 1: Human–model Pearson correlations without temporal shift (

r_{0}

) and at the lag maximizing correlation (

r^{*}

, exploratory). Lags (in seconds, in parentheses) indicate model ratings leading human ratings.

Figure 2: Human–model and inter-model agreement in ToM (a) and valence (b) ratings across three animated movies. Each cell shows the Pearson correlation coefficient between the row and column judges (Human, LLaDA-V, Qwen3-VL, Gemma 3). Human–model values are lag-optimized correlations ( $r^{*}$ , exploratory; see Table 1 ); inter-model values are unshifted. Diagonal values reflect self-correlation. Colors are centered at $r=0$ , with dark red/dark blue denoting positive/negative correlations.

Figure 3: Rating distributions and human–model agreement. (a) Distributions of continuous, frame-wise ToM (left) and valence (right) ratings across three movies for human raters and three MLLMs. (b) Human–model Pearson correlations ( $r^{*}$ ) for ToM ( $x$ -axis) and valence ( $y$ -axis). Colors denote models, shapes denote movies, and stars indicate model means. The solid line shows the least-squares fit.

Figure 4: Effect of temporal context on human-model alignment. Pearson $r^{*}$ between human and model ratings as a function of the length of narrative context (previous scene descriptions) injected into the rating prompt, from no context (“base”) to 16 seconds of lookback, shown separately for each movie and measure.

Figure 5: Mean Pearson $r$ between predicted and held-out fMRI responses per ROI for human ratings and three models. Rows show three movies; columns show ToM and valence. ROIs are grouped by canonical cortical networks and AAL subcortical regions. Black dots indicate FDR-significant ROIs ( $q<0.05$ ).

Figure 6: Spatial overlap of FDR-significant encoding across rating sources. Glass-brain projections were generated using Nilearn [ 1 ] , showing the number of rating sources (Human, LLaDA-V, Qwen3-VL, and Gemma 3) with significant encoding performance (FDR $q<0.05$ ) in each of the 100 Schaefer cortical ROIs. White indicates no significant sources; progressively darker yellow-to-orange shades indicate 1–4 significant sources (14 AAL subcortical ROIs were excluded). Black dots mark the centroids of ROIs significant for human ratings. ROIs significant for at least one MLLM but not for human ratings reflect model-only encoding; counts of two or more indicate overlap across sources.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,096 papers

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?
May 21, 2026Caixin Kang, Tianyu Yan, Sitong Gong +8LLM Grounding Personality Modeling in Language Models
1The University of Tokyo · 2Shanda AI Research Tokyo · 3Dalian University of Technology
Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation
May 1, 2026Prabal Shrestha, Bohan Jiang, Haoning Xue +2Multimodal Large Language Models Language Model Bias Evaluation
Department of Computer Science, Boise State University, Boise, ID, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA · Department of Communication, University of Utah, Salt Lake City, UT, USA
How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans
Aug 10, 2026Hasan Mahmud, Khawaja Abaid Ullah, Mohammad Javad Khojasteh +2Human Preference Evaluation Gender Bias in Language Models
School of Communication, Rochester Institute of Technology · Department of Electrical and Microelectronic Engineering, Kate Gleason College of Engineering, Rochester Institute of Technology Rochester, NY 14623, USA

May 21, 2026J/K move · Enter open · S save

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

Caixin Kang, Tianyu Yan, Sitong Gong, Mingfang Zhang+7

1The University of Tokyo · 2Shanda AI Research Tokyo · 3Dalian University of Technology

LLM Grounding Personality Modeling in Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) are increasingly deployed in human-facing roles where personality perception is critical, yet existing benchmarks evaluate this capability solely on numerical Big Five score prediction, leaving open whether models truly perceive personality through behavioral understanding or merely prejudge through superficial pattern matching. We address this gap with three contributions. (i) A new task: we formalize Grounded Personality Reasoning (GPR), which requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding. (ii) A new dataset: we release MM-OCEAN (1,104 videos, 5,320 MCQs), produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs. (iii) Benchmark and analysis: we design a three-tier evaluation (rating, reasoning, grounding) plus four sample-level failure-mode metrics: Prejudice Rate (PR), Confabulation Rate (CR), Integration-failure Rate (IR), and Holistic-grounding Rate (HR), and benchmark 27 MLLMs (13 closed, 14 open). The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%. These findings expose a disconnect between getting the right score and reasoning for the right reason, charting a roadmap for grounded social cognition in MLLMs.

Abstract

Multimodal Large Language Models

Explore similar work

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation

How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans