Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies

Yiran Huang; Zeyong Zhang; Bharat Biswal; Xin Di; Mengjia Xu

cs.AIOct 15, 2026

Brain-Aligned Multimodal LLMs for Social and Emotional Understanding of Cartoon Movies

Authors: Yiran Huang, Zeyong Zhang, Bharat Biswal, Xin Di, Mengjia Xu

Organizations: Department of Data Science, New Jersey Institute of Technology, Newark, NJ 07102, USA · Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA

Abstract

Multimodal large language models (MLLMs) offer scalable annotations of social and affective content, but their temporal agreement with human judgments and neural predictive value require further evaluation. We compared diffusion-based and autoregressive MLLMs with continuous human Theory of Mind (ToM) and affective valence ratings across three short cartoon movies. Behavioral analyses assessed temporally aligned Pearson correlations, rating distributions, and the effects of preceding scene descriptions. Ratings from each source were separately used in subject-specific regression models to predict regional fMRI responses. Model ratings captured aspects of human temporal variation despite systematic distributional differences, with generally stronger alignment for valence than ToM. Preceding context improved alignment in some conditions, depending on the model and movie. In The Present, more regions showed significant encoding for MLLM ratings than for human ratings, whereas effects were more limited in the other movies. Spatial overlap across rating sources occurred mainly among MLLMs; overlap between human and model ratings was limited and involved primarily visual, dorsal-attention, and posterior default-mode regions. These findings support MLLMs as scalable tools for generating brain-predictive annotations of social and emotional content, while highlighting systematic rating differences and model- and stimulus-dependent alignment with human judgments.

Figures & tables

Figure 1: Overview of the brain-alignment framework for evaluating multimodal LLMs (MLLMs) on social and emotional understanding of cartoon movies. Detailed prompts are available at https://github.com/Yiran-H/functional-MRI-Movie-MLLMs

		LLaDA-V		Qwen3-VL		Gemma 3
Measure	Movie	$r_{0}$	$r^{*}$ (lag)	$r_{0}$	$r^{*}$ (lag)	$r_{0}$	$r^{*}$ (lag)
ToM	TP	.418	.418 (0.0)	.063	.063 (0.0)	.292	.331 (1.6)
	DM	.034	.054 (2.4)	.213	.252 (3.2)	$-$ .078	$-$ .051 (6.4)
	PC	.366	.428 (4.0)	.523	.609 (4.0)	.240	.277 (4.0)
Valence	TP	.548	.554 (0.8)	.401	.411 (0.8)	.370	.393 (1.6)
	DM	.416	.423 (4.0)	.641	.662 (4.8)	.615	.639 (4.8)

Table 1: Human–model Pearson correlations without temporal shift (

r_{0}

) and at the lag maximizing correlation (

r^{*}

, exploratory). Lags (in seconds, in parentheses) indicate model ratings leading human ratings.

Figure 2: Human–model and inter-model agreement in ToM (a) and valence (b) ratings across three animated movies. Each cell shows the Pearson correlation coefficient between the row and column judges (Human, LLaDA-V, Qwen3-VL, Gemma 3). Human–model values are lag-optimized correlations ( $r^{*}$ , exploratory; see Table 1 ); inter-model values are unshifted. Diagonal values reflect self-correlation. Colors are centered at $r=0$ , with dark red/dark blue denoting positive/negative correlations.

Figure 3: Rating distributions and human–model agreement. (a) Distributions of continuous, frame-wise ToM (left) and valence (right) ratings across three movies for human raters and three MLLMs. (b) Human–model Pearson correlations ( $r^{*}$ ) for ToM ( $x$ -axis) and valence ( $y$ -axis). Colors denote models, shapes denote movies, and stars indicate model means. The solid line shows the least-squares fit.

Figure 4: Effect of temporal context on human-model alignment. Pearson $r^{*}$ between human and model ratings as a function of the length of narrative context (previous scene descriptions) injected into the rating prompt, from no context (“base”) to 16 seconds of lookback, shown separately for each movie and measure.

Figure 5: Mean Pearson $r$ between predicted and held-out fMRI responses per ROI for human ratings and three models. Rows show three movies; columns show ToM and valence. ROIs are grouped by canonical cortical networks and AAL subcortical regions. Black dots indicate FDR-significant ROIs ( $q<0.05$ ).

Figure 6: Spatial overlap of FDR-significant encoding across rating sources. Glass-brain projections were generated using Nilearn [ 1 ] , showing the number of rating sources (Human, LLaDA-V, Qwen3-VL, and Gemma 3) with significant encoding performance (FDR $q<0.05$ ) in each of the 100 Schaefer cortical ROIs. White indicates no significant sources; progressively darker yellow-to-orange shades indicate 1–4 significant sources (14 AAL subcortical ROIs were excluded). Black dots mark the centroids of ROIs significant for human ratings. ROIs significant for at least one MLLM but not for human ratings reflect model-only encoding; counts of two or more indicate overlap across sources.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,096 papers

May 21, 2026

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

Multimodal Large Language Models (MLLMs) are increasingly deployed in human-facing roles where personality perception is critical, yet existing benchmarks evaluate this capability solely on numerical Big Five score prediction, leaving open whether models truly perceive personality through behavioral understanding or merely prejudge through superficial pattern matching. We address this gap with three contributions. (i) A new task: we formalize Grounded Personality Reasoning (GPR), which requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding. (ii) A new dataset: we release MM-OCEAN (1,104 videos, 5,320 MCQs), produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs. (iii) Benchmark and analysis: we design a three-tier evaluation (rating, reasoning, grounding) plus four sample-level failure-mode metrics: Prejudice Rate (PR), Confabulation Rate (CR), Integration-failure Rate (IR), and Holistic-grounding Rate (HR), and benchmark 27 MLLMs (13 closed, 14 open). The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%. These findings expose a disconnect between getting the right score and reasoning for the right reason, charting a roadmap for grounded social cognition in MLLMs.

Caixin Kang, Tianyu Yan, Sitong Gong +8

1The University of Tokyo · 2Shanda AI Research Tokyo · 3Dalian University of Technology

LLM Grounding Personality Modeling in Language Models Multimodal Model Evaluation

Ask PDF

May 1, 2026

Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation

Multimodal large language models (MLLMs) have shown strong performance on objective tasks such as video understanding and reasoning. However, it remains unclear whether they can approximate subjective human responses, which depend not only on content comprehension but also on individuals' social contexts. To address this gap, we evaluate MLLMs as synthetic participants in an emerging task: assessing perceived sensory engagement with short videos. Grounded in the Perceived Message Sensation Value (PMSV) framework, we compare ratings from recruited human participants and profile-conditioned MLLM simulations (n=673) using a 17-item scale measuring emotional arousal, dramatic impact, and novelty. We find that even leading MLLMs (Gemini 3 Flash and Qwen 3 Omni) show limited agreement with human participants. The models exhibit distinct downward mean-shift and central-tendency biases in their rating distributions. They both introduce and flatten subgroup differences, while showing inconsistent sensitivity to participant profiles. Prompting strategies affect these metrics differently, modestly improving some aspects while worsening others. These results highlight both the challenges and opportunities of developing MLLMs as synthetic participants in video-based research. Data and code: https://github.com/MINDLab25/mllm-human-simulation-eval

Prabal Shrestha, Bohan Jiang, Haoning Xue +2

Department of Computer Science, Boise State University, Boise, ID, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA · Department of Communication, University of Utah, Salt Lake City, UT, USA

Multimodal Large Language Models Language Model Bias Evaluation Multimodal Model Evaluation

Ask PDF

Aug 10, 2026

How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans

Large language models (LLMs) are increasingly used to perform subjective evaluations traditionally made by humans, yet their validity as social judges remains unclear. This paper examines whether LLMs can assess social attraction from theory-grounded persona profiles constructed from ten psychological and relational constructs and organized into three tiers: socially attractive, socially mixed, and socially unattractive. We examine LLM ratings in two studies and compare them with human judgments in a third study. In Study 1, 34 LLMs rated 12 profiles across three repeated runs. Although some models tended to give higher or lower ratings overall, they showed strong stability across runs, consistent three-tier ordering, and high agreement in relative profile ordering. Study 2 examined sensitivity to gender presentation using six matched name-and-pronoun profile pairs and a separate pronoun-only test with a gender-neutral name, finding no significant effects in either analysis. In Study 3, 198 human participants evaluated the six matched profiles from Study 2. Their ratings reproduced the three-tier structure and followed a profile ordering consistent with that of the LLMs. However, LLMs rated attractive profiles more positively and unattractive profiles more negatively than humans, while neither group showed a significant overall effect of gender presentation.

Hasan Mahmud, Khawaja Abaid Ullah, Mohammad Javad Khojasteh +2

School of Communication, Rochester Institute of Technology · Department of Electrical and Microelectronic Engineering, Kate Gleason College of Engineering, Rochester Institute of Technology Rochester, NY 14623, USA

Human Preference Evaluation Gender Bias in Language Models LLM Evaluation

Ask PDF

Abstract

Multimodal Large Language Models

Explore similar work

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation

How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans