Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions
Authors: Ning E, Changde Du, Yizhuo Lu, Huiguang He
Organizations: School of Biomedical Engineering, ShanghaiTech University, Shanghai, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Multimodal Large Language Models (MLLMs) have achieved impressive performance across a wide range of reasoning tasks, yet it remains unclear whether their internal representations and decision criteria align with the latent structure of human cognition. In this work, we propose a novel framework to align MLLMs with the multidimensional mental representations underlying human similarity judgements. We build on a previously established behavior-derived semantic embedding space, in which objects are represented along 66 interpretable dimensions derived from 4.70 million human Odd-One-Out (O1O) judgements. For each triplet, we infer the most salient decision dimension and convert it into a dimension-grounded linguistic rationale. This enables MLLMs to learn both the final choice and a justification consistent with the inferred behavioral criterion. Experimental results demonstrate that our approach improves model consistency with human similarity judgements while retaining competitive performance on standard multimodal benchmarks. Furthermore, using searchlight Representational Similarity Analysis (RSA) on an independent fMRI Natural Scenes Dataset (NSD), we observe increased representational alignment between model and human. Ultimately, our findings point to a data-driven route for incorporating human cognitive structure into MLLMs alignment. The code is available here.
Figures & tables
Figure 1: Overview of triplet-level semantic dimension attribution. (a) Human similarity judgements are collected via crowdsourcing. Then learning a representational embedding space via SPoSE yields the multidimensional mental representations of natural objects. (b) For one triplet, we compute the original softmax probability based on the entire representational embedding for all pairs. We then iteratively prune each dimension from the representational embedding space and compute the changes in softmax probabilities relative to those obtained with the full embedding as the relevance score for that dimension. By sorting these relevance scores in descending order, we establish a hierarchical ranking of dimensional importance. The top-1 dimension yielding the maximum probability change is regarded as the most salient semantic dimension driving the human decision-making.
Figure 2: Pipeline for dimension-grounded reasoning enhancement. This pipeline illustrates the process of reasoning enhancement via LLM. For one triplet with their textual descriptions and the most salient semantic dimension d∗ driving human decision-making, we integrate them into the structured prompt Prompt(T,cap(T),d∗) . LLM is then utilized exclusively to enhance reasoning. By injecting the dimension d∗ , we guide the LLM to perform targeted reasoning enhancement. This effectively converts discrete cognitive criteria for decision-making into explicit CoT rationale r(d∗) .
Figure 3: (a) Overview of hybrid dataset. The total dataset includes three subsets: Single-image (126K samples), Multi-image (102K samples), and Cognitive-Alignment (80K samples). (b) Examples. The examples include Spot the Difference, Image Edit Instruction, Visual Storytelling, LLaVAR, COCO Caption, and Odd-One-Out tasks.
Model
MMMU val (%)
MMBench en,test (%)
Qwen3-VL
Baseline
CogAlign
Baseline
CogAlign
4B
48.44
51.56
82.96
81.39
8B
50.67
54.22
83.91
83.86
30B-A3B
53.22
57.56
87.44
83.86
Table 1: Performance comparison of baseline and aligned models. CogAlign yields consistent accuracy improvements on MMMUval , while incurring a slight performance degradation on MMBenchen, test across all model scales. Details are provided in Table 9 (Appendix H.3 ).
Model
Baseline (%)
CogAlign (%)
Centaur
70B
51.96
—
Direct
CoT
Ours
Qwen3-VL
4B
41.48 ± 0.30
53.37 ± 0.30
56.81 ± 0.29
8B
40.46 ± 0.29
52.28 ± 0.30
56.99 ± 0.29
30B-A3B
37.09 ± 0.28
55.38 ± 0.30
56.21 ± 0.29
Noise Ceiling
68.74 ± 1.07
Table 2: Human consistency evaluation across baseline and aligned models. The baseline models are evaluated under two inference paradigms: Direct (predicting option immediately) and CoT (thinking step-by-step prior to decision-making). The proposed alignment framework consistently outperforms both the Direct and the more capable CoT baselines across all model scales (noise ceiling is 68.74 ( Hebart et al., 2023 ) ).
Figure 4: Cortical projection of representational alignment improvements across subjects. The heatmaps visualize the voxel-wise difference in Δ~RSA(c) scores between the aligned and baseline Qwen3-VL-4B-Instruct model. Darker colors indicate better representational alignment between models and human brain. Consistent positive shifts are observed, demonstrating that our alignment framework robustly shifts the MLLMs internal representational geometry to be more human-like.
Figure 5: Stability of alignment shifts across different extraction depths. The cortical projections display the alignment improvements Δ~RSA(c) of subject 1 computed using features from the last layer, the second-to-last layer, and the concatenation of the last four layers. Darker regions reflect better alignment improvement. The spatial distribution of these positive shifts remains consistent regardless of the extraction depth, confirming that the achieved representational alignment with human cognition is robust and not a selective layer choice.
Model
S-CoT
d∗ -CoT
FS-CoT
CogAlign
Qwen3-VL
4B
53.37
49.01
52.70
56.81
Table 3: Prompting versus alignment. We compare three inference-time prompting strategies (S-CoT, d∗ -CoT, and FS-CoT) with our alignment framework. Notably, providing semantic dimensions in the prompt during inference time negatively impacts model performance, whereas our CogAlign achieves the best overall results. The chance level is 33.33%. See Appendix H.4 for details.
Model
Baseline (CoT)
CogAlign (Ours)
Qwen2.5-VL
7B
56.63
59.29
Gemma3
12B
55.77
57.28
Table 4: Ablation on architectural generalizability.
Figure 6: Comparison of choice-based RSMs between Qwen2.5-VL-7B and human.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Generator
D
F
L
N
Qwen2.5-72B-Instruct (Ours)
93.37
73.11
70.45
54.73
DeepSeek-V4-Flash
96.21
81.06
69.89
62.12
Appendix
Table 5: Rationale quality across generators.
Task
Dataset
Samples
Ratio (%)
Single-image Scenarios
Caption (13.5K)
COCO
13,501
4.36
ALLaVA-4V (69.98K)
ALLaVA-VFLAN
19,990
6.46
ALLaVA-LAION
49,990
16.16
LLaVAR (43.17K)
LLaVAR-GPT4
43,167
13.96
Multi-image Scenarios
Appendix
Table 6: Hybrid dataset composition.
Table 13
Qwen3-VL
Model
4B
8B
30B-A3B
Baseline
CogAlign
Baseline
CogAlign
Baseline
CogAlign
A. Overall
82.96
81.39
83.91
83.86
87.44
83.86
B. General Capabilities
AR
82.99
85.76
80.90
87.85
87.15
87.50
CP
87.58
85.44
89.08
87.15
88.65
86.30
Appendix
Table 9: Detailed performance comparison of MMBench (en,test).
Metric
Baseline (CoT)
Dimension-only
CogAlign (Ours)
Accuracy
53.37
42.81
56.81
Choice-based RSA
81.79
44.01
85.01
Appendix
Table 10: Ablation on supervision format during fine-tuning Qwen3-VL-4B-Instruct.
Model
Baseline (CoT)
Generic CoT
Choice-only
CogAlign (ours)
Qwen2.5-VL-7B-Instruct
56.63
—
56.21
59.29
Qwen3-VL-4B-Instruct
53.37
47.57
—
56.81
Appendix
Table 11: Comparison of Choice-only, Generic CoT, and ours supervision signals.
Axis
Semantic Concept
Axis
Semantic Concept
0
metallic or artificial
1
food-related
2
animal-related
3
textile
4
plant-related
5
house-related or furnishing-related
6
valuable or precious
7
transportation-related or movement-related
8
body-related or people-related
9
wood-related or brown
10
electronics or technology
11
colorful or playful
Appendix
Table 12: Overview of 66 semantic dimension concepts.
The human ability to translate diverse perceptual and linguistic inputs into structured behavior has been thought to rest on learning robust representations of concepts. The rapid advancement of transformer-based large language models (LLMs) has surfaced a diversity of computational ingredients relevant for model building - architectures, fine-tuning methods, and training datasets among others - yet it remains unclear which are most crucial for developing human-like conceptual representations. Further, most current benchmarks are ill-suited to measuring representational alignment, making LLMs' scores on them unreliable for assessing whether they are progressing as cognitive models. We address these limitations by evaluating over 75 models on a triplet similarity task, a method well established in cognitive science for measuring conceptual representations, using concepts from the THINGS database. We find that instruction fine-tuning and larger attention head dimensionality are among the strongest predictors of human alignment, while activation function choice, multimodal pretraining, and parameter size have limited influence on alignment. Correlations between alignment scores and existing benchmark scores reveal that while some benchmarks (e.g., BigBenchHard) better capture representational alignment than others (e.g., MUSR), none fully accounts for the variance in human-model alignment, demonstrating their insufficiency. Taken together, our findings highlight key computational ingredients for advancing LLMs as models of human conceptual representation and address a key gap in LLM evaluation.
Zach Studdiford, Timothy T. Rogers, Kushin Mukherjee +1
University of Wisconsin–Madison · Stanford University
Multimodal Large Language Models (MLLMs) are increasingly deployed in human-facing roles where personality perception is critical, yet existing benchmarks evaluate this capability solely on numerical Big Five score prediction, leaving open whether models truly perceive personality through behavioral understanding or merely prejudge through superficial pattern matching. We address this gap with three contributions. (i) A new task: we formalize Grounded Personality Reasoning (GPR), which requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding. (ii) A new dataset: we release MM-OCEAN (1,104 videos, 5,320 MCQs), produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs. (iii) Benchmark and analysis: we design a three-tier evaluation (rating, reasoning, grounding) plus four sample-level failure-mode metrics: Prejudice Rate (PR), Confabulation Rate (CR), Integration-failure Rate (IR), and Holistic-grounding Rate (HR), and benchmark 27 MLLMs (13 closed, 14 open). The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%. These findings expose a disconnect between getting the right score and reasoning for the right reason, charting a roadmap for grounded social cognition in MLLMs.
Caixin Kang, Tianyu Yan, Sitong Gong +8
1The University of Tokyo · 2Shanda AI Research Tokyo · 3Dalian University of Technology
Visual search is a fundamental cognitive ability. This study investigates whether Multimodal Large Language Models (MLLMs) exhibit human-like difficulty signatures in visual search tasks. We compared search performance of humans (n = 1,250) and MLLMs using identical 2D and 3D stimuli across different set sizes. Both groups showed efficient performance in feature searches, most clearly when the target had a unique color, but performance degradation in conjunction searches as set sizes increased. Additionally, we found strong correlations between human and MLLM error rates (ρ=0.82), which suggests that MLLMs are sensitive to similar objective complexities, such as stimulus heterogeneity. However, differences were found as well: whereas humans invested extra search time to respond accurately on target-absent trials, MLLMs exhibited extreme present/absent response biases in complex searches. We conclude that MLLMs replicate high-level human performance signatures, yet their underlying computations differ significantly.
Renchi Zhang, Joost C. F. de Winter, Dimitra Dodou +2
Faculty of Mechanical Engineering, Delft University of Technology, Delft, The Netherlands