Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions
Authors: Ning E, Changde Du, Yizhuo Lu, Huiguang He
Organizations: School of Biomedical Engineering, ShanghaiTech University, Shanghai, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Multimodal Large Language Models (MLLMs) have achieved impressive performance across a wide range of reasoning tasks, yet it remains unclear whether their internal representations and decision criteria align with the latent structure of human cognition. In this work, we propose a novel framework to align MLLMs with the multidimensional mental representations underlying human similarity judgements. We build on a previously established behavior-derived semantic embedding space, in which objects are represented along 66 interpretable dimensions derived from 4.70 million human Odd-One-Out (O1O) judgements. For each triplet, we infer the most salient decision dimension and convert it into a dimension-grounded linguistic rationale. This enables MLLMs to learn both the final choice and a justification consistent with the inferred behavioral criterion. Experimental results demonstrate that our approach improves model consistency with human similarity judgements while retaining competitive performance on standard multimodal benchmarks. Furthermore, using searchlight Representational Similarity Analysis (RSA) on an independent fMRI Natural Scenes Dataset (NSD), we observe increased representational alignment between model and human. Ultimately, our findings point to a data-driven route for incorporating human cognitive structure into MLLMs alignment. The code is available here.
Figures & tables
Figure 1: Overview of triplet-level semantic dimension attribution. (a) Human similarity judgements are collected via crowdsourcing. Then learning a representational embedding space via SPoSE yields the multidimensional mental representations of natural objects. (b) For one triplet, we compute the original softmax probability based on the entire representational embedding for all pairs. We then iteratively prune each dimension from the representational embedding space and compute the changes in softmax probabilities relative to those obtained with the full embedding as the relevance score for that dimension. By sorting these relevance scores in descending order, we establish a hierarchical ranking of dimensional importance. The top-1 dimension yielding the maximum probability change is regarded as the most salient semantic dimension driving the human decision-making.
Figure 2: Pipeline for dimension-grounded reasoning enhancement. This pipeline illustrates the process of reasoning enhancement via LLM. For one triplet with their textual descriptions and the most salient semantic dimension d∗ driving human decision-making, we integrate them into the structured prompt Prompt(T,cap(T),d∗) . LLM is then utilized exclusively to enhance reasoning. By injecting the dimension d∗ , we guide the LLM to perform targeted reasoning enhancement. This effectively converts discrete cognitive criteria for decision-making into explicit CoT rationale r(d∗) .
Figure 3: (a) Overview of hybrid dataset. The total dataset includes three subsets: Single-image (126K samples), Multi-image (102K samples), and Cognitive-Alignment (80K samples). (b) Examples. The examples include Spot the Difference, Image Edit Instruction, Visual Storytelling, LLaVAR, COCO Caption, and Odd-One-Out tasks.
Model
MMMU val (%)
MMBench en,test (%)
Qwen3-VL
Baseline
CogAlign
Baseline
CogAlign
4B
48.44
51.56
82.96
81.39
8B
50.67
54.22
83.91
83.86
30B-A3B
53.22
57.56
87.44
83.86
Table 1: Performance comparison of baseline and aligned models. CogAlign yields consistent accuracy improvements on MMMUval , while incurring a slight performance degradation on MMBenchen, test across all model scales. Details are provided in Table 9 (Appendix H.3 ).
Model
Baseline (%)
CogAlign (%)
Centaur
70B
51.96
—
Direct
CoT
Ours
Qwen3-VL
4B
41.48 ± 0.30
53.37 ± 0.30
56.81 ± 0.29
8B
40.46 ± 0.29
52.28 ± 0.30
56.99 ± 0.29
30B-A3B
37.09 ± 0.28
55.38 ± 0.30
56.21 ± 0.29
Noise Ceiling
68.74 ± 1.07
Table 2: Human consistency evaluation across baseline and aligned models. The baseline models are evaluated under two inference paradigms: Direct (predicting option immediately) and CoT (thinking step-by-step prior to decision-making). The proposed alignment framework consistently outperforms both the Direct and the more capable CoT baselines across all model scales (noise ceiling is 68.74 ( Hebart et al., 2023 ) ).
Figure 4: Cortical projection of representational alignment improvements across subjects. The heatmaps visualize the voxel-wise difference in Δ~RSA(c) scores between the aligned and baseline Qwen3-VL-4B-Instruct model. Darker colors indicate better representational alignment between models and human brain. Consistent positive shifts are observed, demonstrating that our alignment framework robustly shifts the MLLMs internal representational geometry to be more human-like.
Figure 5: Stability of alignment shifts across different extraction depths. The cortical projections display the alignment improvements Δ~RSA(c) of subject 1 computed using features from the last layer, the second-to-last layer, and the concatenation of the last four layers. Darker regions reflect better alignment improvement. The spatial distribution of these positive shifts remains consistent regardless of the extraction depth, confirming that the achieved representational alignment with human cognition is robust and not a selective layer choice.
Model
S-CoT
d∗ -CoT
FS-CoT
CogAlign
Qwen3-VL
4B
53.37
49.01
52.70
56.81
Table 3: Prompting versus alignment. We compare three inference-time prompting strategies (S-CoT, d∗ -CoT, and FS-CoT) with our alignment framework. Notably, providing semantic dimensions in the prompt during inference time negatively impacts model performance, whereas our CogAlign achieves the best overall results. The chance level is 33.33%. See Appendix H.4 for details.
Model
Baseline (CoT)
CogAlign (Ours)
Qwen2.5-VL
7B
56.63
59.29
Gemma3
12B
55.77
57.28
Table 4: Ablation on architectural generalizability.
Figure 6: Comparison of choice-based RSMs between Qwen2.5-VL-7B and human.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Generator
D
F
L
N
Qwen2.5-72B-Instruct (Ours)
93.37
73.11
70.45
54.73
DeepSeek-V4-Flash
96.21
81.06
69.89
62.12
Appendix
Table 5: Rationale quality across generators.
Task
Dataset
Samples
Ratio (%)
Single-image Scenarios
Caption (13.5K)
COCO
13,501
4.36
ALLaVA-4V (69.98K)
ALLaVA-VFLAN
19,990
6.46
ALLaVA-LAION
49,990
16.16
LLaVAR (43.17K)
LLaVAR-GPT4
43,167
13.96
Multi-image Scenarios
Appendix
Table 6: Hybrid dataset composition.
Table 13
Qwen3-VL
Model
4B
8B
30B-A3B
Baseline
CogAlign
Baseline
CogAlign
Baseline
CogAlign
A. Overall
82.96
81.39
83.91
83.86
87.44
83.86
B. General Capabilities
AR
82.99
85.76
80.90
87.85
87.15
87.50
CP
87.58
85.44
89.08
87.15
88.65
86.30
Appendix
Table 9: Detailed performance comparison of MMBench (en,test).
Metric
Baseline (CoT)
Dimension-only
CogAlign (Ours)
Accuracy
53.37
42.81
56.81
Choice-based RSA
81.79
44.01
85.01
Appendix
Table 10: Ablation on supervision format during fine-tuning Qwen3-VL-4B-Instruct.
Model
Baseline (CoT)
Generic CoT
Choice-only
CogAlign (ours)
Qwen2.5-VL-7B-Instruct
56.63
—
56.21
59.29
Qwen3-VL-4B-Instruct
53.37
47.57
—
56.81
Appendix
Table 11: Comparison of Choice-only, Generic CoT, and ours supervision signals.
Axis
Semantic Concept
Axis
Semantic Concept
0
metallic or artificial
1
food-related
2
animal-related
3
textile
4
plant-related
5
house-related or furnishing-related
6
valuable or precious
7
transportation-related or movement-related
8
body-related or people-related
9
wood-related or brown
10
electronics or technology
11
colorful or playful
Appendix
Table 12: Overview of 66 semantic dimension concepts.