cs.CLOct 30, 2026

Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions

Authors: Ning E, Changde Du, Yizhuo Lu, Huiguang He

Organizations: School of Biomedical Engineering, ShanghaiTech University, Shanghai, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China · School of Future Technology, University of Chinese Academy of Sciences, Beijing, China · 

Abstract

Multimodal Large Language Models (MLLMs) have achieved impressive performance across a wide range of reasoning tasks, yet it remains unclear whether their internal representations and decision criteria align with the latent structure of human cognition. In this work, we propose a novel framework to align MLLMs with the multidimensional mental representations underlying human similarity judgements. We build on a previously established behavior-derived semantic embedding space, in which objects are represented along 66 interpretable dimensions derived from 4.70 million human Odd-One-Out (O1O) judgements. For each triplet, we infer the most salient decision dimension and convert it into a dimension-grounded linguistic rationale. This enables MLLMs to learn both the final choice and a justification consistent with the inferred behavioral criterion. Experimental results demonstrate that our approach improves model consistency with human similarity judgements while retaining competitive performance on standard multimodal benchmarks. Furthermore, using searchlight Representational Similarity Analysis (RSA) on an independent fMRI Natural Scenes Dataset (NSD), we observe increased representational alignment between model and human. Ultimately, our findings point to a data-driven route for incorporating human cognitive structure into MLLMs alignment. The code is available here.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Uncovering the Computational Ingredients of Human-Like Representations in LLMs

    Oct 1, 2025Zach Studdiford, Timothy T. Rogers, Kushin Mukherjee +1LLM EvaluationLLM Fine-Tuning

  2. Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

    May 21, 2026Caixin Kang, Tianyu Yan, Sitong Gong +8LLM GroundingPersonality Modeling in Language Models

  3. Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMs

    Oct 4, 2026Renchi Zhang, Joost C. F. de Winter, Dimitra Dodou +2Multimodal Large Language ModelsVisual Search