cs.AINov 19, 2026

MM-SystemOneBias: Evaluating Bias in Multimodal Jev-Style System One Decision Models

Authors: Jinzhe Li, Qi Wang, Dandan Guo, Li Liu, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University · International Center of Future Science, Jilin University · King Abdullah University of Science and Technology (KAUST) · School of Big Data and Software Engineering, Chongqing University · 

Abstract

When multimodal decision models determine actions, moderation outcomes, or quality judgments, their reliability depends on preserving semantic decisions across presentation changes, ignoring irrelevant additions, and distinguishing information consistent with task evidence from information that conflicts with it. Existing evaluations study decision-interface biases and multimodal model behavior in separate settings, leaving no common protocol to compare multimodal decision models with their backbones on practical tasks. We introduce MM-SystemOneBias, which evaluates 13 models on 5,323 original instances from nine public datasets and 11 task scenarios using six paired interventions: option order, option label, external judgment, image--text description, irrelevant information, and rating scale. Across seven matched decision-model--backbone pairs, decision models are more stable under option-label, irrelevant-information, and rating-scale changes in all seven pairs; six pairs also show lower option-order flip rates. They incur smaller accuracy losses under incorrect external judgments in all seven pairs and conflicting descriptions in six. In six pairs, they gain less accuracy from correct external judgments and show less favorable accuracy changes under consistent descriptions. Among pairs with unchanged semantic decisions, mean absolute changes in answer probabilities range from 4.56 to 11.55 percentage points, revealing input effects that accuracy alone misses.

Figures & tables

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models

    Oct 31, 2026Alejandro Mozo Quesada, Sergio Gálvez Rojas, Francisco López ValverdeAlgorithmic BiasLanguage Model Bias Evaluation

  2. MMDM: Measuring Massive Multitask Decision Making

    Nov 10, 2026Junhao Shen, Qing Zong, Yue Guo +3LLM Decision-MakingSystem One Decision Models

  3. How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs

    Oct 13, 2026Yanrui Du, Danyang Zhao, Sendong Zhao +6Clinical Decision-MakingLLM Reliability