Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts

Ruijia Li; Mingzi Zhang; Zengyi Yu; Yuang Wei; Bo Jiang

cs.AIApr 11, 2026

Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts

Authors: Ruijia Li, Mingzi Zhang, Zengyi Yu, Yuang Wei, Bo Jiang

Organizations: School of Computer Science and Technology, East China Normal University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · Faculty of Education, East China Normal University

Abstract

As Vision-Language Models (VLMs) become integral to educational decision-making, ensuring their fairness is paramount. However, current text-centric evaluations neglect the visual modality, leaving an unregulated channel for latent social biases. To bridge this gap, we present Edu-MMBias, a systematic auditing framework grounded in the tri-component model of attitudes from social psychology. This framework diagnoses bias across three hierarchical dimensions: cognitive, affective, and behavioral. Utilizing a specialized generative pipeline that incorporates a self-correct mechanism and human-in-the-loop verification, we synthesize contamination-resistant student profiles to conduct a holistic stress test on state-of-the-art VLMs. Our extensive audit reveals critical, counter-intuitive patterns: models exhibit a compensatory class bias favoring lower-status narratives while simultaneously harboring deep-seated health and racial stereotypes. Crucially, we find that visual inputs act as a safety backdoor, triggering a resurgence of biases that bypass text-based alignment safeguards and revealing a systematic misalignment between latent cognition and final decision-making. The contributions of this paper are available at: https://anonymous.4open.science/r/EduMMBias-63B2.

Figures & tables

Figure 1: The Multidimensional Taxonomy of Edu-MMBias. The benchmark spans five primary categories, Gender, Race, SES, Hobbies, and Health, providing a comprehensive landscape of social attributes to systematically audit latent biases in VLMs.

Figure 2: Overview of the EduMMBias evaluation framework, comprising cognitive, affective, and behavioral assessment modules.

Figure 3: The construction pipeline featuring the AI self-correct mechanism and human-in-the-loop audit for data synthesis and validation.

Figure 4: Forest plots of cognitive bias distribution across multiple attributes and models. The x-axis represents the bias probability, where a value of 0.5 denotes perfect parity, and deviations indicate model preferences toward specific attribute values. Asterisks ( $*$ , $**$ , $***$ ) denote statistical significance at $p<0.05$ , $p<0.01$ , and $p<0.001$ levels, respectively.

Figure 5: Forest plots of affective bias distribution across multiple attributes and models. The x-axis represents the normalized score, where a value of 0.5 denotes perfect parity (neutrality).

Figure 6: Forest plots of cognitive bias distribution across multiple attributes and models. The x-axis represents the bias probability, where a value of 0.5 denotes perfect parity.

Figure 7: Regression analysis of behavioral decision bias against cognitive and affective precursors. The top panels plot BBI against CBI, while the bottom panels plot it against ABI. Each point represents a specific model-attribute pair. Solid lines indicate linear regression fits with 95% confidence intervals.

Dimension	Modality	CBS *	BBS *	Sig.
Race	Text-only	0.012	0.009	ns
	VLM	0.215	0.187
SES	Text-only	0.045	0.063	ns
	VLM	0.172	0.154	**
Health	Text-only	0.038	0.045	ns
	VLM	0.141	0.128

Table 1: Comparison of Bias Severity Across Modalities

Figure 8: Scaling Law of Bias Severity: Qwen3 vs LLaVA.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Full Name	Developer	Size	Language	Category
Closed-source Models
GPT-4o	gpt-4o-2024-11-20	OpenAI	N/A	Multilingual	Full-scale
GPT-5.2	gpt-5.2-preview	OpenAI	N/A	Multilingual	Full-scale
Gemini-3	gemini-3-flash-preview-thinking	Google	N/A	Multilingual	Full-scale
Grok-4	grok-4-fast-reasoning	xAI	N/A	Multilingual	Full-scale
Open-source Models

Appendix

Table 2: The list of evaluated vision-language models categorized by openness. ‘Category‘ classifies the model’s scale based on the parameter count.

Panel of Figure 9 — Figure 9: The prompt template used for the IAT experiment. The design elicits the model’s implicit cognitive associations by forcing a rapid categorization based on student profiles. Model confidence serves as a proxy for the strength of these automatic associations.

Panel of Figure 10 — Figure 10: The prompt template used for the AMP experiment. The design utilizes contextual priming , where the model is explicitly instructed to ignore the first image (the prime) and evaluate the second (the target). However, the instruction to rely on "immediate visual impression" facilitates the affective spillover effect, allowing the implicit bias toward the prime to influence the evaluation of the neutral target.

Panel of Figure 11 — Figure 11: The prompt template used for the Resume Audit experiment. The model evaluates two visually distinct student profiles with equivalent textual qualifications within a given educational scenario, forcing a decision that reveals potential behavioral biases.

Panel of Figure 12 — Figure 12: The structured prompt template used for DALL $\cdot$ E 3 to generate student profile images. The base prompt is enriched with specific metadata attributes to ensure a full-factorial design for diverse and controlled image synthesis.

Panel of Figure 13 — Figure 13: The prompt template used for GPT-5 to conduct an automated AI audit of generated student profile images. The model checks for metadata consistency, detects biases, and identifies quality issues, providing actionable feedback for iterative refinement.

Panel of Figure 14 — Figure 14: The prompt template used for GPT-5 to construct diverse education scenarios. By integrating academic subject and social context seeds, the model expands anchor questions into neutral, context-rich scenarios while mitigating social desirability bias.

Panel of Figure 15 — Figure 15: The "AI Jury" audit protocol. To ensure the Baseline Purity of the target stimuli, candidate images generated by DALL-E 3 undergo a rigorous cross-model validation. The prompt explicitly asks for a ternary classification. The Strict Consensus Protocol dictates that only images unanimously classified as "Neutral" by all participating models (GPT-5, Claude, Gemini, etc.) are included in the final benchmark dataset.

Figure 16: Attribute-Adjective Alignment for GPT-4o.

Figure 17: Attribute-Adjective Alignment for GPT-5.

Figure 18: Attribute-Adjective Alignment for Gemini-3-flash.

Figure 19: Attribute-Adjective Alignment for Grok-4.

Figure 20: Attribute-Adjective Alignment for Qwen3-VL-32B.

Figure 21: Attribute-Adjective Alignment for LLaVA-v1.6-13B.

Figure 22: Attribute-Adjective Alignment for Qwen3-VL-8B.

Figure 23: Attribute-Adjective Alignment for LLaVA-Critic-7B.

Figure 24: Attribute-Adjective Alignment for Qwen3-VL-4B.

Figure 25: Attribute-Adjective Alignment for LLaVA-OV-4B.

Cultural Bias in VLMs

See all 14 papers

DeepBias: Adaptive In-depth Probing of Social Biases in LVLMs
Jul 13, 2026Anqi Li, Jie Zhang, Zhongqi Wang +4VLM Evaluation Cultural Bias in VLMs
Interpretable Debiasing of Vision-Language Models for Social Fairness
Feb 27, 2026Na Min An, Yoonna Jang, Yusuke Hirota +3Cultural Bias in VLMs VLM Interpretability
KAIST AI · University of Copenhagen · NVIDIA
Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models
Aug 30, 2026Yusuke Hirota, Michael Ross Boone, Arun George Zachariah +4Language Model Bias Evaluation VLM Evaluation
NVIDIA

Jul 13, 2026J/K move · Enter open · S save

DeepBias: Adaptive In-depth Probing of Social Biases in LVLMs

Anqi Li, Jie Zhang, Zhongqi Wang, Songkai Xue+3

VLM Evaluation Cultural Bias in VLMs Language Model Bias Evaluation LLM Safety Evaluation

Open paper AskPDF arXiv

While Large Vision-Language Models (LVLMs) demonstrate remarkable capabilities, they remain highly susceptible to embedded social biases. Existing bias evaluation protocols predominantly rely on static datasets, which provide only a superficial assessment, as their fixed test cases cannot adaptively evolve to measure the true depth and limits of model vulnerabilities. We introduce DeepBias, an adaptive framework for the in-depth probing of social biases in LVLMs with carefully designed agents. Our approach operates through a dynamic ''generation-evolution-probing'' loop. First, a generative ProposerAgent synthesizes test data and is iteratively updated via Direct Preference Optimization (DPO) based on the target LVLM's responses, exploring model-specific failure modes. Second, an autonomous skill-driven DiggerAgent rewrites each test data across multiple probing turns, adaptively selecting from a curated skill library of deepening and rewriting strategies. At each turn, this process is conditioned on the model's previous response, enabling progressively deeper biases to be exposed. Furthermore, we build a benchmark named DeepBiasBench using our framework. By employing an ensemble of five diverse state-of-the-art LVLMs as anchors, the benchmark captures vulnerabilities shared across architectures. Comprehensive experiments demonstrate the effectiveness of our framework and show that DeepBias provides a challenging benchmark for in-depth bias evaluation, establishing an evolutionary paradigm for LVLM safety assessment.

Abstract

Cultural Bias in VLMs

Explore similar work

DeepBias: Adaptive In-depth Probing of Social Biases in LVLMs

Interpretable Debiasing of Vision-Language Models for Social Fairness

Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models