Beyond Proxy Metrics: MLLMs as Human Surrogates for Explainable AI Evaluation
Organizations: AI Research, JP Morgan, 25 Bank Street, London, UK · Mistral AI, London, UK
Abstract
Explainable AI (XAI) has expanded into a broad and impactful research area, but the fundamental question of how best to evaluate explanation techniques has proven frustratingly elusive. Methodologies typically revolve around either proxy metrics which tell us little about the utility of an explanation, or human evaluation which---while generally considered the gold standard---is expensive, difficult to scale, and challenging to reproduce. In this paper, we investigate if Multimodal Large Language Models (MLLMs) can act as surrogate human participants to automate the evaluation of XAI. To do this, we first propose a ``purpose-grounded'' evaluation framework with metrics applicable to both MLLMs and human users alike. Then, we offer some conceptual insights as to why and when we can trust these models to automate this process. Lastly, we conduct extensive empirical testing across five purpose-grounded areas such as model improvement and fourteen popular MLLMs. Results indicate that all models trend positively, but OpenAI and Qwen models generally show higher alignment compared to other variants, which would be the recommended choices in practice. We expect our insights not to replace user testing, but rather to help XAI researchers complement their current evaluation methodologies and move the field away from heavy reliance on proxy metrics for measuring explanation utility.
Figures & tables
| Model | Teaching | Reliance | Engagement | Model improvement | Recourse | Overall |
| GPT-5 mini | 1.00 [0.50, 1.00] | 1.00 [0.50, 1.00] | 0.90 [0.84, 0.90] | 1.00 [0.50, 1.00] | 0.50 [0.37, 0.61] | 0.88 [0.66, 0.90] |
| Qwen3.5 122B | 1.00 [0.50, 1.00] | 0.87 [-0.50, 1.00] | 0.81 [0.81, 0.81] | 1.00 [-1.00, 1.00] | 0.50 [0.38, 0.60] | 0.83 [0.26, 0.87] |
| Claude 4.5 Sonnet | 1.00 [0.00, 1.00] | 0.50 [-0.50, 1.00] | 0.75 [0.75, 0.81] | 1.00 [-1.00, 1.00] | 0.52 [0.41, 0.62] | 0.76 [0.28, 0.86] |
| GPT-5.5 | 1.00 [0.50, 1.00] | 0.50 [-0.50, 1.00] | 0.84 [0.78, 0.84] | 0.50 [-1.00, 1.00] | 0.73 [0.64, 0.79] | 0.71 [0.41, 0.91] |
| GPT-5 | 1.00 [1.00, 1.00] | 0.50 [0.50, 1.00] | 0.81 [0.75, 0.90] | 0.50 [0.50, 1.00] | 0.74 [0.65, 0.80] | 0.71 [0.69, 0.92] |
| GPT-4o mini | 0.50 [-0.50, 0.50] | 0.50 [0.50, 1.00] | 0.87 [0.72, 0.93] | 1.00 [0.50, 1.00] | 0.29 [0.14, 0.43] | 0.63 [0.40, 0.75] |
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
| Control prompt for appropriate reliance |
| Pretend you are a participant in a user study with some technical machine learning knowledge being asked to assist in a study to examine people’s reliance on AI systems with or without explanations for the AI’s predictions. The domain is Adult Census and the features in order are: {feature_names} The possible labels are: {class_names} Here are 5 randomly sampled examples of training data to help you understand the domain: {five_demonstration_instances} , and here are their labels: {five_demonstration_labels} Your task is to look at a new test instance, and decide what you think the real label is. Here is the new test instance: {row.test_instance} Outline your reasoning process step by step, and give your final answer as one of the possible class labels in the tags <answer>...</answer> . |
| Saliency prompt for appropriate reliance |
| Pretend you are a participant in a user study with some technical machine learning knowledge being asked to assist in a study to examine people’s reliance on AI systems with or without explanations for the AI’s predictions. The domain is Adult Census and the features in order are: {feature_names} The possible labels are: {class_names} Here are 5 randomly sampled examples of training data to help you understand the domain: {five_demonstration_instances} , and here are their labels: {five_demonstration_labels} Your task is to look at a new test instance, and decide what you think the real label is. Here is the new test instance: {row.test_instance} Here is the AI prediction for the new test instance: {row.test_predicted_label} In addition, here is a feature saliency explanation for the AI prediction where you see each feature, and its contribution (positive or negative) towards the prediction: {round_numbers_in_string(row.lime_explanation)} Outline your reasoning process step by step, and give your final answer as one of the possible class labels in the tags <answer>...</answer> . |
| Example-based prompt for appropriate reliance |
| Pretend you are a participant in a user study with some technical machine learning knowledge being asked to assist in a study to examine people’s reliance on AI systems with or without explanations for the AI’s predictions. The domain is Adult Census and the features in order are: {feature_names} The possible labels are: {class_names} Here are 5 randomly sampled examples of training data to help you understand the domain: {five_demonstration_instances} , and here are their labels: {five_demonstration_labels} Your task is to look at a new test instance, and decide what you think the real label is. Here is the new test instance: {row.test_instance} Here is the AI prediction for the new test instance: {row.test_predicted_label} In addition, here is an explanation for the prediction in the form of the two nearest neighbors from the training data that the AI learned from: The first nearest neighbor is {row.neighbor1_instance} , its label is ’ {row.neighbor1_true_label} ’, and the AI’s prediction of this nearest neighbor is ’ {row.neighbor1_predicted_label} ’ The second nearest neighbor is {row.neighbor2_instance} , its label is ’ {row.neighbor2_true_label} ’, and the AI’s prediction of this nearest neighbor is ’ {row.neighbor2_predicted_label} ’ Outline your reasoning process step by step, and give your final answer as one of the possible class labels in the tags <answer>...</answer> . |
| Pretend you are a participant in a user study designed to (1) teach you about a complex causal domain, and (2) get you to make a prediction on a novel test instance given what you learned. |
| The data domain has 5 features and is structured as follows: - X1, X2: Binary variables (0 or 1) - X3, X4, X5: Continuous POSITIVE variables (representing quantities > 0) - Y: Outcome (0 or 1) |
| You are now going to learn about the underlying domain by studying the following information: Here are [len(examples)] training examples, and their labels Y to help you understand the domain: [formatted_examples] |
| Here is the new test instance: [test_instance] |
| Now, given all this information, make an informed prediction on what Y is for this test instance. Explain your reasoning briefly, and give your final answer as 0 or 1 in the tags <answer>…</answer>. |
| Pretend you are a participant in a user study designed to (1) teach you about a complex causal domain, and (2) get you to make a prediction on a novel test instance given what you learned. |
| The data domain has 5 features and is structured as follows: - X1, X2: Binary variables (0 or 1) - X3, X4, X5: Continuous POSITIVE variables (representing quantities > 0) - Y: Outcome (0 or 1) |
| You are now going to learn about the underlying causal graph by studying the following explanation: Here are [len(examples)] training examples, and a causal explanation for their classification, and their labels Y to help you learn the underlying causal graph, study this carefully: [formatted_examples] |
| Here is the new test instance: [test_instance] |
| Now, given all this information, and what you have learned from the explanation, make an informed prediction on what Y is for this test instance. Explain your reasoning briefly, and give your final answer as 0 or 1 in the tags <answer>…</answer>. |
| Pretend you are a participant in a user study designed to (1) teach you about a complex causal domain, and (2) get you to make a prediction on a novel test instance given what you learned. |
| The data domain has 5 features and is structured as follows: - X1, X2: Binary variables (0 or 1) - X3, X4, X5: Continuous POSITIVE variables (representing quantities > 0) - Y: Outcome (0 or 1) |
| You are now going to learn about the underlying causal graph by studying the following explanation: Here are [len(examples)] training examples, counterfactual versions of them where Y changes to the opposite class, and their labels Y to help you learn the underlying causal graph, study this carefully: [formatted_examples] |
| Here is the new test instance: [test_instance] |
| Now, given all this information, and what you have learned from the explanation, make an informed prediction on what Y is for this test instance. Explain your reasoning briefly, and give your final answer as 0 or 1 in the tags <answer>…</answer>. |
| Prompt for System Engagement |
| Pretend you are a human user of a movie recommender system. Imagine that you have $7 and a free evening coming up. You are considering going to the theater to see a movie, but only if there is a movie worth seeing. To determine if there is a movie worth seeing, you consult MovieLens for a personalized movie recommendation. MovieLens recommends one movie, and provides an explanation to justify the recommendation. The movie recommended was: {movie_title} Here is its explanation for the recommendation: {explanation} The movies are rated on a scale of 1-5. Based on the provided explanation, please indicate on a scale of 1-7 how likely you are to see the movie. First, outline your reasoning step by step, and put your answer in between the tags <answer>...</answer> It is very important you follow this exact format. |
| Prompt for System Engagement – No Explanation |
| Pretend you are a human user of a movie recommender system. Imagine that you have $7 and a free evening coming up. You are considering going to the theater to see a movie, but only if there is a movie worth seeing. To determine if there is a movie worth seeing, you consult MovieLens for a personalized movie recommendation. MovieLens recommends one movie. The movie recommended was: {movie_title} Please indicate on a scale of 1-7 how likely you are to see the movie. First, outline your reasoning step by step, and put your answer in between the tags <answer>...</answer> It is very important you follow this exact format. |
| Imagine you are a human applying for a loan. You are at a financial institution undergoing a loan assessment. An AI system determines approval or denial based on your profile data. |
| The requested loan’s purpose, amount and duration are shown below. Monetary amounts are in Deutsche Mark (DM), as recorded in this historical dataset. |
| YOUR CURRENT PROFILE: [profile with readable categories and units] |
| An AI system has rejected your loan application. The following changes would make that same AI system accept your revised profile. All other profile details stay the same. |
| AI Explanation: [numbered changes from original to proposed values] |
| Your Task: Please answer the following two questions using an integer between 1–7 where 1=“Strongly no”, 7=“Strongly yes”, and 4=“Neutral”. |
| 1. Is the AI System’s plan a reasonable explanation for the rejection of your loan application? |
| Optimization prompt structure |
| ROLE: You are a medical professional optimizing a diabetes prediction ML model by configuring training data. CURRENT STATUS - Baseline accuracy: {baseline_accuracy} - Current best accuracy: {current_best} - Improvement so far: {delta} - Original samples: {n_orig} , Current: {n_current} - Available features: {feature_list} CURRENT BEST CONFIGURATION {formatted_config} DATA-CENTRIC SIGNALS (for DCE/HYB) - Class balance: pos %, neg % - Highly correlated pairs (if any) - Data quality scores: zeros, outliers, balance, correlation, overall - Per-feature summary: min, mean, max, p5, p95, skew, zeros %, outliers % MODEL-CENTRIC SIGNALS (for MCE/HYB) - Feature importance ranking - Top decision rules for Class 1 and 0 (confidence, samples) OBSERVED FEATURE RANGES - feat : [min, max], … HISTORY {formatted_recent_iterations} TASK Propose a NEW configuration to beat {current_best} . Return your REASONING followed by CONFIGURATION JSON: {"FeatureName": { "include": true/false, "min": number_or_null, "max": number_or_null } } |
| MLLM Prompt |
| In Herlocker et al.’s 2000 paper “Explaining Collaborative Filtering Recommendations”, what did users rate { insert_method_name } on a 1–7 scale? Give your answer in between the tags <answer>…</answer> |
| Model | New best | Accurate feedback (%) | Feedback problem (%) | Tabular error (%) | Visual mismatch/claims | One-case change |
| Claude 4.5 Sonnet | 4/30 | 83.3 | 13.3 | 23.3 | 6/16 | 10/30 |
| Claude 4.6 Sonnet | 3/30 | 86.7 | 13.3 | 20.0 | 0/8 | 10/30 |
| Claude 4.8 Opus | 1/30 | 80.0 | 20.0 | 13.3 | 0/15 | 4/30 |
| Gemma 3 27B | 3/30 | 70.0 | 23.3 | 30.0 | 2/8 | 4/30 |
| Gemma 4 31B | 1/30 | 70.0 | 23.3 | 6.7 | 0/12 | 8/30 |
| GPT-4o | 4/30 | 46.7 | 23.3 | 30.0 | 1/15 | 2/30 |
| Label | New best at attempt 4 | Attempt-5 accuracy | Attempt-5 gain over baseline |
| Accurate feedback use | 0.08 [-0.01, 0.15] | 0.06 [-0.04, 0.16] | 0.14 [0.04, 0.23] |
| Feedback problem | -0.03 [-0.11, 0.05] | 0.02 [-0.07, 0.10] | -0.04 [-0.13, 0.03] |
| Tabular error | -0.01 [-0.10, 0.09] | -0.07 [-0.17, 0.03] | -0.09 [-0.19, 0.02] |
| Visual mismatch | 0.00 [-0.06, 0.10] | 0.02 [-0.05, 0.08] | 0.01 [-0.04, 0.05] |
| Condition | Positive rate | Modal | Flips | ||
| Original | Edited | agreement | (of 50) | (2-prop.) | |
| Control | 0.421 | 0.417 | 1.00 | 0 | 0.918 |
| Causal | 0.409 | 0.376 | 0.98 | 1 | 0.341 |
| Counterfactual | 0.495 | 0.472 | 0.98 | 1 | 0.520 |
| All | 0.441 | 0.422 | — | 2 | 0.335 |
| Code | Theme | Human quotation (participant) | Rating; direction |
| T1 | Small-change criticism | “Almost same” (P006) | A: 1; |
| T1 | “That’s it?” (P446) | A: 2; | |
| T1 | “Not particularly different compared to the current situation” (P706) | A: 2; | |
| T2 | Large gap justifies rejection | “The career is too different” (P567) | A: 7; |
| T2 | “I think it’s a solid reason, so I don’t blame it” (P739) | A: 6; | |
| T3 | Personal shortcoming | “Because I realize how much I’m not enough” (P732) | A: 5; |
| Study/domain | Human ordering | Study/domain | Human ordering |
| Teaching | Counterfactual Control | Engagement | E1 E21 |
| Teaching | Counterfactual Causal | Engagement | E2 E4 |
| Teaching | Causal Control | Engagement | E2 E5 |
| Reliance—Adult | NN Control | Engagement | E2 E11 |
| Reliance—Adult | NN Saliency | Engagement | E2 E21 |
| Reliance—Adult | Control Saliency | Engagement | E4 E11 |
Explainability Evaluation
Explainability evaluation tests whether AI explanations are faithful, stable, understandable, useful, and reliable across models and settings.
See all 347 papers