Multimodal large language models (MLLMs) are increasingly used as evaluators, yet their reliability in professional assessment tasks that require expert judgment remains unclear. We investigate this challenge in the context of assessing psychotherapeutic interpersonal skills and introduce MIS-Bench, a Multimodal Interpersonal Skills (MIS) benchmark comprising 996 psychotherapy response videos annotated across 8 dimensions of Facilitative Interpersonal Skills. Across 9 MLLMs with multiple modality and prompting settings, we find that current models show only modest agreement with human experts, inconsistent gains from multimodal input, and limited benefits from reasoning-based prompting. To mitigate this gap, we propose MIS-RAFT, a regression-aware fine-tuning method inspired by RAFT and tailored to fine-grained interpersonal skill scoring at one-decimal precision. MIS-RAFT addresses the mismatch between autoregressive token prediction and scalar-valued expert assessment, significantly improving agreement with human ratings. Overall, MIS-Bench reveals a clear gap between general multimodal capability and expert-level interpersonal judgment, while MIS-RAFT offers a promising path toward more reliable model-based assessment.
Figures & tables
Figure 1: Overview of MIS-Bench. Each benchmark instance is built from a psychotherapy response video and associated transcript and rubric information. Models are evaluated with 4 modality configurations and 2 prompting styles, and predict one scalar score for each FIS dimension. Performance is measured by ICC(3) and MSE.
Modality
Feature
Shape / Type
Face
Action units
T×8
Face
Facial landmarks (98 points, x/y)
T×196
Face
Gaze direction
T×2
Face
Detection confidence
T×2
Audio
Wav2Vec 2.0 frame embeddings
T×768
Audio
Prosody
T×15
Table 1: Released feature structure of MIS-Bench. The public release contains de-identified feature representations rather than raw audiovisual recordings.
VF
H/PE
Per.
Emo.
WAU
Emp.
ABC
ARRR
Human ICC(3,1)
64.0
53.0
54.0
51.0
50.0
43.0
51.0
53.0
95% CI
[61.0, 68.0]
[49.0, 58.0]
[49.0, 58.0]
[46.0, 55.0]
[45.0, 54.0]
[38.0, 48.0]
[46.0, 55.0]
[48.0, 57.0]
Table 2: Human single-rater agreement ceiling on the 625 fully rated videos. Values are single-measure ICC(3,1) scores reported in percentage points; brackets denote 95% confidence intervals. Higher is better.
Model
LLM Size
A
V
T
VF
H/PE
Per.
Emo.
WAU
Emp.
ABC
ARRR
Open-source MLLMs
Qwen2-Audio-7B-Instruct ( Chu et al., 2024 )
7B
✓
✓
-6.0
17.1
8.4
0.9
8.4
2.3
9.1
0.1
Qwen2.5-VL-7B-Instruct ( Bai et al., 2025 )
7B
✓
✓
2.8
39.9
29.9
16.5
36.4
28.8
33.0
27.4
Qwen2.5-Omni-7B ( Xu et al., 2025a )
7B
✓
✓
✓
-3.1
19.4
15.3
11.0
18.6
23.0
26.1
20.1
Qwen3-Omni-30B-A3B-Instruct ( Xu et al., 2025b )
30B
✓
✓
✓
21.2
24.6
25.1
14.8
22.8
24.6
18.6
22.7
InternVL3.5-8B-HF ( Wang et al., 2025 )
8B
✓
✓
5.9
32.4
16.6
14.7
28.9
22.9
26.8
17.6
Table 3: Overall performance comparison on MIS-Bench using dimension-level ICC (%). For each model, we report the dimension-level ICC values from the configuration that achieves the highest overall score. This overall score is computed by averaging the eight predicted dimension scores for each video and comparing the resulting average against the corresponding average of the human ratings. If a model output is unparseable for any of the eight dimensions, that video is excluded when determining the overall score, with a mean exclusion rate of 1.5%. The top three performing results in each dimension are highlighted in red (1st), blue (2nd), and yellow (3rd) backgrounds, respectively.
Figure 2: Dimension-level ICC heatmaps on MIS-Bench. We report every supported model × modality × prompting combination across four modality configurations and two prompting strategies. In each panel, rows correspond to models and columns correspond to the eight FIS dimensions. Higher ICC indicates better agreement with human ratings, and blank cells indicate configurations unsupported by a given model. The complete grid enables controlled comparisons among models within a common configuration.
Figure 3: MIS-RAFT scoring pipeline for one-decimal score prediction.
FIS Dimension
Raw
Fine-tuning
MIS-RAFT
VF
−3.8 [ −12.5 , 2.0 ]
1.2 [ −16.3 , 18.9 ]
−0.7 [ −16.0 , 12.6 ]
H/PE
37.6 [ 24.0 , 49.5 ]
21.7 [ 12.3 , 31.2 ]
50.7 [ 38.0 , 62.3 ]
Per.
−2.0 [ −6.6 , 0.0 ]
1.8 [ −15.7 , 16.6 ]
33.6 [ 20.2 , 45.9 ]
Emo.
−0.0 [ −0.0 , 0.0 ]
−13.5 [ −29.4 , 4.1 ]
17.0 [ 0.6 , 31.1 ]
WAU
12.9 [ −1.6 , 27.3 ]
−12.2 [ −22.9 , 0.0 ]
31.4 [ 14.9 , 46.6 ]
Emp.
43.6 [ 28.2 , 55.9 ]
3.7 [ −7.6 , 15.3 ]
35.5 [ 19.1 , 50.7 ]
Table 4: ICC(3,1) by FIS dimension under raw inference, standard fine-tuning, and MIS-RAFT fine-tuning for the Qwen2.5-VL backbone. Values are reported in percentage points; brackets report 95% percentile bootstrap confidence intervals computed over videos using 10,000 bootstrap resamples. Higher is better, and the best ICC in each row is shown in bold.
Figure 4: Radar chart of ICC scores across eight FIS dimensions.
Humans possess an innate ability to understand fine-grained interpersonal relationships, which is central to everyday social interactions. Although such reasoning is inherently multimodal, it remains largely unexplored by existing multimodal large language models (MLLMs). To address this gap, we introduce PIVOTS, the first benchmark built from Social-IQ 2.0 and YouTube data to evaluate MLLMs' ability to predict bidirectional interpersonal relationship dimensions grounded in established psychology research. In addition, PIVOTS includes auxiliary tasks that assess models' ability to identify and leverage the critical visual cues underlying such predictions. We evaluate both proprietary and open-source MLLMs and conduct detailed ablation studies to analyze the effects of visual modalities and explicit social role information in conversational utterances. We further examine how joint and pairwise prediction settings benefit MLLMs in scoring bidirectional PIVOTS dimensions. Project page: https://ciossayin.github.io/pivots-bench/
Shuxiang Zhang, Yiting Yin, Wenxuan Song +2
Sun Yat-sen University · University of Michigan · Tsinghua University
Frontier language models are increasingly marketed as omni systems that can perceive and respond across modalities. Existing evaluation frameworks, however, focus almost exclusively on bimodal understanding, typically text plus one other modality. We propose the Modality Maturity Index (MMI), a benchmark designed to evaluate the multimodal capabilities of large language models across five modalities (text, image, audio, video and document) and combinations of up to three modalities in both inputs and outputs. MMI consists of 893 questions, each carefully crafted to require the model to demonstrate its understanding of multiple input modalities and to generate responses that incorporate various output formats. The questions are designed to be self-contained, with clear expectations for the correct modality or mix of modalities required for an accurate response. Every MMI prompt carries human-authored rubric criteria for each output modality expected in the response; a model's MMI Value expresses the average of the per-modality scores for each prompt. Because low scores can reflect either failure to generate a modality (lack of presence) or failure to generate correct content, we introduce also a supplementary Modality Presence Score (MPS), a per-prompt F1 over the expected output modalities. Applying MMI to five frontier multimodal models, we find that the MPS ranges from only 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4). Given the low availability of returned modalities to even grade, we report MPS as our main result pending model improvements. To assess the viability of judging output correctness with LLM judges and rubrics, we run a separate experiment with custom generation tools. On the assets that generates, we find that an LLM judge applying the rubrics agrees with rubric-blind human annotators (who score the outputs directly and never see the criteria) on 70.8% of judgments.
Using Multimodal Large Language Models (MLLMs) as judges to achieve precise and consistent evaluations has gradually become an emerging paradigm across various domains. Evaluating the capability and reliability of MLLM-as-a-judge systems is therefore essential for ensuring trustworthy assessment. Existing judge benchmarks categorize samples by task types but fail to capture the fundamental judgment capabilities required for reliable evaluation. In this work, we introduce M-JudgeBench, a ten-dimensional capability-oriented benchmark designed to comprehensively assess the judgment abilities of MLLMs. Our benchmark decomposes evaluation into pairwise Chain-of-Thought (CoT) comparison, length bias avoidance, and process error detection tasks, jointly covering ten fine-grained subtasks. This design enables diagnosis of model reliability across reasoning styles, response lengths, and cross-model variations. Systematic evaluation uncovers the systematic weaknesses in existing MLLM-as-a-judge systems. To address this issue, we further propose Judge-MCTS, a data construction framework generating pairwise reasoning trajectories with various correctness and length. Using Judge-MCTS, we construct an MCTS-augmented dataset and train M-Judger, a series of strong judge models. Extensive experiments demonstrate the superiority of M-Judger on existing judge benchmarks as well as M-JudgeBench. Overall, our work establishes a more principled foundation for evaluating MLLM-as-a-judge through M-JudgeBench and Judge-MCTS framework, paving the way for future research on judge model evaluation and capability-driven judge training.