FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation

Hang Ren; Peidong Wang; Yichen Gao; Wen Zhang; Jiayang Yu; Xiaocui Yang; Shi Feng

cs.CLOct 14, 2026

FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation

Authors: Hang Ren, Peidong Wang, Yichen Gao, Wen Zhang, Jiayang Yu, Xiaocui Yang, Shi Feng

Organizations: School of Computer Science and Engineering, Northeastern University, China

Abstract

In recent years, with advances in multimodal interaction technologies and the growing demand for intelligent emotional support, Multimodal Emotional Support Conversation (MESC) has emerged as an important task. However, existing systems still suffer from shallow responses and inappropriate support strategies due to weak semantic feedback, cross-dimension reward compensation, and coarse reasoning supervision. To address these challenges, we propose FACES, a fine-grained gated semantic alignment framework for MESC. Specifically, FACES integrates multi-source rewards to construct multidimensional semantic and functional alignment signals, employs a non-compensatory gated fusion mechanism to reduce cross-dimension reward offset, and leverages structured Chain-of-Thought (CoT) anchors to perform token-level credit assignment. Under a controlled response-length protocol, FACES improves the average MLLM-based evaluation score from 3.61 to 4.04 and achieves an 80.22% pairwise win rate over SOTA. Human evaluation under the same protocol yields a 75.3% preference rate. Results on three external datasets further support transferability. The code and dataset are available at https://github.com/OCT-ARR/arr.

Figures & tables

Figure 1: A MESC example. Both methods use flexible thinking to adaptively select contextually relevant reasoning aspects, while FACES adds reward alignment, recognizing the user’s feeling of being misunderstood and asking a targeted follow-up question.

Figure 2: Workflow of FACES. After SFT and format and outcome warm-up, the policy samples structured CoT candidates $y_{i}\in Y(x)$ . CLIP and MiniLM score visual and emotion spans, while an MLLM judge (Qwen3-VL-8B-Instruct) scores situation, strategy, and outcome. Candidates failing the format check receive zero reward; valid candidates undergo non-compensatory fusion. Group ranking, positive-only advantage scaling, and token-level credit drive clipped GRPO updates for supportive responses.

Model	B-2 $\uparrow$	R-L $\uparrow$	M. $\uparrow$	F ${}_{\text{BERT}}$ $\uparrow$	Dist-3 $\uparrow$	DIFF-E. $\downarrow$
BlenderBot-Joint ∗	4.85	15.25	–	85.50	–	–
BBMHR ∗	1.31	15.38	–	86.60	–	–
SMES ∗	5.13	15.42	–	86.80	–	–
Qwen2.5-VL ∗	9.77	20.30	14.05	86.35	84.01	0.8744
SMES_Qwen2.5 ∗	9.24	19.88	13.16	85.62	92.61	0.7577
FIRES (Baseline) ∗	10.18	22.76	19.54	86.26	98.93	0.7209

Table 1: Automatic evaluation results on the MESC-FT test set. The superscript ∗ denotes results from prior work or corresponding baseline reports. “–” denotes unavailable results. Best results are shown in bold.

Dataset	Judge	FACES Win	GPT-4o Win	Tie
MESC-FT	Qwen2.5-VL-32B	72.0%	16.0%	12.0%
MESC-FT	GLM-4.6V-Flash	69.8%	17.8%	12.4%
MESC-FT	Human	76.0%	14.0%	10.0%
MELD-ESC300	Qwen2.5-VL-32B	66.0%	14.0%	20.0%
MELD-ESC300	GLM-4.6V-Flash	63.0%	16.7%	20.3%
MELD-ESC300	Human	72.0%	13.3%	14.7%

Table 3: Cap-15 pairwise comparison between FACES and directly prompted GPT-4o.

Dataset	Exact Agreement	Cohen’s $\kappa$
MESC-FT	74.7%	0.316
MELD-ESC300	71.7%	0.441

Table 4: Sample-level agreement between final human preferences and Qwen judge majority preferences for the FACES and FIRES comparison under cap-15.

Model	B-2 $\uparrow$	R-L $\uparrow$	M. $\uparrow$	F ${}_{\text{BERT}}$ $\uparrow$	Dist-3 $\uparrow$	DIFF-E. $\downarrow$
w/o multi-source reward	10.46	22.31	19.67	86.55	98.62	0.4189
w/o non-comp. gating	9.82	20.84	20.02	86.79	99.18	0.4057
w/o fine-grained credit	8.56	18.37	18.59	86.02	98.95	0.4902
Ours (Full)	9.68	21.09	20.62	87.14	99.63	0.3643

Table 5: Contribution-level ablation results on the MESC-FT test set. All metrics are higher-is-better except DIFF-E.

Model	B-2 $\uparrow$	R-L $\uparrow$	M. $\uparrow$	F ${}_{\text{BERT}}$ $\uparrow$	Dist-3 $\uparrow$	DIFF-E. $\downarrow$
FIRES (Baseline)	6.32	17.39	11.43	85.39	90.72	0.8192
Ours (Full)	5.59	16.57	15.16 †	86.23 †	94.07 †	0.7664 †

Table 6: Automatic evaluation results on the MELD-ESC300 test set. All metrics are higher-is-better except DIFF-E. † indicates a statistically significant improvement over FIRES under paired bootstrap resampling (

p<0.05

Method	MLLM Avg. $\uparrow$
FIRES (Baseline)	3.61
DPO	3.62
SimPO	3.77
RLOO	3.83
Ours (Full)	4.04

Table 7: Comparison of optimization methods on MESC-FT under the same backbone, SFT initialization, prompts, and length-controlled MLLM protocol.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Policy backbone	Qwen2.5-VL-7B-Instruct
Reward critic	Qwen3-VL-8B-Instruct
SFT learning rate	$5\times 10^{-5}$
GRPO learning rate	$1\times 10^{-5}$
Optimizer	AdamW
SFT / warm up / FACES epochs	3 / 3 / 3

Appendix

Table 8: Core training configuration.

Item	Value
Primary evaluation judge	Qwen2.5-VL-32B-Instruct-AWQ
Cross-family evaluation judge	GLM-4.6V-Flash
Local generation and Qwen judge
Decoding strategy	Greedy decoding
Maximum new tokens	256
Temperature	0

Appendix

Table 9: Inference-time generation and automatic evaluation settings, with judge-specific configurations reported separately.

Component	Range	Role in reward fusion
Format $r_{i}^{\mathrm{Fmt}}$	$\{0,1\}$	Hard gate in the full FACES reward stage; invalid outputs receive 0 in final reward fusion.
Visual $r_{i}^{\mathrm{Vis}}$	$[0,1]$	$1/3$ in cognitive reward
Emotion $r_{i}^{\mathrm{Emo}}$	$[0,1]$	$1/3$ in cognitive reward
Situation $r_{i}^{\mathrm{Sit}}$	$[0,1]$	$1/3$ in cognitive reward
Strategy $r_{i}^{\mathrm{Str}}$	$[0,1]$	Multiplicative gate
Outcome $r_{i}^{\mathrm{Out}}$	$[0,1]$	Multiplicative gate

Appendix

Table 10: Reward components and non-compensatory fusion rule.

Dimension	ICC(2,1)	ICC(2,3)	Human $\rho$
Visual	–	–	0.61
Emotion	–	–	0.68
Situation	0.74	0.90	0.66
Strategy	0.79	0.92	0.71
Outcome	0.77	0.91	0.74
Gated reward vs. Overall Quality	–	–	0.72

Appendix

Table 11: Reliability of the training-time reward system. Visual and Emotion are deterministic CLIP and MiniLM scores, so prompt-level ICC is not applicable. The last column reports reward–human correlations, all significant with

p<0.001

. Higher values indicate stronger reliability or alignment.

Pipeline	Wall-time (h)	GPU-h	Relative cost
FIRES	17.2	34.4	$1.00\times$
FACES	42.2	84.4	$2.45\times$

Appendix

Table 12: Training cost over 100 stable GRPO steps. Lower values are better for all three measures.

Metric	FACES	FIRES
Latency (s) $\downarrow$	$15.828\pm 2.820$	$13.752\pm 2.145$
Output length	184.7 tokens	128.9 tokens
Throughput	11.67 tokens/s	9.37 tokens/s
Peak memory	20.130 GiB	20.130 GiB

Appendix

Table 13: Deployment-time inference cost measured on 100 samples using one NVIDIA RTX A6000. Latency is reported per sample. Reward models are used only during training.

Representative Training-time Reward Prompt

You are a strict evaluator for multimodal empathetic dialogue. Evaluate the candidate using the dialogue context and the corresponding reference annotations. Your judgment must consider all of the following: 1. semantic consistency with the corresponding reference part; 2. functional alignment with the intended role of the reference part; 3. appropriateness to the current dialogue context and available evidence. Score the candidate on the following three dimensions using integer scores from 1 to 5: - Situation: whether the candidate situation inference is semantically consistent with the reference situation, functionally appropriate for understanding the user’s condition, and grounded in the available evidence. - Strategy: whether the candidate support strategy is semantically aligned with the reference strategy, functionally appropriate for the user’s needs, and psychologically safe in the current context. - Outcome: whether the final response is semantically and functionally aligned with the reference response, while remaining empathetic, helpful, and contextually appropriate. Scoring rule: - 5 = great - 4 = good - 3 = acceptable - 2 = weak - 1 = poor - Penalize overly long, verbose, repetitive, or formulaic candidates with redundant or padded wording. If a candidate is substantially longer than the reference and the extra content does not improve semantic or functional alignment, lower the corresponding score(s). [CONTEXT] {context} [CANDIDATE_SITUATION] {candidate_situation} [REFERENCE_SITUATION] {reference_situation} [CANDIDATE_STRATEGY] {candidate_strategy} [REFERENCE_STRATEGY] {reference_strategy} [CANDIDATE_RESPONSE] {candidate_response} [REFERENCE_RESPONSE] {reference_response} Return JSON only in the following format: {"situation_score": 1, "strategy_score": 1, "outcome_score": 1}

Appendix

Table 14: Representative prompt template for training-time MLLM reward scoring. We use three semantically equivalent prompt variants in the actual reward-scoring pipeline.

Rule	Description
JSON-only output	The judge must return a valid JSON object and no extra explanation.
Score range	Pointwise scores are integers from 1 to 5, where 5 indicates the best quality.
Pairwise labels	Pairwise comparison returns one of A , B , or TIE .
Model anonymity	Candidate names are hidden from the judge and replaced with Candidate A and Candidate B.
Think section	Present in the base templates; removed in length-controlled variants.
Verbosity penalty	The judge is instructed to penalize overly long, repetitive, verbose, or formulaic responses when the extra content does not improve helpfulness, empathy, or contextual alignment.

Appendix

Table 15: Common rules for MLLM-based automatic evaluation.

Metric	Evaluation focus	JSON key
Empathy	Whether the final response expresses appropriate empathy, emotional validation, and supportive wording.	empathy_text
Multimodal appropriateness	Whether the response is consistent with the dialogue context and available visual evidence.	multimodal_ appropriateness
Context-Response Cons.	Whether the final response is consistent with the dialogue context, visual evidence, and support direction.	context_response_ consistency

Appendix

Table 16: Pointwise MLLM evaluation metrics.

Representative Pointwise Evaluation Prompt

You are an impartial evaluator for multimodal emotional support dialogue. Evaluate the candidate output according to the specified metric. Use the dialogue context, any attached visual evidence, the model’s think section if available, and the final response. If the think section is missing, it will be shown as Not provided . Your judgment must consider: 1. whether the candidate is appropriate to the user’s emotional state and dialogue context; 2. whether it is grounded in the available multimodal evidence; 3. whether it avoids unsupported, unsafe, overly verbose, repetitive, or formulaic content; 4. whether it satisfies the specific evaluation metric. [EVALUATION_METRIC] {metric_name} [METRIC_DEFINITION] {metric_definition} [CONTEXT] {context} [MODEL_THINK] {think} [CANDIDATE_RESPONSE] {response} Score the candidate from 1 to 5. Return JSON only in the following format: {json_score_format}

Appendix

Table 17: Representative pointwise MLLM evaluation prompt. We use three semantically equivalent prompt variants for each pointwise metric.

Representative Pairwise Preference Prompt

You are an impartial judge for multimodal emotional support dialogue. Compare Candidate A and Candidate B based on the dialogue context, any attached visual evidence, the provided think sections if available, and the final responses. The model identities are hidden. If a think section is missing, it will be shown as Not provided . Choose the better candidate according to the following criteria: 1. empathy and emotional validation; 2. contextual relevance to the user’s situation; 3. consistency with the visual evidence when available; 4. appropriateness and safety of the support strategy; 5. coherence between the think section and the final response; 6. avoidance of unnecessarily long, repetitive, verbose, or formulaic wording. If both candidates are comparable in overall quality, choose TIE . [CONTEXT] {context} [CANDIDATE A THINK] {think_a} [CANDIDATE A RESPONSE] {response_a} [CANDIDATE B THINK] {think_b} [CANDIDATE B RESPONSE] {response_b} Return JSON only in the following format: {"winner": "A"} The value of winner must be one of A , B , or TIE .

Appendix

Table 18: Representative pairwise MLLM preference prompt. We use three semantically equivalent prompt variants, bidirectional candidate swapping, and majority voting to reduce prompt-order bias.

Dataset	View	A Win	B Win	Tie
MESC-FT test set	cap-30	426	11	13
MESC-FT test set	cap-25	411	18	21
MESC-FT test set	cap-15	361	53	36
MELD-ESC300	cap-30	287	7	6
MELD-ESC300	cap-25	279	11	10
MELD-ESC300	cap-15	186	29	85

Appendix

Table 19: Controlled pairwise evaluation results. Each cell reports majority-vote decisions. Model A denotes Ours (Full), and Model B denotes FIRES (Baseline).

Dataset	View	$n$	Qwen2.5-VL-32B	GLM-4.6V-Flash	GLM FACES Share Excl. Ties
MESC-FT	cap-15	450	80.22/11.78/8.00	84.9/6.0/9.1	93.4% ( $p=2.0{\times}10^{-81}$ )
MESC-FT	cap-25	450	91.3/4.0/4.7	97.1/1.3/1.6	–
MESC-FT	response-only	450	100.0/0.0/0.0	99.6/0.2/0.2	–
StickerConv	cap-15	300	63.7/18.3/18.0	64.3/19.4/16.3	76.8% ( $p=4.0{\times}10^{-18}$ )
AvaMERG	cap-15	300	79.3/11.3/9.4	65.7/16.0/18.3	80.4% ( $p=1.3{\times}10^{-22}$ )

Appendix

Table 20: Cross-judge pairwise preferences (%). Both judges evaluate the same candidate pairs. Cap-

N

views truncate final responses to at most

N

words; response-only uses no word cap.

Dataset	Opponent	$n$	Qwen2.5-VL-32B	GLM-4.6V-Flash
MESC-FT	GPT-4o	450	72.0/16.0/12.0	69.8/17.8/12.4
MESC-FT	Claude-Haiku-4.5	450	74.0/14.4/11.6	71.3/16.3/12.4
MESC-FT	Gemini-2.5-Flash-Lite	450	72.7/15.3/12.0	70.2/17.1/12.7
MESC-FT	GLM-4.1V-Thinking-FlashX	450	76.0/12.7/11.3	72.9/14.7/12.4
MESC-FT	Doubao-1.6-Vision	450	76.7/12.0/11.3	74.0/13.8/12.2
MESC-FT	Kimi-K2.5	450	71.6/16.2/12.2	69.3/18.0/12.7

Appendix

Table 21: Cap-15 blind pairwise comparisons between FACES and directly prompted strong MLLMs. Each cell reports FACES win/opponent win/tie percentages.

Dataset	FACES Win	GPT-4o Win	Tie
MESC-FT	342 (76.0%)	63 (14.0%)	45 (10.0%)
MELD-ESC300	216 (72.0%)	40 (13.3%)	44 (14.7%)

Appendix

Table 22: Human pairwise preference between FACES and GPT-4o under the cap-15 blind protocol.

Criterion	Guideline
Empathy	Whether the response recognizes and appropriately addresses the user’s emotion.
Multimodal appropriateness	Whether the response is consistent with the dialogue context and visual evidence.
Strategy suitability	Whether the support strategy fits the user’s current needs without evident risk.
Helpfulness	Whether the response provides practical supportive value.
Tie rule	Select a winner only when one response is clearly better overall. Select Tie when neither response has a clear overall advantage, including cases where the responses have different strengths and weaknesses.

Appendix

Table 23: Criteria and decision rule for human pairwise preference annotation.

Dataset	FACES Win	FIRES Win	Tie	$\boldsymbol{\kappa}$
MESC-FT	339 (75.3%)	63 (14.0%)	48 (10.7%)	0.62

Appendix

Table 24: Final human preferences between FACES and FIRES on the 450-instance MESC-FT test set.

\kappa

denotes Fleiss’ kappa.

Dataset	FACES Win	FIRES Win	Tie	$\boldsymbol{\kappa}$
MELD-ESC300	211 (70.3%)	44 (14.7%)	45 (15.0%)	0.58

Appendix

Table 25: Final human preferences between FACES and FIRES on the 300-instance MELD-ESC300 benchmark.

\kappa

denotes Fleiss’ kappa.

Dataset	$n$	Human F/FIRES/Tie	MLLM F/FIRES/Tie	Exact Agreement	Cohen’s $\kappa$
MESC-FT	450	339/63/48	361/53/36	74.7% (336/450)	0.316
MELD-ESC300	300	211/44/45	186/29/85	71.7% (215/300)	0.441

Appendix

Table 26: Agreement between final human preferences and Qwen2.5-VL-32B-Instruct-AWQ majority preferences for the FACES–FIRES comparison under cap-15. F denotes FACES ; preference distributions are reported as counts.

Model	B-2 $\uparrow$	R-L $\uparrow$	M. $\uparrow$	F ${}_{\text{BERT}}$ $\uparrow$	Dist-3 $\uparrow$	DIFF-E. $\downarrow$
w/o format	10.91	21.84	15.15	85.03	80.47	0.4293
w/o visual	8.89	20.59	19.92	86.51	98.74	0.4073
w/o emotion	9.44	19.89	19.37	86.95	97.92	0.6108
w/o situation	9.97	19.62	19.04	86.24	97.74	0.5918
w/o strategy	9.23	20.27	17.92	84.86	97.05	0.5251
Ours (Full)	9.68	21.09	20.62	87.14	99.63	0.3643

Appendix

Table 27: Additional dimension-level ablation results on the MESC-FT test set. All metrics are higher-is-better except DIFF-E.

Figure 3: Normalized radar chart of dimension-level ablations. Larger values indicate better performance; DIFF-E. is reversed. FACES remains most balanced across metrics.

Dimension	Targeted failure mode	Responsible span	Scoring signal
Visual	Incorrect or missing use of visual evidence	Visual-perception span	CLIP similarity
Emotion	Missing or incorrect affect recognition	Emotion-understanding span	MiniLM similarity
Situation	Incorrect attribution of the user’s condition	Situation-understanding span	MLLM judgment
Strategy	Inappropriate or unsafe support strategy	Strategy-planning span	MLLM judgment; multiplicative gate
Outcome	Unhelpful or misaligned final response	Final-response span	MLLM judgment; multiplicative gate

Appendix

Table 28: Roles of the five reward dimensions. The taxonomy follows the MESC-FT/FIRES flexible-thinking structure and adds Outcome to capture final-response failures.

Aggregation	Definition	Top-Str. $\uparrow$	Top-Out. $\uparrow$	Critical-Fail $\downarrow$
Linear	Equal-weight mean over five dimensions	0.325	0.328	2.4%
Geometric mean	Geometric mean over five dimensions	0.313	0.317	0.0%
Tchebycheff	Minimum over five dimensions	0.314	0.311	0.0%
Gated (Ours)	mean(Visual, Emotion, Situation) $\times$ Strategy $\times$ Outcome	0.343	0.346	0.0%

Appendix

Table 29: Offline comparison of reward aggregation schemes. Top-Str. and Top-Out. denote mean Strategy and Outcome scores among top-25% candidates.

Diagnostic	Value	Linear	Gated (Ours)
High-risk region fraction	65.0%	–	–
Mean reward in $\mathcal{Q}_{\mathrm{HL}}$	–	0.4465	0.0096
Global mean reward	–	0.4320	0.0349
$\mathcal{Q}_{\mathrm{HL}}$ / global mean	–	103.4%	27.6%
Groups assigning maximum advantage to a critical failure	–	14.9%	11.1%

Appendix

Table 30: Reward-geometry diagnostics for candidates with strong cognitive signals but weak Strategy or Outcome quality.

Variant	Composite $\uparrow$	Change
Full model	50.6	–
Without span alignment	45.2	$-10.7\%$

Appendix

Table 31: Supplementary diagnostic for credit assignment to semantic spans.

$\boldsymbol{\tau_{v}}$	B-2 $\uparrow$	R-L $\uparrow$	M. $\uparrow$	F ${}_{\text{BERT}}$ $\uparrow$	Dist-3 $\uparrow$	DIFF-E. $\downarrow$	MLLM Avg. $\uparrow$
0.50	9.91	21.36	20.19	86.84	99.18	0.3984	3.94
0.60	9.74	21.16	20.48	87.03	99.42	0.3726	4.01
0.65	9.65	21.02	20.35	86.99	99.46	0.3795	3.98
0.70	9.56	20.84	20.07	86.83	99.29	0.4017	3.93

Appendix

Table 32: Sensitivity analysis of the CLIP visual threshold

\tau_{v}

on the MESC-FT validation set.

Statistic	Value
Total samples	300
strict samples	208
relaxed samples	92
Frames per sample	5
Maximum context turns	6
Emotion: neutral	232

Appendix

Table 37: Summary statistics of MELD-ESC300 . Strict denotes samples whose immediately preceding utterance already exhibits negative affect; relaxed denotes samples where the immediately preceding utterance does not, but one of the three most recent preceding utterances does. For all samples, the visual input is taken from the video of the immediately preceding utterance, from which five frames are uniformly extracted.

Subset	$n$	Emotional Need	Relevant Response	Supportive Function	Overall Validity
Strict	208	204 (98.1%)	201 (96.6%)	196 (94.2%)	193 (92.8%)
Relaxed	92	89 (96.7%)	87 (94.6%)	83 (90.2%)	81 (88.0%)
Overall	300	293 (97.7%)	288 (96.0%)	279 (93.0%)	274 (91.3%)

Appendix

Table 38: Expert audit of MELD-ESC300 construction validity.

Model	B-2 $\uparrow$	R-L $\uparrow$	M. $\uparrow$	F ${}_{\text{BERT}}$ $\uparrow$	Dist-3 $\uparrow$	DIFF-E. $\downarrow$
GPT-4o	1.41	5.73	13.04	95.35	54.53	0.5053
Claude-Haiku-4.5	1.68	6.81	15.12	95.56	64.18	0.7507
Gemini-2.5-Flash-Lite	2.47	7.82	13.47	96.02	56.19	0.2994
GLM-4.1V-Thinking-FlashX	1.82	6.23	13.60	95.74	69.25	0.8126
Doubao-1.6-Vision	1.70	6.67	11.80	95.65	74.42	0.7057
Kimi-K2.5	1.63	6.06	14.47	95.60	73.10	0.8056

Appendix

Table 40: Automatic evaluation of FACES and six directly prompted MLLMs on MELD-ESC300. Higher values are better for all metrics except DIFF-E. The best result for each metric is shown in bold.

Dataset	View	$n$	FACES Win	FIRES Win	Tie	FACES Share Excl. Ties	Sign-test $p$
StickerConv	cap-15	300	63.7%	18.3%	18.0%	77.6% (72.4–82.9%)	$8.9{\times}10^{-19}$
AvaMERG	cap-15	300	79.3%	11.3%	9.4%	87.5% (83.5–91.5%)	$7.2{\times}10^{-39}$
StickerConv	cap-25	300	75.7%	15.0%	9.3%	–	–
AvaMERG	cap-25	300	85.0%	8.7%	6.3%	–	–
StickerConv	response-only	300	82.3%	12.0%	5.7%	–	–
AvaMERG	response-only	300	90.7%	5.7%	3.6%	–	–

Appendix

Table 41: External zero-shot blind pairwise evaluation on StickerConv and AvaMERG.

Dataset	Model	B-2 $\uparrow$	R-L $\uparrow$	M. $\uparrow$	F ${}_{\text{BERT}}$ $\uparrow$	Dist-3 $\uparrow$	DIFF-E. $\downarrow$
StickerConv	FACES	20.13	25.46	34.08	95.99	68.19	0.4382
StickerConv	FIRES	8.55	22.02	17.94	96.24	60.51	0.4715
AvaMERG	FACES	5.80	11.64	19.42	96.11	74.55	0.5127
AvaMERG	FIRES	5.28	14.48	13.71	96.62	67.11	0.5406

Appendix

Table 42: Automatic evaluation on StickerConv and AvaMERG.

Figure 4: Qualitative comparison of FACES and FIRES. Panels (A)–(B) show MESC-FT successes in standard and cap-15 settings; (C)–(E) show transfer successes on MELD-ESC300, StickerConv, and AvaMERG, respectively; and (F) shows a failure caused by a misread visual cue.

Emotional Support Conversation

Emotional support conversation studies how people and AI systems provide empathetic, safe help through dialogue, including counseling, peer support, and companionship.

See all 60 papers

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data
Sep 24, 2026Wenhao Li, Zhibin Wu, Chong Xiao +1Missing-Modality Learning Multimodal Robustness
Software School, Shandong University · Shenzhen Loop Area Institute
Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis
Jul 30, 2026Wei Chen, Junkai Li, Tongguan Wang +4Multimodal Large Language Models Multimodal Classification
Huazhong Agricultural University Wuhan, China · Hubei University Wuhan, China · Engineering Research Center of Intelligent Technology for Agriculture, Hubei Engineering Technology Research Center of Agricultural Big Data, Key Laboratory of Smart Farming for Agricultural Animals.
MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition
Sep 5, 2026Pengfei Shao, Jisheng Dang, Jiawen Fang +7Multimodal Sentiment Analysis Multimodal Emotion Recognition
School of Information Science and Technology, Guangdong University of Foreign Studies, Guangdong, China · School of Information Science and Engineering, Lanzhou University, Lanzhou, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangdong, China +2

Sep 24, 2026J/K move · Enter open · S save

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Wenhao Li, Zhibin Wu, Chong Xiao, Qiangchang Wang

Software School, Shandong University · Shenzhen Loop Area Institute

Missing-Modality Learning Multimodal Robustness Cross-Modal Alignment Multimodal Sentiment Analysis

Open paper AskPDF arXiv

Recent research on Multimodal Sentiment Analysis (MSA) has focused on learning from language, visual, and acoustic modalities with incomplete data to infer human sentiment. Most studies typically compensate for missing information by reconstructing modality features or designing complicated fusion mechanisms. However, these methods still suffer from spurious generation and noisy guidance due to the lack of high-level semantic grounding in partially observed multimodal evidence. To address these issues, we propose SemMSA, a latent semantic-aided framework that constructs rich sentiment-relevant semantics with LLMs, fully integrating with all modalities via anchor-free spectral alignment. It mainly consists of Cross-modal Semantic Refinement (CSR) and Cross-modal Spectral Alignment (CSA). Specifically, CSR first adaptively extracts visual and acoustic representations by corresponding adapters to form a unified multimodal prefix with language in the frozen LLM embedding space. It then iteratively produces continuous discriminative semantic states through a token-efficient latent refinement process without decoding explicit text. Next, CSA simultaneously aligns the refined semantics with all modalities by enhancing the dominant spectral component of their kernel Gram matrix. This captures global nonlinear dependencies among all representations without relying on a predefined anchor modality. In addition, an instance-level spectral separation constraint preserves cross-sample discriminability and mitigates representation collapse. Extensive experiments on SIMS, MOSI, and MOSEI benchmarks demonstrate that SemMSA achieves state-of-the-art performance.

Abstract

Emotional Support Conversation

Explore similar work

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis

MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition