VISTA: Value-Informed Event Appraisal for Multimodal Emotion Conflict

Jiale Dai; Liuxian Ma; Xiaoke Niu; Wenjing Zhang; Huiying Zhao; Zhaoxiang Liu; Shiguo Lian; Guojie Song

cs.AISep 29, 2026

VISTA: Value-Informed Event Appraisal for Multimodal Emotion Conflict

Authors: Jiale Dai, Liuxian Ma, Xiaoke Niu, Wenjing Zhang, Huiying Zhao, Zhaoxiang Liu, Shiguo Lian, Guojie Song

Organizations: State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · College of Artificial Intelligence, Tsinghua University · China United Network Communications Group Co., Ltd.

Abstract

Conflicting emotional cues can be individually valid: a subdued voice may reflect a blocked goal while a smile satisfies a social obligation. Their interpretation depends on what the event means to the person. We introduce VISTA (Value-Informed Semantic Trust Arbitration), a learned seven-field appraisal interface that conditions modality arbitration on concerns, event relations, and expression conditions while retaining a joint-evidence residual. A log-odds decomposition separates emotion expectation from cue diagnosticity, motivating an interface that lets appraisal change how evidence is interpreted. With a shared Qwen2.5-Omni-7B backbone and matched training examples and steps, VISTA reaches 64.5% conflict accuracy on CA-MER, improving on modality gating by 2.5 percentage points on conflict and 0.2 on consistency. Shuffling appraisal across scenes or removing its decision connection reduces this benefit. A common frozen-backbone probe reaches 0.600 macro CCC for appraisal readout, compared with 0.505 for emotion-only fine-tuning. Evaluations across five benchmarks connect recognition under increasing conflict with appraisal readout and downstream decision use. Together, the analyses and experiments support scene-specific appraisal as an intermediate representation that helps interpret conflicting emotional evidence.

Figures & tables

Figure 1 : The same cues can have different meanings under different concerns. This constructed comparison holds the observed event and cues fixed while changing the supplied goal of the candidate on the left. Goal congruence changes with that goal; the shared public setting supplies a condition on expression. The interpretations are conditional readings, not model predictions.

Figure 2 : An explicit appraisal interface for multimodal recognition. Appraisal sets modality weights

\alpha

; prediction combines weighted unimodal logits with joint evidence. Replacing appraisal at fixed evidence produces the readout

\tilde{p}

. The lower panels show this diagnostic and the supervision roles.

Method	Video-aligned	Audio-aligned	Conflict	Consistent	Overall
Qwen2.5-Omni Base	49.0	60.0	54.5	68.0	59.0
Emotion-SFT	55.0	65.0	60.0	73.0	64.3
Generic-CoT-SFT	57.0	66.0	61.5	74.0	65.7
Modality-Gate-SFT	58.0	66.0	62.0	74.0	66.0
VISTA	61.0	68.0	64.5	74.2	67.7

Table 1 : CA-MER accuracy (%). Conflict averages the video-aligned and audio-aligned subsets. The four trained models share their common-stage examples and optimization schedule. Complete output-validity results and protocol-reassessed external methods are in Appendix G .

Figure 3 : Conflict resolution beyond stronger reasoning and gating. (a) Subgroup accuracy. (b) Conflict and consistent gains;

S

is their difference. (c) The audio–video accuracy gap. (d) External methods re-evaluated under the common output interface. Points show aggregate scores; gains and gaps use percentage points.

Method	Direction	Class change	$\Delta p$	$\Delta$ log-odds	Stability
Emotion-SFT	62.0	24.0	0.050	0.201	91.0
Generic-CoT-SFT	66.0	28.0	0.070	0.281	90.0
Modality-Gate-SFT	63.0	25.0	0.055	0.221	91.0
VISTA	75.0	36.0	0.120	0.484	92.0

Table 2: Directed response and selective stability. Direction is agreement with the annotated target-probability direction; class change counts label flips. Both use 150 relevant pairs; stability uses 150 irrelevant rewrites. Rates are percentages;

\Delta p

and

\Delta

log-odds retain native scales.

Figure 4 : From scene-specific appraisal to selective responses. (a) Accuracy losses under appraisal and training variants. (b) Native modality weights; triangles show the gate’s dominant-modality weight. (c) Direction agreement, label changes, and irrelevant-rewrite stability on two separate 150-pair sets. Native weights are distinct from masking-derived diagnostics.

	EmoMM accuracy		CH-SIMS v2 Acc2		MELD
Method	Conflict	Conflict + missing	Overall	Q4	wF1
Qwen2.5-Omni Base	46.5	38.0	79.4	70.0	59.55
Emotion-SFT	47.5	38.0	84.0	77.0	65.49
Generic-CoT-SFT	48.0	38.6	84.3	77.8	65.69
Modality-Gate-SFT	49.0	40.0	84.8	79.2	65.74
VISTA	52.0	43.5	85.9	81.5	66.94

Table 3: Recognition across conditions and tasks (%). EmoMM uses no benchmark-specific adaptation; CH-SIMS v2 and MELD use task-adapted models (Base is zero-shot). Acc2 is binary accuracy, Q4 the highest-conflict group, and wF1 weighted F1.

Figure 5 : Conflict strength, frozen evaluation, and appraisal readout. (a) Task-adapted CH-SIMS v2 binary accuracy by conflict group. (b) EmoMM gains without benchmark-specific adaptation. (c) THERADIA four-dimensional probe CCC on frozen backbones. (d) Ten-label emotion-intensity regression CCC with different appraisal access; human appraisal is an oracle.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Type	Values
$g$	Short text	Goal or concern supported by the scene
$c$	Categorical	promotes , obstructs , irrelevant
$e$	Categorical	expected , violated , uncertain
$a$	Categorical	self , other , environment , shared , unknown
$k$	Continuous	$[0,1]$ coping/control value
$n$	Four binary entries	politeness , identity , status , relationship

Appendix

Table 4 : Typed appraisal interface. These value spaces distinguish scene semantics, expressive conditions, and missing information. Each norm bit has its own validity mask.

Field	Evidence anchor	Distinction to preserve
Goal / concern	Requests, commitments, prior choices, or stated priorities in the available context	The current object of concern, such as obtaining this role, is more specific than a general value such as achievement.
Goal congruence	The observed outcome together with evidence about the relevant goal	Whether the event supports or obstructs the goal; the same external outcome can have different signs for different concerns.
Expectation	Prior plans, predictions, promises, or established patterns	Expectedness concerns anticipation. An undesired event can be expected, and a desirable event can be surprising.
Agency	Actions, decisions, attributions, and identified participants	Who or what produced the outcome; identifying a responsible agent does not by itself establish intent or blame.
Coping / control	Available remedies, remaining choices, resources, or finality of the decision	The person’s capacity to alter or manage the consequences, distinct from responsibility for causing them.
Norm / social relevance	Roles, relationships, agreed procedures, audience, and obligations	Which interpersonal standards bear on this situation, including whether an outcome or an expression is socially expected.

Appendix

Table 5 : A semantic reading guide for the seven fields. Evidence anchors identify relevant observations; the final column specifies the relation to interpret.

Approach	Organizing signal	Decision role	Evaluation focus
Wang et al. (2026)	Expectations and their violations	Predict emotion labels and shifts	Emotion labels and transition dynamics
AG-CTR 2 ( Chu et al., 2026 )	Event understanding, appraisal, and coping	Retrieve support using appraisal chains	Support generation and retrieval-query quality
CHASE ( Sun et al., 2026a )	Modality hidden states and attention preferences	Detect conflict and steer head-level attention	Recognition under conflict and missingness
VISTA	Concerns, event relations, and expression conditions	Condition modality arbitration and affect prediction	Conflict gains, correspondence, and decision access

Appendix

Table 6 : Related approaches by decision role. Selected methods organized by the information used, the decision it informs, and the evaluation target.

Figure 6 : The context changes what a cue can mean. Each constructed pair holds the described surface observation fixed while changing the goal, audience, or allocation rule. The contrasts illustrate appraisal relations; they are not evaluated predictions.

Figure 7 : A context-dependent cue in a constructed probability model. Both conditions have prior satisfaction probability

0.4

and positive-display likelihood

0.8

under satisfaction. The likelihood under disappointment changes from

0.2

0.6

, giving posteriors

8/11

and

8/17

. The log-odds plane separates emotional expectation from cue diagnosticity; its boundary is

b+\ell=0

. These are exact analytical calculations, not model predictions or fitted probabilities.

Contrast	Reference retained	Dependency examined
Cross-sample shuffle	The current input and recognition task	Does another sample’s appraisal supply the same useful interpretation as the scene’s own appraisal?
Within-emotion shuffle	The current input and the donor’s emotion identity	Does event-specific correspondence matter beyond the donor appraisal’s association with the same class?
Appraisal with no decision connection	Auxiliary appraisal supervision	Does appraisal improve prediction when its forward connection to the decision is removed?
Generic semantic bottleneck	An intermediate semantic route to the same recognition task	Does organizing content as event appraisal contribute beyond the supplied semantic alternative?
Rationale / internal-state corruption	One of verbal presentation and internal appraisal is retained while the other is altered	Is the observed dependence stronger on displayed explanation text or on the internal appraisal information?
Relevant change / irrelevant paraphrase	The pairing identifies what event meaning should change or remain invariant	Does the response follow a relevant change while remaining stable to a reformulation of the same meaning?

Appendix

Table 7 : Interpretive roles of the intervention families. Each row identifies a distinct question about the appraisal-conditioned decision pathway.

Figure 8 : Appraisal replacement in the implemented weighted readout. With the checkpoint, all evidence representations

H

, and available modalities fixed, reference and replacement appraisals produce different weights over the same unimodal logits. The joint-evidence residual is shared. For any class pair, the resulting logit-margin change is the weight change dotted with the branch margins; its magnitude is bounded by their range times half the

\ell_{1}

weight change. The comparison describes how appraisal-dependent arbitration changes a class preference.

Dataset	Public context	Role in this paper
CA-MER	1,500 examples: 500 video-aligned, 500 audio-aligned, 500 consistent	Primary comparison of conflict and consistent performance
EmoMM	4,000 base examples from Chinese CH-SIMS v2.0 and English CMU-MOSI; multimodal and unimodal sentiment annotations	Alignment, conflict, missingness, and their combination
CH-SIMS v2.0	4,402 labeled and 10,161 unlabeled Chinese clips, with multimodal and unimodal sentiment information	Binary accuracy (Acc2) across four conflict groups
THERADIA	2,735 affect-annotated clips from interactions during cognitive exercises, with appraisal annotations	Frozen-representation appraisal probes and emotion-intensity regression
MELD	13,708 utterances across 1,433 dialogues; official train/development/test sizes of 9,989/1,109/2,610	Ordinary seven-class conversational emotion recognition

Appendix

Table 8 : Public dataset context and the role of each evaluation. Dataset composition and the evaluation question are listed separately.

Input	Canonical	Input	Canonical
anger	angry	sadness	sad
happiness	happy	worried	worry
surprised	surprise	doubtful	doubt
fearful; afraid	fear	contemptuous	contempt

Appendix

Table 9: Fixed synonym normalization for the CA-MER interface. Canonical labels pass through unchanged. The mapping is applied after JSON extraction and string normalization.

Comparison / population	Disagreement	Eligible clips	Rate (%)
Any text–audio–video pair	1,117	2,281	48.97
Text–audio	800	2,281	35.07
Text–video	945	2,281	41.43
Audio–video	602	2,281	26.39
Any pair: train	676	1,368	49.42
Any pair: validation	231	456	50.66

Appendix

Table 10: Public-label audit of cross-modal disagreement on CH-SIMS. All rows use Equation 34 , with a restricted modality pair for the three pairwise rows. Pairwise rows overlap and are not summed. The final three rows partition the full corpus by the released split.

Quantity	Reported evidence	Population and interpretation
Three-modal disagreement	$48.97\%$ (1,117/2,281)	Our public-label audit of CH-SIMS: at least two unimodal polarity labels differ; Table 10 .
Image–text inconsistency	$42.5\%$ / $26.0\%$	Filtered MVSA-Single / Multiple; neutral versus positive or negative modality labels ( Pan & Meng, 2024 ) .
Text–joint disagreement	$36.31\%$	UniC video reviews: complement of 63.69% text-only versus multimodal label agreement ( Du et al., 2025 ) .
Strong-conflict prevalence	$15.0\%$ (661/4,402)	Labeled CH-SIMS v2.0 clips screened at $\delta>1$ ; corpus frequency ( Wang & Wu, 2025 ) .
Conflict difficulty	$23.53$ pp	MulT binary accuracy: 89.13% aligned, 65.60% conflict; selected DiffEmo tests, 173 examples per group ( Wang & Wu, 2025 ) .
Primary benchmark mix	1,000/1,500 conflict	CA-MER’s equal allocation across two conflict directions and consistency; evaluation design ( Han et al., 2025 ) .

Appendix

Table 11 : Quantitative motivation with explicit denominators. A public-label audit, published corpus analyses, and selected evaluation subsets answer different questions. Arithmetic gaps are calculated from the cited values.

Figure 9 : A constructed illustration of the public set-matching metric. After semantic matching, three illustrative clips give equal-clip Acc

7/9

, recall

13/18

, and their arithmetic mean

3/4

. Pooling memberships instead gives

2/3

for both metrics. This public-protocol example is separate from the normalized-interface experiments reported in this paper.

Comparator	$\Delta_{\mathrm{conf}}$	$\Delta_{\mathrm{cons}}$	$S$
Emotion-SFT	4.5	1.2	3.3
Generic-CoT-SFT	3.0	0.2	2.8
Modality-Gate-SFT	2.5	0.2	2.3

Appendix

Table 12: CA-MER conflict specificity (pp). The same comparator is used for both gains. Contrasts are calculated before rounding the displayed accuracies.

Comparator	$\Delta_{v}$	$\Delta_{a}$	$\Delta_{\mathrm{cons}}$	$S_{v}$	$S_{a}$	$2d_{B}$
Emotion-SFT	6.0	3.0	1.2	4.8	1.8	3.0
Generic-CoT-SFT	4.0	2.0	0.2	3.8	1.8	2.0
Modality-Gate-SFT	3.0	2.0	0.2	2.8	1.8	1.0

Appendix

Table 13: Directional control contrasts (pp). Calculated from the displayed subgroup means in Table 14 . The final column is the reduction in the audio-minus-video disparity.

Method	Visual	Audio	Conflict Avg	Consistent	Overall	Invalid
Base	49.0	60.0	54.5	68.0	59.0	0.5
Emotion-SFT	55.0	65.0	60.0	73.0	64.3	0.2
Generic-CoT-SFT	57.0	66.0	61.5	74.0	65.7	0.4
Modality-Gate-SFT	58.0	66.0	62.0	74.0	66.0	0.4
VISTA	61.0	68.0	64.5	74.2	67.7	0.7

Appendix

Table 14: Complete CA-MER results (%). All five core models use Qwen2.5-Omni-7B and the same normalized output interface. Conflict Avg is

(A_{v}+A_{a})/2

Method	Conflict Avg (%)	VISTA gain
MoSEAR	61.5	3.0
CHASE	61.0	3.5
VISTA	64.5	—

Appendix

Table 15: External methods under the normalized CA-MER interface. These reevaluations are separate from the methods’ originally published open-vocabulary scores. Differences are in pp.

Method	Consistent	Conflict	Missing	Conflict + missing	Macro Avg
Base	52.1	46.5	43.3	38.0	45.0
Emotion-SFT	55.0	47.5	44.0	38.0	46.1
Generic-CoT-SFT	55.5	48.0	44.2	38.6	46.6
Modality-Gate-SFT	55.6	49.0	45.1	40.0	47.4
VISTA	56.3	52.0	46.8	43.5	49.7
CHASE (reevaluated)	53.2	51.3	49.3	44.5	49.6

Appendix

Table 16: Complete EmoMM accuracy (%). The five core models use frozen transfer. CHASE is a separate protocol reevaluation with prior EmoMM supervision. Macro Avg weights conditions equally; overlapping missingness conditions prevent interpreting it as full-sample overall accuracy.

Method	Q1	Q2	Q3	Q4	Overall Acc2	Q4 MAE $\downarrow$
Base	87.5	82.5	77.5	70.0	79.4	0.460
Emotion-SFT	90.0	86.5	82.5	77.0	84.0	0.365
Generic-CoT-SFT	89.8	86.7	82.8	77.8	84.3	0.355
Modality-Gate-SFT	89.7	86.8	83.5	79.2	84.8	0.337
VISTA	90.2	87.2	84.5	81.5	85.9	0.315

Appendix

Table 17: CH-SIMS v2.0 by conflict strength. Q1–Q4 use increasing conflict intensity with boundaries set on training data. Acc2 is binary accuracy in percent; Q4 MAE retains its original scale. Base is zero-shot; the four trained methods receive task adaptation.

Comparator	$g_{1}$	$g_{2}$	$g_{3}$	$g_{4}$	$E_{B}$	$\beta_{B}$
Emotion-SFT	0.2	0.7	2.0	4.5	4.3	1.42
Generic-CoT-SFT	0.4	0.5	1.7	3.7	3.3	1.11
Modality-Gate-SFT	0.5	0.4	1.0	2.3	1.8	0.60

Appendix

Table 18: CH-SIMS gains calculated from Table 17 . Gains and endpoint contrasts are in pp; slopes are in pp per group index.

Condition	Conflict	Consistent	$\Delta_{\mathrm{conf}}$	$S$
Full VISTA	64.5	74.2	0.0	3.3
Within-emotion appraisal shuffle	63.7	74.0	$-0.8$	2.7
Cross-sample appraisal shuffle	62.0	73.5	$-2.5$	1.5
Field-name randomization	64.2	74.2	$-0.3$	3.0
Auxiliary only; decision connection cut	62.5	74.0	$-2.0$	1.5
No expression regulation $r$	63.6	74.0	$-0.9$	2.6

Appendix

Table 19: Complete CA-MER mechanism controls. Accuracy is in percent; changes and specificity are in pp.

S

is always relative to Emotion-SFT. “Auxiliary only” retains appraisal supervision but cuts its decision connection. The

\lambda_{\mathrm{cf}}=0

row repeats the full configuration.

Method	Conflict subset	Visual weight	Audio weight
Modality-Gate-SFT	Visual-aligned	0.37	—
Modality-Gate-SFT	Audio-aligned	—	0.43
VISTA	Visual-aligned	0.42	0.30
VISTA	Audio-aligned	0.25	0.47

Appendix

Table 20: Reported native modality weights

\alpha

. Dashes mark gate weights for which no separate value is available; these entries are not inferred from normalization. These are weights on their original scale.

Prompt	Conflict Avg	Consistent	Invalid	Output tokens
Direct	54.5	68.0	0.5	8
Modality decomposition	55.8	69.0	0.8	210
Generic conflict CoT	56.4	69.0	1.0	290
Appraisal prompt	56.0	68.5	1.8	440

Appendix

Table 21: Complete frozen-model prompt comparison. Recognition and invalid-output rates are percentages; output length is the mean token count.

Figure 10 : What prompting supplies before training. (a) Conflict and consistent accuracy of four frozen-backbone prompts, with output length and invalid-output rates. (b) Grounded-appraisal rates of

52.0\%

for Direct and

60.0\%

for Appraisal prompt in a separate explanation audit. That audit locks the predicted label before eliciting the second-pass explanation.

Method	Direction	Label change	$\Delta p$	$\Delta$ log-odds	Rewrite stability
Emotion-SFT	62.0	24.0	0.050	0.201	91.0
Generic-CoT-SFT	66.0	28.0	0.070	0.281	90.0
Modality-Gate-SFT	63.0	25.0	0.055	0.221	91.0
VISTA	75.0	36.0	0.120	0.484	92.0

Appendix

Table 22: Complete paired intervention results. Direction, label change, and rewrite stability are percentages. Probability and log-odds differences retain their original scales. Valid interventions and irrelevant rewrites each contain 150 pairs.

Intervention type	Pairs	Direction compliance (%)
Goal congruence	25	84.0
Expression regulation	25	81.0
Coping / control	25	68.0
Social norm	25	69.0

Appendix

Table 23: Available VISTA intervention-type results. Each of the six types has 25 pairs; type-specific percentages are available for the four listed types. The remaining two types contribute to the 150-pair aggregate but have no separately reported rates.

Method	Novelty	Pleasantness	Goal	Coping	Macro CCC
Base	0.400	0.470	0.490	0.520	0.470
Emotion-SFT	0.440	0.500	0.530	0.550	0.505
Generic-CoT-SFT	0.470	0.530	0.560	0.570	0.533
Modality-Gate-SFT	0.470	0.530	0.560	0.580	0.535
VISTA	0.540	0.590	0.620	0.650	0.600

Appendix

Table 24: THERADIA appraisal readout: CCC by dimension. All models use the same four-output linear probe on a frozen backbone. Macro CCC averages the four dimensions.

Macro CCC $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$
0.600	0.086	0.111	0.658	0.625

Appendix

Table 25: Aggregate VISTA appraisal-probe metrics on THERADIA. All quantities retain their native numerical scale.

Figure 11 : From readable appraisal to useful appraisal. (a) Five frozen backbone representations under a common four-dimensional linear probe, evaluated by dimension-wise and macro CCC. (b) Downstream regression of ten emotion intensities under four appraisal conditions. Model-generated appraisal gives $0.450$ macro CCC, versus $0.415$ with auxiliary supervision alone; human appraisal is a privileged test-time reference.

Appraisal condition	Macro CCC	Gain over none
No appraisal	0.390	0.000
Auxiliary only; decision connection cut	0.415	0.025
Model-generated appraisal	0.450	0.060
Human appraisal (oracle)	0.480	0.090

Appendix

Table 26: THERADIA downstream emotion-intensity regression. The task predicts ten named emotion intensities; the metric is macro CCC. Human appraisal is an oracle condition, separate from inference using model-generated appraisal.

Candidate disposition	Count	Retained source	Count
Automatically accepted	2,872	CH-SIMS v2.0	2,123
Accepted after human adjudication	1,231	THERADIA	866
Retained with partial-field masking	3,897	MELD	5,011
Rejected	2,256
All candidates	10,256	All retained	8,000

Appendix

Table 27: Complete pseudo-appraisal data accounting. The left block partitions all candidates; the right block partitions the retained training set by source. Partial masking retains an example while masking unsupported fields.

Field	Valid $N$	Coverage	Sampling consistency	Grounded
Goal / concern $g$	240	60.0	80.0	76.0
Goal congruence $c$	296	74.0	85.0	81.0
Expectation $e$	172	43.0	76.0	67.0
Agency $a$	260	65.0	86.0	84.0
Coping / control $k$	192	48.0	81.0	73.0
Social norm $n$	136	34.0	78.0	68.0

Appendix

Table 28: Seven-field audit. Coverage is the valid count divided by 400 candidates. Sampling consistency and grounded proportion are percentages with their field-specific denominators. Multiple fields may be valid in the same example;

n

retains the reported field-level aggregate.

Figure 12 : Supervision from selection to decision quality. (a) Coverage, sampling consistency, grounded proportion, and valid count for all seven fields in the 400-candidate audit. (b) Disposition of all 10,256 candidates and dataset sources of the 8,000 retained examples. (c) Label-blind versus label-visible teacher diagnostics, with the latter trained in an isolated copy. Percentages, field-consistency scores, and mutual information retain their distinct units.

Teacher condition	Consistency	Grounded	Copying	MI (bits)	Dev Acc2	Conflict Acc2
Label-blind	0.53	73.0	3.0	0.120	84.5	80.5
Label-visible, isolated	0.65	67.0	21.0	0.240	85.0	81.2

Appendix

Table 29: Label-visibility diagnostic. Field consistency retains the reported score scale; grounded proportion, copying, and Acc2 are percentages. MI is the mean bias-corrected field–label mutual information in bits (Equation 51 ). Both development evaluations hide true labels.

Method	Params (M)	GPU-h	Peak GiB	P50 (s)	P95 (s)	Tokens
Emotion-SFT	10.12	32.0	42.0	1.2	4.0	8
Generic-CoT-SFT	10.12	40.0	45.0	13.5	26.0	360
Modality-Gate-SFT	12.19	64.0	52.0	6.0	13.0	96
VISTA	12.39	72.0	56.0	15.5	31.0	360

Appendix

Table 30: Complete resource accounting. Training columns concern the common stage; inference is measured on one A100 at batch size 1. GPU-hours are per seed and peak memory is GiB per GPU. P50 and P95 are inference latency percentiles in seconds.

Figure 13 : Resources and error groups. (a) All six reported training and inference resource measurements for the four trained methods. (b) The five available error-category shares, separated by the three paired-model error groups that define their denominators. These shares are not a complete error partition and are not pooled across groups.

Error group	Reported error category	Share (%)
Only Emotion-SFT wrong	Conflict-source identification	24.0
Only Emotion-SFT wrong	Expression regulation / sarcasm	22.0
Only VISTA wrong	Goal / concern	11.0
Only VISTA wrong	Goal congruence / Expectation	12.0
Both models wrong	Insufficient context	20.0

Appendix

Table 31: Available paired-error audit results. Each percentage uses the group in its first column as the denominator. The table lists the reported categories rather than a full partition; entries must not be summed across error groups or interpreted as corpus prevalence.

Method	Weighted-F1	Macro-F1	Accuracy	$\Delta$ wF1
Base	59.55	43.73	60.59	$-5.93$
Emotion-SFT	65.49	50.65	66.48	0.00
Generic-CoT-SFT	65.69	51.12	66.64	$+0.20$
Modality-Gate-SFT	65.74	50.84	66.73	$+0.25$
VISTA	66.94	52.89	67.85	$+1.46$

Appendix

Table 32: Complete MELD results (%). Differences are in pp relative to Emotion-SFT and are calculated before rounding, so they need not equal subtraction of the displayed rounded means.

Emotion	Emotion-SFT	VISTA
Disgust	18.0	21.0
Fear	20.0	23.0
Sadness	39.0	41.5
Neutral	—	81.7

Appendix

Table 33: Available MELD class-recall measurements (%). The table includes the available numerical recall values; a dash marks an unreported comparator value.

Setting	Final configuration
Backbone	Qwen2.5-Omni-7B Thinker
Frozen components	Audio encoder, visual encoder, and Talker
Retained training examples	8,000
Epochs / optimization steps	3 / 375
Effective batch size	64
LoRA targets	q_proj , k_proj , v_proj , o_proj in 28 layers

Appendix

Table 34: Common-stage training configuration. These settings apply to all four trained core methods. Additional task adaptation and the common THERADIA probe are specified separately.

Evaluation	Base	Four trained methods
CA-MER	Frozen Base	Common-stage trained models; normalized output interface
EmoMM	Frozen Base	Common-stage checkpoint; no EmoMM adaptation
CH-SIMS v2.0	Zero-shot	Separate task adaptation; train-defined conflict groups
MELD	Zero-shot	Separate task adaptation for seven emotion classes
THERADIA probe	Frozen backbone + common probe	Frozen backbone + common probe

Appendix

Table 35: Evaluation-stage use of the core models. The labels identify distinct experimental protocols rather than a single checkpoint condition for every task.

Method	CA conflict	CA consistent	EmoMM conflict	SIMS Q4	CCC	MELD wF1
Base	54.5	68.0	46.5	70.0	0.470	59.55
Emotion-SFT	60.0	73.0	47.5	77.0	0.505	65.49
Generic-CoT-SFT	61.5	74.0	48.0	77.8	0.533	65.69
Modality-Gate-SFT	62.0	74.0	49.0	79.2	0.535	65.74
VISTA	64.5	74.2	52.0	81.5	0.600	66.94

Appendix

Table 36: Complete six-metric overview. Classification and F1 values are percentages; THERADIA CCC retains its native scale. CA-MER conflict is the macro average of its two directions, CH-SIMS Q4 is binary accuracy, and THERADIA is the four-dimensional probe macro CCC.

Evidence family	Question and complete contents	Location
Core settings and definitions	Backbone, shared training, task adaptation, units, and six-metric overview	Tables 34 – 36 ; this section
CA-MER recognition	All subsets, overall accuracy, invalid output, external reevaluations	Tables 14 , 15
Conflict specificity	All three control contrasts and directional decomposition	Tables 12 , 13
EmoMM	All four conditions and CHASE reevaluation	Table 16
CH-SIMS v2.0	All conflict groups, overall Acc2, Q4 MAE	Table 17
MELD	Weighted-F1, macro-F1, accuracy, unrounded differences, class recall	Tables 32 , 33

Appendix

Table 37: A guide to the complete experimental evidence. All experiment families and their numerical results.

Explore similar work

Aug 8, 2026cs.LG

CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognition

Multimodal emotion recognition often treats self-reported labels as reliable supervision while overlooking self-report unreliability and cross-modal conflict. We propose \textbf{CONFER}, a graph-based conflict-aware evidence negotiation framework for weakly supervised multimodal emotion recognition. CONFER represents each modality expert as a node with a predictive belief, boundary-based uncertainty, and runtime reliability estimated from historical out-of-fold performance and current-sample uncertainty. Uncertainty-aware compatibility and reliability-directed asymmetric edge weights govern iterative message-passing negotiation, followed by peer-supported prediction readout. Conflict reduction, residual disagreement, and mean modality uncertainty further characterize three regimes---Consensus, Dissent, and Ambiguity---for sample-specific weak-label calibration. We evaluate CONFER on AMIGOS, MAHNOB-HCI, and DEAP under subject-dependent 10-fold and strict leave-one-subject-out (LOSO) protocols. CONFER achieves competitive performance, reaching \textbf{0.873} accuracy on AMIGOS-V and \textbf{0.854} accuracy on MAHNOB-V under strict LOSO evaluation. Further analyses show larger negotiation gains on high-conflict samples and improved robustness to weak-label corruption, indicating that cross-modal conflict provides useful information for both directional modality coordination and supervision-reliability estimation.

Bojing Hou, Ruohao Li, Yitong Zhu +2

The Hong Kong University of Science and Technology (Guangzhou)

Emotion Recognition Cross-Modal Conflicts Sessions

Ask PDF

May 6, 2026cs.MM

To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition

Multimodal emotion recognition (MER) benefits from combining text, audio, and vision, yet standard fusion often fails when modalities conflict. Crucially, conflicts differ in resolvability: benign conflicts stem from missing, weak, or ambiguous cues and can be mitigated by cross-modal calibration, while severe conflicts arise from intrinsically contradictory (e.g., sarcasm) or misleading signals, for which forced fusion may amplify errors. Recognizing this, we propose Dual-Path Conflict Resolution (DCR), a unified framework that learns when to fuse and when to drop modalities. Path I (Affective Fusion Distiller, AFD) performs reverse distillation from audio/visual teachers to a textual student using temporally weighted class evidence, thereby enhancing representation-level calibration and improving fusion when alignment is beneficial. Path II (Affective Discernment Agent, ADA) formulates MER as a contextual bandit that selects among fusion and unimodal predictions based on a dual-view state and a calibration-aware reward, enabling decision-level arbitration under irreconcilable conflicts without requiring per-modality reliability labels. By taking into account the full multimodal context and coupling soft calibration with hard arbitration, DCR reconciles conflicts that can be aligned while bypassing misleading modalities when fusion is harmful. Across five benchmarks covering both dialogue-level and clip-level MER, DCR consistently outperforms competitive baselines or achieves highly competitive results. Further ablations, conflict-specific subset evaluation, and modality-selection analysis verify that AFD and ADA are complementary and jointly improve robust conflict-aware emotion recognition.

Yangchen Yu, Qian Chen, Jia Li +5

School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Computing and Information Systems, Singapore Management University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore +1

Emotion Recognition Cross-Modal Modalities Ambiguity

Ask PDF

Sep 3, 2026cs.CL

Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory

Emotion recognition benchmarks often predict one emotion per text, missing many real-world scenarios where two people arrive at opposing emotions from a single shared event. For example, a child kicks the seat in front of her in excitement while the passenger ahead grows angry. We introduce CHIARO, a 1,000 human-annotated sentence benchmark for contrastive emotion inference grounded in appraisal theory. Each scene describes one causal trigger eliciting a positive emotion in one person and a negative emotion in the other, drawn from a ten-class taxonomy. We benchmark seven frontier LLMs and four off-the-shelf emotion classifiers. The strongest LLM reaches 67.3 macro-F1, well below human agreement, while existing emotion classifiers score near chance. Beyond evaluation, CHIARO also serves as a training signal. When combined with an existing emotion corpus, the resulting downstream classifier improves on CHIARO itself and on six of ten external emotion benchmarks, which positions our dataset as a complementary signal for emotion recognition.

Divyesh Bommana, Mohammad Saim, Tianyu Jiang

University of Cincinnati

Emotion Recognition Emotion Subjectivity

Ask PDF