Small multimodal models can misjudge emotion when words, voice, and visual cues conflict. We present EmoHarness, which runs such a model, the frozen student, inside a harness: the prompts, settings, and verification code around its calls. A human-written starting harness, the scaffold, numbers the student's visual observations, transcript, and prosody descriptions as citable evidence and has the student weigh candidate emotions against it. In Meta-Harness Evolving (MHE), a stronger LLM, the Builder, reads the student's errors and execution traces offline and edits the harness; static checks and validation gates decide which edits are kept, and the Builder never sees evaluation items. On 5,507 MME-Emotion items, EmoHarness raises task-macro Recognition by 7.6-13.4 points over a single video-only call for all four students and both Builders. Transcripts and prosody descriptions supply most of this boost (+5.38 to +10.71); the scaffold changes the audio-augmented call by -0.79 to +2.54, partly through extra multi-label predictions, and evolved edits add at most +0.95, although no evolved increment survives Holm correction. The harness's emotion context also transfers to IntentQA, with small gains for three of four students, none of which survives correction. EmoHarness thus boosts small multimodal students chiefly by supplying and organizing audio-derived evidence.
Figures & tables
Figure 1: Emo Harness at a glance. (a) A selected held-out clip (Gemma-4-E4B; Figure 4 a): posture suggests neutral , the words negative . Video, Audio (video plus transcript and prosody text), and Shared (the human-written scaffold) miss the reference; both MHE harnesses answer negative . (b) Offline, a Builder LLM (GPT-5.6 Luna or GPT-6.1 Sol) edits the harness from the frozen student’s failures and execution traces; checks and validation gates promote edits over four generations. (c) Task-macro Recognition contrasts (VL: Qwen3-VL; Q3.5: Qwen3.5-4B).
Figure 2: How Meta-Harness Evolving builds a harness and how the frozen student runs it. (a) Offline, the Builder turns the incumbent’s construction failures and traces into at most four candidate edits per generation; static checks, a proposal reviewer, and the gates of Eq. 3 promote the best candidate or retain the incumbent, and after four generations the harness is frozen. (b) At inference the frozen student makes four calls (at most six with retries) around deterministic verification and a fixed controller. The example is the fine-grained sentiment (FG-SA) clip of Appendix Figure 11 a: the evidence and the quoted output come from the GPT-5.6 Luna harness on Qwen3-VL-4B; the Propose rows, the confusions in (a), and c1–c4 are illustrative.
Seen task types
Held-out task types
Summary
ER-
ER-
Noise-
FG-
ML-
FG-
Held-
Lab
Wild
SA
ER
ER
ER
SA
IR
Seen
out
Macro
95% CI
Video
8.5
36.2
41.4
56.4
23.6
38.8
13.2
21.5
28.7
30.7
29.95
28.8–31.1
Audio
23.2
38.0
63.3
65.0
19.2
41.7
30.8
32.0
41.5
37.7
39.15
37.8–40.5
Shared
28.8
35.5
59.9
56.6
26.1
48.9
26.7
41.2
41.4
39.9
40.47
39.0–41.9
MHE GPT-5.6 Luna
29.4
35.0
60.6
57.2
26.2
48.7
27.8
40.6
41.7
40.1
40.68
39.2–42.1
Table 1: Main results on MME-Emotion. Recognition (%) per task type and task-macro averages (Seen, Held-out, Macro) on 5,507 items, for each student under Video and Audio (one call each), the Shared scaffold, and the harnesses evolved by Meta-Harness Evolving (MHE) with each Builder (about four calls each). Seen task types occur in construction; held-out types do not. FG-ER and ML-ER use recall-like Recognition. Shaded rows: MHE; bold : best per student and column (ties unmarked). 95% CI: task-stratified bootstrap interval of the macro score (10,000 resamples). † No candidate passed the gates, so the harness equals Shared. No MHE − Shared difference survives Holm correction over eight tests (smallest adjusted p=.192 ). Differences reported in the text and figures use unrounded scores and can differ by 0.01 (0.1 in one-decimal columns) from differences of the displayed values.
Figure 3: Recognition paths per student. (a) Seen- versus held-out-type macro Recognition (%; Table 1 ): arrows run Video (open circle) → Audio (amber diamond) → Shared (gray); amber numbers beside each name: Audio − Video macro gain (pp); dashed lines: equal overall Macro. (b) MHE − Shared, zoomed (origin = Shared; ticks 10 × finer than in (a)); violet circle: GPT-5.6 Luna; blue square: GPT-6.1 Sol; changes below 0.1 points have no arrow; labels and dashed lines give the macro change. † Harness equals Shared. Short names and rounding as in Table 2 .
Figure 4: Case study: MHE versus baselines on Gemma-4-E4B. The input is drawn once: instruction, transcript, and five of our seven sampled frames per clip (the student saw 2 fps), subtitled with the words spoken within 0.12 s of each frame (every condition except Video receives the transcript); lettered boxes mark the regions that cards cite. Each card gives one condition’s answer (green: correct; red: wrong), and the tags between cards name what each condition adds. Note glyphs: green check (supported); red cross (misread, unsupported, or wrong); amber note (audio-derived); gray dash (limitation). The last MHE note gives the answer of the GPT-6.1 Sol harness; [v i ]/[a j ]: evidence IDs. The edit panel quotes the rule GPT-5.6 Luna added to the review prompt, with verbatim Shared and MHE excerpts. Cases are selected, not representative.
VL-8B
VL-4B
Q3.5
Gemma
Frames
70.99
67.48
79.90
57.31
Audio
72.63
68.42
77.84
60.59
+ Emotion MHE Luna
73.10
70.29
76.34 †
61.62
+ Emotion MHE Sol
72.87
69.92
76.43
61.95
Table 2: Downstream QA accuracy (%). Inputs as in Section 4.3 ; 2,134 IntentQA test questions. Shaded rows add MHE emotion context; bold : best per student (ties unmarked). Luna: GPT-5.6 Luna; Sol: GPT-6.1 Sol. VL-8B/VL-4B: Qwen3-VL-8B/4B; Q3.5: Qwen3.5-4B; Gemma: Gemma-4-E4B. No Audio+Emotion − Audio contrast survives Holm correction over eight tests (smallest adjusted p=.085 ; Appendix Table 5 ). Differences reported in the text and figures use unrounded scores and can differ by 0.01 from differences of the displayed values. † Retained scaffold: this context is generated by Shared.
Figure 5: Emotion context transfers to IntentQA. Accuracy change (pp) from adding the harness’s emotion context to Audio. Bars: mean over 4 students × 2 Builders (green: gain; red: loss); dots: the 8 individual changes. Rows: all questions, then the official types (temporal: TN+TP). The mean change is positive for every type; none survives Holm correction (Appendix Table 6 ).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Dv Rec. (%)
Seen
Student
Scored
Prom.
Start
Final
eval.
USD
VL-8B
14
2
44.92
47.46
+0.10
0.41
VL-4B
10
2
45.94
47.97
+0.20
0.49
Q3.5
10
0
41.12
41.12
0.00 †
0.23
GPT-5.6 Luna
Gemma
12
2
42.89
43.65
+0.10
0.40
VL-8B
12
1
44.92
45.94
+0.30
1.51
Appendix
Table 3: MHE construction records. Validation Recognition is a flat item mean over 394 items, distinct from evaluation task-macro Recognition. Every run has four generations. Scored: proposals scored on Dv ; Prom.: promotions. Proposals screened out by static checks or the proposal reviewer before validation: 5 (GPT-5.6 Luna) and 9 (GPT-6.1 Sol). USD covers logged Builder, reviewer, and proxy-judge calls, excluding local GPU work. Seen eval.: gain over Shared as a flat item mean on the 2,963 seen-type evaluation items, comparable to Final − Start. VL-8B/VL-4B: Qwen3-VL-8B/4B; Q3.5: Qwen3.5-4B; Gemma: Gemma-4-E4B. † Retained scaffold with identical labels.
Figure 6: MHE trajectories on the validation set. Incumbent validation Recognition (item mean over 394 items) per generation for each Builder; markers are candidates scored on validation (filled: promoted; open: rejected by the gates); × marks proposals screened out by the checks or the reviewer, drawn at the panel floor and never scored. S: Shared scaffold; diamond: run with no promotion.
Figure 7: Validation gains overstate evaluation gains. Gain of each evolved harness over Shared on validation (item mean over 394 items) and on evaluation (task macro over 5,507 items). Connected points describe the same harness under two summaries, not two estimates of one quantity.
Builder
Δ
95% CI
pHolm
GPT-5.6 Luna
+0.21
−0.18 to +0.59
1.000
VL-8B
GPT-6.1 Sol
+0.06
−0.20 to +0.29
1.000
GPT-5.6 Luna
+0.95
+0.12 to +1.82
.192
VL-4B
GPT-6.1 Sol
+0.25
+0.01 to +0.49
.262
GPT-5.6 Luna
0.00 †
0.00 to 0.00
1.000
Q3.5
GPT-6.1 Sol
+0.03
−0.10 to +0.16
1.000
Appendix
Table 4: Recognition contrasts with intervals. Task-macro Recognition differences (pp) with pointwise 95% intervals. (a) MHE − Shared, with task-stratified paired bootstrap intervals (10,000 resamples) and Holm-adjusted p . No difference survives Holm correction over eight tests (smallest adjusted p=.192 ). † Retained scaffold with identical labels. (b) Audio − Video and Shared − Audio, with intervals from the same procedure (pointwise; not part of a Holm family). VL-8B/VL-4B: Qwen3-VL-8B/4B; Q3.5: Qwen3.5-4B; Gemma: Gemma-4-E4B.
Figure 8: Per-task MHE − Shared changes. Differences (pp) for each Builder. Both panels share one zero-centered color scale, and the horizontal rule separates the three seen task types from the five held-out types. Exact scores are in Table 1 .
Figure 9: Decomposing recognition gains. Task-macro contrasts (pp) on 5,507 MME-Emotion items. (a) Audio-derived text adds 5.4–10.7 points, and the Shared scaffold changes Audio by between −0.8 and +2.5 . (b) MHE adds at most +0.95 over Shared (note the narrower axis), and no contrast survives Holm correction over eight tests. Whiskers are pointwise 95% paired bootstrap intervals; the diamond marks the run that retained the scaffold.
Student
Builder
Δ
95% CI
pHolm
VL-8B
GPT-5.6 Luna
+0.47
−1.15 to +2.05
1.000
GPT-6.1 Sol
+0.23
−1.33 to +1.77
1.000
VL-4B
GPT-5.6 Luna
+1.87
+0.47 to +3.27
.085
GPT-6.1 Sol
+1.50
+0.05 to +2.97
.344
Q3.5
GPT-5.6 Luna
−1.50 †
−3.05 to +0.09
.395
GPT-6.1 Sol
−1.41
−2.91 to +0.15
.426
Appendix
Table 5: Effect of emotion context on downstream QA. Audio+Emotion − Audio accuracy (pp) on the 2,134 IntentQA test questions, with pointwise 95% intervals from a paired bootstrap clustered by video (10,000 resamples). No contrast survives Holm correction over eight tests (smallest adjusted p=.085 ). † Retained scaffold: this context is generated by Shared.
Figure 10: Emotion context added to Audio on IntentQA. Accuracy change (pp); whiskers: pointwise 95% paired intervals clustered by video. No contrast survives Holm correction (eight tests). Frames and audio text are identical in both conditions; the gray diamond (Qwen3.5-4B, GPT-5.6 Luna) is the retained Shared scaffold’s context.
VL-8B
VL-4B
Q3.5
Gemma
Stratum
Luna
Sol
Luna
Sol
Luna
Sol
Luna
Sol
Why (CW)
+1.04
+0.48
+1.36
+1.04
−1.68
−1.76
+0.96
+1.44
n=1,250
−1.0 to +3.1
−1.5 to +2.4
−0.6 to +3.3
−0.9 to +2.9
−3.6 to +0.2
−3.6 to +0.2
−1.1 to +3.2
−0.6 to +3.6
How (CH)
−2.23
−2.51
+2.51
+3.90
−0.84
−0.56
+0.84
+1.11
n=359
−6.0 to +1.4
−6.2 to +1.2
−0.6 to +5.7
+0.3 to +7.4
−4.6 to +2.8
−4.3 to +3.1
−2.9 to +4.6
−2.6 to +4.9
Temporal
+0.95
+1.52
+2.67
+0.95
−1.52
−1.14
+1.33
+1.33
Appendix
Table 6: Emotion context by IntentQA question type. Audio+Emotion − Audio accuracy (pp) within exploratory strata given by the official types of the 2,134 test questions, never by outcomes: causal-why (CW), causal-how (CH), and temporal (TN+TP, next and previous); every stratum is reported. Gray: pointwise 95% interval from a paired bootstrap clustered by video (10,000 resamples); n : questions in the stratum. No contrast survives Holm correction within its stratum (8 tests: 4 students × 2 Builders; smallest adjusted p=.278 ). Luna: GPT-5.6 Luna; Sol: GPT-6.1 Sol. VL-8B/VL-4B: Qwen3-VL-8B/4B; Q3.5: Qwen3.5-4B; Gemma: Gemma-4-E4B.
Figure 11: Additional cases. Selected, not representative; letters, note glyphs, and subtitles as in Figure 4 (no headline tags or edit panels); merged cards report two conditions that agree. (a) Qwen3-VL-4B, FG-SA: the GPT-5.6 Luna harness reaches the reference negative ; GPT-6.1 Sol’s does not. (b) Qwen3-VL-8B, ER-Lab: the correct Excited fits the evolved ranking code, while the written explanation favors Happiness .
Emotion recognition in conversation has been widely studied, but applying Large Language Models (LLMs) to continuous dimensional emotion evaluation in multimodal dialogue remains largely unexplored. We propose an LLM-based framework that performs discrete emotion recognition and Valence-Arousal-Dominance (VAD) dimensional evaluation on IEMOCAP, incorporating acoustic cues as natural language descriptions following the SpeechCueLLM approach. We evaluate six models spanning the LLaMA, GPT, and Qwen families under zero-shot prompting, few-shot prompting, and LoRA fine-tuning. LoRA fine-tuned LLaMA models substantially outperform prompt-engineered GPT models on both tasks despite GPT's larger scale, a gap we attribute to domain adaptation rather than model capacity. Our best model achieves a Valence CCC of 0.7822, a new state-of-the-art on IEMOCAP. Ablation studies confirm that textual audio descriptions meaningfully improve smaller models (+3.5 to 3.6 weighted F1) while contributing little for the largest model, suggesting audio cues are most valuable when linguistic capacity is limited. The performance asymmetry across VAD dimensions closely mirrors the annotator agreement hierarchy in IEMOCAP's own annotations.
Multimodal Emotion Recognition in Conversation (MERC) combines lexical, acoustic, visual and conversational cues, but accepting audio and video does not establish their effective or robust use. We audit off-the-shelf Omni models while holding the target transcript and prior textual context fixed and adding, removing or replacing target audio and video. In the primary evaluation of Qwen2.5-Omni-7B, Baichuan-Omni-1.5 and MiniCPM-o 2.6 on MELD and the full five-session six-class IEMOCAP set, T+A improves Accuracy in all six model-dataset pairs, with four gains significant after false discovery rate (FDR) correction. Across three source mappings, unrelated A/V (RandA/V) lowers Accuracy relative to Full in all 18 comparisons; all 18 remain significant under dialogue-cluster inference and BH-FDR correction, both on common-valid pairs and when invalid/OOM outcomes are treated as incorrect. Comparisons with modality-omission baselines further distinguish replacement penalties from mismatch-specific harm. A Qwen/MELD case study finds significantly greater gold-vs-best-candidate margin recovery in Late than Early or Middle under decision-query audio knockout, relative to an edge-count-matched Video-Sham control, using Holm-corrected tests. These results distinguish genuine-modality benefit from sensitivity and vulnerability to misleading input.
We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge, which aims to predict six continuous emotion intensity dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy, from in-the-wild multimodal video clips. We propose a staged multimodal framework that combines textual, acoustic, and visual representations, with an optional motion branch. Our approach first trains modality-specific encoders independently and then fuses their learned representations through a lightweight regressor with modality dropout and controlled encoder adaptation. Across our submitted systems, the best validation performance is obtained by the text--audio--vision--motion fusion model under the expanded 4:1 split, achieving an average Pearson correlation of 0.4722. Although the motion branch yields only very slight gains, its behavior can be interesting to study. Our team was placed third in the EMI challenge, achieving an average Pearson correlation of 0.57 for the test set. Overall, we provide a practical and reproducible baseline for EMI prediction.