Small multimodal models can misjudge emotion when words, voice, and visual cues conflict. We present EmoHarness, which runs such a model, the frozen student, inside a harness: the prompts, settings, and verification code around its calls. A human-written starting harness, the scaffold, numbers the student's visual observations, transcript, and prosody descriptions as citable evidence and has the student weigh candidate emotions against it. In Meta-Harness Evolving (MHE), a stronger LLM, the Builder, reads the student's errors and execution traces offline and edits the harness; static checks and validation gates decide which edits are kept, and the Builder never sees evaluation items. On 5,507 MME-Emotion items, EmoHarness raises task-macro Recognition by 7.6-13.4 points over a single video-only call for all four students and both Builders. Transcripts and prosody descriptions supply most of this boost (+5.38 to +10.71); the scaffold changes the audio-augmented call by -0.79 to +2.54, partly through extra multi-label predictions, and evolved edits add at most +0.95, although no evolved increment survives Holm correction. The harness's emotion context also transfers to IntentQA, with small gains for three of four students, none of which survives correction. EmoHarness thus boosts small multimodal students chiefly by supplying and organizing audio-derived evidence.
Figures & tables
Figure 1: Emo Harness at a glance. (a) A selected held-out clip (Gemma-4-E4B; Figure 4 a): posture suggests neutral , the words negative . Video, Audio (video plus transcript and prosody text), and Shared (the human-written scaffold) miss the reference; both MHE harnesses answer negative . (b) Offline, a Builder LLM (GPT-5.6 Luna or GPT-6.1 Sol) edits the harness from the frozen student’s failures and execution traces; checks and validation gates promote edits over four generations. (c) Task-macro Recognition contrasts (VL: Qwen3-VL; Q3.5: Qwen3.5-4B).
Figure 2: How Meta-Harness Evolving builds a harness and how the frozen student runs it. (a) Offline, the Builder turns the incumbent’s construction failures and traces into at most four candidate edits per generation; static checks, a proposal reviewer, and the gates of Eq. 3 promote the best candidate or retain the incumbent, and after four generations the harness is frozen. (b) At inference the frozen student makes four calls (at most six with retries) around deterministic verification and a fixed controller. The example is the fine-grained sentiment (FG-SA) clip of Appendix Figure 11 a: the evidence and the quoted output come from the GPT-5.6 Luna harness on Qwen3-VL-4B; the Propose rows, the confusions in (a), and c1–c4 are illustrative.
Seen task types
Held-out task types
Summary
ER-
ER-
Noise-
FG-
ML-
FG-
Held-
Lab
Wild
SA
ER
ER
ER
SA
IR
Seen
out
Macro
95% CI
Video
8.5
36.2
41.4
56.4
23.6
38.8
13.2
21.5
28.7
30.7
29.95
28.8–31.1
Audio
23.2
38.0
63.3
65.0
19.2
41.7
30.8
32.0
41.5
37.7
39.15
37.8–40.5
Shared
28.8
35.5
59.9
56.6
26.1
48.9
26.7
41.2
41.4
39.9
40.47
39.0–41.9
MHE GPT-5.6 Luna
29.4
35.0
60.6
57.2
26.2
48.7
27.8
40.6
41.7
40.1
40.68
39.2–42.1
Table 1: Main results on MME-Emotion. Recognition (%) per task type and task-macro averages (Seen, Held-out, Macro) on 5,507 items, for each student under Video and Audio (one call each), the Shared scaffold, and the harnesses evolved by Meta-Harness Evolving (MHE) with each Builder (about four calls each). Seen task types occur in construction; held-out types do not. FG-ER and ML-ER use recall-like Recognition. Shaded rows: MHE; bold : best per student and column (ties unmarked). 95% CI: task-stratified bootstrap interval of the macro score (10,000 resamples). † No candidate passed the gates, so the harness equals Shared. No MHE − Shared difference survives Holm correction over eight tests (smallest adjusted p=.192 ). Differences reported in the text and figures use unrounded scores and can differ by 0.01 (0.1 in one-decimal columns) from differences of the displayed values.
Figure 3: Recognition paths per student. (a) Seen- versus held-out-type macro Recognition (%; Table 1 ): arrows run Video (open circle) → Audio (amber diamond) → Shared (gray); amber numbers beside each name: Audio − Video macro gain (pp); dashed lines: equal overall Macro. (b) MHE − Shared, zoomed (origin = Shared; ticks 10 × finer than in (a)); violet circle: GPT-5.6 Luna; blue square: GPT-6.1 Sol; changes below 0.1 points have no arrow; labels and dashed lines give the macro change. † Harness equals Shared. Short names and rounding as in Table 2 .
Figure 4: Case study: MHE versus baselines on Gemma-4-E4B. The input is drawn once: instruction, transcript, and five of our seven sampled frames per clip (the student saw 2 fps), subtitled with the words spoken within 0.12 s of each frame (every condition except Video receives the transcript); lettered boxes mark the regions that cards cite. Each card gives one condition’s answer (green: correct; red: wrong), and the tags between cards name what each condition adds. Note glyphs: green check (supported); red cross (misread, unsupported, or wrong); amber note (audio-derived); gray dash (limitation). The last MHE note gives the answer of the GPT-6.1 Sol harness; [v i ]/[a j ]: evidence IDs. The edit panel quotes the rule GPT-5.6 Luna added to the review prompt, with verbatim Shared and MHE excerpts. Cases are selected, not representative.
VL-8B
VL-4B
Q3.5
Gemma
Frames
70.99
67.48
79.90
57.31
Audio
72.63
68.42
77.84
60.59
+ Emotion MHE Luna
73.10
70.29
76.34 †
61.62
+ Emotion MHE Sol
72.87
69.92
76.43
61.95
Table 2: Downstream QA accuracy (%). Inputs as in Section 4.3 ; 2,134 IntentQA test questions. Shaded rows add MHE emotion context; bold : best per student (ties unmarked). Luna: GPT-5.6 Luna; Sol: GPT-6.1 Sol. VL-8B/VL-4B: Qwen3-VL-8B/4B; Q3.5: Qwen3.5-4B; Gemma: Gemma-4-E4B. No Audio+Emotion − Audio contrast survives Holm correction over eight tests (smallest adjusted p=.085 ; Appendix Table 5 ). Differences reported in the text and figures use unrounded scores and can differ by 0.01 from differences of the displayed values. † Retained scaffold: this context is generated by Shared.
Figure 5: Emotion context transfers to IntentQA. Accuracy change (pp) from adding the harness’s emotion context to Audio. Bars: mean over 4 students × 2 Builders (green: gain; red: loss); dots: the 8 individual changes. Rows: all questions, then the official types (temporal: TN+TP). The mean change is positive for every type; none survives Holm correction (Appendix Table 6 ).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Dv Rec. (%)
Seen
Student
Scored
Prom.
Start
Final
eval.
USD
VL-8B
14
2
44.92
47.46
+0.10
0.41
VL-4B
10
2
45.94
47.97
+0.20
0.49
Q3.5
10
0
41.12
41.12
0.00 †
0.23
GPT-5.6 Luna
Gemma
12
2
42.89
43.65
+0.10
0.40
VL-8B
12
1
44.92
45.94
+0.30
1.51
Appendix
Table 3: MHE construction records. Validation Recognition is a flat item mean over 394 items, distinct from evaluation task-macro Recognition. Every run has four generations. Scored: proposals scored on Dv ; Prom.: promotions. Proposals screened out by static checks or the proposal reviewer before validation: 5 (GPT-5.6 Luna) and 9 (GPT-6.1 Sol). USD covers logged Builder, reviewer, and proxy-judge calls, excluding local GPU work. Seen eval.: gain over Shared as a flat item mean on the 2,963 seen-type evaluation items, comparable to Final − Start. VL-8B/VL-4B: Qwen3-VL-8B/4B; Q3.5: Qwen3.5-4B; Gemma: Gemma-4-E4B. † Retained scaffold with identical labels.
Figure 6: MHE trajectories on the validation set. Incumbent validation Recognition (item mean over 394 items) per generation for each Builder; markers are candidates scored on validation (filled: promoted; open: rejected by the gates); × marks proposals screened out by the checks or the reviewer, drawn at the panel floor and never scored. S: Shared scaffold; diamond: run with no promotion.
Figure 7: Validation gains overstate evaluation gains. Gain of each evolved harness over Shared on validation (item mean over 394 items) and on evaluation (task macro over 5,507 items). Connected points describe the same harness under two summaries, not two estimates of one quantity.
Builder
Δ
95% CI
pHolm
GPT-5.6 Luna
+0.21
−0.18 to +0.59
1.000
VL-8B
GPT-6.1 Sol
+0.06
−0.20 to +0.29
1.000
GPT-5.6 Luna
+0.95
+0.12 to +1.82
.192
VL-4B
GPT-6.1 Sol
+0.25
+0.01 to +0.49
.262
GPT-5.6 Luna
0.00 †
0.00 to 0.00
1.000
Q3.5
GPT-6.1 Sol
+0.03
−0.10 to +0.16
1.000
Appendix
Table 4: Recognition contrasts with intervals. Task-macro Recognition differences (pp) with pointwise 95% intervals. (a) MHE − Shared, with task-stratified paired bootstrap intervals (10,000 resamples) and Holm-adjusted p . No difference survives Holm correction over eight tests (smallest adjusted p=.192 ). † Retained scaffold with identical labels. (b) Audio − Video and Shared − Audio, with intervals from the same procedure (pointwise; not part of a Holm family). VL-8B/VL-4B: Qwen3-VL-8B/4B; Q3.5: Qwen3.5-4B; Gemma: Gemma-4-E4B.
Figure 8: Per-task MHE − Shared changes. Differences (pp) for each Builder. Both panels share one zero-centered color scale, and the horizontal rule separates the three seen task types from the five held-out types. Exact scores are in Table 1 .
Figure 9: Decomposing recognition gains. Task-macro contrasts (pp) on 5,507 MME-Emotion items. (a) Audio-derived text adds 5.4–10.7 points, and the Shared scaffold changes Audio by between −0.8 and +2.5 . (b) MHE adds at most +0.95 over Shared (note the narrower axis), and no contrast survives Holm correction over eight tests. Whiskers are pointwise 95% paired bootstrap intervals; the diamond marks the run that retained the scaffold.
Student
Builder
Δ
95% CI
pHolm
VL-8B
GPT-5.6 Luna
+0.47
−1.15 to +2.05
1.000
GPT-6.1 Sol
+0.23
−1.33 to +1.77
1.000
VL-4B
GPT-5.6 Luna
+1.87
+0.47 to +3.27
.085
GPT-6.1 Sol
+1.50
+0.05 to +2.97
.344
Q3.5
GPT-5.6 Luna
−1.50 †
−3.05 to +0.09
.395
GPT-6.1 Sol
−1.41
−2.91 to +0.15
.426
Appendix
Table 5: Effect of emotion context on downstream QA. Audio+Emotion − Audio accuracy (pp) on the 2,134 IntentQA test questions, with pointwise 95% intervals from a paired bootstrap clustered by video (10,000 resamples). No contrast survives Holm correction over eight tests (smallest adjusted p=.085 ). † Retained scaffold: this context is generated by Shared.
Figure 10: Emotion context added to Audio on IntentQA. Accuracy change (pp); whiskers: pointwise 95% paired intervals clustered by video. No contrast survives Holm correction (eight tests). Frames and audio text are identical in both conditions; the gray diamond (Qwen3.5-4B, GPT-5.6 Luna) is the retained Shared scaffold’s context.
VL-8B
VL-4B
Q3.5
Gemma
Stratum
Luna
Sol
Luna
Sol
Luna
Sol
Luna
Sol
Why (CW)
+1.04
+0.48
+1.36
+1.04
−1.68
−1.76
+0.96
+1.44
n=1,250
−1.0 to +3.1
−1.5 to +2.4
−0.6 to +3.3
−0.9 to +2.9
−3.6 to +0.2
−3.6 to +0.2
−1.1 to +3.2
−0.6 to +3.6
How (CH)
−2.23
−2.51
+2.51
+3.90
−0.84
−0.56
+0.84
+1.11
n=359
−6.0 to +1.4
−6.2 to +1.2
−0.6 to +5.7
+0.3 to +7.4
−4.6 to +2.8
−4.3 to +3.1
−2.9 to +4.6
−2.6 to +4.9
Temporal
+0.95
+1.52
+2.67
+0.95
−1.52
−1.14
+1.33
+1.33
Appendix
Table 6: Emotion context by IntentQA question type. Audio+Emotion − Audio accuracy (pp) within exploratory strata given by the official types of the 2,134 test questions, never by outcomes: causal-why (CW), causal-how (CH), and temporal (TN+TP, next and previous); every stratum is reported. Gray: pointwise 95% interval from a paired bootstrap clustered by video (10,000 resamples); n : questions in the stratum. No contrast survives Holm correction within its stratum (8 tests: 4 students × 2 Builders; smallest adjusted p=.278 ). Luna: GPT-5.6 Luna; Sol: GPT-6.1 Sol. VL-8B/VL-4B: Qwen3-VL-8B/4B; Q3.5: Qwen3.5-4B; Gemma: Gemma-4-E4B.
Figure 11: Additional cases. Selected, not representative; letters, note glyphs, and subtitles as in Figure 4 (no headline tags or edit panels); merged cards report two conditions that agree. (a) Qwen3-VL-4B, FG-SA: the GPT-5.6 Luna harness reaches the reference negative ; GPT-6.1 Sol’s does not. (b) Qwen3-VL-8B, ER-Lab: the correct Excited fits the evolved ranking code, while the written explanation favors Happiness .