Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio
Authors: Siddharth Ravi, Anders R. Bargum, Hugo Anselme, Patrizio Cugia di Sant'Orsola, Alberto Cardaci, Flora Kincso Parragi, Vincent Borowiec, Line H. Clemmensen
Organizations: Interhuman AI · Universidad de Alicante · École Polytechnique · Institut Supérieure d’Electronique de Paris · University of Copenhagen
Recognizing social signals such as interest and hesitation requires interpreting context-dependent behaviors that may co-occur in conversation. Existing affective computing benchmarks rarely evaluate such co-occurring signals, and conventional multi-label F1 overlooks semantic relationships and annotator disagreement. We introduce Social Signal Bench, a 47-hour densely annotated benchmark for the evaluation of social signal recognition and engagement, comprising two datasets: Social Signal Bench Video (25 hours), covering ten social signals in audiovisual recordings, and Social Signal Bench Audio (22 hours), covering six signals in audio-only recordings. Each dataset pairs a public training set of online conversational video, of which 10 hours are expert-annotated and the remainder pseudo-labeled, with a private expert-labeled test set spanning four conversational domains. We also propose Evidence-Weighted F1 (EW-F1), which weights reference labels by annotator evidence and assigns graded credit to semantically related predictions through optimal per-clip matching. Controlled perturbations demonstrate that EW-F1 distinguishes semantically related errors from unrelated ones, unlike conventional F1. Under a shared evaluation protocol, 7B open models post-trained on Social Signal Bench outperform general-purpose frontier models on video and match them on audio.
Figures & tables
Figure 1: Several social signals annotated on one minute of conversation. Three trained raters mark when each signal occurs. Each line is one rater’s span: dark lines mean the rater was highly confident, light lines moderately confident. The blue band shows how strong the combined evidence is at each moment, and the bars on the right give each signal’s overall evidence weight (Sec. 3.4 ). Evidence weights keep all four signals, and one highly confident rater is enough for full weight. Engagement is labeled separately and not shown.
Figure 2: Social Signal Bench construction and evaluation. Each card is a pipeline step, and its grey text describes that step. Expert annotations (top lane) pass through taxonomy-guided review and an evidence merge that gives each reference signal a weight pg . A teacher trained on proprietary data annotated with the same protocol pseudo-labels additional online video for the public training set (bottom lane). The private test sets hold expert labels only. EW-F1 uses the same taxonomy for graded credit and weights each reference signal by pg .
Dataset
Target
Unit and scale
Modalities
Reference and disagreement
IEMOCAP
Categorical and dimensional emotion [ 9 ]
10,039 turns; ∼ 12 h; 10 actors
AV, text, mocap
Multiple raters; categorical agreement reported
MELD
Emotion, sentiment [ 49 ]
13,708 utterances; TV dialogue
AV, text
Three raters; majority label; Fleiss’ κ=0.43
CMU-MOSEI
Sentiment, six emotion intensities [ 5 ]
23,453 sentences; 1,000 speakers
AV, text
Three crowd raters; averaged ratings
M3ED
Seven emotions, blends [ 69 ]
24,449 utterances; 56 TV series
AV, text
Three raters; majority; Fleiss’ κ=0.59
RECOLA
Valence, arousal, social behavior [ 51 ]
3.8 h annotated; 46 participants
AV, ECG, EDA
Six raters; continuous traces
NoXi
Engagement, interaction descriptors [ 10 ]
84 sessions; 25 h
AV, depth
Continuous and discrete annotations
Table 1: Comparison with related multimodal corpora. Units differ between rows (turns, utterances, sentences, sessions, clips), so the table compares task and annotation design, not scale or difficulty.
Figure 4
Split
Labels
Clips
Hours
Sources
Empty
Social Signal Bench Video
Train (public)
Expert
4,321
10.0
319
37.2%
Train (public)
Pseudo
3,480
9.5
453
19.0%
Test (private)
Expert
2,372
5.6
85
47.2%
Total
10,173
25.1
857
33.3%
Social Signal Bench Audio
Table 2: Composition of Social Signal Bench. The training sets are public; the test sets are private and span all four domains. Empty : clips with no annotated signal at any probability.
Figure 6
Figure 7: Reference results. EW-F1 and engagement macro-F1 on the private test sets of Social Signal Bench Video (left) and Social Signal Bench Audio (right) under the shared harness and evaluator. The first row of each panel joins the untuned backbone to the same model post-trained on Social Signal Bench. The datasets use different signal subsets, so scores compare only within a panel. Supplement Table S10 lists full model names and access.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: From rater spans to the evidence-weighted reference (schematic). (1) Trained raters label spans independently, each with a confidence. (2) In every one-second bin, each vote earns support from peers under the taxonomy credit; a vote with no support in its own or an adjacent cluster is cluster-severe. (3) An expert reviews only cluster-severe votes. (4) Reviewed votes are merged per signal and time slice; the slice score rises with both confidence and the number of raters, and a clip’s evidence weight pg is its overlap-weighted mean.
Signal
Cluster
Perceived behavior
Agreement
Agreement
Visible alignment or affirmation
Confidence
Confidence
Assured, steady stance toward one’s message
Disengagement
Disengagement
Withdrawal of attention or involvement
Engagement
Engagement & Interest
Sustained, active connection to task or partner
Interest
Engagement & Interest
Motivated attention to something new
Disagreement
Skept. & Disagr.
Open opposition to another’s position
Appendix
Table S1: The twelve-signal taxonomy, its seven clusters, and the perceived behavior that defines each signal. Engagement and Disengagement form a separate engagement head; EW-F1 scores the other ten signals.
Audiovisual
Audio-only
Metric
Online
Dyadic
Online
Dyadic
Raw exact-label disagreement
0.354
0.402
0.482
0.549
Raw cluster-weighted disagr.
0.272
0.342
0.376
0.433
Reviewed cluster-weighted disagr.
0.275
0.341
0.371
0.418
Annotations unchanged by review
88.0%
93.4%
90.3%
68.7%
Cluster-severe votes removed
19.6%
16.2%
7.0%
18.4%
Appendix
Table S2: Disagreement before and after expert review on the annotation audit cohort. Lower disagreement is better.
Construction
Clips
Purity
Time rec.
Event rec.
Social Signal Bench Video
Sentence timestamps
4,076
64.5
—
—
Joint, online video
4,480
94.2
98.2
96.4
Joint, added domains
4,542
98.0
99.3
97.4
Social Signal Bench Audio
Joint, online video
6,108
96.8
98.9
97.0
Appendix
Table S3: Agreement between clip labels (social signals) and temporal annotations (%) on the full annotated corpus, before benchmark selection.
Test set
Clips
Rec.
A
B
Sales
Dyad.
Empty
Social Signal Bench Video
2,372
85
45
30
6
4
47.2%
Social Signal Bench Audio
1,094
47
24
17
5
1
29.3%
Appendix
Table S4: Private test sets. Rec. : source recordings, with the recordings per domain (online video A and B, sales calls, dyadic conversations). Empty : clips with no annotated signal at any probability among the scored signals.
Table 13Table 14
Teacher
Clips
Recordings
EW-F1
95% CI
Video
758
62
54.8
[50.4, 61.0]
Audio
604
106
65.4
[62.2, 69.1]
Appendix
Table S9: Pseudo-label quality of the teacher models against expert annotations.
System
Access
EW-F1 ↑
Eng. MF1 ↑
Social Signal Bench Video
Muse Spark 1.3
API
40.9
43.0
Gemini 3.8 Flash
API
40.6
35.1
Qwen3.5-Omni-Plus
open
40.6
40.0
Gemini 4 Argon
API
40.4
39.4
GPT-6.1 Sol
API
39.6
33.5
Appendix
Table S10: Social Signal Bench leaderboard (main paper Figure 7 ) on the 2,372-clip test set of Social Signal Bench Video (85 recordings) and the held-out test set of Social Signal Bench Audio under the shared harness and evaluator. Systems are ranked by EW-F1 within each dataset; Eng. MF1 is engagement macro-F1. The datasets use different signal subsets, so scores should only be compared within each dataset.
System
EW-F1
Micro-F1
Macro-F1
Uniform
Qwen2.5-Omni-7B (post-trained)
44.0 (1)
39.2 (1)
31.0 (3)
44.0 (1)
Muse Spark 1.3
40.9 (2)
35.6 (4)
29.8 (4)
40.9 (5)
Gemini 3.8 Flash
40.6 (3)
35.2 (5)
29.4 (5)
41.2 (4)
Qwen3.5-Omni-Plus
40.6 (4)
38.6 (2)
33.1 (1)
41.9 (3)
Gemini 4 Argon
40.4 (5)
37.1 (3)
32.8 (2)
42.3 (2)
GPT-6.1 Sol
39.6 (6)
34.7 (6)
28.7 (6)
40.5 (6)
Appendix
Table S11: The Social Signal Bench Video leaderboard (2,372 clips) under EW-F1, exact micro- and macro-F1, and EW-F1 with uniform evidence weights. Ranks in parentheses.
System
EW-F1
Micro-F1
Macro-F1
Uniform
Qwen2-Audio-7B-Instruct (post-trained)
58.1 (1)
54.7 (1)
47.4 (1)
58.8 (1)
Gemini 3.1 Pro
54.0 (2)
51.2 (2)
44.7 (2)
55.3 (2)
Gemini 3.8 Flash
51.0 (3)
47.1 (3)
43.2 (3)
52.4 (3)
Qwen3.5-Omni-Plus
47.5 (4)
44.1 (4)
42.0 (4)
49.0 (4)
GPT-Audio
44.4 (5)
42.3 (5)
37.8 (5)
45.8 (5)
Kimi-Audio-7B-Instruct
29.8 (6)
26.7 (6)
20.8 (6)
31.2 (6)
Appendix
Table S12: The Social Signal Bench Audio leaderboard (1,094 clips) under EW-F1, exact micro- and macro-F1 over the six audio signals, and EW-F1 with uniform evidence weights.