cs.CVNov 17, 2026

Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio

Authors: Siddharth Ravi, Anders R. Bargum, Hugo Anselme, Patrizio Cugia di Sant'Orsola, Alberto Cardaci, Flora Kincso Parragi, Vincent Borowiec, Line H. Clemmensen

Organizations: Interhuman AI · Universidad de Alicante · École Polytechnique · Institut Supérieure d’Electronique de Paris · 

Abstract

Recognizing social signals such as interest and hesitation requires interpreting context-dependent behaviors that may co-occur in conversation. Existing affective computing benchmarks rarely evaluate such co-occurring signals, and conventional multi-label F1 overlooks semantic relationships and annotator disagreement. We introduce Social Signal Bench, a 47-hour densely annotated benchmark for the evaluation of social signal recognition and engagement, comprising two datasets: Social Signal Bench Video (25 hours), covering ten social signals in audiovisual recordings, and Social Signal Bench Audio (22 hours), covering six signals in audio-only recordings. Each dataset pairs a public training set of online conversational video, of which 10 hours are expert-annotated and the remainder pseudo-labeled, with a private expert-labeled test set spanning four conversational domains. We also propose Evidence-Weighted F1 (EW-F1), which weights reference labels by annotator evidence and assigns graded credit to semantically related predictions through optimal per-clip matching. Controlled perturbations demonstrate that EW-F1 distinguishes semantically related errors from unrelated ones, unlike conventional F1. Under a shared evaluation protocol, 7B open models post-trained on Social Signal Bench outperform general-purpose frontier models on video and match them on audio.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?

    Aug 13, 2026Koen P. de Vries, Xavier Alameda-Pineda, Estefanía Talavera +1Supervised Fine-TuningMultimodal Large Language Models

  2. FriendBench: Benchmarking Dyadic Familiarity Inference in Humans and Multimodal Large Language Models

    Jul 31, 2026Jeffrey M. Girard, Jason Z. Zheng, Jacqueline R. Vertino +2ClassificationMultimodal Model Evaluation

  3. GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?

    Oct 10, 2026Zonghuan Li, Hayley Hung, Chirag RamanMultimodal Model EvaluationMultimodal Understanding