LG-GER: Language-Guided Group Emotion Recognition via Multimodal Evidence Distillation

Ahmed Shehab Khan; Zhiyuan Li; Yan Tong

cs.CVAug 24, 2026

LG-GER: Language-Guided Group Emotion Recognition via Multimodal Evidence Distillation

Authors: Ahmed Shehab Khan, Zhiyuan Li, Yan Tong

Organizations: University of South Carolina Columbia, South Carolina, USA

Abstract

Inferring the collective emotional state of a group of people from a single image, a task known as group emotion recognition (GER), requires integrating spatially distributed cues such as faces, poses, interactions, and scene context. Current methods rely on detector-driven multi-stream pipelines. These are trained with only image-level supervision that lacks guidance on which regions matter or how strongly each contributes. We propose LG-GER, a language-guided distillation framework that uses a multimodal large language model (MLLM) to generate dense, spatially grounded evidence, i.e., bounding boxes paired with emotion signals and confidence scores, for the training images. This structured evidence is distilled into a single vision-language model (VLM) backbone through four complementary losses: classification, region-text grounding, spatial emotion, and spatial confidence regression. At inference, LG-GER requires no detectors, no MLLM, and no multi-stream fusion, making GER practical for real-time and resource-constrained deployment. LG-GER has been evaluated on two benchmark GER datasets (GroupEmoW and GAF~3.0) and achieves competitive or superior results compared to state-of-the-art methods that require detection and multi-stream processing at inference.

Figures & tables

Figure 1. The ambiguity of isolated visual cues in group emotion recognition. (a, b) Flowers appear at both a wedding (positive) and a funeral (negative). (c, d) Facial expressions alone are insufficient: costumes obscure intent in (c), while scene context reveals positive emotion despite an occluded face in (d). (e, f) Raised fists signal celebration with a trophy (e) but anger during a protest (f). Reliable GER requires integrating faces, objects, scene layout, and interactions.

Figure 2. An example of MLLM-generated annotations. Each evidence region includes a bounding box, type, cue description $c_{i}$ , emotion signal $s_{i}$ , and signal strength $\tau_{i}$ .

Figure 3. An overview of the LG-GER framework. The VLM backbone produces patch features $\mathbf{F}_{\text{patch}}$ and pooled representations $\mathbf{f}_{\text{pool}}$ . The classification path uses $\mathcal{L}_{\text{cls}}$ on the pooled representation. During training, patch features within evidence bounding boxes are mean-pooled and supervised by $\mathcal{L}_{\text{rtxt}}$ , $\mathcal{L}_{\text{emo}}$ , and $\mathcal{L}_{\text{conf}}$ via the spatial supervision path.

Figure 4. The Emotion Adapter remaps VLM text embeddings into an emotion-discriminative space via triplet loss.

Figure 5. Spatial supervision heads. Patch features $\mathbf{F}_{\text{patch}}$ within each evidence bounding box are mean-pooled into region features $\mathbf{r}_{i}$ , which are supervised by three losses: region-text grounding ( $\mathcal{L}_{\text{rtxt}}$ ), spatial emotion ( $\mathcal{L}_{\text{emo}}$ ), and spatial confidence ( $\mathcal{L}_{\text{conf}}$ ).

Figure 6. t-SNE of cue-text embeddings. (a) Raw SigLIP2 embeddings. (b) After Emotion Adapter.

Method	GEW (%)	GAF 3.0 (%)
GNN ( Guo et al., 2020 )	89.14	79.08
CARAN ( Khan et al., 2021 )	90.18	79.13
FANet ( Wang et al., 2024b )	90.06	—
DCAT ( Xie et al., 2023 )	90.47	79.20
VSIM ( Zhu et al., 2025 )	91.18	81.42
FEHSS ( Li et al., 2025 )	91.77	—

Table 1. Comparison with the state-of-the-art methods. Specifically, test accuracy is reported on GroupEmoW; post-challenge validation accuracy is reported on GAF 3.0. All SOTA methods require detection-based multi-stream inference. “—” indicates the method was not evaluated on that dataset.

Figure 7. Grad-CAM attention maps for the baseline and LG-GER on GroupEmoW test images. Red/blue in the difference maps indicate gained/lost attention. Top and middle: LG-GER correctly focuses on emotionally informative regions. Bottom: a failure case where focused attention misclassifies a neutral scene.

Variant	GEW (%)	GAF 3.0 (%)
Gemini 3.0 Flash (zero-shot) ( Google DeepMind, 2025 )	84.7	75.0
(a) $\mathcal{L}_{\text{cls}}$	91.32	83.06
(b) $\mathcal{L}_{\text{cls}}+\mathcal{L}_{\text{rtxt}}^{\dagger}$	91.25	83.16
(c) $\mathcal{L}_{\text{cls}}+\mathcal{L}_{\text{rtxt}}$	91.76	83.73
(d) $\mathcal{L}_{\text{cls}}+\mathcal{L}_{\text{rtxt}}+\mathcal{L}_{\text{emo}}$	91.88	83.89
(e) $\mathcal{L}_{\text{cls}}+\mathcal{L}_{\text{rtxt}}+\mathcal{L}_{\text{conf}}$	91.88	83.78

Table 2. Ablation study on SigLIP 2 ViT-L/16-256. Each row indicates which losses are active.

\dagger

uses raw VLM text embeddings (no Emotion Adapter); all other rows use the Emotion Adapter. Row (f) is the full LG-GER method.

Figure 8. Confusion matrices on GroupEmoW test (top) and GAF 3.0 validation (bottom) for the baseline (Row a), Gemini zero-shot, and LG-GER (Row f) in Table 2 .

Direction	Acc (%)	Gap
GEW $\to$ GEW (in-domain)	92.39	—
GAF $\to$ GEW	88.23	$-$ 4.16
GAF $\to$ GAF (in-domain)	84.08	—
GEW $\to$ GAF	79.54	$-$ 4.54

Table 3. Cross-dataset evaluation. LG-GER is trained on one benchmark and evaluated on the other without fine-tuning. GEW = GroupEmoW, GAF = GAF 3.0. All entries are accuracy (%).

Explore similar work

Variational Encoder--Multi-Decoder (VE-MD) for Privacy-by-functional-design (Group) Emotion Recognition
Apr 2, 2026Anderson Augusma, Dominique Vaufreydaz, Fédérique LetuéEmotion Recognition Privacy-Preserving ML
UGA, LIG, M-PSI · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK, 38000 Grenoble, France +2
Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach
May 27, 2026Anderson AugusmaMultimodal Emotion Recognition Privacy-Preserving ML
Thèse de Doctorat de l’Université Grenoble Alpes
Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?
Jul 14, 2026Kaiwen Zheng, Junchen Fu, Wenhao Deng +3Efficient Multimodal Inference Multimodal Large Language Models
University of Glasgow United Kingdom · Institute of Computing Technology, Chinese Academy of Sciences China · School of Artificial Intelligence, Shandong University China +1

Apr 2, 2026J/K move · Enter open · S save

Variational Encoder--Multi-Decoder (VE-MD) for Privacy-by-functional-design (Group) Emotion Recognition

Anderson Augusma, Dominique Vaufreydaz, Fédérique Letué

UGA, LIG, M-PSI · Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, 38000 Grenoble, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK, 38000 Grenoble, France+2

Emotion Recognition Privacy-Preserving ML Variational Autoencoders Multimodal Emotion Recognition

Open paper AskPDF arXiv

Group Emotion Recognition (GER) aims to infer collective affect in social environments such as classrooms, crowds, and public events. Many existing approaches rely on explicit individual-level processing, including cropped faces, person tracking, or per-person feature extraction, which makes the analysis pipeline person-centric and raises privacy concerns in deployment scenarios where only group-level understanding is needed. This research proposes VE-MD, a Variational Encoder-Multi-Decoder framework for group emotion recognition under a privacy-aware functional design. Rather than providing formal anonymization or cryptographic privacy guarantees, VE-MD is designed to avoid explicit individual monitoring by constraining the model to predict only aggregate group-level affect, without identity recognition or per-person emotion outputs. VE-MD learns a shared latent representation jointly optimized for emotion classification and internal prediction of body and facial structural representations. Two structural decoding strategies are investigated: a transformer-based PersonQuery decoder and a dense Heatmap decoder that naturally accommodates variable group sizes. Experiments on six in-the-wild datasets, including two GER and four Individual Emotion Recognition (IER) benchmarks, show that structural supervision consistently improves representation learning. More importantly, the results reveal a clear distinction between GER and IER: optimizing the latent space alone is often insufficient for GER because it tends to attenuate interaction-related cues, whereas preserving explicit structural outputs improves collective affect inference. In contrast, projected structural representations seem to act as an effective denoising bottleneck for IER. VE-MD achieves state-of-the-art performance on GAF-3.0 (up to 90.06%) and VGAF (82.25% with multimodal fusion with audio). These results show that preserving interaction-related structural information is particularly beneficial for group-level affect modeling without relying on prior individual feature extraction. On IER datasets using multimodal fusion with audio modality, VE-MD outperforms SOTA on SamSemo (77.9%, adding text modality) while achieving competitive performances on MER-MULTI (63.8%), DFEW (70.7%) and EngageNet (69.0).

Abstract

Explore similar work

Variational Encoder--Multi-Decoder (VE-MD) for Privacy-by-functional-design (Group) Emotion Recognition

Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach

Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?