EgoGroups: A Benchmark For Detecting Social Groups of People in the Wild

Jeffri Murrugarra-Llerena; Pranav Chitale; Zicheng Liu; Kai Ao; Yujin Ham; Guha Balakrishnan; Paola Cascante-Bonilla

cs.CVMar 23, 2026

EgoGroups: A Benchmark For Detecting Social Groups of People in the Wild

Authors: Jeffri Murrugarra-Llerena, Pranav Chitale, Zicheng Liu, Kai Ao, Yujin Ham, Guha Balakrishnan, Paola Cascante-Bonilla

Organizations: Stony Brook University · Rice University

Abstract

Social group detection, or the identification of humans involved in reciprocal interpersonal interactions (e.g., family members, friends, and customers and merchants), is a crucial component of social intelligence needed for agents transacting in the world. The few existing benchmarks for social group detection are limited by low scene diversity and reliance on third-person camera sources (e.g., surveillance footage). Consequently, these benchmarks generally lack real-world evaluation on how groups form and evolve in diverse cultural contexts and unconstrained settings. To address this gap, we introduce EgoGroups, a first-person view dataset that captures social dynamics in cities around the world. EgoGroups spans 65 countries covering low, medium, and high-crowd settings under four weather/time-of-day conditions. We include dense human annotations for person and social groups, along with rich geographic and scene metadata. Using this dataset, we performed an extensive evaluation of state-of-the-art VLM/LLMs and supervised models on their group detection capabilities. We found several interesting findings, including VLMs and LLMs can outperform supervised baselines in a zero-shot setting, while crowd density and cultural regions clearly influence model performance.

Figures & tables

Figure 1: EgoGroups is a dataset that captures social group behavior from walking-tour videos in cities around the world. We show culturally diverse settings with large groups of people or isolated pedestrians, in videos with a narrow, constantly moving field-of-view with frequent motion blur and occlusions.

Figure 2: Statistics of EgoGroups . Distribution of clips by region (left), number of annotated groups (middle) and by weather (right).

Panel of Figure 3 — Figure 3: Annotation samples in EgoGroups . We show group detections with varying annotator confidence levels (red = lowest, blue = highest) and corresponding fine-grained annotations (individuals marked in gray).

Figure 4: Overview of the pipeline for social grouping . Our process involves bounding along with track_ID to generate 3D metadata for an effective prompting.

Model			Scattered						Moderate						Crowded						All
Model			G 1	G 2	G 3	G 4	G 5	AP	G 1	G 2	G 3	G 4	G 5	AP	G 1	G 2	G 3	G 4	G 5	AP	AP
Cosmos- Reason2	8B	VLM	57.18	74.42	76.36	75.35	80.6	72.78	28.92	45.28	59.54	68.02	68.55	54.06	12.8	30.12	50.05	61.53	73.18	45.53	51.07
Cosmos- Reason2	8B	LLM	60.72	71.03	72.84	70.49	75.0	70.02	32.47	49.61	65.81	62.54	76.02	57.29	20.58	40.56	55.9	59.69	65.15	48.38	53.38
Qwen2.5	32B	VLM	58.94	83.37	86.92	81.6	92.27	80.62	30.28	62.72	77.55	84.21	79.38	66.83	19.11	52.42	73.26	76.05	71.56	58.48	63.37
	32B	LLM	63.71	85.28	82.38	79.51	62.73	74.72	40.49	73.14	75.28	73.02	70.62	66.51	30.45	64.66	72.69	68.27	64.38	60.09	63.54
	72B	VLM	60.53	85.1	85.11	79.17	84.54	78.89	37.18	66.91	79.14	78.45	82.76	68.89	25.89	59.79	73.79	77.81	74.18	62.29	66.00

Table 1: Fine-grained performance comparison on EgoGroups dataset segmented by crowd density: scattered, moderate and crowded. Model combination indicates the backbone family, number of parameters, and modality.

G_{1}

–

G_{5}

represent AP scores for group sizes 1–5 + , with AP showing the average performance across all groups.

Model			Scattered			Moderate			Crowded			All
Model			Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1	F1
Cosmos- Reason2	8B	VLM	50.44	49.82	50.1	25.92	16.16	19.9	11.22	5.41	7.3	17.64
Cosmos- Reason2	8B	LLM	44.73	40.6	42.47	22.04	14.3	17.33	11.75	6.61	8.45	15.93
Qwen2.5	32B	VLM	51.4	61.87	56.12	27.05	26.18	26.58	14.35	13.69	14.01	23.49
	32B	LLM	51.22	64.05	56.88	29.8	33.25	31.41	17.5	21.74	19.38	27.70
	72B	VLM	52.77	65.65	58.49	32.76	33.54	33.1	21.0	22.25	21.6	29.97

Table 2: Fine-grained performance comparison on EgoGroups dataset segmented by crowd density: scattered, moderate and crowded. Model combination indicates the backbone family, number of parameters, and modality. Precision, Recall, and F1 scores measure the false positive rate of predicted group IDs (without individuals).

Model			AF	AN	CA	EU	GE	LA	LE	ME	NE	SA	O
Cosmos- Reason2	8B	VLM	47.19	50.49	54.01	51.69	40.07	51.12	52.98	47.05	49.86	52.86	58.89
Cosmos- Reason2	8B	LLM	45.95	56.89	55.25	54.59	50.95	54.17	54.40	49.18	51.21	52.95	53.37
Qwen2.5	32B	VLM	61.99	64.63	63.57	58.82	60.48	66.55	64.09	57.98	71.75	65.75	50.00
	32B	LLM	61.87	65.01	62.16	61.69	62.84	66.89	63.84	59.75	69.36	63.28	61.63
	72B	VLM	58.64	67.24	65.78	64.54	67.06	70.06	67.69	59.42	71.45	67.30	65.80
	72B	LLM	61.94	65.22	61.88	62.32	64.28	65.04	62.61	57.33	67.31	64.49	54.21

Table 3: Fine-grained performance comparison on the EgoGroups dataset, segmented by GLOBE regions: African (AF), Anglo (AN), Confucian Asia (CA), Eastern Europe (EU), Germanic Europe (GE), Latin America (LA), Latin Europe (LE), Middle East (ME), Nordic Europe (NE), South-East Asia (SA), Other (O). Model combination indicates the backbone family, number of parameters, and input modality. We report each region’s AP score individually, and compute MAP as the average across all regions. The best and worst values per row are highlighted in red and blue, respectively.

Model			Group Detection (AP)						Group ID Prediction
Model			G 1	G 2	G 3	G 4	G 5	AP	Precision	Recall	F1
JLSG [ 19 ]			8.0	29.30	37.5	65.40	67.0	41.4	-	-	-
JRDB-Act [ 20 ]			81.40	64.80	49.10	63.20	37.20	59.2	-	-	-
DVT3 [ 60 ]			-	-	-	-	-	-	61.16	31.06	41.19
Cosmos- Reason2	8B	VLM	44.88	34.7	30.96	36.83	55.0	40.47	19.83	5.83	9.02
Cosmos- Reason2	8B	LLM	48.47	36.19	29.05	31.71	56.26	40.34	24.7	6.93	10.82

Table 4: Fine-grained performance comparison on the JRDB dataset. Model combination indicates the backbone family, number of parameters, and input modality.

G_{1}

–

G_{5}

represent AP scores for group sizes 1–5 + , with AP showing the average performance across all groups. Precision, Recall, and F1 scores measure the false positive rate of predicted group IDs (without individuals).

Figure 5: Qualitative comparison of group activities predicted by VLMs for various scenarios.

Figure 6: Per-region activity count heatmap across GLOBE region by Qwen2.5-VL 72B Instruct. Column-wise color intensity reflects the relative frequency of each activity within each region.

Panel of Figure 7 — Figure 7: Qualitative results of our approach across frames with varying crowd densities. From left to right: scattered, moderate, and crowded. Detections with the same color are judged to be in the same social group.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Value	Confidence Level	Description
1	Not confident at all	Very uncertain about group membership
2	Low confidence	Frequent doubts in labeling
3	Moderately confident	Some uncertainty remains
4	Confident	Mostly sure about group membership
5	Very confident	Certain about group detection correctness

Appendix

Table 5: Confidence scale for group membership labeling.

Panel of Figure 8 — Figure 8: Coarse annotation tool used for labeling groups.

Panel of Figure 9 — Figure 9: Fine-grained annotation tool used for labeling groups.

Panel of Figure 10 — Figure 10: Examples of coarse (left) and fine-grained (right) annotations.

Figure 11: Regional distribution of activity recognition across VLMs for “riding motorcycle”, “moving goods”, and “cooking” .

Figure 12: Regional distribution of activity recognition across VLMs for “commute”, “using phone”, and “socializing” .

Figure 13: Distribution of proximity activities (Hugging and Hand holding) grouped by GLOBE regions. Inside a region, each pair lines represent predictions from a different model.

Panel of Figure 14 — Figure 14: Comparison of Semantic Activity Maps across five different models.

Model			Scattered						Moderate						Crowded
Model			G 1	G 2	G 3	G 4	G 5	AP	G 1	G 2	G 3	G 4	G 5	AP	G 1	G 2	G 3	G 4	G 5	AP
Cosmos- Reason2	2B	VLM	76.21	59.89	61.84	54.86	45.0	59.56	58.4	51.44	55.98	48.75	57.55	54.42	54.69	46.57	49.09	49.07	48.8	49.64
	2B	LLM	71.01	59.22	62.99	60.42	37.12	58.15	55.68	48.81	56.9	62.92	52.45	55.35	48.73	41.51	51.45	52.77	55.81	50.05
	8B	VLM	57.18	74.42	76.36	75.35	80.6	72.78	28.92	45.28	59.54	68.02	68.55	54.06	12.8	30.12	50.05	61.53	73.18	45.53
	8B	LLM	60.72	71.03	72.84	70.49	75.0	70.02	32.47	49.61	65.81	62.54	76.02	57.29	20.58	40.56	55.9	59.69	65.15	48.38
Qwen2.5	3B	VLM	60.06	76.93	71.16	65.63	53.64	65.48	38.75	61.15	68.92	61.92	56.49	57.45	39.46	55.47	62.98	60.16	61.66	55.94

Appendix

Table 6: Full fine-grained experiments performance comparison on EgoGroups dataset segmented by crowd density: scattered, moderate and crowded. Model combination indicates the backbone family, number of parameters, and modality. G 1 –G 5 represent AP scores for group sizes 1–5+, where AP is the average performance across groups.

Model			Scattered			Moderate			Crowded
Model			Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1
Cosmos- Reason2	2B	VLM	49.03	25.24	33.29	33.97	17.3	22.88	25.88	12.77	17.05
	2B	LLM	38.24	21.5	27.52	32.46	17.74	22.92	26.56	14.07	18.37
	8B	VLM	50.44	49.82	50.1	25.92	16.16	19.9	11.22	5.41	7.3
	8B	LLM	44.73	40.6	42.47	22.04	14.3	17.33	11.75	6.61	8.45
Qwen2.5	3B	VLM	46.41	54.8	50.23	32.77	33.8	33.25	26.81	29.27	27.98

Appendix

Table 7: Full fine-grained experiments performance comparison on EgoGroups dataset segmented by crowd density: scattered, moderate and crowded. Model combination indicates the backbone family, number of parameters, and modality. Precision, Recall, and F1 scores measure the false positive rate of predicted group IDs.

Model			G 1	G 2	G 3	G 4	G 5	AP	Precision	Recall	F1
Cosmos- Reason2	2B	VLM	57.49	49.88	52.72	49.29	51.65	52.21	31.37	15.75	20.96
	2B	LLM	52.55	46.24	54.35	57.03	53.26	52.68	30.07	14.01	17.64
	8B	VLM	21.09	40.74	55.68	65.4	72.46	51.07	23.83	14.01	17.64
	8B	LLM	27.27	47.32	60.96	61.9	69.47	53.38	20.59	12.99	15.93
Qwen2.5	3B	VLM	40.9	59.96	65.69	61.21	59.11	57.37	31.16	33.56	32.32
	3B	LLM	53.35	66.57	63.23	52.78	48.08	56.8	28.58	34.78	31.35

Appendix

Table 8: Full fine-grained experiments performance comparison on the EgoGroups dataset. Model combination indicates the backbone family, number of parameters, and input modality. G 1 –G 5 represent AP scores for group sizes 1–5+, with AP showing the average performance across groups. Precision, Recall, and F1 scores measure the false positive rate of predicted group IDs.

Model			AF	AN	CA	EU	GE	LA	LE	ME	NE	SA	O
Cosmos- Reason2	2B	VLM	57.13	53.18	53.67	56.16	44.94	47.93	51.12	49.50	46.30	50.67	50.64
	2B	LLM	48.85	51.74	54.61	53.88	51.64	58.28	49.75	50.54	57.31	53.96	53.51
	8B	VLM	47.19	50.49	54.01	51.69	40.07	51.12	52.98	47.05	49.86	52.86	58.89
	8B	LLM	45.95	56.89	55.25	54.59	50.95	54.17	54.40	49.18	51.21	52.95	53.37
Qwen2.5	3B	VLM	55.45	58.24	58.29	58.09	56.34	54.06	55.75	55.14	60.74	56.32	54.53
	3B	LLM	55.86	58.69	55.46	56.71	59.75	57.48	54.51	53.88	57.93	56.55	57.71

Appendix

Table 9: Full fine-grained experiments performance comparison on the EgoGroups dataset, segmented by GLOBE regions. Model combination indicates the backbone family, number of parameters, and input modality. We report each region’s AP score individually, and compute MAP as the average across all regions. The best and worst values per row are highlighted in red and blue, respectively.

Model			Group Detection (AP)						Group ID Prediction
Model			G 1	G 2	G 3	G 4	G 5	AP	Precision	Recall	F1
JLSG [ 19 ]			8.0	29.30	37.5	65.40	67.0	41.4	-	-	-
JRDB-Act [ 20 ]			81.40	64.80	49.10	63.20	37.20	59.2	-	-	-
DVT3 [ 60 ]			-	-	-	-	-	-	61.16	31.06	41.19
Cosmos- Reason2	2B	VLM	75.59	45.6	33.17	32.3	28.3	42.99	16.41	2.33	4.09
	2B	LLM	51.93	39.46	36.07	36.91	42.05	41.28	17.06	3.49	5.79

Appendix

Table 10: Fine-grained performance comparison on the JRDB dataset. Model combination indicates the backbone family, number of parameters, and input modality.

G_{1}

–

G_{5}

represent AP scores for group sizes 1–5 + , with AP showing the average performance across all groups. Precision, Recall, and F1 scores measure the false positive rate of predicted group IDs.

Panel of Figure 15 — Figure 15: Qualitative results across frames with varying crowd densities. From left to right: (a) scattered, (b) moderate, and (c) crowded. From top to bottom: ground truth, Nvidia-Cosmos2 8B, Qwen2.5 72B, Qwen3 30B, and Gemini-3-Pro. Detections with the same color are judged to be in a social group.

Panel of Figure 16 — Figure 16: Prompt instruction and input/output field descriptions processed by LLM.

Panel of Figure 17 — Figure 17: Prompt instruction and input/output field descriptions processed by VLMs.

Panel of Figure 18 — Figure 18: Prompt instruction and input/output field descriptions processed by VLM for the results in Figure 13 .

Explore similar work

LG-GER: Language-Guided Group Emotion Recognition via Multimodal Evidence Distillation
Aug 24, 2026Ahmed Shehab Khan, Zhiyuan Li, Yan TongEmotion Recognition Multimodal Emotion Recognition
University of South Carolina Columbia, South Carolina, USA
Socialality Anchors: Towards Group-bounded Trajectory Prediction
Sep 29, 2026Ziqian Zou, Conghao Wong, Qinmu Peng +1Pedestrian Trajectory Prediction Multi-Agent Trajectory Prediction
Huazhong University of Science and Technology, Wuhan, 430074, Hubei, China.
IndoorCrowd: A Multi-Scene Dataset for Human Detection, Segmentation, and Tracking with an Automated Annotation Pipeline
Apr 2, 2026Sebastian-Ion Nae, Radu Moldoveanu, Alexandra Stefania Ghita +1Instance Segmentation Multi-Object Tracking
National University of Science and Technology Politehnica Bucharest, Romania · Expleo, Romania

Aug 24, 2026J/K move · Enter open · S save

LG-GER: Language-Guided Group Emotion Recognition via Multimodal Evidence Distillation

Ahmed Shehab Khan, Zhiyuan Li, Yan Tong

University of South Carolina Columbia, South Carolina, USA

Emotion Recognition Multimodal Emotion Recognition VLM Distillation

Open paper AskPDF arXiv

Inferring the collective emotional state of a group of people from a single image, a task known as group emotion recognition (GER), requires integrating spatially distributed cues such as faces, poses, interactions, and scene context. Current methods rely on detector-driven multi-stream pipelines. These are trained with only image-level supervision that lacks guidance on which regions matter or how strongly each contributes. We propose LG-GER, a language-guided distillation framework that uses a multimodal large language model (MLLM) to generate dense, spatially grounded evidence, i.e., bounding boxes paired with emotion signals and confidence scores, for the training images. This structured evidence is distilled into a single vision-language model (VLM) backbone through four complementary losses: classification, region-text grounding, spatial emotion, and spatial confidence regression. At inference, LG-GER requires no detectors, no MLLM, and no multi-stream fusion, making GER practical for real-time and resource-constrained deployment. LG-GER has been evaluated on two benchmark GER datasets (GroupEmoW and GAF~3.0) and achieves competitive or superior results compared to state-of-the-art methods that require detection and multi-stream processing at inference.

Abstract

Explore similar work

LG-GER: Language-Guided Group Emotion Recognition via Multimodal Evidence Distillation

Socialality Anchors: Towards Group-bounded Trajectory Prediction

IndoorCrowd: A Multi-Scene Dataset for Human Detection, Segmentation, and Tracking with an Automated Annotation Pipeline