COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention

Zonghuan Li; Litian Li; Arthur Mercier; Gara Dorta; Balint Dioszegi; Jose Morales-Vargas; Chenxu Hao; Ivan Kondyurin

cs.HCJun 2, 2026

COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention

Authors: Zonghuan Li, Litian Li, Arthur Mercier, Gara Dorta, Balint Dioszegi, Jose Morales-Vargas, Chenxu Hao, Ivan Kondyurin,

Organizations: Delft University of Technology · University of Greenwich · Instituto Tecnológico de Costa Rica · University of Hamburg ·

Abstract

COSI-Lab presents a multimodal multi-sensor dataset of a real interdisciplinary scientific workshop with 32 academics at an international conference. It captures ecologically valid social interactions in a weakly scripted setting consisting of two 30-minute mingling sessions with real professional and social consequences. We argue that intelligent systems could better handle subjective perceptions by modeling their multiplicity not as label noise but as an explainable perspective-driven reasoning process. We focus on the Apparent Intent Inference (AII) problem as determined by ex-situ observers and conceptualize intentions to be independent of the future. We contribute a novel annotation process for AII that accounts for a perceiver's own interpretative tendencies, quantitative and qualitative analyses of intent narratives with respect to diversity, plausibility and grounding, benchmark tasks for AII and relevant contextual factors, speech quality audio for all participants as well as privacy-preserving multi-modal environmental and wearable sensor data, enabling lexical and nonverbal behavior analysis; and the coupling of self-reported participant goals (30 minute to 3 hour) with annotated AII (seconds).

Figures & tables

Figure 1: COSI-Lab: The conference living lab for studying multi-perspective social intentions.

	Data capture					Desiderata
Dataset	Scenario (Scripting)	Intention Representation	Narrative Annotation	Modalities	Annotation Size	D1	D2	D3	D4
PIE++ ( Khindkar et al., 2024 )	Pedestrian-Vehicle (Strong)	Categorical	Multi-annot. prediction	Video	5k instances of 1-2 sec.	$\checkmark$	$\checkmark$	$\times$	N/A
PSI ( Jing et al., 2025 )	Pedestrian-Vehicle (Strong)	Categorical Free-form	Multi-annot. subjective	Video	196 vids of 15 sec.	$\checkmark$	$\checkmark$	$\checkmark$	N/A
IntentQA ( Li et al., 2023 )	Skilled activities (Strong)	Categorical guided Q&A	Multi-annot. agreement	Video	5k vids of 44 sec.	$\times$	( $\checkmark$ )	( $\checkmark$ )	$\checkmark$
Ego-Exo4D ( Grauman et al., 2024 )	Skilled activities (Strong)	Expert free-form/reflections	Multi-perspective & Multi-annotator	Video, eye, IMU	5k takes of 2.6 min.	$\checkmark$	$\times$	$\checkmark$	$\checkmark$
I2M ( Umagami et al., 2026 )	Daily living (Strong)	Free-form descriptions	Single-perspective logs	Video, IMU	215 seqs of 2.8 min.	$\times$	$\times$	$\times$	$\checkmark$

Table 1: Comparison of human behavior datasets with intention and reasoning annotations.

Figure 3: Interaction structure and survey-reported goals across the two mingling sessions.

Figure 4: Conversation topic composition and redirection uptake by session and group size. M1/M2 denotes Mingle 1/2. Panel A: topic-family shares (T1: background; T2: research domains; T3: methods/data; T4: team/organization; T5: networking; T6: personal; T7: gender/diversity; T8: other). Panel B: realized uptake among valid topic-redirection attempts.

Figure 5: Left: Semantic similarity of the annotations and presence of emotions, cues, beliefs, intentions, and all 4 simultaneously. Right: Conversation group detection results (mean $\pm$ std over camera-wise 5-fold tests) and successful topic-transition detection (mean $\pm$ std over five folds).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Dataset file structure. For more information, refer to the corresponding README.md files.

Figure 7: Example calibration footage for camera geometry estimation. The left panel shows a per-camera ChArUco-board recording for intrinsic calibration, and the right panel shows multi-camera footage used to estimate extrinsic camera poses after installation.

Survey wave	Cleaned rows	Missing ID rows	Duplicate ID rows	Unique IDs
Pre-workshop	34	0	0	34
Workshop	33	2	0	31
Post-workshop	32	0	0	32
Post-event	28	1	2	25

Appendix

Table 2: Availability of respondent-level matching IDs across survey waves. Cleaned response rows do not include column headers. Duplicate rows refer to extra rows sharing an already observed matching ID within the same survey.

Survey comparison	Matched unique IDs
Pre-workshop $\cap$ workshop	23
Pre-workshop $\cap$ post-workshop	22
Pre-workshop $\cap$ post-event	16
Workshop $\cap$ post-workshop	29
Workshop $\cap$ post-event	19
Post-workshop $\cap$ post-event	18

Appendix

Table 3: Exact cross-wave matching coverage based on the constructed respondent-level matching ID.

Width	600 mm
Height	400 mm
Rows	4
Columns	6
Checker width	90 mm
Dictionary	DICT_5X5_100
OpenCV legacy pattern	True

Appendix

Table 4: Charuco boards details

Figure 8: Illustration of the process of benchmarking task 2. (a)Segmentation mask from SAM3 model; (b)Inferred keypoints from VitPose; (c)Transferred 3D keypoints in space; (d)Illustration of the conversation group detection task.

Figure 9: Displayed a screen grab of the UI of Covfee.

	Description			Explanation
Group	PAS	Wrong clip	AUC	PAS	Wrong clip	AUC
Held-out human	0.457	0.395	0.68	0.464	0.398	0.72
Paraphrased human	0.440	0.366	0.71	0.470	0.391	0.72
Atypical human	0.334	0.318	0.54	0.349	0.323	0.56
gemma	0.368	0.317	0.63	0.402	0.347	0.67
qwen3omni30b	0.402	0.358	0.63	0.425	0.375	0.66

Appendix

Table 5: PAS for known groups (HTS, max, matched pools; generic templates are scored against every clip). Wrong clip : the same items scored against a random other clip, one draw per item. It therefore differs slightly from the 20-draw baseline used in the text. AUC: probability that an item’s score on its own clip exceeds a score on a wrong clip (

0.5

= no clip information). Clip-weighted means.

Metric	Result
Validation windows	12
Composition MAE	12.1
Composition RMSE	19.8
Mean total-variation distance	0.483
Pearson correlation	0.555
Concordance correlation coefficient	0.522

Appendix

Table 6: Human–human agreement for topic-composition annotation. Composition errors are reported in percentage points.

Metric	Result
Composition MAE	12.1
Composition RMSE	21.6
Mean total-variation distance	0.484
Dominant-topic accuracy	33.3%
Topic-presence precision	0.939
Topic-presence recall	0.585

Appendix

Table 7: Local LLM performance for topic composition against the human-adjudicated reference. Composition errors are reported in percentage points.

Judgment	Annotators	Eligible $N$	$\alpha$	Agreement
Stage 1: ATTEMPT vs. NOT_ATTEMPT	2	22	0.366	68.2%
Stage 2: VALID vs. INVALID	3	36	0.094	57.7%
Stage 2: SUCCESS vs. FAILURE	up to 3	18	$-0.170$	61.5%

Appendix

Table 8: Human–human reliability for topic-redirection and uptake coding. Eligible

N

denotes cases with sufficient non- UNCERTAIN human judgments for the corresponding analysis.

Task	Reference $N$	Accuracy	Macro-F1
Stage 1: ATTEMPT vs. NOT_ATTEMPT	15	0.800	0.762
Stage 2: VALID vs. INVALID	33	0.424	0.388
Stage 2: SUCCESS vs. FAILURE	11	0.909	0.906

Appendix

Table 9: Local LLM performance for topic-redirection and uptake coding. Reference sample sizes differ because uncertain or unresolved cases are excluded, and Stage 2 outcome evaluation is conditional on validity.

trained	cam06	cam08	cam10	cam01	cam03
cam06	$0.775\pm 0.048$	$0.721\pm 0.076$	$0.653\pm 0.206$	$0.740\pm 0.154$	$0.688\pm 0.127$
cam08	$0.726\pm 0.084$	$0.860\pm 0.041$	$0.820\pm 0.133$	$0.767\pm 0.145$	$0.682\pm 0.087$
cam10	$0.572\pm 0.101$	$0.766\pm 0.111$	$0.664\pm 0.210$	$0.608\pm 0.187$	$0.708\pm 0.149$
cam01	$0.402\pm 0.139$	$0.692\pm 0.092$	$0.603\pm 0.062$	$0.809\pm 0.062$	$0.705\pm 0.119$
cam03	$0.411\pm 0.289$	$0.680\pm 0.384$	$0.502\pm 0.317$	$0.632\pm 0.202$	$0.706\pm 0.394$

Appendix

Table 10: Conversation group detection results on DANTE Swofford et al. (2020) with

T=1

(top) and

T=2/3

(bottom)

trained	cam06	cam08	cam10	cam01	cam03
cam06	$0.574\pm 0.177$	$0.699\pm 0.108$	$0.537\pm 0.210$	$0.631\pm 0.154$	$0.608\pm 0.171$
cam08	$0.588\pm 0.149$	$0.765\pm 0.087$	$0.679\pm 0.148$	$0.512\pm 0.129$	$0.442\pm 0.092$
cam10	$0.648\pm 0.115$	$0.793\pm 0.038$	$0.696\pm 0.159$	$0.428\pm 0.196$	$0.514\pm 0.105$
cam01	$0.297\pm 0.140$	$0.520\pm 0.028$	$0.480\pm 0.131$	$0.844\pm 0.110$	$0.557\pm 0.056$
cam03	$0.532\pm 0.175$	$0.773\pm 0.069$	$0.680\pm 0.182$	$0.636\pm 0.155$	$0.807\pm 0.143$

Appendix

Table 11: Conversation group detection results on LSTM-based model Tan et al. (2022) with

T=1

(top) and

T=2/3

(bottom)

Class	Source	Samples
Positive	Successful topic-redirection attempts	316
Negative	Failed topic-redirection attempts	148
Negative	Ordinary conversation	928
Total		1,392

Appendix

Table 12: Sample composition for Benchmark 3.

Model	Input	AUROC	AUPRC
LR (initiator)	KP	$0.556\pm 0.026$	$0.291\pm 0.029$
	ACC	$0.546\pm 0.056$	$0.274\pm 0.041$
	KP+ACC	$0.568\pm 0.043$	$0.296\pm 0.049$
TCN (initiator)	KP	$\mathbf{0.596\pm 0.074}$	$\mathbf{0.334\pm 0.086}$
	ACC	$0.483\pm 0.049$	$0.218\pm 0.029$
	KP+ACC	$0.565\pm 0.036$	$0.286\pm 0.058$

Appendix

Table 13: Benchmark 3 results for successful topic-transition detection using the primary +10 s post-turn observation horizon. Values are the unweighted mean

\pm

sample standard deviation across five held-out test folds.

Model	Input	+2 s	+5 s	+10 s
LR	KP	$0.536\pm 0.031$	$0.545\pm 0.025$	$0.556\pm 0.026$
	ACC	$0.516\pm 0.042$	$0.525\pm 0.052$	$0.546\pm 0.056$
	KP+ACC	$0.544\pm 0.034$	$0.555\pm 0.039$	$0.568\pm 0.043$
TCN	KP	$\mathbf{0.572\pm 0.061}$	$0.549\pm 0.067$	$\mathbf{0.596\pm 0.074}$
	ACC	$0.503\pm 0.070$	$0.488\pm 0.068$	$0.483\pm 0.049$
	KP+ACC	$0.570\pm 0.050$	$\mathbf{0.596\pm 0.043}$	$0.565\pm 0.036$

Appendix

Table 14: AUROC for successful topic-transition detection using different post-turn observation horizons. Values are the unweighted mean

\pm

sample standard deviation across five held-out test folds.

Model	Input	+2 s	+5 s	+10 s
LR	KP	$0.272\pm 0.044$	$0.277\pm 0.040$	$0.291\pm 0.029$
	ACC	$0.264\pm 0.034$	$0.267\pm 0.042$	$0.274\pm 0.041$
	KP+ACC	$0.282\pm 0.037$	$0.293\pm 0.040$	$0.296\pm 0.049$
TCN	KP	$\mathbf{0.309\pm 0.032}$	$0.275\pm 0.032$	$\mathbf{0.334\pm 0.086}$
	ACC	$0.229\pm 0.029$	$0.242\pm 0.048$	$0.218\pm 0.029$
	KP+ACC	$0.284\pm 0.033$	$\mathbf{0.349\pm 0.063}$	$0.286\pm 0.058$

Appendix

Table 15: AUPRC for successful topic-transition detection using different post-turn observation horizons. Values are the unweighted mean

\pm

sample standard deviation across five held-out test folds. The positive-class prevalence is 0.227.

Model	Input	+2 s	+5 s	+10 s
LR	KP	$0.524\pm 0.073$	$0.528\pm 0.073$	$0.534\pm 0.037$
	ACC	$\mathbf{0.557\pm 0.026}$	$\mathbf{0.568\pm 0.029}$	$\mathbf{0.563\pm 0.025}$
	KP+ACC	$0.547\pm 0.071$	$0.544\pm 0.057$	$0.552\pm 0.032$
TCN	KP	$0.519\pm 0.020$	$0.528\pm 0.041$	$0.520\pm 0.027$
	ACC	$0.504\pm 0.051$	$0.508\pm 0.082$	$0.502\pm 0.078$
	KP+ACC	$0.541\pm 0.031$	$0.538\pm 0.042$	$0.549\pm 0.079$

Appendix

Table 16: AUROC for the secondary topic-redirection attempt-detection task using different post-turn observation horizons. Values are the unweighted mean

\pm

sample standard deviation across five held-out test folds.

Model	Input	+2 s	+5 s	+10 s
LR	KP	$0.382\pm 0.066$	$0.383\pm 0.072$	$0.387\pm 0.056$
	ACC	$0.384\pm 0.034$	$\mathbf{0.401\pm 0.040}$	$0.398\pm 0.034$
	KP+ACC	$\mathbf{0.395\pm 0.065}$	$0.399\pm 0.062$	$\mathbf{0.411\pm 0.054}$
TCN	KP	$0.357\pm 0.023$	$0.380\pm 0.046$	$0.373\pm 0.035$
	ACC	$0.354\pm 0.029$	$0.357\pm 0.053$	$0.342\pm 0.039$
	KP+ACC	$0.393\pm 0.030$	$0.383\pm 0.040$	$0.392\pm 0.047$

Appendix

Table 17: AUPRC for the secondary topic-redirection attempt-detection task using different post-turn observation horizons. Values are the unweighted mean

\pm

sample standard deviation across five held-out test folds. The positive-class prevalence is 0.333.

Intent Classification

Intent classification assigns user utterances or behaviors to goals, supporting dialogue systems, search, safety decisions, and proactive assistance.

See all 60 papers

GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?
Oct 10, 2026Zonghuan Li, Hayley Hung, Chirag RamanMultimodal Model Evaluation Multimodal Understanding
Department of EEMCS, Delft University of Technology, Delft, Netherlands
MoCA: Implicit Social Context Analysis
Aug 6, 2026Wenhao Xu, Kaiwen Zhang, Hao Li +10Abductive Reasoning Multimodal Large Language Models
National University of Singapore, Singapore, Singapore. · Wuhan University, Wuhan, China. · University of Oxford, Oxford, United Kingdom.
Intent Signal Theory: A Computational Framework for Intent-State Control in Human-AI Interaction
May 24, 2026Gang PengHuman-Centered AI User Modeling
Huizhou Lateni AI Technology Co., Ltd., Huizhou, China · Huizhou University, Huizhou, China

Oct 10, 2026J/K move · Enter open · S save

GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?

Zonghuan Li, Hayley Hung, Chirag Raman

Department of EEMCS, Delft University of Technology, Delft, Netherlands

Multimodal Model Evaluation Multimodal Understanding Temporal Reasoning in Language Models Multimodal Grounding

Open paper AskPDF arXiv

Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to specific facial, gestural, or prosodic signals that shape social meaning. Current multimodal LLM benchmarks largely evaluate final-answer understanding over complete episodes, rewarding post-hoc recognition of social meaning rather than the unfolding perception required for real-time interaction. Drawing on interactionally embedded Gestalt principles of multimodal human communication, we introduce GesBench: a human-grounded benchmark for cue-level social perception in multimodal LLMs. GesBench uses cumulative sub-utterance clips to measure when models form stable interpretations of speaker intention and social affordance, and human gaze-guided corruption to evaluate whether regions attended by humans are functionally important for model judgments. Built from more than 6000 utterance-level clips from scripted and unscripted conversational datasets, GesBench includes human annotations of unfolding judgments and gaze on a representative subset. Across open and closed multimodal models, we find a clear human-model gap: humans form predictive understandings early and revise more often, while models often hedge or repeat across different clips. GesBench shifts social evaluation from whether models eventually recognize meaning to when and where that meaning is formed. Our codes are publicly available at https://anonymous.4open.science/status/GesBench-share-212C.

Abstract

Intent Classification

Explore similar work

GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?

MoCA: Implicit Social Context Analysis

Intent Signal Theory: A Computational Framework for Intent-State Control in Human-AI Interaction