Inclusive AI for Group Interactions: Predicting Gaze-Direction Behaviors in People with Intellectual and Developmental Disabilities

Giulia Huang; Maristella Matera; Micol Spitale

cs.HCMar 15, 2026

Inclusive AI for Group Interactions: Predicting Gaze-Direction Behaviors in People with Intellectual and Developmental Disabilities

Authors: Giulia Huang, Maristella Matera, Micol Spitale

Organizations: Politecnico di Milano, Italy

Abstract

Artificial agents that support human group interactions hold great promise, especially in sensitive contexts such as well-being promotion and therapeutic interventions. However, current systems struggle to mediate group interactions involving people who are not neurotypical. This limitation arises because most AI detection models (e.g., for turn-taking) are trained on data from neurotypical populations. This work takes a step toward inclusive AI by addressing the challenge of eye contact detection, a core component of non-verbal communication, with and for people with Intellectual and Developmental Disabilities. First, we introduce a new dataset, Multi-party Interaction with Intellectual and Developmental Disabilities (MIDD), capturing atypical gaze and engagement patterns. Second, we present the results of a comparative analysis with neurotypical datasets, highlighting differences in class imbalance, speaking activity, gaze distribution, and interaction dynamics. Then, we evaluate classifiers ranging from SVMs to FSFNet, showing that fine-tuning on MIDD improves performance, though notable limitations remain. Finally, we present the insights gathered through a focus group with six therapists to interpret our quantitative findings and understand the practical implications of atypical gaze and engagement patterns. Based on these results, we discuss data-driven strategies and emphasize the importance of feature choice for building more inclusive human-centered tools.

Figures & tables

Fig. 1: Illustration of the recording setup for the MIDD dataset with position labels. Cameras are indicated with green, microphone with blue.

Class Label	Direction	Count	%
0	None	1,972	66.6%
1	Right	342	11.5%
2	Front	334	11.3%
3	Left	314	10.6%
Total		2,962	100%

TABLE I: Class distribution of eye contact labels in the MIDD Dataset

Fig. 2: Eye contact class distribution comparison: MIDD vs. MultiMediate

Dataset	Eye contacts	Speaking Events	Rate
MultiMediate	1,043	1,626	64.2%
MIDD	128	365	35.1%

TABLE II: Proportion of speaker eye contact during speaking

Model	Train Accuracy	Val Accuracy
FSFNet	.89	.82
SVC	.63	.57
XGBoost	.73	.62

TABLE III: Training and Validation Accuracy for FSFNet, SVC, and XGBoost on MultiMediate Dataset

Model	Train Acc.	Val Acc.	F1 Score	Score
FSFNet (base)	.64	.82	.71	.75
FSFNet (aug)	.60	.82	.74	.76
SVC	.67	.69	.44	.52
XGBoost	.43	.70	.40	.50

TABLE IV: Performance of FSFNet, SVC, and XGBoost on the MIDD Dataset

Features	SVC			XGBoost
Features	Acc	Macro $F_{1}$	Score	Acc	Macro $F_{1}$	Score
{ gaze_of }	.61	.27	.38	.62	.25	.37
{ gaze_of , is_speaker }	.66	.25	.39	.61	.25	.37
{ gaze_of , is_speaker , pose_of }	.67	.37	.47	.68	.37	.47
{ gaze_of , pose_of }	.69	.43	.52	.68	.37	.47
{ is_speaker , pose_of }	.67	.31	.43	.69	.40	.50

TABLE V: Validation performance of SVC and XGBoost across feature subsets

Model		Acc	Macro $F_{1}$	Score
Individual Models
	FSFNet (original)	.64	.53	.57
	SVC	.60	.29	.39
	XGBoost	.62	.30	.40
FSFNet + SVC
	Ensemble (F 0.7 /S 0.3 )	.67	.54	.58

TABLE VI: MIDD Test‐Set Performance for Individual Models and Ensembles

Therapist insight	Relation to quantitative findings
Heterogeneous interaction patterns	High variability in speaking activity and turn-taking; some participants dominate while others remain passive, explaining skewed distributions and class imbalances.
Critical role of scaffolding	Educators guide turn-taking through gaze and posture; aligns with higher engagement or gaze alignment in certain sessions, suggesting models should account for contextual support.
Multimodal communication	Reliance on gestures, facial expressions, symbolic tools, and touch indicates that gaze-only models may miss meaningful communication, supporting the use of multimodal features (gaze + head pose + speaker identity).
Repetitive or idiosyncratic movements	May be incorrectly interpreted as disengagement in quantitative metrics, explaining false negatives in engagement detection.
Intentional silence	Silence combined with gaze or subtle expressions can represent deliberate communication, highlighting the need for context-sensitive cues in modeling engagement.
Implications for modeling	Supports the use of multimodal features, domain adaptation, and context-aware labeling; explains why fine-tuning on IDD-specific datasets like MIDD improves performance but limitations remain.

TABLE VII: Mapping of therapist insights to quantitative findings

Gaze Estimation

Gaze estimation predicts where a person is looking from eye, face, head-pose, and scene cues.

See all 66 papers

Channelling, Coordinating, Collaborating: A Three-Layer Framework for Disability-Centered Human-Agent Collaboration
Mar 27, 2026Lan Xiao, Catherine HollowayHuman-AI Collaboration Human-AI Co-Creation
Global Disability Innovation Hub UCL Interaction Centre University College London London, United Kingdom
INTERACT: An AI-Driven Extended Reality Framework for Accesible Communication Featuring Real-Time Sign Language Interpretation and Emotion Recognition
Apr 7, 2026Nikolaos D. Tantaroudas, Andrew J. McCracken, Ilias Karachalios +1Sign Language Translation
Institute of Communications and Computer Systems (ICCS), Iroon Polytechneiou 9, 15773 Zografou, Athens, Greece · DASKALOS-APPS, 183 Rue de l’Abbé Griffon, 01960 Péronnas, France · National Technical University of Athens, Leof. Alimou, Katechaki, Zografou, 15772 Athens, Greece +1
"I followed what felt right, not what I was told": Autonomy, Coaching, and Recognizing Bias Through AI-Mediated Dialogue
Mar 11, 2026Atieh Taheri, Hamza El Alaoui, Patrick Carrington +1Human-AI Interaction
Carnegie Mellon University, Pittsburgh, Pennsylvania, USA

Mar 27, 2026J/K move · Enter open · S save

Channelling, Coordinating, Collaborating: A Three-Layer Framework for Disability-Centered Human-Agent Collaboration

Lan Xiao, Catherine Holloway

Global Disability Innovation Hub UCL Interaction Centre University College London London, United Kingdom

Human-AI Collaboration Human-AI Co-Creation Human-Centered AI

Open paper AskPDF arXiv

AI accessibility tools have mostly been designed for individual use, helping one person overcome a specific functional barrier. But for many people with disabilities, complex tasks are accomplished through collaboration with others who bring complementary abilities, not solitary effort. We propose a three-layer framework, Channelling, Coordinating, and Co-Creating, that rethinks AI's role in ability-diverse collaboration: establishing shared informational ground across abilities, mediating workflows between collaborators with different abilities, and contributing as a bounded partner toward shared goals. Grounded in the Ability-Diverse Collaboration framework, grounding theory, and Carlile's 3T framework, it extends the ``agents as remote collaborators'' vision by centring the collaborative, interdependent ways people with disabilities already work.

Abstract

Gaze Estimation

Explore similar work

Channelling, Coordinating, Collaborating: A Three-Layer Framework for Disability-Centered Human-Agent Collaboration

INTERACT: An AI-Driven Extended Reality Framework for Accesible Communication Featuring Real-Time Sign Language Interpretation and Emotion Recognition

"I followed what felt right, not what I was told": Autonomy, Coaching, and Recognizing Bias Through AI-Mediated Dialogue