INTERACT: An AI-Driven Extended Reality Framework for Accesible Communication Featuring Real-Time Sign Language Interpretation and Emotion Recognition
Authors: Nikolaos D. Tantaroudas, Andrew J. McCracken, Ilias Karachalios, Evangelos Papatheou
Organizations: Institute of Communications and Computer Systems (ICCS), Iroon Polytechneiou 9, 15773 Zografou, Athens, Greece · DASKALOS-APPS, 183 Rue de l’Abbé Griffon, 01960 Péronnas, France · National Technical University of Athens, Leof. Alimou, Katechaki, Zografou, 15772 Athens, Greece · Exeter Small-Scale Robotics Laboratory, Engineering Department, University of Exeter, Exeter EX4 4QF, UK
Video conferencing has become central to professional collaboration, yet most platforms offer limited support for deaf, hard-of-hearing, and multilingual users. The World Health Organisation estimates that over 430 million people worldwide require rehabilitation for disabling hearing loss, a figure projected to exceed 700 million by 2050. Conventional accessibility measures remain constrained by high costs, limited availability, and logistical barriers, while Extended Reality (XR) technologies open new possibilities for immersive and inclusive communication. This paper presents INTERACT (Inclusive Networking for Translation and Embodied Real-Time Augmented Communication Tool), an AI-driven XR platform that integrates real-time speech-to-text conversion, International Sign Language (ISL) rendering through 3D avatars, multilingual translation, and emotion recognition within an immersive virtual environment. Built on the CORTEX2 framework and deployed on Meta Quest 3 headsets, INTERACT combines Whisper for speech recognition, NLLB for multilingual translation, RoBERTa for emotion classification, and Google MediaPipe for gesture extraction. Pilot evaluations were conducted in two phases, first with technical experts from academia and industry, and subsequently with members of the deaf community. The trials reported 92% user satisfaction, transcription accuracy above 85%, and 90% emotion-detection precision, with a mean overall experience rating of 4.6 out of 5.0 and 90% of participants willing to take part in further testing. The results highlight strong potential for advancing accessibility across educational, cultural, and professional settings. An extended version of this work, including full pilot data and implementation details, has been published as an Open Research Europe article [Tantaroudas et al., 2026a].
Figures & tables
Figure 1: INTERACT system architecture showing the integration among CORTEX2 infrastructure components and INTERACT AI modules. The diagram traces the data flow from audio input through the Mediation Gateway to each AI service module and ultimately to the XR rendering layer.
Figure 2: Speech-to-Text Processing Pipeline illustrating the audio chunking, Whisper model inference, and text assembly stages that produce the final transcription output stream.
Figure 3: Multilingual Translation Flow diagram depicting source text input, language detection, NLLB model inference, and translated output distribution to downstream modules.
Figure 4: International Sign Language gesture extraction pipeline. From left to right: original ISL video frame; MediaPipe Holistic landmark detection overlay; extracted 3D coordinate array used to drive avatar animations.
Figure 5: 3D Avatar performing ISL signs within the virtual environment, showing the avatar from multiple angles during signing sequences.
Figure 6: Emotion Analysis Output displaying transcribed text with emotion labels and corresponding avatar facial expression modifications.
Figure 7: Meeting Summarisation Workflow showing transcript accumulation, BART model processing, and formatted summary output including key points, decisions, and action items.
Figure 8: Virtual Office Environment depicting the complete meeting space with participant seating positions and signing avatar placement within the immersive XR setting.
Figure 9: Rainbow SDK Integration within Unity showing the SDK configuration and communication flow between conferencing infrastructure and AI processing modules.
Figure 10: Avatar Animation and Emotion Analysis within the VR scene whilst hearing individuals communicate using Rainbow SDK, demonstrating the real-time ISL rendering pipeline.
Characteristic
Demo 1 (May 2025)
Demo 2 (June 2025)
Number of participants
4
6
Participant type
Technical experts
Deaf community stakeholders
VR experience: None
40%
40%
VR experience: Once/twice
40%
40%
VR experience: Moderate
20%
20%
ISL fluency
Varied
Native/near-native
Table 1: Participant Demographics in the two validation demonstrations.
KPI
Target
Achieved
User satisfaction
≥ 85%
92%
Transcription accuracy
≥ 85%
> 85%
Emotion detection precision
≥ 85%
90%
Concurrent users supported
1,000
1,000
Transcription latency
< 2 s
1.2 s
Translation latency
< 2 s
0.8 s
Table 2: Key Performance Indicators (KPIs) achieved during pilot validation.
Evaluation Dimension
1
2
3
4
5
Overall usability
0
0
1
3
6
Transcription quality
0
0
1
4
5
Avatar sign clarity
0
0
2
4
4
Emotion feedback usefulness
0
0
1
3
6
Willingness to use again
0
0
1
2
7
Table 3: Distribution of participant responses across key evaluation dimensions ( N=10 ).
Figure 11: Load Testing Performance Graphs depicting throughput, latency, and error rates across different load levels, confirming zero failures up to 1,000 concurrent connections.
This work introduces a modular platform that brings together six AI services, automatic speech recognition via OpenAI Whisper, multilingual translation through Meta NLLB, speech synthesis using AWS Polly, emotion classification with RoBERTa, dialogue summarisation via flan t5 base samsum, and International Sign (IS) rendering through Google MediaPipe. A corpus of IS gesture recordings was processed to derive hand landmark coordinates, which were subsequently mapped onto three dimensional avatar animations inside a virtual reality (VR) environment. Validation comprised technical benchmarking of each AI component, including comparative assessments of speech synthesis providers and multilingual translation models (NLLB 200 and EuroLLM 1.7B variants). Technical evaluations confirmed the suitability of the platform for real time XR deployment. Speech synthesis benchmarking established that AWS Polly delivers the lowest latency at a competitive price point. The EuroLLM 1.7B Instruct variant attained a higher BLEU score, surpassing NLLB. These findings establish the viability of orchestrating cross modal AI services within XR settings for accessible, multilingual language instruction. The modular design permits independent scaling and adaptation to varied educational contexts, providing a foundation for equitable learning solutions aligned with European Union digital accessibility goals.
N. D. Tantaroudas, A. J. McCracken, I. Karachalios +1
Institute of Communications and Computer Systems (ICCS), Iroon Polytechneiou 9, 15773 Zografou, Athens, Greece · DASKALOS-APPS, 183 Rue de l’Abbé Griffon, 01960 Péronnas, France · National Technical University of Athens, Leof. Alimou, Katechaki, Zografou 15772, Athens, Greece +1
Vision-Language-Action (VLA) models enable robots to execute manipulation tasks from natural-language instructions grounded in visual observations. However, existing VLA interfaces primarily rely on speech or text input, limiting accessibility for deaf, hard-of-hearing, and speech-impaired users. We present SignVLA, a real-time sign-language-guided VLA framework for accessible human-robot interaction. The system introduces a modular sign-to-text interface that converts visual sign gestures into semantic instructions compatible with downstream VLA policies. Given video streams, SignVLA extracts hand landmark features and employs an attention-enhanced Long Short-Term Memory (LSTM) network to capture temporal gesture dynamics for alphabet- and command-level sign recognition. A temporal stabilization module further improves prediction consistency in real-time interaction settings.The generated instruction sequence is then passed to a downstream VLA policy for sign-conditioned robotic manipulation. Experimental results demonstrate stable real-time sign recognition and successful execution of manipulation tasks driven by sign-language inputs. Our findings suggest that lightweight temporal sign recognition can serve as an effective and practical accessibility layer for multimodal embodied intelligence.
Ningwei Bai, Xinyu Tan, Harry Gardner +6
Department of Computer Science, University College London.
Sign language is the primary language for many Deaf and Hard-of-Hearing (DHH) signers, yet most conversational AI systems still mediate interaction through spoken or written language. This spoken-language-centered interface can limit access for signers for whom spoken or written language is not the most accessible medium, motivating direct sign-to-sign conversational modeling. However, sentence-level sign video data are expensive to collect and annotate, leaving existing sign translation and production models with limited vocabulary coverage and weak open-domain generalization. We address this bottleneck by constructing continuous sign conversations from isolated signs: large-scale labeled isolated clips are collected as lexically grounded motion primitives and recomposed into sign-language-ordered utterances derived from existing dialogue corpora. We introduce SignaVox-W, which provides, to our knowledge, the largest labeled isolated-sign vocabulary to date, and SignaVox-U, a continuous 3D sign conversation dataset built from SignaVox-W. To bridge structural mismatch between spoken and signed languages, we use a retrieval-guided spoken-to-gloss translator; to bridge independently collected isolated clips, we propose BRAID, a diffusion Transformer that performs duration alignment and co-articulatory boundary inpainting. With the resulting data, we train SignaVox, a direct sign-to-sign conversational model that generates 3D body, hand, and facial motion responses from prior signing context without spoken-language text or externally provided glosses at inference time. Quantitative and qualitative evaluations show improved isolated-to-continuous motion quality, stronger response-level semantic alignment, and scalable signer-centered interaction that better supports visual-spatial articulation.
Youngmin Kim, Kyobin Choo, Jiwoo Park +4
1Yonsei University · 2LG Electronics · 3Emory University