Immersion

Follow this topic Ask about this topic

Momentum

0 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 18

Sep 30, 2026cs.CV

Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality

Advancements in augmented reality (AR) continue to foster innovative solutions, facilitating novel methodologies within educational systems, healthcare delivery, and risk-mitigation protocols. However, optimizing for end-user immersion and comfort remains challenging, as AR head-mounted displays contend with constrained scene geometry, spatial jitter, and temporal instability. User studies are the standard AR evaluation method for visual quality, but their cost, diminishing scalability, and inflexibility pose bottlenecks during iterative application design. To address this problem, we present an automated framework for AR content evaluation and refinement, built on vision-language models (VLMs), to evaluate and predict the visual fidelity of AR scenes as perceived by users. First, we introduce RateAR, a benchmark of AR images and videos collected across diverse scenes and environmental conditions, with good-to-excellent reliability (ICC(2,5) >= .90) across perceptual factors, including object placement, scale, and shadow consistency. Subsequently, we evaluate eleven commercial VLMs on the crafted benchmark. Results support that VLM-based quality predictions strongly correlate with human subjective judgments, achieving Spearman's rank-order correlations of up to 0.8695. An ablation study further suggests that, compared to other prompting strategies, our contextual prompting yields better alignment with human ratings while balancing introduced complexity cues. Building on these findings, we construct an automated AR content adjustment system and conduct a 21-participant user study. More than 90% of participants found that the system improved placement and size coherence of virtual content.

Elias Rotondo, Lin Duan, Yanming Xiu +3

Duke University

Augmented Reality Perceptual Quality Immersion Visual Fidelity Contextual Cues

Sep 29, 2026cs.CV

HelixWorld: A Real-time Interactive Audio-Visual World Model

World simulation is inherently multisensory, demanding synchronized visual and acoustic dynamics in real time. Yet prevailing interactive world models remain strictly silent, focusing exclusively on visual rendering and control while overlooking the acoustic dimension. We present HelixWorld, a real-time interactive audio-visual world model where visual scenes and camera-grounded spatial stereo sound co-evolve natively under user interaction. We curate a high-fidelity spatial audio-visual dataset with true stereo acoustics and metric camera poses, upon which we pre-train a bidirectional teacher conditioned on 6-DoF camera trajectories and user actions. To enable low-latency causal interaction, we distill the teacher into a few-step streaming student via an online trajectory distillation loss, sustaining drift-free joint audio-visual rollouts at 24 FPS on a single GPU. Furthermore, we formalize spatial-acoustic consistency and introduce HelixBench to evaluate whether synthesized sound fields faithfully track dynamic viewpoint motion. Extensive experiments demonstrate that HelixWorld matches state-of-the-art silent world models in visual fidelity and responsiveness, while significantly surpassing existing baselines in camera-aligned spatial-acoustic immersion.

Lei Ke, Jiahao Pan, Zeyue Tian +13

The Hong Kong University of Science and Technology · Noiz AI

Video World Models Spatial Audio Audio-Visual Consistency Immersion Sound Source Localization World Models+1

Sep 28, 2026cs.SD

Enabling Immersive Audio-Visual Experience from Any Video

Most videos capture only a narrow field of view and provide no spatial audio, limiting the sense of immersion they can provide. Recent video generation models can expand perspective videos into panoramic ones, but do not provide the corresponding spatial soundscape. Without spatially consistent audio, these expanded visual worlds remain incomplete. This paper presents OmniDream, a training-free framework that transforms a silent monocular video into an immersive audiovisual experience, where viewers can freely look around while sounds remain spatially aligned with the visual scene. At the core of OmniDream is an object-centric audio representation that disentangles each sound source's intrinsic audio content from its scene-dependent acoustic effects, enabling independent audio generation, physics-based simulation of propagation effects, and flexible spatial audio rendering. Experiments show improved audio-visual alignment, spatial correctness, and perceptual immersiveness over baselines. Examples are available on https://huggingface.co/spaces/CuriousAlien000/spatial-audio-360-demo

Zitong Lan, Mutian Tong, Jiatao Gu +1

University of Pennsylvania

Spatial Audio Immersion Dream Monocular Video

Aug 10, 2026cs.HC

Intuitive Directional Sense Presentation to the Torso Using McKibben-Based Surface Haptic Sensation in Immersive Space

In recent years, systems that utilize immersive space have been developed in various fields. Immersive spaces often contain considerable amounts of visual information; therefore, users often fail to obtain their desired information. Therefore, various methods have been developed to guide users toward haptic sensations. However, many of these methods have limitations in terms of the intuitive perception of haptic sensation and require practice for familiarization with haptic sensation. Fabric actuators are wearable haptic devices that combine fabric and McKibben artificial muscles to provide wearers with surface haptic sensation. These sensations can be provided to a wide area of the body with intuitive perception, instead of only to a part of the body. This paper presents a novel air pressure adjustment method for whole-body motion guidance using surface haptic sensations provided by a wearable fabric actuator. The proposed system can provide users with a directional sense without visual information in an immersive space. The effectiveness of the proposed system was evaluated through subject experiments and statistical data analysis. Finally, a directional sense presentation was conducted for users performing micromanipulations in a mixed-reality space to demonstrate the applicability of the proposed system for teleoperation.

Kenta Yokoe, Tadayoshi Aoyama, Yuki Funabora +2

Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya, Aichi, 464-8603, Japan · Department of Mechanical Systems Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi 464-8603, Japan · Center for One Medicine Innovative Translational Research, Gifu University, Yanagido 1-1, Gifu, Gifu 501-1193, Japan +1

Haptic Feedback Immersion Soft Pneumatic Actuators Whole-Body Motion

Aug 6, 2026cs.HC

A Multi-Layer System for Ultra-High-Resolution Static 360-Degree Telepresence

360-degree video telepresence offers strong immersive potential but remains constrained by the limited resolution of current capture and display hardware. Many telepresence installations feature fixed viewpoints and largely static scenes, yet optimization strategies tailored to such setups have received limited attention. We present a multi-layer, ultra-high-resolution system for static 360-degree telepresence that combines an 8K panoramic camera with a rotatable 4K pan-tilt-zoom (PTZ) camera. Our approach builds a three-layer representation: (1) a tile-based ultra-high-resolution panoramic background, generated by offline stitching high-detail 4K PTZ scans onto the base 8K panorama to achieve effective resolution beyond native capture, and represented as a set of spatial tiles; (2) a dynamic update layer that composites foreground motions from the 8K stream via real-time high-resolution background matting; and (3) a region-of-interest 4K layer that streams a real-time PTZ view of the selected region and additionally updates the corresponding background tiles over time. We evaluate the proposed system through comparisons with representative video super-resolution approaches and a user study assessing perceived detail and immersive experience. Our results indicate that tile-based background refinement, together with user-guided updates, provides a practical way to balance panoramic fidelity and interactivity in static 360-degree telepresence.

Jiapeng Chi, Gerd Bruder, Carsten Neumann +2

VARLab, University of Central Florida, Orlando, FL 32816, USA

Omnidirectional Images Immersion High-Resolution Visual Fidelity

Jul 8, 2026cs.RO

Immersive Social Interaction with VR and LLM-Assisted Humanoids

Humanoid robots can extend human presence to remote, constrained, or hazardous environments, but existing teleoperation interfaces often require physically demanding motion tracking or cognitively demanding low-level control. This paper presents an immersive teleoperation framework that integrates voice-controlled locomotion, VR-based manipulation, and bidirectional social interaction for whole-body humanoid control. Using Apple Vision Pro, the operator receives egocentric visual feedback, issues natural-language locomotion commands, and teleoperates the robot's arms and dexterous hands through wrist and finger tracking. An LLM-assisted voice-control module converts spoken instructions into high-level locomotion commands, while the manipulation module retargets human hand motions to the robot through inverse kinematics and PD control. The system also records multimodal data, including egocentric RGB observations, voice/text commands, joint states, hand motions, and eye-gaze signals, supporting future imitation learning and autonomy. We evaluate the framework on a Unitree H1 humanoid equipped with dexterous hands in manipulation and social interaction tasks. Results show that novice users can successfully operate the system after brief familiarization, achieving 80% success in object manipulation and 70% success in a social cube-passing task. These results demonstrate the potential of immersive, language-assisted teleoperation as an accessible interface for humanoid interaction, remote assistance, and multimodal data collection.

Niraj Pudasaini, Geeta Chandra Raju Bethala, Pranav Doma +2

New York University Abu Dhabi (NYUAD), UAE

Teleoperation Full-Body Humanoid Control Humanoid Immersion Egocentric Vision

Jul 4, 2026cs.HC

CoGen3D: An Agentic Human-AI Co-Design Pipeline for 3D Asset Generation for Virtual Reality

Creating 3D assets for virtual reality requires modeling expertise, which restricts the authorship of immersive experiences. Existing generative AI tools rely on unconstrained, command-driven prompting, lacking the conversational scaffolding needed for users to articulate their intent and validate designs prior to rendering. To address this, we introduce CoGen3D, an agentic human-AI co-design pipeline that proactively guides users through conversational intent elicitation, a concept image confirmation, and image-to-3D generation that directly deploys to immersive scenes. We evaluated this system through a user study (N=120) across six affectively diverse immersive scenes, observing 60 Design group participants who co-created 3D assets for the scenes, and 60 Validation group participants who experienced the scenes with generated assets. Our findings show that co-designed assets are associated with higher scene engagement and shifted affective responses, while participants generally preferred concept images over the final 3D assets, with no increased leniency toward degradation in their own creations. Analysis of the human-AI conversations further shows that target environments shape users' conversational patterns. Our results suggest that our staged, intent-based co-design can democratize virtual reality authoring and shift immersive content creation from technical execution toward collaborative spatial design.

Weiwei Jiang, Wanyu He, Zheyu Tan +5

Nanjing University of Information Science and Technology, Nanjing, China · Japan Advanced Institute of Science and Technology, Nomi, Japan · The University of Sydney, Sydney, Australia +3

3D Assets Virtual Reality Immersion 3D Generation

Jul 3, 2026cs.CV

Towards Standardized Light Field Quality Assessment: Hybrid Subjective Benchmarking and Objective Metric Evaluation

Benchmarking immersive media coding solutions, especially in the standardization context, requires reliable and reproducible subjective quality assessment (QA) procedures, along with objective quality metrics that remain accurate across different distortion types. This paper presents a standardized workflow for light field QA, developed and deployed in the context of JPEG Pleno standardization activities, which integrates benchmark generation, a hybrid subjective evaluation, and objective metric analysis into a common workflow. The benchmark is designed to encompass not only traditional coding-only artifacts but also distortions that arise in processing pipelines in which light field encoding is accompanied with view synthesis and reconstruction techniques. A hybrid subjective method is proposed enabling fine-grained assessment by combining reference-anchored quality rating with targeted pairwise refinement in perceptually ambiguous regions. The reliability of subjective scores is verified using statistical consistency analyses between observers of two cohorts. Finally, a large set of objective metrics is systematically evaluated in terms of global prediction accuracy, local agreement in ambiguous quality regions, and robustness across distortion families. The results show that several metrics achieve strong agreement for coding-only stimuli, but their performance consistently drops when view synthesis distortions are included. The analysis further highlights the importance of view-pooling strategy in the design of future light field quality metrics. The work provides a reproducible and standardization-ready framework for fine-grained light field QA, while identifying key limitations of current objective metrics under emerging coding pipelines. The subjectively annotated dataset is publicly available at https://plenodb.jpeg.org/lfqa/objectivecfp.

Saeed Mahmoudpour, Mylene C. Q. Farias, Gi-Mun Um +6

Dept. of Electronics and Informatics, Vrije Universiteit Brussel, Belgium · IMEC, Kapeldreef 75, Leuven, B-3001, Belgium

Image Quality Assessment Immersion

Jul 1, 2026cs.CV

Sign in the Air to Unlock: An Interface for authentication in Virtual and Augmented Reality Powered by Point-Voxel Cross-Attention Network

Significant advancement of immersive technologies such as Virtual and Augmented Reality (VR/AR) and their integration into diverse aspects of modern life need authentication interfaces that are secure, intuitive, and compatible with embodied interaction. Traditional methods such as passwords, PINs, and device-based logins, break immersion and rely on external hardware. Recent 3D-specific behavioral approaches, such as hand-gesture, eye-tracking, and electroencephalography (EEG)-based methods, offer promising alternatives but often require specialized sensors or constrain natural movement, limiting usability in dynamic environments. We present Sign in the Air to Unlock, an in-air signature interface that enables users to authenticate by signing naturally in 3D space which is a familiar, personal, and reproducible gesture. To realize this interface, we design a point-voxel Cross-Attention Network (PV-Net) that jointly models local motion dynamics and global spatial structure from 3D trajectories. The model is evaluated on two datasets: the public DeepAirSig dataset (1,800 signatures from 40 users) and ImmAirsig, a new dataset collected using Meta Quest 2 in immersive VR (880 samples from 22 users). PV-Net achieves an Equal Error Rate of 2.5% on DeepAirSig and 76% classification accuracy on ImmAirSig. These findings highlight the potential of 3D behavioral interfaces for seamless, user-centric authentication that merges security with natural interaction in immersive environments.

Neda Abdolrahimi, Thiru Siddharth, Frank Sicong chen +1

Syracuse University, 900 S Crouse Ave, Syracuse, New York, USA, 13244

Authentication Immersion Augmented Reality Hand Gesture Recognition Cross-Attention

Jun 28, 2026cs.CV

D $^{2}$ R $^{2}$ OSR: Degradation-Disentangled Representation for Real-World Omnidirectional Image Super-Resolution

With the growing demand for immersive visual experiences, high-quality omnidirectional images (ODIs) have become increasingly important. However, limitations in imaging devices and transmission bandwidth often lead to low-resolution ODIs, hindering the rendering of fine-grained 360° details, especially in the presence of real-world degradations and geometric distortions. Existing real-world super-resolution (Real-SR) methods are inadequate for ODIs, as their degradation models fail to account for the complex imaging pipeline involving fisheye capture and Equirectangular Projection (ERP), introducing severe aliasing and projection-specific distortions. To address these challenges, we propose D

^{2}

R

^{2}

OSR, a Degradation-Disentangled Representation framework for Real-world Omnidirectional image Super-Resolution. D

^{2}

R

^{2}

OSR explicitly models degradations arising from both fisheye imaging and ERP projection, guided by two key insights: (1) projection priors play a critical role in shaping real-world degradations, and (2) human perception in immersive environments is inherently viewpoint-centric. Accordingly, we introduce a Perspective Projection Representation (PPR) operating alongside the ERP branch to capture viewpoint-aware features, together with a Degradation-Specific Module (DSM) that jointly models ERP-induced geometric distortions and PPR-specific real-world degradations. Extensive experiments demonstrate that D

^{2}

R

^{2}

OSR achieves state-of-the-art performance and produces visually compelling, high-fidelity omnidirectional Real-SR results while maintaining favorable computational efficiency for low-resource deployment.

Hongyu An, Xinfeng Zhang, Xu Fan +3

School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · ByteDance Inc., Shenzhen, China · School of Computer Science, Peking University, Beijing, China

Real-World Image Super-Resolution Low-Resolution Visual Degradation Distortion Immersion Real-World

Jun 18, 2026cs.AI

Augmenting Game AI with Deep Reinforcement Learning

Immersion in video games depends not only on graphics, audio, and game mechanics, but also on the quality of in-game characters. Producing believable characters, or game AI, remains a significant challenge as behavioral complexity is hard to capture with hand-coded systems. Game AI is a source of immersion and engagement; however, the limitations stemming from the challenges of creating game AI often lead to frustration and the breaking of the illusion of realism within the game. The introduction of machine learning models opens the door to creating more believable, authentic, and relatable characters in games. The promise is that they either learn from interacting with the game, or from player data, to develop true human-like behavior. In this paper, we envision more applications of reinforcement learning for game AI in the future. For this to materialize, current research limitations are prohibitive to broad deployment across game genres. Therefore, we propose a framework for training reinforcement learning models with a set of requirements in mind that are suited towards game AI and game development. We present examples of games with reinforcement learning-augmented game AI and describe the practicalities of deploying player-facing machine learning agents in modern games. Furthermore, we identify bottlenecks and hard problems in these areas, which we believe offer promising research directions to accelerate the adoption of machine learning in game AI for the video game industry.

Alessandro Sestini, Joakim Bergdahl, Amir Baghi +3

Electronic Arts (EA), Stockholm, Sweden

May 29, 2026cs.HC

EMORSION: Examining the Impact of Audio Parameters on Emotional Responses and Immersion in Film

EMORSION is an exploratory proof-of-concept study examining how film audio design shapes audience emotion and immersion in acinema setting. Four film scenes were selected across the horror (2) and drama (2) genres, balanced between mainstream and independent productions. For each scene, multiple alternative audio mixes were created by systematically manipulating three core aspects of audio design, frequency (pitch), dynamics (loudness), and directionality (spatial placement). Three audience groups viewed the scenes, with each group exposed to one manipulated mix alongside a control mix for each scene. Audience responses were assessed through a triangulated multimodal framework combining self-reported emotion and immersion via a questionnaire, physiological measures including heart rate monitoring, and video-based motion tracking. The protocol successfully captured measurable, interpretable differences across audio conditions, indicating that even subtle changes in audio design can shape emotional perception and immersion. Unconventional mixes tended to produce greater variability in audience interpretation, while conventional immersive mixes were associated with stronger cross-audience agreement. These findings establish the feasibility of the EMORSION protocol and motivate larger-scale studies to characterise the role of specific audio parameters in shaping audience experience.

Nelly Garcia, Ruby Crocker, Bleiz M Del Sette +4

Queen Mary University of London

Audience Immersion Emotional Regulation Impact Analysis

May 21, 2026cs.RO

Quantifying Full-Body Immersion

Humanity is at the forefront of yet another digital revolution, where the lines between real and virtual worlds are dissolving, reshaping how we perceive and interact with our surroundings. In this context, we introduce a transformative paradigm for immersive virtual experiences centered around whole-body kinetic interactions. Our approach redefines immersion through three distinct levels: audio-visual immersion, capturing sensory realism; physical immersion, delivering haptic feedback; and full-body immersion (FBI), where dynamic bodily interaction integrates seamlessly with virtual environments. At the core of this innovation lies a scalable, distributable platform based on modular robotic surface units inspired by the adaptive designs of nature. These units enable the rendering of immersive environments at any scale, from intimate personal experiences to expansive multi-user settings, dynamically adapting to interactions in real-time. The modular system distributes force, shape, and motion feedback throughout entire spaces, replicating the physical characteristics of the environment and enabling new depth of engagement through FBI. By combining scalability, adaptability, and dynamic physical engagement, this framework bridges the gap between real and virtual worlds. It offers an unprecedented level of immersion where users can engage their entire bodies in symbiotic interactions with the virtual space. This work not only advances immersive technology but also redefines how humans and virtual environments coexist, setting a foundation for a new era of human-environment synthesis.

Alihan Bakir, Ekrem Yüksel, Fabio Zuliani +3

Reconfigurable Robotics Lab, EPFL, MED 1 2313 Building MED Station 9, Lausanne, 1015, Vaud, Switzerland.

Immersion Whole-Body Motion

May 20, 2026cs.RO

Flying Together: Human-Guided Immersive Shared Control for Aerial Robot Teams in Unknown Environments

While autonomous multi-robots can achieve safe and coordinated navigation, they often struggle to adapt to unforeseen conditions and to capture operator-driven objectives in unstructured environments. We present a Virtual Reality (VR)-based shared control framework for teams of drones operating in constrained and unknown environments, enabling real-time, user-guided exploration. At the core of our approach is a novel, user-guided motion-primitive-based planner that computes continuous, collision-free trajectories while continuously integrating operator input. This planner is coupled with an admittance controller, allowing the operator to flexibly influence team behavior and guide drones toward regions of interest that autonomous planners may overlook. The system supports mixed-reality operations with both physical and simulated drones, and implements a bilateral VR-based interface, allowing the operator to guide the robot team via migration points while receiving immediate visual feedback of the team state. Experimental results show that shared control improves obstacle avoidance, maintains inter-agent spacing, and reduces operator effort, demonstrating the feasibility and advantages of immersive, human-in-the-loop multi-robot navigation.

Lou De Bel-Air, Luca Morando, Ruitao Chen +7

New York University, NY 10012, USA · Ecole Polytechnique Federale de Lausanne, Station 9 CH-1015 Lausanne, Switzerland · University of California Berkeley, Department of Electrical Engineering and Computer Sciences, Berkeley, CA 94720, USA

Aerial Robots Autonomous Drones Dynamic Environments Immersion Robot Systems

May 7, 2026cs.CV

Closing the Loop: Unified 3D Scene Generation and Immersive Interaction via LLM-RL Coupling

Recent advances in large language models (LLMs) have significantly improved language-driven 3D content generation, but most existing approaches still treat scene generation and user interaction as separate processes, limiting the adaptability and immersive potential of interactive multimedia systems. This paper presents a unified framework that closes the loop between language-driven 3D scene generation and immersive user interaction. Given natural language instructions, the system first constructs structured scene representations using LLMs, and then optimizes spatial layouts via reinforcement learning under geometric and semantic constraints. The generated environments are deployed in a virtual reality setting to facilitate HRI-in-the-loop, where user interactions provide continuous feedback to align generated content with human perception and usability. By tightly coupling generation and interaction, the proposed framework enables more responsive, adaptive, and realistic multimedia experiences. Experiments on the ALFRED benchmark demonstrate state-of-the-art performance in task-based scene generation. Furthermore, qualitative results and user studies show consistent improvements in immersion, interaction quality, and task efficiency, highlighting the importance of closed-loop integration of generation and interaction for next-generation multimedia systems. Our project page can be found at https://proj-showcase.github.io/h3ds/.

Anh H. Vo, Sungyo Lee, Phil-Joong Kim +2

Department of Computer Engineering, Sejong University, Seoul, Republic of Korea

3D Scene Generation Immersion 3D Scene Understanding Human-In-The-Loop Loop Fast-Slow

Feb 6, 2026cs.LG

Hybrid Feedback-Guided Optimal Learning for Wireless Interactive Panoramic Scene Delivery

Immersive applications such as virtual and augmented reality impose stringent requirements on frame rate, latency, and synchronization between physical and virtual environments. To meet these requirements, an edge server must render panoramic content, predict user head motion, and transmit a portion of the scene that is large enough to cover the user viewport while remaining within wireless bandwidth constraints. Each portion produces two feedback signals: prediction feedback, indicating whether the selected portion covers the actual viewport, and transmission feedback, indicating whether the corresponding packets are successfully delivered. Prior work models this problem as a multi-armed bandit with two-level bandit feedback, but fails to exploit the fact that prediction feedback can be retrospectively computed for all candidate portions once the user head pose is observed. As a result, prediction feedback constitutes full-information feedback rather than bandit feedback. Motivated by this observation, we introduce a two-level hybrid feedback model that combines full-information and bandit feedback, and formulate the portion selection problem as an online learning task under this setting. We derive an instance-dependent regret lower bound for the hybrid feedback model and propose AdaPort, a hybrid learning algorithm that leverages both feedback types to improve learning efficiency. We further establish an instance-dependent regret upper bound that matches the lower bound asymptotically, and demonstrate through measurements on an end-to-end testbed that AdaPort outperforms state-of-the-art learning-based baselines as well as the heuristic minimum scene delivery scheme.

Xiaoyi Wu, Juaren Steiger, Bin Li +1

IEEE/ACM TRANSACTIONS ON NETWORKING

Immersion Omnidirectional Images Optimal Learner Edge Platforms Transmission Bandits+1

Feb 4, 2026cs.CV

SalFormer360: a transformer-based saliency estimation model for 360-degree videos

Saliency estimation has received growing attention in recent years due to its importance in a wide range of applications. In the context of 360-degree video, it has been particularly valuable for tasks such as viewport prediction and immersive content optimization. In this paper, we propose SalFormer360, a novel saliency estimation model for 360-degree videos built on a transformer-based architecture. Our approach is based on the combination of an existing encoder architecture, SegFormer, and a custom decoder. The SegFormer model was originally developed for 2D segmentation tasks, and it has been fine-tuned to adapt it to 360-degree content. To further enhance prediction accuracy in our model, we incorporated Viewing Center Bias to reflect user attention in 360-degree environments. Extensive experiments on the three largest benchmark datasets for saliency estimation demonstrate that SalFormer360 outperforms existing state-of-the-art methods. In terms of Pearson Correlation Coefficient, our model achieves 8.4% higher performance on Sport360, 2.5% on PVS-HM, and 18.6% on VR-EyeTracking compared to previous state-of-the-art.

Mahmoud Z. A. Wahba, Francesco Barbato, Sara Baldoni +1

Department of Information Engineering, University of Padova, Via Gradenigo 6b, 35131, Padua, Italy

Saliency Visual Geometry Grounded Transformer Immersion

Jul 23, 2024cs.RO

On the Use of AI-Driven Immersive Digital Technologies for Designing and Operating UAVs

Uncrewed Aerial Vehicles (UAVs) offer agile, cost-effective, and efficient solutions for communication relay networks. However, their modeling and control are challenging, and the mismatch between simulations and actual conditions limits real-world deployment; while maintaining adequate situational awareness remains essential for safe operation. Several studies have proposed integrating the operation of UAVs with immersive digital technologies such as Digital Twin (DT) and Extended Reality (XR) to overcome these challenges. This paper provides a comprehensive overview of the latest research and developments involving immersive digital technologies for UAVs. We explore the use of Machine Learning (ML) techniques, particularly Deep Reinforcement Learning (DRL), to improve the capabilities of DT for UAV systems, and present a case study of a DT-driven DRL pipeline that couples bidirectional physical-digital synchronisation with online recursive least-squares channel calibration for UAV resource allocation. We identify and discuss key research gaps, and propose countermeasures based on Generative AI (GAI), emphasizing the significant role of AI in advancing DT technology for UAVs. Furthermore, we review and discuss how XR technology can transform UAV operations with the support of GAI, and examine its practical challenges. Finally, we propose future research directions to further develop the application of immersive digital technologies for UAV operation.

Yousef Emami, Mohammadhossein Homaei, Miguel Gutierrez Gaitan +2

Unmanned Aerial Vehicles Immersion