Designing Digital Humans with Ambient Intelligence
Organizations: JPMorganChase New York, New York, United States
Abstract
Digital humans are lifelike virtual agents capable of natural conversation and are increasingly deployed in domains like retail and finance. However, most current digital humans operate in isolation from their surroundings and lack contextual awareness beyond the dialogue itself. We address this limitation by integrating ambient intelligence (AmI) - i.e., environmental sensors, IoT data, and contextual modeling - with digital human systems. This integration enables situational awareness of the user's environment, anticipatory and proactive assistance, seamless cross-device interactions, and personalized long-term user support. We present a conceptual framework defining key roles that AmI can play in shaping digital human behavior, a design space highlighting dimensions such as proactivity levels and privacy strategies, and application-driven patterns with case studies in financial and retail services. We also discuss an architecture for ambient-enabled digital humans and provide guidelines for responsible design regarding privacy and data governance. Together, our work positions ambient intelligent digital humans as a new class of interactive agents powered by AI that respond not only to users' queries but also to the context and situations in which the interaction occurs.
Figures & tables
| Layer | Examples | Supported Roles | Domains |
| Information Retrieval | |||
| Physical sensing | Cameras, microphones, proximity sensors | R1, R3 | Banking, retail, hospitality |
| Cross-device interaction | Mobile app state, kiosk events | R1, R4 | Public services, retail, banking |
| Infrastructural platforms | CRM, policy engines, risk systems | R2, R3, R5 | Banking, healthcare, public services |
| System Actuation | |||
| Conversational | Speech, text, human-like avatar | R4, R5 | All domains |
| Low Context Richness | High Context Richness | |
|---|---|---|
| High Initiative | Eager Uninformed Agent. Acts proactively but lacks environmental grounding. May volunteer suggestions that are irrelevant or poorly timed, increasing the risk of user annoyance or trust erosion. | Ambient Digital Human. Fuses multi-layer context with proactive, safeguarded action. Anticipates needs, coordinates across channels, and acts within governed boundaries, the target archetype of this work. |
| Low Initiative | Conventional Conversational Agent. Responds to explicit user prompts using dialogue history alone. Effective for simple question-answering but unable to adapt to situational cues or coordinate across devices. | Context-Aware Assistant. Observes rich environmental and enterprise signals but waits for user requests before acting. Reduces redundant questions and personalizes responses, yet misses opportunities for anticipatory support. |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Engine | Key Digital Human Capabilities |
|---|---|
| Unreal Engine 5 (Epic Games) | Nanite virtualized geometry, Lumen GI, MetaHuman Creator, strand-based hair (Groom), Chaos physics, Pixel Streaming [ 58 ] |
| Unity HDRP | Physical light units, SSS skin shading, SSGI, strand-based hair, broad cross-platform deployment [ 175 ] |
| CryEngine (Crytek) | SVOGI, multi-layer skin shaders, SSAO (originator), volumetric rendering, performance-capture integration [ 40 ] |
| NVIDIA Omniverse | USD-based collaboration, RTX ray tracing, advanced soft-body and physics simulation [ 133 ] |
| O3DE (Open 3D Foundation) | Open-source, CryEngine-derived, integrated cloud services [ 5 ] |
| Godot Engine | Open-source, Vulkan-based rendering, accessible entry point [ 71 ] |
Human-AI Interaction
Human-AI interaction studies how people and AI systems communicate, collaborate, make decisions, and shape one another over time.
See all 329 papers