Prior beliefs about whether a virtual human (VH) is controlled by a human or a computer can shape how users interpret and respond to its behaviour. As emerging hybrid VH systems allow control to transfer between human operators and computer programs during an ongoing interaction, it remains unclear whether perceived agency updates with the actual controller or continues to reflect users' prior beliefs and first impressions. We investigated this question in a within-subject VR study with 24 participants across four conditions: always computer-controlled, computer-to-human, human-to-computer, and always human-controlled. We designed an interaction task in which participants acted as interviewers, asking the VHs a fixed set of questions. Before each interaction, participants were informed whether the VH was human- or computer-controlled, matching the actual controller at the beginning, but were unaware of possible control transitions. We found that perceived agency was strongly influenced by the controller presented at the beginning of the interaction, with no evidence of updating following the control transitions. Social evaluations including likeability and co-presence were generally higher in conditions where human control occurred later and covered a greater proportion of the interaction. The results suggest that prior beliefs and first impressions continued to shape perceived agency, whereas social evaluations were more sensitive to the VH's actual behaviour over the interaction.
Figures & tables
Figure 1: First-person interface perspectives. (a) The semi-transparent panel reminds participants of the predefined interview questions. The VH in the figure is controlled by the experimenter and can be either an agent or an avatar. (b) The red text indicates the control transition status, while the semi-transparent panel provides the scripted responses. The VH in the figure is an avatar controlled by the participant.
Figure 2: Different appearances of the VH used in the experiment.
Figure 4: Time-weighted mean slider values in the Before and After segments across C1, C2, C3, and C4. Brackets indicate significant pairwise comparisons ( ∗p<.05 , ∗∗p<.01 ).
Figure S1 : A screenshot showing the interface included both a third-person Unity scene-view recording of the VH they interacted with and a first-person game-view perspective and the slider used in Phase II.
Before
After
Condition
Gaze at VH (%)
Head displacement (m)
Gaze at VH (%)
Head displacement (m)
C1
48.908 (67.755)
0.028 (0.045)
45.314 (65.433)
0.037 (0.043)
C2
62.660 (42.482)
0.046 (0.024)
53.294 (62.329)
0.059 (0.064)
C3
21.227 (55.306)
0.040 (0.036)
32.719 (62.469)
0.045 (0.054)
C4
45.653 (66.274)
0.044 (0.028)
47.079 (46.194)
0.054 (0.055)
Table S1 : Descriptive statistics for gaze and head displacement across conditions and segments. Values are reported as median (IQR).
Measure
C1
C2
C3
C4
Questionnaire measures
Co-presence
5.250 (2.208)
5.917 (1.208)
5.500 (1.875)
6.333 (1.375)
Attention allocation
4.000 (1.583)
4.667 (1.833)
4.167 (1.875)
5.583 (1.750)
Likeability
4.333 (1.333)
5.000 (1.417)
4.667 (2.000)
6.000 (1.083)
Perceived agency
3.833 (2.333)
4.667 (2.750)
5.667 (2.250)
6.167 (1.750)
Memory recall
3.000 (1.000)
2.500 (1.000)
3.000 (1.000)
3.000 (1.000)
Table S2 : Descriptive statistics for the subjective measures across conditions. Values are reported as median (IQR), with n=24 per condition. Questionnaire scores ranged from 1 to 7, except memory recall, which ranged from 0 to 3. Slider values ranged from 0 to 1.
Can people distinguish between human and AI agency in humanoid teleoperation? To explore this question, we developed Ghost-in-the-Loop, a teleoperation framework that supports both human-operated and AI-generated control of a robot's voice, facial expressions, and gestures while maintaining a consistent embodiment. We conducted a preliminary online study (N=50) in which participants viewed short interaction clips generated by either a Human Operator or an AI Control and judged the perceived source of control. Results suggest that participants often struggled to distinguish between the two conditions in brief interactions. Qualitative responses indicate that judgments were primarily influenced by perceived naturalness, temporal coordination, and consistency across speech, facial expression, and gesture. These findings provide initial insights into agency perception in embodied human--AI communication and motivate future investigations of blended human--AI telepresence systems.
Xiang Li, Koya Dendo, Keigo Minamida +3
Sony Computer Science Laboratories Kyoto, Japan · University of Cambridge Cambridge, United Kingdom · The University of Tokyo Tokyo, Japan
Prototyping and evaluating human-robot teaming (HRT) scenarios in the real-world is costly. Virtual simulation of HRT scenarios has been adopted as an alternative to conducting user studies in the real-world to investigate user perceptions, behaviors, and performance during human-robot interactions. The consistency of human behavior between the real and virtual-worlds is integral to the validity of utilizing such virtual experimentation. This paper presents a user study examining the difference in human behavior between an HRT scenario conducted in a virtual vs real environment. We employed mixed-methods to examine team performance and human factors during the HRT. Our quantitative results showed a significant difference in workload between the two modalities. Our qualitative analysis expanded on the quantitative results and found differences in participants' strategy, their mental model of robots, and the type of trust they had for robots between modalities.
Sean Dallas, Absalat Getachew, Motaz AbuHijleh +4
Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309, United States · College of Innovation & Technology, University of Michigan-Flint, Flint, MI 48502, United States · U.S. Army DEVCOM Ground Vehicle Systems Center (GVSC), United States
As embodied agents take on increasingly social and relational roles in VR, visual realism and embodiment alone may be insufficient; users must also perceive these agents as emotionally attuned, supportive, and humanlike. Prior work suggests that verbal attunement and nonverbal mimicry can each improve users' social evaluations of embodied agents. However, behavioral mimicry has largely been studied outside of real-time, conversational AI interactions, leaving limited understanding of how users respond when an agent simultaneously generates contextually responsive dialogue and adapts its nonverbal behavior during an immersive conversation. To address this gap, we developed an embodied AI counselor that combines conversational AI with real-time facial-expression and posture mimicry, while producing either verbally attuned or neutral responses. We evaluated the system in a 2 X 2 within-subjects study with 20 participants, manipulating verbal attunement and behavioral mimicry. Results showed that verbal attunement was the most reliable driver of perceived empathy. Behavioral mimicry showed a marginal relationship with perceived humanness, while greater mimicry exposure showed preliminary, exploratory positive associations with empathy, positivity, and humanness, particularly among female participants. Together, these findings show that multimodal synchrony is not a simple additive strategy for designing empathic conversational agents in VR and underscore the need to consider how verbal and nonverbal behaviors are combined during real-time interaction.