Seeing Eye to Eye: Enabling Cognitive Alignment Through Shared First-Person Perspective in Human-AI Collaboration

Zhuyu Teng; Pei Chen; Yichen Cai; Ruoqing Lu; Zhaoqu Jiang; Jiayang Li; Weitao You; Lingyun Sun

cs.HCMar 13, 2026

Seeing Eye to Eye: Enabling Cognitive Alignment Through Shared First-Person Perspective in Human-AI Collaboration

Authors: Zhuyu Teng, Pei Chen, Yichen Cai, Ruoqing Lu, Zhaoqu Jiang, Jiayang Li, Weitao You, Lingyun Sun

Organizations: College of Computer Science and Technology, Zhejiang University, Hangzhou, China

Abstract

Despite advances in multimodal AI, current vision-based assistants often remain inefficient in collaborative tasks. We identify two key gulfs: a communication gulf, where users must translate rich parallel intentions into verbal commands due to the channel mismatch , and an understanding gulf, where AI struggles to interpret subtle embodied cues. To address these, we propose Eye2Eye, a framework that leverages first-person perspective as a channel for human-AI cognitive alignment. It integrates three components: (1) joint attention coordination for fluid focus alignment, (2) revisable memory to maintain evolving common ground, and (3) reflective feedback allowing users to clarify and refine AI's understanding. We implement this framework in an AR prototype and evaluate it through a user study and a post-hoc pipeline evaluation. Results show that Eye2Eye significantly reduces task completion time and interaction load while increasing trust, demonstrating its components work in concert to improve collaboration.

Figures & tables

Figure 1. An example of the Eye2Eye collaborative loop applied to a family book organization task. Wearing AR glasses, the user hesitates while deciding where to place a book in response to the AI assistant’s suggestion. The AI assistant interprets implicit cues from eye gaze and hand-object interaction (joint attention), infers intent, and consults accumulated common ground (update and retrieve memory) to refine its understanding of the user’s personalized organization rules. Based on this, it generates situated feedback through multimodal outputs (e.g., voice prompts, visual overlays), suggesting a personalized classification (e.g., creating a new “Children’s Books” area), rather than suggesting generic categories. The closed loop process illustrates how Eye2Eye transforms shared perception into shared understanding and coordinated action. A composite image illustrating the "Eye2Eye" concept in a book classification task. Left: The user's first-person view through AR glasses. The user holds a book, and the system highlights the book with a bounding box and detects the user's gaze point. A voice prompt bubble shows the user asking, "I should put this book in…?". Right: A system diagram showing the collaborative loop. The loop connects the user's inputs~(Eye Gaze, Hand-Object Interaction) to three AI processing stages: (1) Joint Attention~(Identify Objects), (2) Common Ground~(Retrieve & Update Memory), and (3) Situated Feedback~(Classify Actions & Select Strategy), which then feeds back to the user via AI Feedback.

Challenge Type	Subcategory	Example in Furniture Assembly
Communication gulf	Attention mismatch	The user points to a side panel in the corner and says “put this here,” but the AI assistant defaults to the central wooden board, giving incorrect guidance.
	Over-verbalization burden	The user asks “Which screw should I use?” and the AI assistant responds with a long explanation of screw material and threading standards, instead of simply saying “use the 5mm screw.”
Understanding gulf	Intervention blindness	The user hesitates between using a drill or a screwdriver, pausing with their hand above the tools. The AI assistant fails to interpret the hesitation as a cue for timely guidance.
	Common ground drift	The user decides to mount the drawer on the left for personal convenience, but the AI assistant continues to insist on the default right-side placement specified in the manual.

Table 1. Typical challenges in collaborating with wearable AI assistants, taking the furniture assembly task as examples.

Figure 2. Framework overview of Eye2Eye . The framework illustrates the bidirectional cognitive loop between humans (left) and AI (right), mediated by three core components. The process flows from (1) joint attention, aligning and interpreting the focus of attention, to (2) common ground, enriching and updating shared memory, and then to (3) situated feedback, acting and reflecting based on context. This feedback loop, in turn, updates the shared memory and can proactively generate new attention cues. A triangular flow diagram representing the \textit{Eye2Eye} framework's cognitive loop centered on a Shared First-Person Perspective. It illustrates a bidirectional cycle between three components: Joint Attention, Common Ground, and Situated Feedback. Clockwise Flow: Human cues~(Gaze, Speech, Gesture) initiate the information flow~($I \rightarrowII \rightarrowIII$), driving the system to perceive and act1111. Counter-Clockwise Flow: AI reflection and memory updates~($III \rightarrowII \rightarrowI$) cycle back to refine the system's understanding based on user feedback. — Figure 2. Framework overview of Eye2Eye . The framework illustrates the bidirectional cognitive loop between humans (left) and AI (right), mediated by three core components. The process flows from (1) joint attention, aligning and interpreting the focus of attention, to (2) common ground, enriching and updating shared memory, and then to (3) situated feedback, acting and reflecting based on context. This feedback loop, in turn, updates the shared memory and can proactively generate new attention cues. A triangular flow diagram representing the *Eye2Eye* framework's cognitive loop centered on a Shared First-Person Perspective. It illustrates a bidirectional cycle between three components: Joint Attention, Common Ground, and Situated Feedback. Clockwise Flow: Human cues~(Gaze, Speech, Gesture) initiate the information flow~( $I \rightarrowII \rightarrowIII$ ), driving the system to perceive and act1111. Counter-Clockwise Flow: AI reflection and memory updates~( $III \rightarrowII \rightarrowI$ ) cycle back to refine the system's understanding based on user feedback.

Figure 3. Bidirectional attention in a shared first-person perspective: AI captures and infers human’s attention from explicit and implicit cues, subsequently aligning their attention and providing guidance through multimodal feedback. A matrix of illustrations showing how attention is exchanged. Left~(Human Cues): Shows explicit cues~(User pointing, asking ``What's this?'') and implicit cues~(User gazing at an object, grasping or touching a box). Right~(AI Cues): Shows how AI responds. Visual feedback places a virtual label next to the object; Auditory feedback provides a spoken explanation (``These are Italian coffee beans…''); Textual feedback displays a floating caption~(``Coffee Beans''). — Figure 3. Bidirectional attention in a shared first-person perspective: AI captures and infers human’s attention from explicit and implicit cues, subsequently aligning their attention and providing guidance through multimodal feedback. A matrix of illustrations showing how attention is exchanged. Left~(Human Cues): Shows explicit cues~(User pointing, asking `What's this?'') and implicit cues~(User gazing at an object, grasping or touching a box). Right~(AI Cues): Shows how AI responds. Visual feedback places a virtual label next to the object; Auditory feedback provides a spoken explanation (`These are Italian coffee beans…''); Textual feedback displays a floating caption~(``Coffee Beans'').

Figure 4. The technical pipeline of the implemented prototype: (1) attention trigger and interaction event understanding; (2) retrieving and revising common ground for cognitive alignment; and (3) executing the feedback strategy. A technical flowchart showing the data processing pipeline from input to output. Input: A video stream showing a user holding a bag of coffee beans. Component I (Joint Attention): The system processes video frames, speech ("How should I add coffee beans?"), and segmentation masks to detect the "Coffee Beans" label and user intent. Component II (Common Ground): The system retrieves relevant "Object Cards" (e.g., Coffee Machine, Coffee Hopper) to contextualize the current state. Component III (Situated Feedback): A VLM generates a strategy (Action Planning) and outputs multimodal feedback~(Voice: "You can open the bean hopper…", Visual: highlighting the hopper lid).

Figure 5. Details of the dynamic, self-correcting memory unit structure: each analysis of the current state ( $c_{state}$ ) retrieves existing object cards to either create a new common ground unit or update an existing one, enabling persistent accumulation. Updates to the AI response are based on reflection of user feedback from interactions, achieving cognitive refinement. A diagram detailing the "Object Card" dynamic memory structure. It shows a sequence where a new interaction updates an existing memory card. Initial State: A card for ``Coffee Hopper'' exists. Update Process: The user interacts with the hopper. The system analyzes the ``Current State'' and ``User Intent''~(changing from adding powder to adding beans). Result: The Object Card is updated with new ``User_feedback_details'' and a revised ``AI Response'' strategy, demonstrating how the system learns from correction. — Figure 5. Details of the dynamic, self-correcting memory unit structure: each analysis of the current state ( $c_{state}$ ) retrieves existing object cards to either create a new common ground unit or update an existing one, enabling persistent accumulation. Updates to the AI response are based on reflection of user feedback from interactions, achieving cognitive refinement. A diagram detailing the "Object Card" dynamic memory structure. It shows a sequence where a new interaction updates an existing memory card. Initial State: A card for `Coffee Hopper'' exists. Update Process: The user interacts with the hopper. The system analyzes the` Current State'' and `User Intent''~(changing from adding powder to adding beans). Result: The Object Card is updated with new` User_feedback_details'' and a revised ``AI Response'' strategy, demonstrating how the system learns from correction.

Figure 6. Mapping of situation types to feedback strategy modalities and categories, with representative examples for each case. A table summarizing the mapping logic between four general situation categories and their corresponding feedback modalities. Object Recognition: Uses Text label + Short voice for quick identification~(e.g., naming attributes). Error Checking: Uses Visual overlay + Detailed voice for spatial correction and explanation~(e.g., fixing installation errors). Knowledge Recall: Uses Text label + Detailed voice for delivering factual information~(e.g., providing rationale). Action Planning: Uses Visual overlay + Short voice for immediate spatial guidance~(e.g., procedural steps). The table further details specific sub-categories and provides concrete examples for each case.

Figure 7. Experimental tasks for evaluating human-AI collaboration, including coffee machine operation, book classification, and circuit board troubleshooting. A composite image displaying the three experimental tasks designed to evaluate the system. Task 1~(Left): Coffee Machine Operation. A semi-automatic espresso machine is shown with labels pointing to key components: the Bean Hopper~(top), Water Tank~(side), and Control Panel~(front). This represents a procedural task. Task 2~(Middle): Book Classification. A collection of books is laid out on a surface. Callouts indicate three classification criteria: Language~(English/Chinese), Appearance~(cover style), and Subject~(content). This represents a classification task based on subjective rules. Task 3~(Right): Circuit Board Troubleshooting. A breadboard containing electronic circuits is shown. Labels identify the components to be inspected: LED lights, Buttons, and Resistors. This represents an inspection task requiring visual search.

Task Type	Condition	Completion Time(s) $\downarrow$		Error Rate(%) $\downarrow$		Clarification Cost $\downarrow$		Interaction Turns $\downarrow$
		$EMM$	$SE$	$EMM$	$SE$	$EMM$	$SE$	$EMM$	$SE$
Procedural	Baseline	430.00	29.90	11.02	1.47	1.27	0.22	16.34	0.97
	System	365.00	24.70	5.07	1.15	0.87	0.18	12.54	0.80
Classification	Baseline	309.00	20.90	14.50	1.90	1.68	0.25	12.76	0.81
	System	354.00	23.90	5.10	1.17	0.55	1.34	12.34	0.79
Inspection	Baseline	310.00	20.90	7.39	1.55	0.64	0.11	9.83	0.68

Table 2. Task performance metrics across conditions and task types. Values represent estimated marginal means (

EMMs

) and standard errors (

SEs

) derived from LME/GLMM (

\downarrow

indicates lower values denote better performance).

Figure 8. Collaborative performance comparison between Eye2Eye and the baseline system assessed using linear mixed-effects models. (A) Growth curves of cumulative error rates over interaction rounds. (B) Subjective ratings of intent expression modalities (gaze, gesture, speech), showing estimated marginal means (EMMs) and 95% CI. (A) Line Charts~(Error Rate): Three charts showing "Cumulative Error Rate" over interaction rounds. The "System" line (Eye2Eye, yellow) remains significantly lower and flatter than the "Baseline" line (gray) across Overall, Procedural, and Inspection tasks, indicating \textit{Eye2Eye} prevents error accumulation. (B) Dot Plots~(Modality Preference): User ratings for Gaze, Gesture, and Speech. Speech is consistently rated highest~(most effective) across all tasks. Gaze and Gesture show wider variance, with Gaze being rated highly in the Classification task. — Figure 8. Collaborative performance comparison between Eye2Eye and the baseline system assessed using linear mixed-effects models. (A) Growth curves of cumulative error rates over interaction rounds. (B) Subjective ratings of intent expression modalities (gaze, gesture, speech), showing estimated marginal means (EMMs) and 95% CI. (A) Line Charts~(Error Rate): Three charts showing "Cumulative Error Rate" over interaction rounds. The "System" line (Eye2Eye, yellow) remains significantly lower and flatter than the "Baseline" line (gray) across Overall, Procedural, and Inspection tasks, indicating *Eye2Eye* prevents error accumulation. (B) Dot Plots~(Modality Preference): User ratings for Gaze, Gesture, and Speech. Speech is consistently rated highest~(most effective) across all tasks. Gaze and Gesture show wider variance, with Gaze being rated highly in the Classification task.

Figure 9. Subjective ratings of collaboration experience assessed using Mann-Whitney U tests. (A) Boxplots of NASA-TLX workload scores, showing medians, interquartile ranges (IQR), and whiskers (outliers shown as points). (B) Distributions of individual responses on five collaboration measures (7-point Likert scale: fluency, intrusiveness, copresence, shared awareness, performance trust), with mean values ± SE indicated. (A) Boxplots (NASA-TLX): Comparison of workload scores. The "System" (yellow) shows lower medians (better) than "Baseline" (gray) for Mental Demand, Physical Demand, Temporal Demand, Effort, and Frustration. Performance score is higher for the System. (B) Dot Plots (Collaboration Quality): 7-point Likert scale ratings. The "System" (blue dots) shows a strong shift towards "Strongly Agree" (scores 6-7) for Copresence, Shared Awareness, and Performance Trust compared to the Baseline.

Figure 10. Data preparation and evaluation workflow for the offline ablation study. The pipeline illustrates how 18 source clips from the user study were processed across four conditions to generate 720 outputs, which were then evaluated by 12 raters to yield a final dataset of 2,880 ratings.

Figure 11. Ratings for the accuracy and usefulness across different pipeline conditions. (A) Overall accuracy and usefulness ratings with estimated marginal means (EMMs) and 95% CI. (B) Temporal robustness trends over interaction with mean scores. (A) Bar Charts: Comparison of Accuracy and Usefulness across four conditions: Ours, w/o-JA (No Joint Attention), w/o-CG-SF (No Common Ground/Feedback), and Baseline. "Ours" (yellow bar) is significantly higher than all other conditions, which are roughly equal. (B) Line Charts: Temporal robustness. The "Ours" line maintains high accuracy and usefulness over time (frames 1-10), while all ablated versions show a declining trend, proving the stability of the full framework.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	Ours	w/o-JA	w/o-CG-SF	w/o-JA-CG-SF (baseline)
Memory	Can access and update object cards	Can access and update object cards	Cannot access or update object cards	Cannot access or update object cards
Attention	Uses joint attention alignment	Removes joint attention alignment, but still uses target bounding boxes and object nodes to maintain object-level context	Uses joint attention alignment	Removes joint attention alignment
Input Stream	Context, image frames, bounding boxes, content of object nodes, memory updates	Context, image frames, bounding boxes, content of object nodes, memory updates	Context, image frames, bounding boxes	Context, image frames, bounding boxes
Trigger Threshold	Fixates on the same object for $\geq$ 6 seconds,, or triggers when overlap threshold 0.85 is reached	Fixates on the same object for $\geq$ 6 seconds, or triggers when overlap threshold 0.85 is reached	None	None
Feedback Strategy	Produces situational intention predictions, multimodal suggestion prompts, and memory-update feedback	Produces situational intention predictions, multimodal suggestion prompts, and memory-update feedback	Produces only situational intention predictions and multimodal suggestion prompts	Produces only situational intention predictions and multimodal suggestion prompts
Output	Updates object cards according to memory-update feedback; logs process data; sends multimodal suggestions to the frontend	Updates object cards according to memory-update feedback; logs process data; sends multimodal suggestions to the frontend	Logs process data; sends multimodal suggestions to the frontend	Logs process data; sends multimodal suggestions to the frontend

Appendix

Table 3. Configurations of full Eye2Eye pipeline (ours) and the ablation variants.

Explore similar work

CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views
Jul 7, 2026Alexey Gavryushin, Dingxi Zhang, Zhao Huang +8Human Activity Recognition Theory of Mind
1ETH Zurich, Switzerland · 2MPI for Informatics, Germany · 3Microsoft Switzerland +2
From Gaze to Guidance: Interpreting and Adapting to Users' Cognitive Needs with Multimodal Gaze-Aware AI Assistants
Apr 9, 2026Valdemar Danry, Javier Hernandez, Andrew Wilson +2Multimodal Agents Human-AI Collaboration
Microsoft Research Cambridge, MA, USA · MIT Media Lab Cambridge, MA, USA · Microsoft Research Cambridge, Massachusetts, USA
Alignment-Process-Outcome: Rethinking How AIs and Humans Collaborate
Mar 9, 2026Haichang Li, Anjun Zhu, Arpit NarechaniaHuman-AI Collaboration Human-AI Decision Making
George Mason University, Fairfax, Virginia, USA · Simon Fraser University, Burnaby, British Columbia, Canada · The Hong Kong University of Science and Technology, Hong Kong S.A.R., China

Jul 7, 2026J/K move · Enter open · S save

CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views

Alexey Gavryushin, Dingxi Zhang, Zhao Huang, Alexandros Delitzas+7

1ETH Zurich, Switzerland · 2MPI for Informatics, Germany · 3Microsoft Switzerland+2

Human Activity Recognition Theory of Mind Egocentric Video Understanding

Open paper AskPDF arXiv

Human-human collaboration is a fundamental aspect of everyday life, essential to success in a wide range of goal-directed activities from household tasks to professional teamwork. While much research has focused on modeling coordination and task execution, the cognitive processes that support such collaboration, particularly Theory of Mind (the ability to infer the mental states of others), remain difficult to study in natural settings. To address this gap, we introduce a novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios. The dataset integrates multi-perspective video, high-quality audio, gaze tracking, and 3D scene and object scans, with annotations for shared attention to objects, social cues and interactions between agents, as well as agent-object interactions. We establish benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction, enabling research on multimodal perception, proactive assistance, and collaborative planning. By providing temporally aligned, richly annotated multimodal data, CoMind facilitates the development and evaluation of AI systems capable of modeling complex social interactions and reasoning about human behaviors in collaborative environments. Our dataset and benchmarks are made available at https://comind.ethz.ch/.

Abstract

Explore similar work

CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views

From Gaze to Guidance: Interpreting and Adapting to Users' Cognitive Needs with Multimodal Gaze-Aware AI Assistants

Alignment-Process-Outcome: Rethinking How AIs and Humans Collaborate