Organizations: Ant Group Shanghai, China · Zhejiang University Hangzhou, China · Ant Group Beijing, China · Ant Group Hangzhou, China · The Chinese University of Hong Kong Hong Kong, China
The EgoLink Challenge at ACM Multimedia 2026 benchmarks egocentric multimodal intelligence linking first-person perception, social reasoning, and interactive action. Unlike benchmarks centered on visible objects and activities, EgoLink evaluates whether models can interpret emotional and social cues, resolve visually grounded references, and convert user requests into verified tool-mediated actions. Its two tracks cover social reasoning in egocentric video and interactive agents that complete service tasks through dialogue, tool use, and closed-loop execution. Together, they test whether perception, reasoning, and decision-making remain coupled under both offline question answering and state-changing interaction. We highlight a multi-stage Track 1 data pipeline for multiple-choice question (MCQ) generation, chain-of-thought generation, automatic quality grading, and video-contribution filtering. We summarize the task design, evaluation metrics, rules, leaderboards, and representative solutions. The benchmark combines objective answer scoring with process- and result-based evaluation, enabling analysis of both final correctness and execution quality. The best Track 1 system achieved 0.9263 accuracy, while the best Track 2 system achieved 0.844 joint success and 0.916 micro accuracy. Participant reports reveal a shared principle: strong systems make intermediate evidence explicit through candidate verification, option-robust reasoning, scenario-aware planning, tool canonicalization, and state-aware correction before answering or acting. These findings highlight the importance of process verification and evidence-grounded reasoning for reliable egocentric multimodal intelligence.
Figures & tables
Figure 1. EgoLink challenge overview.
Figure 2. Video-Contribution Filtering for Track 1. Candidates are screened to reduce questions answerable without visual evidence.
Metric
Description
Track 1 accuracy
Overall top-1 MCQ accuracy over the evaluation questions.
Dimension scores
Scores for emotion, causality, intent, and summarization question groups.
Track 2 joint success
Task-level success requiring both correct tool process and correct database result.
Micro accuracy
Call-level tool accuracy for diagnosing fine-grained execution behavior.
Result/tool success
Separate result-based and tool-based scores that expose process-result mismatches.
Table 1. Evaluation signals used in EgoLink.
Rank
Team
Accuracy
1
GXU-LIPE
0.9263
2
Lai
0.9246
3
TeleAI
0.9132
4
EgoMinds
0.9092
5
wjdd
0.9089
5
MR-CAS
0.9089
Table 2. Official Track 1 leaderboard excerpt. Accuracy is the overall top-1 MCQ score reported on the challenge website.
Team
Joint
Micro
Result
Tool
MediaLab
0.844
0.916
0.875
0.854
GML-MM Group
0.823
0.914
0.854
0.823
BigVisionAgentGoGoGo
0.781
0.889
0.896
0.802
uestc_goat
0.729
0.814
0.760
0.740
EgoMinds
0.604
0.733
0.646
0.615
Table 3. Official Track 2 leaderboard excerpt. Joint success is the primary ranking metric.
Team
Track
Main mechanism
Reported evidence
GXU-LIPE
Track 1
Question-conditioned social evidence graph, candidate-wise verification, source-traceable reranking, low-confidence retraction, and all-option evidence calibration.
92.89% validation accuracy and 92.63% official test accuracy.
Lai
Track 1
Cold-start structured reasoning data, elimination-guided reward design, and option-perturbed DAPO that rewards semantic candidate identity rather than displayed option letters.
92.75% validation accuracy and 92.46% official challenge accuracy.
MediaLab
Track 2
Scenario-aware service agent, official-tool canonicalization, evidence ledger, and scope-limited correction for state-changing calls and final replies.
0.844 Joint and 0.916 Micro over five test scenarios.
GML-MM Group
Track 2
Status-aware grounded execution, Reporter summaries for evidence compression, step-specific tool views, and paired visual grounding for occluded references.
0.781 Joint on released retail1 ; 0.823 Joint over five test scenarios.
Table 4. Representative participant teams and their main technical mechanisms.
As AI agents increasingly operate in open, real-world environments, they require a deep synergy of multimodal perception, tool invocation with multi-hop reasoning, and dynamic interaction with users. However, existing benchmarks fail to jointly evaluate these capabilities due to challenges in designing strictly coupled multi-capability tasks, simulating natural and task-constrained user feedback, and ensuring objective evaluation of dynamic interaction. To bridge this gap, we introduce EgoBench, the first interactive multimodal benchmark for tool-using agents. EgoBench comprises 1,590 egocentric-video-grounded tasks covering five daily scenarios, along with a user-agent-tool interactive environment for evaluation. We implement a three-stage synergistic pipeline through which each task is designed to enforce the joint application of visual perception and tool-augmented multi-hop reasoning. We additionally develop a multi-agent simulated user to evaluate agents' interaction capabilities, which generates high-fidelity, task-aligned responses to agents. Furthermore, we establish a deterministic joint validation framework that guarantees objective assessment through process-based and result-based equivalence. Benchmarking eight video-MLLM agents on EgoBench reveals a severe performance ceiling: the best-performing model achieves only a 34.95% average Joint Success Rate across the three interaction modes. Finally, we conduct a multi-dimensional error analysis to disentangle failure modes, exposing capability bottlenecks for advancing future AI agents.
Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their tight coupling in real-world interaction. We propose the Ego-Action-World interaction loop as a unifying principle: the wearer perceives the world, forms goals, acts, changes the world, and uses feedback to guide subsequent behavior. Based on this view, we introduce EgoOmniWorld, a benchmark organized around four complementary interaction structures: Node captures ego, action, and world states; Edge captures directed relations among them; Chain captures linked dependencies; and Loop captures feedback-closed interactions. We develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from egocentric videos, grounds them in visual and auditory evidence, and generates questions according to their interaction structure. Required modalities and single- or multi-turn formats follow naturally from the supporting evidence and dependencies. EgoOmniWorld contains 24,166 QA groups, including 20,284 training and 3,882 test samples. Experiments on representative open- and closed-source multimodal large language models reveal substantial and uneven limitations across interaction structures. Models trained on EgoOmniWorld improve in-domain interaction understanding and generalize to existing egocentric benchmarks. EgoOmniWorld provides a unified foundation for evaluating and advancing egocentric video understanding. Data, code, and models will be released.
The rapid development of Multimodal Large Language Models (MLLMs) has led to growing interest in egocentric video understanding, specifically the ability for MLLMs to recognize fine-grained hand-object interactions, track object state changes over time, and reason about manipulative processes in dynamic environments from a first-person perspective. However, existing egocentric video benchmarks suffer from limited grounded rationale evaluation, offering limited support for fine-grained operation-centric reasoning and rarely examining whether model rationales are grounded in explicit spatio-temporal evidence. To address this gap, we introduce EgoCoT-Bench, a fine-grained egocentric benchmark for grounded and verifiable operation-centric reasoning with explicit step-by-step rationale annotations. Overall, EgoCoT-Bench comprises 3,172 verifiable QA pairs over 351 egocentric videos separated into four task groups for a total of 12 sub-task groups, encompassing perception and retrospection, anticipation, and high-level reasoning. The benchmark is constructed through a spatio-temporal scene graphs (STSG) guided generation framework and is further refined by human annotators to ensure correctness, egocentric relevance and fine-grained quality. Experimental results show continuing difficulties with egocentric fine-grained reasoning and further reveal that many multimodal models produce explanations that are answer-correct, but have evidence that is inconsistent with the answer. We hope EgoCoT-Bench can serve as a useful testbed for grounded and verifiable reasoning in egocentric video understanding. Project page and supplementary materials are available at: https://dstardust.github.io/EgoCoT/.