As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future. To ensure better communication between human users and the multi-agent system, it is crucial to interpret incoming information from agents in parallel and refer to the appropriate context for each query. Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video and correctly aggregating multiple egocentric videos to construct system-level memory. In this work, we first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents. To facilitate research in this direction, we introduce MultiAgent-EgoQA (MA-EgoQA), a benchmark designed to systemically evaluate existing models in our scenario. MA-EgoQA provides 1.7k questions unique to multiple egocentric streams, spanning five categories: social interaction, task coordination, theory-of-mind, temporal reasoning, and environmental interaction. We further propose a simple baseline model for MA-EgoQA named EgoMAS, which leverages shared memory across embodied agents and agent-wise dynamic retrieval. Through comprehensive evaluation across diverse baselines and EgoMAS on MA-EgoQA, we find that current approaches are unable to effectively handle multiple egocentric streams, highlighting the need for future advances in system-level understanding across the agents. The code and benchmark are available at https://ma-egoqa.github.io.
Figures & tables
Figure 1
Figure 2: Examples from MA-EgoQA across five categories. MA-EgoQA is the first multiple embodied agents egocentric video QA benchmark, requiring comprehension of six egocentric videos spanning seven days per query. False options are omitted.
Figure 3: Statistics for MA-EgoQA; (Left) number of samples per category; (Center) day/agent reference counts of all categories; (Right) question type distribution.
Figure 4: Benchmark construction pipeline. QA pairs are generated for each category through its dedicated process, and refined through LLM filtering and manual check.
Figure 5: Overview of EgoMAS. EgoMAS builds an event-based shared memory from multi-agent videos and dynamically retrieve relevant information from it.
Figure 6: Comparison of inference latency and accuracy.
Figure 7Table 8
Figure 9: Case Study: Comparative analysis across models on MA-EgoQA
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Ablation studies on shared memory retrieval size ( n ), agent-wise dynamic retrieval size ( k ), and score threshold ( τ ).
Figure S2 : Accuracy of EgoMAS with limited agents.
Table S0 : Social Interaction (SI) - Single Span category samples
Table S1 : Social Interaction (SI) - Multi Span category samples
Category
Task Coordination, Single Span
Question
How did they resolve concerns about the safety of the power strip?
(A) Lucia conducted a test and decided it was fine for usage.
(B) Shure insisted safety wasn’t a big issue and moved forward without discussion.
(C) Shure reassured everyone it was brand-new and safe to use.
(D) Katrina brought over a different power strip for better safety.
(E) Jake and Alice discussed using a reputable brand and reassured others.