cs.IRApr 2, 2026

Multi-Agent Video Recommenders: Evolution, Patterns, and Open Challenges

Authors: Srivaths Ranganathan, Abhishek Dharmaratnakar, Anushree Sinha, Debanshu Das

Organizations: Google LLC, Mountain View, USA · Google LLC, San Bruno, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Video recommender systems are among the most popular and impactful applications of AI, shaping content consumption and influencing culture for billions of users. Traditional single-model recommenders, which optimize static engagement metrics, are increasingly limited in addressing the dynamic requirements of modern platforms. In response, multi-agent architectures are redefining how video recommender systems serve, learn, and adapt to both users and datasets. These agent-based systems coordinate specialized agents responsible for video understanding, reasoning, memory, and feedback, to provide precise, explainable recommendations. In this survey, we trace the evolution of multi-agent video recommendation systems (MAVRS). We combine ideas from multi-agent recommender systems, foundation models, and conversational AI, culminating in the emerging field of large language model (LLM)-powered MAVRS. We present a taxonomy of collaborative patterns and analyze coordination mechanisms across diverse video domains, ranging from short-form clips to educational platforms. We discuss representative frameworks, including early multi-agent reinforcement learning (MARL) systems such as MMRF and recent LLM-driven architectures like MACRec and Agent4Rec, to illustrate these patterns. We also outline open challenges in scalability, multimodal understanding, incentive alignment, and identify research directions such as hybrid reinforcement learning-LLM systems, lifelong personalization and self-improving recommender systems.

Figures & tables

Figure 1. Illustration of Multi-agent Video Recommender patterns highlighting an example for each pattern in Section 3.
PatternPrimary Evaluation FocusRepresentative MetricsCritical Failure Points & Risks
Hierarchical Orchestration (e.g., MMRF, MMAgentRec)Orchestration Effectiveness: How well does the central agent integrate diverse sub-goals to optimize the primary system objective?Main objective metric (e.g., WatchTime), contribution weights (from attentive mechanism), system-wide latency.Coordinator Bottleneck: The central agent becomes a single point of failure. Conflicting Goals: Auxiliary agents may work at cross-purposes, harming the main objective.
Pipeline-based Modular (e.g., VRAgent-R1, MACRec)End-to-End Task Quality: How well does the final output perform after passing through all sequential stages?Quality of intermediate outputs, error propagation rate, end-to-end latency.Compounding Errors and Brittleness: An error in an early agent (e.g., IP Agent) can degrade the entire chain.
User-Agent Collaboration (e.g., TKGPT)User-Perceived Agency: Does the user feel in control and satisfied with the system’s response to their natural language commands?User satisfaction (SUS scores), task success rate (from user studies), latency from command to feed update.Misinterpretation: The system may misunderstand the user’s (often ambiguous) intent and make drastic, undesirable changes to recommendations.
User Simulation Ensemble (e.g., Agent4Rec)Behavioral Fidelity: How accurately does the simulated agent population replicate the statistical properties of real human users?KL divergence (or similar) between simulated and real interaction distributions; replication of known macro-effects (e.g., filter bubbles).Lack of Generalization: Agents overfit to initialization data and fail to model novel behaviors. Prohibitive Cost: High computational overhead for running thousands of LLM agents.
Table 1. Evaluation of collaborative multi-agent video recommender architectures. Metrics emphasize coordination, user alignment, and computational feasibility.
Figure 2. Challenges and Future Research Directions for Multi-Agent Video Recommendation Systems (MAVRS).
More in this topic

Recommender Systems

Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.

See all 514 papers

Explore similar work

CardsList
May 1, 2026

Scaling Video Understanding via Compact Latent Multi-Agent Collaboration

Multi-modal large language models (MLLMs) advance vision language understanding but face inherent limitations in long-video tasks due to bounded perception context budgets. Existing agentic methods mitigate this via rule-based preprocessing, yet often suffer from information loss, high cost, and reliance on textual intermediates. We propose MACF, an end-to-end Multi-Agent Collaboration Framework that decouples per-agent perception budgets from global video complexity, enabling scalable video understanding while preserving visual fidelity. MACF partitions videos into segments for locally budgeted agents and enables holistic reasoning via an agent-native latent communication protocol. Each agent encodes partial observations into compact, task-sufficient tokens in a shared embedding space, allowing efficient and information-preserving collaboration by a central coordinator. We introduce a curriculum training strategy that progressively enforces semantic alignment, evidence summarization, and cross-agent coordination. Extensive experiments on diverse video understanding benchmarks show that MACF consistently outperforms state-of-the-art MLLMs and multi-agent systems under identical budget constraints, demonstrating the effectiveness of our latent collaboration for scalable video understanding.

Kerui Chen, Jinglu Wang, Jianrong Zhang +3

Microsoft Research Asia · Zhejiang University · Guanming Lab.

Multi-Agent CollaborationLong-Video UnderstandingMulti-Agent CommunicationMulti-Agent LLMsTemporal Video Understanding
AskPDF
Jun 8, 2026

MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding

The dominant paradigm in video retrieval relies on embedding-based full-corpus scanning, which suffers from inherent computational inefficiency and the semantic asymmetry between information-dense videos and sparse textual queries. To bridge this gap, we introduce MAVIS, a novel multi-agent framework that rethinks retrieval as cooperative reasoning rather than brute-force search. MAVIS first bridges the granularity mismatch by parsing raw videos into a Structured Semantic Library, enabling explicit attribute-level indexing. During retrieval, a planner decomposes complex user intents into atomic sub-tasks, dispatching specialized agents to independently nominate candidates. Crucially, MAVIS employs a Logic-aware Debate mechanism with a strict veto protocol, where agents collaboratively prune logical mismatches to identify a compact set of ``controversial'' candidates for fine-grained verification. This agentic workflow effectively bypasses the inefficiency of full-library traversal. Extensive experiments on MSR-VTT, MSVD, and ActivityNet demonstrate that MAVIS achieves competitive performance without task-specific fine-tuning, offering a scalable and interpretable alternative to traditional dual-encoder approaches.

Jie Zhang, Qilang Ye, Hao Zhou +2

School of Computing and Information Technology, Great Bay University · College of Computer Science, Nankai University · Tsinghua Shenzhen International Graduate School, Tsinghua University +1

Video UnderstandingMulti-Agent CollaborationAgentic RetrievalInformation RetrievalVision-Language RetrievalVideo Retrieval
AskPDF
Sep 21, 2026

Agentic Video Understanding: A Survey

As large language models (LLMs) become capable of processing increasingly diverse modalities and longer temporal contexts, an emerging line of work is moving beyond fixed video-language inference toward agentic systems that actively decide what information to inspect, retain, verify, and act upon. This survey reviews video understanding agents: systems that use video as the primary information source and solve understanding tasks through adaptive state construction and action selection. We first formalize an agent loop for video understanding, then address a central question: why do agents matter for video understanding? To answer this, we organize the literature through a challenge-to-design taxonomy, linking context bottlenecks to hierarchical evidence memory, evidence sparsity to active evidence acquisition, temporal causality to state and process tracking, and multimodal ambiguity to role-specialized coordination. We further review state space paradigms, learning paradigms, supervision signals, benchmarks, and evaluation protocols. Finally, we identify open directions toward agentic-native temporal modeling and video-native agents. Project page: https://github.com/DXY0711/Awesome-Agentic-Video-Understanding

Xinyu Deng, Siwen Luo, Daochang Liu

The University of Western Australia

Temporal Video UnderstandingMemory-Augmented Video Understanding
AskPDF