Multimodal Agents

Multimodal agents combine language models with visual, audio, and video perception to answer questions, use tools, and interact with apps, websites, robots, or other environments. Common approaches include planning and tool orchestration, specialized collaborating agents, active evidence gathering, and memory or reusable skills for long tasks; reinforcement learning and execution feedback are increasingly used to improve these behaviors. Benchmarks now test browsing, GUI use, video reasoning, and embodied tasks, often evaluating action traces and process reliability as well as final answers. Despite progress, agents remain brittle over long horizons, and more modalities or more agents do not automatically improve grounding, safety, efficiency, or task success.