Vision-Language Agents

Vision-language agents study how models can use images, video, and language to act in settings such as websites, mobile interfaces, games, robots, and social environments. Common approaches combine visual grounding with planning and action, often adding tools, reusable skills, persistent memory, multi-agent coordination, or reinforcement learning. Benchmarks test both isolated abilities and complete interactive tasks, including GUI navigation, embodied control, spatial reasoning, and real-time response. A central challenge is turning strong local perception and reasoning into reliable long-horizon behavior: agents still struggle with changing scenes, spatial state tracking, memory, and completing all parts of a task.