MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation

Mehrshad Taji; Arad Mahdinezhad Kashani; Iman Ahmadi; AmirHossein Jadidi; Saina Kashani; Babak Khalaj

cs.ROFeb 18, 2026

MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation

Authors: Mehrshad Taji, Arad Mahdinezhad Kashani, Iman Ahmadi, AmirHossein Jadidi, Saina Kashani, Babak Khalaj

Organizations: Department of Electrical Engineering Sharif University of Technology

Abstract

Task planning for robotic manipulation with large language models (LLMs) is an emerging area. Prior approaches rely on specialized models, fine tuning, or prompt tuning, and often operate in an open loop manner without robust environmental feedback, making them fragile in dynamic settings. MALLVI presents a Multi Agent Large Language and Vision framework that enables closed-loop feedback driven robotic manipulation. Given a natural language instruction and an image of the environment, MALLVI generates executable atomic actions for a robot manipulator. After action execution, a Vision Language Model (VLM) evaluates environmental feedback and decides whether to repeat the process or proceed to the next step. Rather than using a single model, MALLVI coordinates specialized agents, Decomposer, Localizer, Thinker, and Reflector, to manage perception, localization, reasoning, and high level planning. An optional Descriptor agent provides visual memory of the initial state. The Reflector supports targeted error detection and recovery by reactivating only relevant agents, avoiding full replanning. Experiments in simulation and real-world settings show that iterative closed loop multi agent coordination improves generalization and increases success rates in zero shot manipulation tasks. Code available at https://github.com/iman1234ahmadi/MALLVI .

Figures & tables

Figure 1: The MALLVi framework architecture. The pipeline processes user prompts through specialized agents: Decompose breaks instructions into atomic steps, Describe provides scene understanding, Perceive processes visual inputs, Ground localizes target objects, Project generates motion trajectories, Think coordinates high-level reasoning, Act executes robotic commands, and Reflect evaluates outcomes to enable iterative refinement and error recovery.

Figure 2: Comparison between single-agent and multi-agent frameworks.

Figure 3: Analysis of specialized agents and their roles in a multi-agent system. Each agent functions at a designated level (high, mid, or low) to address specific components of task execution, including instruction decomposition, memory utilization, object localization, task reasoning, action execution, and closed-loop feedback provision.

Figure 4: Example of our real-world tasks. Stack Blocks, Sort Shape, and Math Operation each combine a specific prompt with a physical environment to assess an agent’s ability to act and solve problems in tangible settings.

Figure 5: A real-world example of the Stack Blocks task. MALLVi is asked to stack the blocks in the order red, blue and green. The wooden block acts as a distraction.

Method	Place Food	Put Shape	Stack Blocks	Shopping List	Put in Mug	Math Ops	Stack Cups	Rearrange Objects
MALMM	75	65	55	70	55	25	50	-
VoxPoser	70	55	40	45	40	15	35	0
ReKep	80	85	75	90	75	60	40	60
Single-Agent	25	10	15	10	30	5	10	0
w/o Reflector	85	60	60	65	55	70	50	45
MALLVi (Ours)	100	95	90	90	80	80	85	75

Table 1: Success rates (%) on 8 real-world tasks with 20 repetitions.

Method	Simple Manipulation	Novel Concepts	Visual Reasoning	Visual Goal Reaching
Wonderful Team	100	85	90	-
CoTDiffusion	86	70	54	44
PERIA	93	78	76	68
Single-Agent	25	10	15	10
w/o Reflector	100	80	30	40
MALLVi (Ours)	100	95	90	73

Table 2: Success rates (%) on VIMABench tasks on 100 repetitions.

Method	Put in Safe	Put in Drawer	Stack Cups	Place Cups	Stack Blocks
PerAct	44	68	0	0	36
Single-Agent	58	73	15	22	42
w/o Reflector	81	89	63	75	78
MALLVi (Ours)	92	94	83	96	90

Table 3: Success rates (%) on RLBench tasks with 100 repetition.

Method	Simple Manipulation	Novel Concepts	Visual Reasoning	Visual Goal Reaching
GPT-4.1-Mini ( )	100	95	95	73
Qwen-3B w/ Qwen-VL	70	54	10	46
Qwen-7B w/ Qwen-VL	85	50	30	62
LLaMA-3.1-8B w/ LLaMA-Vision-3.2-11B	80	50	27	59

Table 4: Success rates (%) on open-source models over 100 repetitions.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Basketball in Hoop	Close Jar	Empty Container	Insert in Peg	Meat off Grill	Open Bottle	Put Block	Rubbish in Bin	Stack Blocks
MALMM	82	76	59	67	88	91	93	81	47
Single-Agent	45	37	34	26	41	78	89	41	22
w/o Reflector	78	67	42	57	82	86	95	83	78
MALLVi (Ours)	89	81	71	66	94	93	100	91	90

Appendix

Table 5: Success rates (%) on RLBench tasks with 100 repetition.

Figure 6: Credits to Wang et al. (2024c) for the figure. Prompts for Visual Manipulation , Novel Nouns , Without Exceeding , and Pick in Order then Restore tasks.

Category	Subtask	MALLVi (Ours)
Simple Manipulation	visual manipulation	100
	scene understanding	100
	rotate	100
Novel Concepts	novel adj	95
	novel noun	100
	novel adj and noun	90

Appendix

Table 6: Detailed success rates (%) for all VIMABench subtasks. This table provides a breakdown of the aggregated scores shown in Table 2 .

Figure 7: The pipeline is instructed to put the banana on the keyboard. The optimal grasp point for the robot’s end-effector is thus determined.

Explore similar work

A Closed-Loop Multi-Agent Framework for Robust Multi-Robot Manipulation
Jul 8, 2026Yi-Xiang He, Lan Wei, Haoming Cen +6Large Language Model-Based Robot Planning Multi-Task Robotic Manipulation
Sun Yat-sen University · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation
Oct 8, 2026Lonnie Tankersley, Noorbakhsh Amiri Golilarz, Soroush KorivandRobotic Manipulation Robot Task Planning
dept. of Aerospace Engineering, Mississippi State University, Starkville, MS · dept. of Computer Science, The University of Alabama, Tuscaloosa, AL · dept. of Mechanical Engineering, Mississippi State University, Starkville, MS
Demonstration-Free Robotic Control via LLM Agents
Jan 28, 2026Brian Y. Tsui, Alan Y. Fang, Tiffany J. HwuRobotic Control Language Model-Based Control
All authors are independent researchers.

Jul 8, 2026J/K move · Enter open · S save

A Closed-Loop Multi-Agent Framework for Robust Multi-Robot Manipulation

Yi-Xiang He, Lan Wei, Haoming Cen, Jian-Jian Jiang+5

Sun Yat-sen University · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China

Large Language Model-Based Robot Planning Multi-Task Robotic Manipulation Multi-Robot Coordination Robotic Manipulation

Open paper AskPDF arXiv

Multi-robot systems provide the parallelism and redundancy necessary for long-horizon tasks, while Large Language Models (LLMs) offer the reasoning capabilities to decompose these objectives into actionable plans. However, effectively grounding this high-level reasoning in physical multi-robot execution remains an open challenge. Existing LLM-based approaches fall mainly into two categories: Single-robot methods achieve robust contact-rich manipulation but lack the coordination mechanisms required for tasks spanning multiple workspaces. Current multi-robot frameworks focus on high-level planning, often treating manipulation as an idealized primitive that fails to account for real-world execution uncertainties. To address this, we propose a hierarchical closed-loop agentic LLM-based framework to ensure robust multi-robot manipulation. Our system consists of three specialized agents: the Planning Agent decomposes instructions into allocated sub-tasks, the Manipulation Agent for each robot executes actions via adaptive tool use, and the Verification Agent closes the loop by monitoring physical outcomes and feeding back semantic corrections. Extensive real-world experiments demonstrate that our framework achieves superior success rates, ensures robust adaptability ranging from single to cross workspace manipulation, and offers a generalizable approach for diverse manipulation tasks.

Abstract

Explore similar work

A Closed-Loop Multi-Agent Framework for Robust Multi-Robot Manipulation

Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation

Demonstration-Free Robotic Control via LLM Agents