InstAgent: A Recursively Self-Improving Agent Operating System for Heterogeneous Robot Collaboration

Haoyang Luo; Junhao Qin; Jingyan Cao; Yi Huang; Guangjie Yang; Xiaoyu Chen; Haotao Lu; Tianxiang Gui

cs.ROOct 16, 2026

InstAgent: A Recursively Self-Improving Agent Operating System for Heterogeneous Robot Collaboration

Authors: Haoyang Luo, Junhao Qin, Jingyan Cao, Yi Huang, Guangjie Yang, Xiaoyu Chen, Haotao Lu, Tianxiang Gui,

Organizations: ShanghaiTech University, Shanghai, China. · InstAdapt, Shanghai, China.

Abstract

Heterogeneous robot teams enable complex, long-horizon tasks by combining complementary sensing, mobility, and manipulation capabilities across multiple embodiments. Yet existing systems struggle to retain execution-time corrections as reusable experience, maintain task-relevant context memory across space and time for reliable handoffs, and explicitly represent the inter-robot dependencies required for structured collaboration. In order to address these challenges, we present InstAgent, a recursively self-improving (RSI) Agent Operating System for heterogeneous robot collaboration. InstAgent organizes team behaviors as agentic behavior trees (BTs) that explicitly model capability assignment, inter-robot dependencies and handoff checks. The RSI mechanism updates these collaboration programs from execution feedback and preserves successful revisions for reuse in future tasks, while shared multimodal spatio-temporal memory provides context for planning and handoff validation. Experiments on 20 real-world multi-robot tasks with three high-level models show consistent gains in task success, first-attempt completion, and execution efficiency.

Figures & tables

System	Collab.	Spatial memory	RSI
System	Collab.	Spatial memory	Experience-Context	Structured-Program
LRLL [ 3 ]	$\times$	$\times$	$\checkmark$	$\checkmark$
LLM-HBT [ 10 ]	$\checkmark$	$\times$	$\times$	$\times$
PhyAgentOS [ 4 ]	$\times$	$\times$	$\checkmark$	$\times$
ABot-AgentOS [ 15 ]	$\times$	$\checkmark$	$\checkmark$	$\times$
HoloAgent-0 [ 7 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\times$

TABLE I: Capability comparison of agentic robotic systems.

\checkmark

: explicitly reported;

\times

: not explicitly reported.

Fig. 2: BT-based RSI from Skill BTs to collaboration programs. (a) Skill BTs provide reusable capability procedures for task composition. (b) HeteroComposer binds retrieved subtrees to task-specific roles and arguments and composes them with inter-robot handoff conditions. (c) The RSI Engine retains validated programs with linked context documents for inheritance and further revision. BT-A, BT-B, and BT-C denote example Skill BTs associated with robots A, B, and C. The service case in Section IV-B reports a human-triggered revision followed by automatic reuse.

Fig. 3: SMAR-T shared memory and the InstAgent memory loop. (a) Multiple robots collect observations in a shared map. (b) Each memory entry couples an image and its caption with visual and caption embeddings, the observing robot’s pose, timestamp, and source identity. (c) InstAgent decomposes the request and selects retrieval tools. In the illustrated example, spatial and temporal constraints define the retrieval scope, followed by visual recall and caption-assisted reranking. The Agent evaluates the returned candidates and their metadata against the request. Insufficient evidence triggers further retrieval; sufficient evidence supports a grounded response and, when requested, task execution through the Coordinator and BT Executive.

GPT-5.6 Sol
Configuration	SR $\uparrow$	FSR $\uparrow$	LC $\uparrow$	$\bar{R}\downarrow$
Simple CoT	25/40	20/40	—	1.60
BT Recovery ( $E_{0}$ )	32/40	30/40	35/40	1.10
Fixed-Experience BT ( $E_{1}$ )	34/40	32/40	37/40	0.85
Full InstAgent ( $E_{2}$ )	38/40	35/40	39/40	0.40
DeepSeek-V4-Pro

TABLE II: Real-robot performance across high-level models, execution configurations, and procedural-experience stages.

Subset	RAVEN	ReMEmbR	Ours	Ours w/o Agent
RealWorld	89.05	76.67	95.24	76.19
Habitat-Sim	80.47	54.99	93.79	63.35
IRS	96.30	92.60	98.15	92.59
IRS-hard	82.90	58.50	82.93	58.54
Overall	84.57	64.49	93.14	69.31

TABLE III: Query-level task success rate (SR, %) on RAVEN-QA. Higher is better.

Agentic AI for Robotics

See all 39 papers

CollabFlow: Recursive Self-Improvement of Agent Collaboration
Sep 29, 2026Xiao Huang, Mingda Zhang, Junming Zhang +5Multi-Agent LLM Systems Multi-Agent Collaboration
The Chinese University of Hong Kong, Shenzhen · Fudan University · University of Oxford
DynaHMRC: Decentralized Heterogeneous Multi-Robot Collaboration for Dynamic Tasks with Large Language Models
Jun 12, 2026Wenhao Yu, Yu'ang Xie, Yifan Duan +5Multi-Agent LLM Systems Multi-Robot Coordination
University of Science and Technology of China (USTC) · University of Macau (UM)
RoboRSI: Stable, efficient, and reusable robot self-evolution in complex real-world environments
Oct 8, 2026Zimo Wen, Yijin Chen, Yuxuan Cao +8Robot Skill Learning Continual Robot Learning

Sep 29, 2026J/K move · Enter open · S save

CollabFlow: Recursive Self-Improvement of Agent Collaboration

Xiao Huang, Mingda Zhang, Junming Zhang, Qiang Huang+4

The Chinese University of Hong Kong, Shenzhen · Fudan University · University of Oxford

Multi-Agent LLM Systems Multi-Agent Collaboration Multi-Agent Systems Self-Improving Agents Multi-Agent Communication

Open paper AskPDF arXiv

Recursive self-improvement (RSI) lets a system improve from its own outcomes; in LLM-based multi-agent systems, Agents refine one another within a task, and outcomes improve how they collaborate across tasks. However, existing multi-agent collaboration leaves this loop open: collaboration is pre-defined at the operator level, topology-only learning keeps verbatim exchange that propagates errors, and reward maximization on a system's own outcomes concentrates on a few teams. To address these challenges, we propose CollabFlow, an RSI system of Learned Agent Collaboration: a trainable Collab-Director constructs teams of complete Agents, a frozen executor runs them, and each round's outcomes retrain the director. Within each round, the edges of a collaboration graph carry protocols of Evidence-Conditioned Communication: a receiver adopts a differing answer only when the sender's evidence is stronger by a margin, so the director learns who communicates and how. Across rounds, we further propose Collaborative Trajectory Balance (CTB), a flow-based objective that credits each team once across its construction orders and targets a reward-proportional distribution over teams, so several good teams stay in play. We also bound how far this self-generated target moves between rounds, which shrinks as records accumulate. On twelve datasets, CollabFlow outperforms all baselines and keeps improving across rounds. Code is available at https://anonymous.4open.science/r/CollabFlow-631E.

Abstract

Agentic AI for Robotics

Explore similar work

CollabFlow: Recursive Self-Improvement of Agent Collaboration

DynaHMRC: Decentralized Heterogeneous Multi-Robot Collaboration for Dynamic Tasks with Large Language Models

RoboRSI: Stable, efficient, and reusable robot self-evolution in complex real-world environments