RACAS: Controlling Diverse Robots With a Single Agentic System

Dylan R. Ashley; Jan Przepióra; Yimeng Chen; Ali Abualsaud; Nurzhan Yesmagambet; Shinkyu Park; Eric Feron; Jürgen Schmidhuber

cs.ROMar 5, 2026

RACAS: Controlling Diverse Robots With a Single Agentic System

Authors: Dylan R. Ashley, Jan Przepióra, Yimeng Chen, Ali Abualsaud, Nurzhan Yesmagambet, Shinkyu Park, Eric Feron, Jürgen Schmidhuber

Organizations: Center of Excellence in Generative AI, King Abdullah University of Science and Technology (KAUST), Saudi Arabia. · Dalle Molle Institute for Artificial Intelligence Research (IDSIA), Switzerland. · Università della Svizzera italiana (USI), Switzerland. · Scuola universitaria professionale della Svizzera italiana (SUPSI), Switzerland. ·

Abstract

Many robotic platforms expose an API through which external software can command their actuators and read their sensors. However, transitioning from these low-level interfaces to high-level autonomous behaviour requires a complicated pipeline, whose components demand distinct areas of expertise. Existing approaches to bridging this gap either require retraining for every new embodiment or have only been validated across structurally similar platforms. We introduce RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three LLM/VLM-based modules (Monitors, a Controller, and a Memory Curator) communicate exclusively through natural language to provide closed-loop robot control. RACAS requires only a natural language description of the robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms. We evaluate RACAS on several tasks using a wheeled ground robot, a recently published novel multi-jointed robotic limb, and an underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential of agentic AI to substantially reduce the barrier to prototyping robotic solutions.

Figures & tables

Fig. 1 : The basic design of the system tries to minimize the extent of changes needed to apply it to different robots. The majority of the complexity of adapting to different robot platforms is offloaded to large AI models augmented with a dynamic structured memory system.

Panel of Figure 2 — Fig. 2 : The three real-world robotic platforms the system from Fig. 1 was applied to.

Fig. 3 : An example of RACAS’s reasoning process during one of the Dingo (real-world) runs.

Panel of Figure 4 — Fig. 4 : Snapshots of the various setups experiments were conducted in. In all cases, the target was never visible from the starting position.

Platform	Morphology	# Cam.	$\|\mathcal{A}\|$	Img. res.
Alhakami et al. Limb	4-DOF arm	4	9	$100\times 100$
Dingo (simulation)	3-DOF Wheeled	3	4	$640\times 480$
Dingo (real-world)	3-DOF Wheeled	1	4	$1280\times 800$
BlueROV2	6-DOF ROV	1	6	$1920\times 1080$

TABLE I : Platform Configurations

Experiment	Baseline	RACAS	Min.
Dingo (simulation)	25.00 $\pm$ 0.00	16.40 $\pm$ 0.96	10
Dingo (real-world)	N/A	25.80 $\pm$ 4.94	7
Alhakami et al. Limb	22.07 $\pm$ 0.98	9.56 $\pm$ 1.50	3
BlueROV2 (small tank)	N/A	14.00 $\pm$ 1.75	5
BlueROV2 (large pool)	N/A	19.20 $\pm$ 6.51	$\approx$ 5

TABLE II : Performance Comparison of RACAS and Baselines (

\pm

standard error)

Fig. 5 : Distribution of steps needed to complete the tasks in each of the settings. Note that random control policies were artificially limited to 25 steps.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Fig. 6 : Blackjack visualization and experimental results.

Agentic AI for Robotics

See all 39 papers

AutoRAS: Learning Robust Agentic Systems with Primitive Representations
Jun 19, 2026Yang Yue, Xuancheng Zhu, Yuyang Ma +7Multi-Agent LLM Systems Adversarial Robustness
Beijing University of Posts and Telecommunications · China Telecom · Guangxi Transportation Science and Technology Group Co., Ltd. +1
ChunkAct: Action Chunking for Agentic Robotics
Oct 13, 2026Bingxin Xu, Wenrui Bao, Xinxin Liu +3Agentic AI for Robotics Robotics
University of Southern California · University of Central Florida · Stony Brook University
RHO: Your Coding Agent is Secretly a Roboticist
Jun 15, 2026Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade +3LLM Agent Harnesses Robotic Manipulation
University of California, Berkeley · Embodied Science · AMD

Jun 19, 2026J/K move · Enter open · S save

AutoRAS: Learning Robust Agentic Systems with Primitive Representations

Yang Yue, Xuancheng Zhu, Yuyang Ma, Guoshun Nan+6

Beijing University of Posts and Telecommunications · China Telecom · Guangxi Transportation Science and Technology Group Co., Ltd.+1

Multi-Agent LLM Systems Adversarial Robustness AI Agent Safety

Open paper AskPDF arXiv

The automated design of agentic systems offers a promising pathway for scaling large language models (LLMs) beyond single-agent reasoning. While prior work has advanced task performance through handcrafted or automatically generated multi-agent workflows, robustness is often treated as an afterthought, leaving systems vulnerable to external adversaries and internal failures. We propose AutoRAS, a framework for the Automated design of Robust Agentic Systems. AutoRAS formulates system design as generating a sequence of symbolic primitives that jointly encode structural connectivity and behavioral actions, and learns to optimize this sequence using execution-derived safety signals and flow-based sequence-level objectives. Extensive experiments show that AutoRAS achieves the best performance in both vanilla and adversarial settings, with the smallest performance degradation under attacks. Further analyses demonstrate strong transferability, stable optimization behavior, stability across primitive sets, and favorable cost trade-offs. Our code is available at this https URL.

Abstract

Agentic AI for Robotics

Explore similar work

AutoRAS: Learning Robust Agentic Systems with Primitive Representations

ChunkAct: Action Chunking for Agentic Robotics

RHO: Your Coding Agent is Secretly a Roboticist