Although large language models have shown strong capabilities in domain-specific code generation, their performance in robotics software development, especially for the Robot Operating System (ROS), remains largely unexplored. ROS code generation presents distinctive challenges, including complex inter-node coordination, middleware configuration, and lifecycle semantics that go far beyond standard coding benchmarks, and many important correctness properties become observable only once migrated nodes are launched and communicate. We present ROSTrans, a benchmark for evaluating LLMs on ROS1-to-ROS2 code migration, a representative and practically important setting for ROS code generation. ROSTrans comprises 108 scaffolded completion tasks drawn from a diverse set of 70+ open-source ROS1 repositories and organized into a hierarchical taxonomy assessing interface-level, behavioral-level and system-level correctness. For every task we establish an executable reference: a verified ROS2 implementation, a runtime test suite that launches the nodes and exercises their topics, services, actions and parameters, and a container ensuring reproducibility. Candidates are scored by injecting only their target files into an otherwise fixed copy of the reference package, so a candidate can neither supply nor weaken the tests it is judged by. Evaluating six frontier LLMs over five stochastic generations per task, we find pass@1 below 10.9%, and that 56.5% of all samples build successfully and then fail at runtime,and 32.0% fail runtime assertions on the migrated interfaces, a class of error no compile-only oracle can detect. Execution-guided repair further improves task success, reaching 49.1% and 47.2% for Claude Opus 4.6 and GPT-5.5 after three rounds, respectively.
Figures & tables
Fig. 1: ROSTrans overview. ROS1 repositories are transformed into 108 tasks via LLM-assisted hollowing, then evaluated via pass rate.
Fig. 2: Pipeline of ROSTrans . (a) Benchmark Construction: coherent file groups are selected from legacy ROS 1 code and hollowed into scaffolded tasks with structured TODO markers across three capability levels. (b) Reference Establishment: each task is translated to ROS 2 and verified by building and running it inside a container, with build and test logs fed back for repair until the task builds and its runtime tests pass. (c) Test Quality Assurance: the hollowed function is replaced by a no-op stub and the task is re-run to check that its tests detect the change. (d) Candidate Evaluation: only the candidate’s target files are injected into an otherwise fixed copy of the reference package, yielding a binary outcome and the stage at which failure occurred.
Fig. 3: Task statistics by capability level. Language categorization is based on target file extensions (mixed tasks are counted as C++). Modifications and hollowed lines are measured via TODO markers in the scaffolded source.
Model
pass@1 (%)
pass@3
pass@5
by level (%)
[95% CI]
(%)
(%)
Intf.
Behav.
Sys.
Claude Opus 4.6
10.9 [5.6,16.9]
12.5
13.0
8.2
11.7
13.7
GPT-5.5
10.0 [5.4,15.2]
14.6
16.7
9.5
12.4
8.6
Gemini 3.1 Pro
7.2 [3.5,11.5]
11.5
13.9
5.5
7.6
9.1
Qwen 3.5 Plus
6.7 [3.0,11.1]
9.3
10.2
6.4
3.4
9.7
GLM-5 Turbo
6.5 [2.6,10.9]
8.2
8.4
4.5
6.2
9.2
TABLE I: pass@ k performance over 108 tasks (five samples/task). Brackets give 95% CIs from bootstrap resampling over the 108 tasks.
Fig. 4: pass@1 across ROS capability categories, in percent. Columns are action server, manipulation, multi-node coordination, navigation, parameter server, perception and control, publisher/subscriber, sensor processing, service client, simulation integration, and state machine; rows are ordered by overall pass@1. Every model fails all service-client tasks, and the spread across categories is wider than the spread across models.
Fig. 5: Outcome of all 3,237 evaluated samples, with failures broken down by stage and cause. More samples build successfully and then fail at runtime than fail to build at all.
Fig. 6: Pass rate by ROS capability category over all models and runs. Rates span 16.4 % to zero, with service-client tasks passed by no sample.
Fig. 7: Case study of a service-client task. The model emits valid ROS2 service-client APIs but collapses the original two-stage interaction into a single call, a semantic failure that only the runtime test detects.
Cross-language migration of large software systems is a persistent engineering challenge, particularly when the source codebase evolves rapidly. We present a methodology for LLM-assisted continuous code translation in which a large language model translates a production Rust codebase (648K LOC, 65 crates) into Python (41K LOC, 28 modules), with public agent benchmarks as the objective function driving iterative refinement. Our subject system is Codex CLI, a production AI coding agent. We demonstrate that: (1) the Python port resolves 59/80 SWE-bench Verified tasks (73.8%) versus Rust's 56/80 (70.0%), and achieves 42.5% on Terminal-Bench versus Rust's 47.5%, confirming near-parity on real-world agentic tasks; (2) benchmark-driven debugging, revealing API protocol mismatches, environment pollution, a silent WebSocket failure mode, and an API 400 crash, is more effective than static testing alone; (3) the architecture supports continuous upstream synchronisation via an LLM-assisted diff-translate-test loop; and (4) the Python port has evolved into a capability superset with 30 feature-flagged extensions (multi-agent orchestration, semantic memory, guardian safety, cost tracking) absent from Rust, while preserving strict parity mode for comparison. Our evaluation shows that for LLM-based agents where API latency dominates, Python's expressiveness yields a 15.9x code reduction with negligible performance cost, while the benchmark-as-objective-function methodology provides a principled framework for growing a cross-language port from parity into an extended platform.
Recent advances in large language models (LLMs) provide robots with contextual reasoning abilities to comprehend human instructions. Yet, current LLM-enabled robots typically depend on cloud-based models or high-performance computing infrastructure, which limit their deployment on robots under unreliable internet environments or with constrained computational resources, such as UAVs and small ground vehicles. Thus, deploying fine-tuned small language models (SLMs) that support onboard deployment offers a promising alternative. This paper introduces Ro-SLM, a framework that enables reliable SLM-driven robot operation by distilling LLMs' knowledge and reasoning. Ro-SLM starts from dataset synthesis by leveraging LLMs to generate diverse task instructions, produce corresponding ground truth code with minimal human assistance, and augment instructions into real-world application scenarios. Ro-SLM is then fine-tuned with the dataset, in which LLM serves as a reward function to guide the training. Extensive experiments on UAV operation tasks demonstrate that Ro-SLM improves the performance of SLM from being incapable of supporting robotic task planning and code generation to achieving performance that approaches LLM.
Wenhao Wang, Yanyan Li, Long Jiao +1
Department of Computer & Information Science, University of Massachusetts Dartmouth · Department of Computer Science & Engineering, California State University San Marcos
End-user robot programming grants users the flexibility to re-task robots in situ, yet it remains challenging for novices due to the need for specialized robotics knowledge. Large Language Models (LLMs) hold the potential to lower the barrier to robot programming by enabling task specification through natural language. However, current LLM-based approaches generate opaque, "black-box" code that is difficult to verify or debug, creating tangible safety and reliability risks in physical systems. We present RoboCritics, an approach that augments LLM-based robot programming with expert-informed motion-level critics. These critics encode robotics expertise to analyze motion-level execution traces for issues such as joint speed violations, collisions, and unsafe end-effector poses. When violations are detected, critics surface transparent feedback and offer one-click fixes that forward structured messages back to the LLM, enabling iterative refinement while keeping users in the loop. We instantiated RoboCritics in a web-based interface connected to a UR3e robot and evaluated it in a between-subjects user study (n=18). Compared to a baseline LLM interface, RoboCritics reduced safety violations, improved execution quality, and shaped how participants verified and refined their programs. Our findings demonstrate that RoboCritics enables more reliable and user-centered end-to-end robot programming with LLMs.
Callie Y. Kim, Nathan Thomas White, Evan He +2
Department of Computer Sciences, University of Wisconsin–Madison, Madison, Wisconsin, USA