LITHE: Bridging Best-Effort Python and Real-Time C++ for Hot-Swapping Robotic Control Laws on Commodity Linux
Authors: He Kai Lim, Tyler R. Clites
Organizations: All authors are with the Department of Mechanical and Aerospace Engineering, University of California Los Angeles, Los Angeles, CA 90095 USA
Modern robotic systems rely on hierarchical control, where a high-level "Brain" (Python) directs a lower-level "Spine" (C++ real-time controller). Despite its necessity, this hierarchy makes it difficult for the Brain to completely rewrite the Spine's immutable control logic, consequently inhibiting fundamental adaptation for different tasks and environments. Conventional approaches require complex middleware, proprietary hardware, or sacrifice real-time performance. We present LITHE (Linux Isolated Threading for Hierarchical Execution), a lightweight software architecture that collapses the robot control hierarchy onto a commodity single-board computer (Raspberry Pi 4B with pi3hat), while maintaining safe frequency decoupling between the Brain and Spine. LITHE integrates strict CPU isolation (isolcpus), lock-free inter-process communication (IPC), and pipelined execution to meet high-frequency deadlines with minimal jitter. By adding multi-threaded dynamic linking, LITHE enables a Python-based Brain to dynamically evolve the logic of a 1kHz C++ Spine without interruption. We validate "functional real-time" system performance with worst-case execution time (WCET) < 100 μs and maximum release jitter (MRJ) < 4 μs under heavy load. We demonstrate a novel application where a large language model (LLM) supervisor performs online system identification to evolve a real-time controller on-the-fly, without interrupting the 1 kHz control loop. In essence, LITHE eliminates the "immutable compiled code" bottleneck for best-effort Brains to synthesize and inject completely new control laws into the real-time Spine. This bridges a critical gap between high-level AI and low-level real-time control to unlock continuous real-time evolution of embodied intelligence in safe, human-in-the-loop systems.
Figures & tables
Fig. 1: Hierarchical control from the Brain, across the Spine, onto different tasks and applications. LITHE enables high-level “best-effort" models and algorithms to evolve and hot-swap low-level real-time controllers without interrupting on-the-fly real-time operation. This empowers continuous, real-time evolution of embodied intelligence.
Fig. 2: LITHE system architecture for evolving real-time robot control on a Raspberry Pi. A top level “Python Brain” is isolated to CPU 2. It can run independently, with best-effort tasks such as deep learning, or in collaboration with an external agentic robot client. The Brain sends commands and trajectories via lock-free inter-process communication to a low-level “C++ Spine” which is isolated on CPU 1. The Spine maintains a functional real-time controller, such as PID or impedance control. Through our hot-swap mechanism, this controller can be hot-swapped on-the-fly, without interrupting real-time. It interacts with robot hardware through an isolated transport layer on CPU 3, based on the open-source pi3hat library for the pi3hat GPIO expansion board. For our simple experiment in Section IV-B , we set up an external agent based on qwen2.5-coder-7b to collaborate with the Brain. Through LITHE, we then implement various hot-swapped controllers that interact with a 1-DOF robot comprised of moteus hardware and a 3D printed arm.
Fig. 3: Example timing diagram of the pipelined (double buffered) execution for Cycle k . Spine dispatches the command uk (computed in the previous cycle) and immediately begins computing the next control law uk+1 while Transport handles the blocking CAN-FD bus transactions, before returning feedback xk+1 for the next cycle. This parallel structure allows the control logic to execute during the hardware transmission delay.
Fig. 4: Jitter distributions of the Spine under heavy synthetic load on the isolated Brain. (a) Execution Times for Spine to dispatch command u[k] under heavy synthetic load (NumPy, Stress-ng). Range is [0 μ s, 100 μ s], with 2 μ s bin size. The worst case execution time (WCET) of transport handoff under NumPy is 87.704 μ s, and for Stress-ng is 98.299 μ s. (b) Release Jitter Distributions under heavy synthetic load (NumPy, Stress-ng). Range is [0 μ s, 4 μ s], with 0.08 μ s bin size. The maximum release jitter (MRJ) for NumPy is 2.997 μ s, and for Stress-ng is 3.110 μ s.
Fig. 5: Dynamic evolution of real-time control law for robot arm through an external decentralized robot agent. The agent is qwen2.5-coder-7b, a large language model (LLM) (a) The agent starts by thinking about our prompts of the system. It formulates probing controllers for system identification, and a final gravity compensation controller based on received telemetry (green arrows). These controllers are sent at approximately t=2,6,13 s (blue arrows), and correspond with increased LITHE jitter due to subsequent compilation processes (red boxes). (b) Execution time for command handoff in each control cycle. Largest time measured was 57.648 μ s (within WCET 98.299 μ s, Figure 4 a). (c) Release Jitter for each control cycle. Maximum jitter measured was 0.703 μ s (within MRJ 3.110 μ s, Figure 4 b). After compilation is complete for each new controller, they are hot-swapped into the Spine at approximately t=5,9,16 s (blue arrows) (d) The robot’s tracking performance across four hot-swapped real-time controllers, at 1000Hz. No real-time deadlines were missed. Note the initial startup delay for t<2.2 s to initialize LITHE. At t=21.5 s, we simulate a brain crash, but LITHE continues to hold the load smoothly against gravity, preventing mechanical collapse.
Vision-Language-Action (VLA) models have achieved impressive performance on diverse embodied tasks. However, deploying VLA models on low-power onboard devices, such as the Jetson Orin, remains challenging due to their high computational complexity, which leads to substantial inference latency and low control frequency. Asynchronous inference can partially mask this latency by parallelizing action execution and subsequent inference, but it introduces two critical issues: perception-execution misalignment and long reaction time. In this paper, we propose EagleVLA, a method for efficient VLA deployment on onboard devices via Foresight-Aligned Asynchronous Correction. To address misalignment, we train a lightweight future correction module that predicts future environment representations, allowing the action expert to predict actions from the future time step. To reduce reaction time, we introduce confidence-based scheduling optimization that adaptively balances VLM and action expert invocations. We also build a llama.cpp-based inference engine tailored for onboard VLA deployment, with system-level optimizations including CUDA graph reuse, GPU-resident intermediate buffering, and flow unrolling. Extensive experiments demonstrate that EagleVLA achieves 14.71x and 9.20x improvements in control frequency compared with naive PyTorch and vla.cpp on Jetson Orin, while outperforming VLASH by 14.8% in success rate on LIBERO benchmark. The code of our asynchronous algorithm is available on https://github.com/PKU-SEC-Lab/EagleVLA, and our efficient llama.cpp-based inference engine is available on https://github.com/PKU-SEC-Lab/EagleVLA-Edge.
Zebin Yang, Qi Wang, Yunhe Wang +6
Peking University · TLAIC · PrimeBot Research Institute +1
Most robotic systems rely on fixed-frequency discrete-time controllers, creating a trade-off between the efficiency of low-frequency control and the responsiveness of high-frequency feedback. As a result, systems typically default to high control rates for robustness, at the cost of wasted inference and unnecessary actuation. Addressing this, we introduce Time-Adaptive Robotic Control (TARC), a reinforcement learning framework in which the policy jointly predicts a control action and its duration of application. TARC learns temporally extended actions by optimizing task performance under soft or hard constraints on the number of control switches, enabling adaptive modulation of control rates. We evaluate TARC on two robotic hardware platforms: a high-speed RC car and the Unitree Go1 quadruped, and on a vision-language action model in simulation, where each query incurs a costly transformer forward pass. Across all settings, TARC matches the performance of high-frequency discrete-time controllers while operating at less than half their control frequency. Unlike fixed-rate controllers, TARC adapts its control frequency online, allocating high-frequency feedback only when required.
Cloud robotics enables robots to offload high-dimensional motion planning and reasoning to remote servers. However, for continuous manipulation tasks requiring high-frequency control, network latency and jitter can severely destabilize the system, causing command starvation and unsafe physical execution. To address this, we propose Speculative Policy Orchestration (SPO), a latency-resilient cloud-edge framework. SPO utilizes a cloud-hosted world model to pre-compute and stream future kinematic waypoints to a local edge buffer, decoupling execution frequency from network round-trip time. To mitigate unsafe execution caused by predictive drift, the edge node employs an ε-tube verifier that strictly bounds kinematic execution errors. The framework is coupled with an Adaptive Horizon Scaling mechanism that dynamically expands or shrinks the speculative pre-fetch depth based on real-time tracking error. We evaluate SPO on continuous RLBench manipulation tasks under emulated network delays. Results show that even when deployed with learned models of modest accuracy, SPO reduces network-induced idle time by over 60% compared to blocking remote inference. Furthermore, SPO discards approximately 60% fewer cloud predictions than static caching baselines. Ultimately, SPO enables fluid, real-time cloud-robotic control while maintaining bounded physical safety.
Chanh Nguyen, Shutong Jin, Florian T. Pokorny +1
Department of Computing Science, Ume˚a University, 90187 Ume˚a, Sweden · School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 10044 Stockholm, Sweden