CARLA-Air: Fly Drones Inside a CARLA World -- A Unified Infrastructure for Air-Ground Embodied Intelligence

Tianle Zeng; Yanci Wen; Hong Zhang

cs.ROMar 30, 2026

CARLA-Air: Fly Drones Inside a CARLA World -- A Unified Infrastructure for Air-Ground Embodied Intelligence

Authors: Tianle Zeng, Yanci Wen, Hong Zhang

Organizations: 1Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology.

Abstract

The convergence of low-altitude economies, embodied intelligence, and air-ground cooperative systems creates growing demand for simulation infrastructure capable of jointly modeling aerial and ground agents within a single physically coherent environment. Existing open-source platforms remain domain-segregated: driving simulators lack aerial dynamics, while multirotor simulators lack realistic ground scenes. Bridge-based co-simulation introduces synchronization overhead and cannot guarantee strict spatial-temporal consistency. We present CARLA-Air, an open-source infrastructure that unifies high-fidelity urban driving and physics-accurate multirotor flight within a single Unreal Engine process. The platform preserves both CARLA and AirSim native Python APIs and ROS 2 interfaces, enabling zero-modification code reuse. Within a shared physics tick and rendering pipeline, CARLA-Air delivers photorealistic environments with rule-compliant traffic, socially-aware pedestrians, and aerodynamically consistent UAV dynamics, synchronously capturing up to 18 sensor modalities across all platforms at each tick. The platform supports representative air-ground embodied intelligence workloads spanning cooperation, embodied navigation and vision-language action, multi-modal perception and dataset construction, and reinforcement-learning-based policy training. An extensible asset pipeline allows integration of custom robot platforms into the shared world. By inheriting AirSim's aerial capabilities -- whose upstream development has been archived -- CARLA-Air ensures this widely adopted flight stack continues to evolve within a modern infrastructure. Released with prebuilt binaries and full source: https://github.com/louiszengCN/CarlaAir

Figures & tables

Panel of Figure 2 — Figure 2: Per-frame inter-process data transfer time as a function of concurrent sensor count. Bridge-based co-simulation [ 17 ] exhibits near-linear growth with sensor count due to cross-process serialization, while CARLA-Air remains effectively constant ( $<0.5$ ms) owing to its single-process architecture.

Panel of Figure 3 — Figure 3: Platform positioning along simulation fidelity and agent domain breadth. CARLA-Air ( $\star$ ) occupies the high-fidelity, multi-domain quadrant without inter-process bridging. Dashed arrows indicate subsumption of upstream capabilities.

Platform	Urban Traffic	Pedes- trians	UAV Flight	Single Process	Shared Renderer	Native APIs	Joint Sensors	Prebuilt Binary	Test Suite	Custom Assets	Open Source
Autonomous Driving
CARLA [ 2 ]	✓	✓	×	✓	✓	✓	×	✓	×	✓	✓
LGSVL [ 13 ]	✓	✓	×	✓	✓	✓	×	✓	×	$\sim$	✓
SUMO [ 8 ]	✓	✓	×	✓	×	✓	×	✓	×	×	✓
MetaDrive [ 7 ]	✓	$\sim$	×	✓	$\sim$	✓	×	×	×	×	✓
VISTA [ 1 ]	$\sim$	×	×	✓	$\sim$	✓	×	×	×	×	✓

Table 1: Consolidated comparison of representative open-source simulation platforms for air-ground embodied intelligence research. ✓ = supported;

\sim

= partial or constrained; × = not supported; — = not applicable.

Platform	Single Proc.	API Kept	Shared Render	Sync Mode	Open Source
TranSimHub [ 17 ]	×	×	×	Msg.	✓
CARLA+SUMO [ 2 , 8 ]	×	$\sim$	×	RPC	✓
AirSim+Gazebo [ 15 , 6 , 9 ]	×	$\sim$	×	Decpl.	✓
AirSim+CARLA b	×	×	×	Decpl.	✓
CARLA-Air (Ours)	✓	✓	✓	Shared	✓

Table 2: Architectural comparison of representative joint and co-simulation platforms. a

Panel of Figure 4 — Figure 4: Runtime architecture of CARLA-Air. A single engine process hosts both simulation backends, each communicating with its respective Python client via an independent RPC server. CARLAAirGameMode acquires ground simulation functionality through class inheritance and integrates the aerial flight actor through composition. All world actors share a single rendering pipeline.

Panel of Figure 5 — Figure 5: Resolving the UE4 single-game-mode constraint. (a) Both backends provide independent game mode classes; assigning either silently discards the other. (b) CARLAAirGameMode inherits all ground functionality from CARLA’s game mode base while composing the aerial flight actor as a spawned world actor.

Figure 6: Coordinate frames of the two simulation backends. The transform $T$ requires only a Z-axis sign flip and a centimeter-to-meter scale factor; the forward ( $X$ / $X_{N}$ ) and rightward ( $Y$ / $Y_{E}$ ) axes are aligned across both conventions.

Profile	Configuration	FPS	VRAM (MiB)	CPU (%)
Standalone baselines
Ground sim only s	3 vehicles + 2 pedestrians; 8 sensors @ $1280\!\times\!720$	$28.4\pm 1.2$	$3{,}821\pm 10$	$31\pm 3$
Aerial sim only s	1 drone; 8 sensors @ $1280\!\times\!720$	$44.7\pm 2.1$	$2{,}941\pm 8\phantom{0}$	$29\pm 3$
Joint workloads
Idle	Town10HD; no actors; no sensors	$60.0\pm 0.4$	$3{,}702\pm 8\phantom{0}$	$12\pm 2$
Ground only	3 vehicles + 2 pedestrians; 8 sensors @ $1280\!\times\!720$	$26.3\pm 1.4$	$3{,}831\pm 11$	$38\pm 4$

Table 3: Frame rate and resource consumption across representative joint workloads (RTX A4000, Town10HD, Epic quality, synchronous mode). Harmonic mean

\pm

1 SD over 2 000 ticks after 200 warm-up.

Panel of Figure 7 — Figure 7: VRAM trace over a 3-hour stability run (357 spawn/destroy cycles, moderate joint configuration, RTX A4000). Early-to-late drift is ${\approx}10$ MiB; linear regression yields $R^{2}=0.11$ , confirming no significant memory accumulation.

Metric	Early (cycles 1–30)	Late (cycles 328–357)
Frame rate (FPS)	$19.9\pm 1.2$	$19.7\pm 1.3$
VRAM (MiB)	$3{,}868\pm 14$	$3{,}878\pm 17$
CPU utilization (%)	$53\pm 5$	$55\pm 5$
API error count	0	0
Crash count	0	0
VRAM regression slope	$0.49$ MiB/cycle, $R^{2}=0.11$

Table 4: Stability endurance results over 3 hours and 357 actor lifecycle cycles (moderate joint configuration).

API	Call	Median ( $\mu$ s)	IQR ( $\mu$ s)
Ground sim	World state snapshot	320	40
Ground sim	Actor transform query	280	35
Ground sim	Actor spawn (+ paired destroy)	1,850	210
Ground sim	Actor destroy	920	95
Aerial sim	Multirotor state query	410	55
Aerial sim	Image capture (1 RGB stream)	3,200	380

Table 5: Round-trip API call latency on the loopback interface (median

\pm

IQR; 5 000 calls after 500 warm-up; RTX A4000; idle scene).

	Workflow	Research Direction	Platform Features Exercised	FPS	Key Outcome
W1	Precision landing	Air-ground cooperation	Tick-sync control, descent planning, cross-frame coordination	${\approx}19$	$<0.5$ m final landing error
W2	VLN/VLA data generation	Embodied navigation	Dual-view sensing, waypoint planning, semantic annotation	—	Cross-view VLN data pipeline
W3	Multi-modal dataset	Perception & dataset	12-stream sync capture, shared tick	${\approx}17$	${\leq}1$ -tick alignment error
W4	Cross-view perception	Perception & dataset	Shared renderer, weather consistency	${\approx}18$	14/14 weather presets verified
W5	RL training env.	RL policy training	Sync stepping, stable resets, cross-domain reward	—	357 reset cycles, 0 crashes

Table 6: Summary of five representative workflows validated on CARLA-Air, mapped to the four research directions from Section 1 .

Panel of Figure 8 — Figure 8: Dual-client architecture shared by all five workflows. A single Python script drives both clients, connected via TCP to independent RPC servers inside the unified engine process.

Metric	Value	Notes
Mean FPS	$19.3$	Harmonic mean
Initial altitude	${\approx}12$ m	Start of descent
Landing duration	${\approx}20$ s	Approach to touchdown
Final horiz. error	$<0.5$ m	Within tolerance band
Initial horiz. error	${\approx}6$ m	At descent start
RPC errors	$0$	Both clients

Table 7: W1 cooperative precision landing results (Town10HD, RTX A4000).

Figure 9: W1: Air-ground cooperative precision landing on a moving vehicle. (a) Time-lapse sequence (

t_{1}

–

t_{5}

) showing the drone (red box) descending toward and landing on a moving ground vehicle through approach, descent, and touchdown phases. (b) 3D trajectory overview: the UAV trajectory (blue) converges onto the vehicle trajectory (orange), with the ground projection (dashed) showing horizontal alignment. (c) Altitude profile over time, illustrating smooth descent from 12 m to ground level. (d) Horizontal convergence error, decreasing from

{\approx}6

m to within the

\pm 0.5

m tolerance band. All data are recorded within CARLA-Air’s synchronous tick loop.

Figure 10: W2: Embodied navigation with aerial reasoning. A UAV autonomously tracks a pedestrian (red box, bottom) through an urban environment using bird’s-eye visual observations. Each frame is annotated with the drone’s chain-of-thought reasoning, illustrating how the agent interprets the scene, anticipates occlusions, and adjusts its flight path to maintain persistent visual contact with the target. All frames are rendered within CARLA-Air’s shared simulation world under consistent lighting and physics.

Metric	Value	Notes
Mean FPS	$17.1$	Harmonic mean
Concurrent streams	12	8 ground + 4 aerial
Records collected	1 000	One per tick
Max alignment dev.	${\leq}1$ tick	Normal disk-write load
RPC errors	$0$	Both clients
Per-tick write latency	$61\pm 9$ ms	Incl. serialization

Table 8: W3 multi-modal dataset collection results (1 000 ticks, 30 vehicles, 10 pedestrians, Town10HD, RTX A4000).

Figure 11: W3: Synchronized multi-modal dataset collection at a single simulation tick. Top row (vehicle perspective): RGB, semantic segmentation, depth, LiDAR bird’s-eye view, surface normals, and instance segmentation. Bottom row (drone perspective): the same six modalities captured simultaneously from the aerial viewpoint. All 12 sensor streams share an identical tick index and are rendered under the same lighting and weather conditions through CARLA-Air’s shared rendering pipeline, guaranteeing per-tick spatial-temporal correspondence without interpolation.

Metric	Value	Notes
Mean FPS	$18.2$	Harmonic mean
Co-registered pairs	500	Aerial depth + ground seg.
Per-tick latency	$52\pm 6$ ms	Full collection loop
Sensor alignment	$0$ ticks	Sync mode guarantee
Weather presets passed	$14/14$	All official presets
RPC errors	$0$	Both clients

Table 9: W4 cross-view perception results (500 ticks, Town10HD, RTX A4000).

Figure 12: W4: Air-ground cross-view perception across diverse environments and weather conditions. Each row shows an aerial RGB view from the drone across six representative CARLA maps (Town 01–05 and Town 10HD); each column corresponds to a different weather preset (Clear Noon, Cloudy Noon, Dense Fog, Hard Rain, Night, Soft Rain, and Sunset).

Figure 13: W5: Reinforcement learning training environment. Top: a drone learns to maintain an optimal aerial observation position above a moving ground vehicle within CARLA-Air’s urban traffic environment. Bottom: the closed-loop RL pipeline. At each synchronous tick, CARLA-Air provides an observation space (drone and vehicle poses, relative distance, surrounding traffic state) to the policy network, which outputs 3D velocity commands as actions. The reward signal encodes tracking accuracy, altitude maintenance, and collision avoidance.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 14: Custom assets imported into CARLA-Air through the extensible asset pipeline. Top: a four-wheeled mobile robot with onboard LiDAR, imported from an external FBX model. Bottom: a custom electric sport car with user-defined vehicle dynamics. Both assets operate within the shared simulation world alongside all built-in CARLA traffic and AirSim aerial agents, and are visible to all sensor modalities.

Service	Protocol	Port
CARLA RPC Server	TCP	2000
CARLA Streaming	UDP	2001
AirSim RPC Server	TCP	41451

Appendix

Table 10: Default network port assignments.

Component	Status
CARLA API	89/89 automated tests passing
AirSim API	Full flight control and sensor access verified
ROS 2 topics	63 total (43 CARLA + 14 AirSim + 6 generic)
Key upstream scripts confirmed working
manual_control.py	CARLA manual driving interface
automatic_control.py	CARLA autopilot demonstration

Appendix

Table 11: API compatibility and test coverage.

File	Modification
CarlaGameModeBase.h	friend declaration for CARLAAirGameMode
CarlaEpisode.h	Sensor tagging visibility: private $\to$ protected
CarlaGameModeBase.cpp	State flag assignment (1 line)
New integration layer (no upstream modification)
CARLAAirGameMode	Unified game mode class; ${\sim}1{,}405$ lines C++

Appendix

Table 12: Upstream CARLA source modifications.

Explore similar work

May 29, 2026

Can Aerial VLA Models Cooperate? Evaluating Closed-Loop Air-Ground Coordination with CARLA-Air

Recent aerial vision-language-action (VLA) models show promising single-UAV capabilities, such as tracking moving objects and navigating to language-specified landmarks. However, it remains unclear whether these capabilities can transfer to air-ground cooperation, where a UAV and a UGV must act jointly in a shared, closed-loop physical world. We study this question with CARLA-Air, a single-process air-ground evaluation environment that unifies CARLA and AirSim inside one Unreal Engine runtime. By sharing the same world state, physics tick, and sensing pipeline, CARLA-Air enables physically consistent UAV--UGV interaction and precise measurement of simulation-timestamp alignment and effective coordination latency. Using CARLA-Air, we evaluate representative aerial VLA and planning baselines on two complementary diagnostic tasks: moving-platform landing and occlusion-recovery escort. The results show that current aerial VLA models can often track or follow a ground partner, but struggle to convert this single-agent competence into stable cooperative behavior. State prompting provides limited benefit, and naive bidirectional interaction fails to consistently improve performance and can amplify errors for most baselines. These findings suggest that, under the tested text-based cue interfaces, zero-shot cooperative air-ground VLA requires three components beyond the current paradigm: explicit partner-state grounding, low-latency action coordination, and team-level objective alignment. Our code is available at https://github.com/louiszengCN/CarlaAir.

Tianle Zeng, Yanci Wen, Xueang Yu +1

Southern University of Science and Technology · Fudan University

VLM Evaluation Aerial Robotics Robotics Simulation Multi-Robot Coordination Vision-Language-Action Models

Ask PDF

Mar 24, 2026

AirSimAG: A High-Fidelity Simulation Platform for Air-Ground Collaborative Robotics

As spatial intelligence continues to evolve, heterogeneous multi-agent systems-particularly the collaboration between Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs), have demonstrated strong potential in complex applications such as search and rescue, urban surveillance, and environmental monitoring. However, existing simulation platforms are primarily designed for single-agent dynamics and lack dedicated frameworks for interactive air-ground collaborative simulation. In this paper, we present AirsimAG, a high-fidelity air-ground collaborative simulation platform built upon an extensively customized AirSim framework. The platform enables synchronized multi-agent simulation and supports heterogeneous sensing and control interfaces for UAV-UGV systems. To demonstrate its capabilities, we design a set of representative air-ground collaborative tasks, including mapping, planning, tracking, formation, and exploration. We further provide quantitative analyses based on these tasks to illustrate the platform effectiveness in supporting multi-agent coordination and cross-modal data consistency. The AirsimAG simulation platform is publicly available at https://github.com/BIULab-BUAA/AirSimAG.

Yangjie Cui, Xin Dong, Boyang Gao +3

School of Aeronautic Science and Engineering, Beihang University, Beijing 100191, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · Institution of Unmanned System, Beihang University, Beijing 100191, China

Robotics Simulation Robotics Multi-Robot Systems Multi-Robot Coordination

Ask PDF

Sep 10, 2026

CARLAverse: A Highly Modular, Distributed, and Multimodal Framework for Human-in-the-Loop Simulation

The development of autonomous driving demands comprehensive testing in mixed-traffic scenarios involving vulnerable road users (VRUs), where purely artificial agents often fail to capture authentic human social negotiations. While human-in-the-loop (HITL) simulators enable safe investigation of these interactions, existing multi-agent platforms struggle with the network latency and synchronization constraints required for high-fidelity haptic feedback. To resolve this, we present CARLAverse, an open-source, multimodal simulation ecosystem. Extending modular hardware abstraction, CARLAverse integrates driving (DrivoCARLA), cycling (CycloCARLA), and pedestrian (WalkoCARLA) simulators into a shared virtual environment. Its core methodological contribution is a distributed physics architecture: latency-critical ego dynamics and high-frequency force feedback are computed locally on client nodes, while a central CARLA server orchestrates non-player character (NPC) physics and global traffic. By decoupling haptic control loops from network bottlenecks, CARLAverse enables scalable, cross-institutional HITL experiments without compromising physical immersion. Code and documentation: https://git.ieem-ka.de/simulator-environments/carlaverse

Patrick Rebling, Philipp Nenninger, Reiner Kriesten

Institute of Energy Efficient Mobility Karlsruhe University of Applied Sciences

Haptic Feedback Autonomous Driving Simulation

Ask PDF