cs.CVMar 17, 2026

When the City Teaches the Car: Label-Free 3D Perception from Infrastructure

Authors: Zhen Xu, Jinsu Yoo, Cristian Bautista, Zanming Huang, Tai-Yu Pan, Zhenzhen Liu, Katie Z Luo, Mark Campbell, 

Organizations: The Ohio State University · Google · Cornell University · Stanford University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Building robust 3D perception for self-driving still relies heavily on large-scale data collection and manual annotation, yet this paradigm becomes impractical as deployment expands across diverse cities and regions. Meanwhile, modern cities are increasingly instrumented with roadside units (RSUs), static sensors deployed along roads and at intersections to monitor traffic. This raises a natural question: can the city itself help train the vehicle? We propose infrastructure-taught, label-free 3D perception, a paradigm in which RSUs act as stationary, unsupervised teachers for ego vehicles. Leveraging their fixed viewpoints and repeated observations, RSUs learn local 3D detectors from unlabeled data and broadcast predictions to passing vehicles, which are aggregated as pseudo-label supervision for training a standalone ego detector. The resulting model requires no infrastructure or communication at test time. We instantiate this idea as a fully label-free three-stage pipeline and conduct a concept-and-feasibility study in a CARLA-based multi-agent environment. With CenterPoint, our pipeline achieves 82.3% AP for detecting vehicles, compared to a fully supervised ego upper bound of 94.4%. We further systematically analyze each stage, evaluate its scalability, and demonstrate complementarity with existing ego-centric label-free methods. Together, these results suggest that city infrastructure itself can potentially provide a scalable supervisory signal for autonomous vehicles, positioning infrastructure-taught learning as a promising orthogonal paradigm for reducing annotation cost in 3D perception.

Figures & tables

Figure 1: Can city infrastructure teach vehicles to perceive? We explore a new paradigm where roadside infrastructure acts as distributed teachers, providing supervision to train ego perception models without manual annotations.
Figure 2: Overview of infrastructure-taught, label-free 3D perception. Stage 1: each RSU learns a location-specialized detector in an unsupervised manner by exploiting temporal consistency from its stationary viewpoint. Stage 2: trained RSUs broadcast their predicted 3D bounding boxes to nearby ego vehicles when their fields of view overlap. Stage 3: the ego vehicle aggregates these predictions as pseudo-labels to train its own detector offline, producing a standalone ego model that no longer requires infrastructure at deployment time.
Figure 3: Sample from the CIVET dataset used in Stage 2 RSU-to-ego broadcasting. The ego vehicle and RSU observe the same traffic scene from different viewpoints with overlapping fields of view.
Figure 4
Figure 6: Cross-evaluation across RSUs. Each RSU detector is trained at its own location and evaluated across all other RSU viewpoints. The heatmap shows that detectors perform strongly within their own trained view but degrade substantially when evaluated at different locations, indicating that RSU detectors are location-specific. Furthermore, detectors trained at RSU viewpoints fail to generalize to the ego viewpoint (last row).
Figure 6
Figure A1 : RSU deployment and coverage in Town 7 and Town 10. Each shaded region denotes the field of view of an RSU. Together, the RSUs provide complementary coverage of the geo-fenced environment, enabling infrastructure-taught supervision across most of the drivable area. One region is shaded slightly darker for visualization.
Figure A2 : RSU deployment and coverage in Town 1 and Town 2. One region is shaded slightly darker for clearer visualization.
Figure A3 : Reference driving paths in Town 7 and Town 10. The ego vehicle follows high-level reference routes designed to traverse most of the drivable area, ensuring broad coverage during data collection.
Figure A4 : Reference driving paths in Town 1 and Town 2.
Figure A5 : Sample data from the CIVET dataset (urban and rural towns). Each column shows paired examples from the urban and rural environments. The two towns exhibit distinct semantics—urban scenes contain smaller vehicles and denser pedestrian/cyclist activity, while rural scenes feature larger vehicles and sparser non-vehicle traffic.
Figure A6 : Sample data from the CIVET dataset (additional towns). These environments are used to evaluate the scalability of the proposed framework.
Figure A7 : RSU placement used in the analysis of the number of RSUs.
Figure A8 : Cross-evaluation across RSUs in the rural town. Compared to the urban setting, RSU detectors exhibit slightly better cross-location generalization due to the simpler traffic semantics in rural environments. Nevertheless, each RSU performs best within its own trained field of view and degrades when evaluated elsewhere. The last row further shows that no single RSU detector transfers well to the ego viewpoint.
Figure A9 : Qualitative results of the ego detector with CenterPoint [ 71 ] . Despite being trained without human labels, the ego detector generates accurate 3D bounding boxes, demonstrating the feasibility of our infrastructure-taught pipeline.
Figure A10 : Failure cases. While both RSU and ego detectors perform well in many cases, RSUs often miss distant objects and the ego detector struggles with small objects.
Figure A11 : Additional qualitative results of the ego detector. [1/2]
Figure A12 : Additional qualitative results of the ego detector. [2/2]
More in this topic

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 536 papers

Explore similar work

CardsList

  1. Infrastructure-based Monocular 3D Vehicle Localization Framework with Experimental Validation

    Sep 1, 2026Akos T. Kopeczi-Bocz, Tian Mi, Gabor Orosz +1Autonomous Driving PerceptionMonocular 3D Object Detection

    Department of Applied Mechanics, Budapest University of Technology and Economics, Budapest, H-1111, Hungary · Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI 48109, USA · Department of Mechanical Engineering, the Department of Civil and Environmental Engineering, and the Transportation Research Institute at the University of Michigan, Ann Arbor, MI 48109, USA

  2. Polis: 3D Self-Supervision at City Scale

    Aug 29, 2026Alexander Rusnak, Sophia Kovalenko, Jingru Wang +3Point Cloud Learning3D Representation Learning

    École Polytechnique Fédérale de Lausanne

  3. Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos

    Feb 25, 2026Matthew Strong, Wei-Jer Chang, Quentin Herau +4Autonomous DrivingAutonomous Driving Perception

    Applied Intuition · Stanford University · UC Berkeley