Road Infrastructure Perception for Autonomous Driving: Taxonomy, Representations, Tasks, and Challenges

Guoyang Zhao; Weiqing Qi; Kai Zhang; Zhihai Bi; Tianshuai Hu; Zeying Gong; Dezong Zhao; Jun Ma

cs.CVOct 24, 2026

Road Infrastructure Perception for Autonomous Driving: Taxonomy, Representations, Tasks, and Challenges

Authors: Guoyang Zhao, Weiqing Qi, Kai Zhang, Zhihai Bi, Tianshuai Hu, Zeying Gong, Dezong Zhao, Jun Ma

Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China. · The Hong Kong University of Science and Technology, Hong Kong SAR, China. · the James Watt School of Engineering, University of Glasgow, Glasgow G12 8QQ, U.K.

Abstract

Road infrastructure perception is a fundamental component of autonomous driving, as it recovers the structured road elements that define roadway geometry, delimit operational space, and encode traffic regulations. Although individual tasks such as lane detection, drivable-space estimation, traffic-sign recognition, and online map construction have been extensively studied, the field remains fragmented across task definitions, sensing assumptions, output representations, datasets, and evaluation protocols. This survey provides a function-oriented synthesis of road infrastructure perception. We first clarify its scope and organize existing studies into three complementary categories: geometric-structural, functional-spatial, and regulatory-semantic infrastructure. Based on this taxonomy, we review sensing modalities, scene representations, representative methods, benchmarks, and evaluation protocols, emphasizing the field's evolution from image-space recognition toward metric, vectorized, topology-aware, occupancy-based, and rule-grounded scene modeling. We further examine how representation choices affect evaluation and downstream usability, revealing limitations of current benchmarks in relation reasoning, temporal consistency, robustness, and planning-oriented assessment. Finally, we discuss open challenges and future directions toward unified infrastructure representation, open-environment generalization, relation- and rule-aware reasoning, and downstream-oriented evaluation. By connecting tasks, representations, and evaluation under a common functional framework, this survey aims to establish road infrastructure perception as a coherent foundation for map-compatible and behavior-relevant autonomous-driving scene understanding.

Figures & tables

Fig. 2: Visual taxonomy of road infrastructure perception. The proposed taxonomy organizes road infrastructure perception into three complementary categories according to the dominant driving-relevant information recovered from road scenes: geometric-structural, functional-spatial, and regulatory-semantic infrastructure.

Category	Standardized Task	Representative Elements	Downstream Relevance
Geometric-Structural	(1) Lane Detection	Lane markings, boundaries, dividers, connectors	Lane keeping, localization, planning
	(2) Road Boundary and Curb Detection	Road boundaries, curbs, edges, medians, shoulders	Road-edge constraint, map alignment, safety
	(3) Lane Topology and Intersection Perception	Lane connectivity, merge/diverge, intersection layout	Routing, maneuver reasoning, online mapping
Functional-Spatial	(4) Drivable Space Perception	Drivable, free, non-drivable, accessible regions	Local planning, collision avoidance, traversability
	(5) Parking Slot Detection	Parking slots, boundaries, structured spaces	Automated parking, low-speed maneuvering, local maps
Regulatory-Semantic	(6) Traffic Sign Recognition	Warning, prohibition, mandatory, speed-limit signs	Rule compliance, speed control, route constraints

TABLE I: Standardized taxonomy of road infrastructure perception tasks for autonomous driving.

Fig. 3: Relationship between sensing modalities and scene representations in road infrastructure perception. Different infrastructure categories impose distinct sensing requirements and representation preferences. Vision is dominant for texture- and semantics-rich elements, LiDAR is critical for geometry- and boundary-oriented perception, while GNSS/IMU and map priors mainly support alignment, temporal consistency, and structured reasoning.

Representation	Encoded Structure	Representative Tasks	Utility and Limitation
Dense raster	Pixel-, point-, voxel-, or grid-level labels	Lane detection; drivable-space; curb detection; road-marking recognition	Fine-grained parsing, but view-dependent, topology-weak, and hard to use for mapping/planning
BEV grid	Metric top-down layout in ego/map-aligned coordinates	Lane detection; drivable-space; parking slot detection; online map perception	Metric reasoning and multi-modal fusion, but sensitive to projection, calibration, and grid resolution
Structured vector	Polylines, splines, boundaries, polygons, and primitives	Lane detection; curb detection; parking slot detection; map-element perception	HD-map and planning compatibility, but sensitive to discontinuity, ordering, and matching errors
Topology-aware graph	Connectivity, adjacency, successor, merge/diverge, and lane-level relations	Lane topology; intersection perception; lane-level map construction	Routing and relation reasoning, but costly to annotate and difficult to evaluate
Occupancy / traversability	Occupancy states, accessibility fields, traversability maps, and cost-map fields	Drivable-space perception; free-space estimation; local planning	Direct planning interface, but ambiguous under occlusion, temporary changes, and weak boundaries
Semantic-symbolic	Rule-bearing symbols, signal states, lane usage, attributes, and constraints	Traffic sign recognition; traffic-light state; road-marking recognition	Rule-compliant behavior, but requires grounding, lane association, and regional adaptation

TABLE II: Scene representation families for road infrastructure perception.

Fig. 4: Task-oriented method paradigms and output representations. Road infrastructure perception tasks differ in target structure and downstream role, while their methodological evolution shows a broad trend from image-space dense prediction toward BEV, vectorized, graph-based, occupancy-oriented, and semantic-symbolic representations.

TABLE III: Representative studies on lane detection. The input icons indicate the input modalities used by each method: denotes camera input, and denotes LiDAR input.

TABLE IV: Representative studies on road boundary and curb detection. The input icons indicate the input modalities used by each method: denotes camera input, denotes LiDAR input, and denotes radar input.

TABLE V: Representative studies on lane topology and intersection perception. The input icons indicate the input modalities or priors used by each method: denotes camera input, denotes LiDAR input, denotes HD map input, denotes GNSS input, and denotes trajectory input.

TABLE VI: Representative studies on drivable space perception. The input icons indicate the input modalities used by each method: denotes camera input, denotes LiDAR input, and denotes radar input.

TABLE VII: Representative studies on parking slot detection. The input icon indicates the input modality used by each method: denotes camera input.

TABLE VIII: Representative studies on traffic sign recognition. The input icons indicate the input modalities used by each method: denotes camera input, and denotes text input.

TABLE IX: Representative studies on traffic light detection and state recognition. The input icons indicate the input modalities used by each method: denotes camera input, and denotes LiDAR input.

TABLE X: Representative studies on road marking recognition. The input icons indicate the input modalities used by each method: denotes camera input, denotes LiDAR input, and denotes text input.

TABLE XI: Representative infrastructure-specific datasets for road infrastructure perception. G , F , and R denote geometric-structural, functional-spatial, and regulatory-semantic infrastructure, respectively. The input icons indicate the input modalities or priors provided by each dataset: denotes camera input, denotes LiDAR input, denotes HD map input, denotes GNSS input, and denotes text input.

Metric Family		Formula	Symbol Definition	Output / Limitation
Classification	Accuracy	$\mathrm{Acc}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}(\hat{y}_{i}=y_{i})$	$N$ : samples; $\hat{y}_{i}$ : predicted label; $y_{i}$ : ground-truth label	Class labels for signs, lights, and markings; biased by frequent classes.
	Precision / Recall / F1	$\mathrm{Prec}=\frac{TP}{TP+FP}$ , $\mathrm{Rec}=\frac{TP}{TP+FN}$ , $\mathrm{F1}=\frac{2\mathrm{Prec}\cdot\mathrm{Rec}}{\mathrm{Prec}+\mathrm{Rec}}$	$TP$ : true positives; $FP$ : false positives; $FN$ : false negatives	Class-level recognition; weak for localization and lane-level relevance.
	Macro-F1 / Balanced Accuracy	$\mathrm{MacroF1}=\frac{1}{C}\sum_{c=1}^{C}\mathrm{F1}_{c}$ , $\mathrm{BAcc}=\frac{1}{C}\sum_{c=1}^{C}\mathrm{Rec}_{c}$	$C$ : classes; $\mathrm{F1}_{c}$ : class-wise F1; $\mathrm{Rec}_{c}$ : class-wise recall	Long-tail signs and rare markings; sensitive to noisy rare-class labels.
Detection	AP / mAP	$\mathrm{AP}_{c}=\int_{0}^{1}p_{c}(r)\,dr$ , $\mathrm{mAP}=\frac{1}{C}\sum_{c=1}^{C}\mathrm{AP}_{c}$	$p_{c}(r)$ : precision-recall curve for class $c$ ; $C$ : classes	Boxes for signs, lights, slots, and markings; weak for rule relevance.
	IoU Matching	$\mathrm{IoU}(B,\hat{B})=\frac{\|B\cap\hat{B}\|}{\|B\cup\hat{B}\|}$	$B$ : ground-truth box or region; $\hat{B}$ : predicted box or region	Object-like regions; unsuitable for thin or elongated infrastructure.
Segmentation	IoU / mIoU	$\mathrm{IoU}_{c}=\frac{TP_{c}}{TP_{c}+FP_{c}+FN_{c}}$ , $\mathrm{mIoU}=\frac{1}{C}\sum_{c=1}^{C}\mathrm{IoU}_{c}$	$TP_{c},FP_{c},FN_{c}$ : pixel / point / grid counts for class $c$ ; $C$ : classes	Masks for roads, curbs, lanes, and markings; weak for boundary shifts.

TABLE XII: Evaluation metric families for road infrastructure perception. The table summarizes representative metrics, formulas, symbol definitions, and their representation-level applicability.

Autonomous Driving Perception

Autonomous driving perception enables vehicles to detect and interpret road users, drivable space, and hazards using onboard and shared sensors.

See all 379 papers

Infrastructure-Centric World Models: Bridging Temporal Depth and Spatial Breadth for Roadside Perception
Apr 19, 2026Siyuan Meng, Chengbo AiWorld Model Learning World Models
Department of Civil and Environmental Engineering, University of Massachusetts Amherst
Bridging Structure and Language: Graph-Based Visual Reasoning for Autonomous Road Understanding
May 20, 2026Lena Wild, Katie Z Luo, Marco PavoneVisual Reasoning Autonomous Driving
KTH Royal Institute of Technology · TRATON · Stanford University +1
When the City Teaches the Car: Label-Free 3D Perception from Infrastructure
Mar 17, 2026Zhen Xu, Jinsu Yoo, Cristian Bautista +7Autonomous Driving 3D Object Detection
The Ohio State University · Google · Cornell University +2

Apr 19, 2026J/K move · Enter open · S save

Infrastructure-Centric World Models: Bridging Temporal Depth and Spatial Breadth for Roadside Perception

Siyuan Meng, Chengbo Ai

Department of Civil and Environmental Engineering, University of Massachusetts Amherst

World Model Learning World Models Autonomous Driving Perception Vision-Language-Action Models World Models for Autonomous Driving

Open paper AskPDF arXiv

World models, generative AI systems that simulate how environments evolve, are transforming autonomous driving, yet all existing approaches adopt an ego-vehicle perspective, leaving the infrastructure viewpoint unexplored. We argue that infrastructure-centric world models offer a fundamentally complementary capability: the bird's-eye, multi-sensor, persistent viewpoint that roadside systems uniquely possess. Central to our thesis is a spatio-temporal complementarity: fixed roadside sensors excel at temporal depth, accumulating long-term behavioral distributions including rare safety-critical events, while vehicle-borne sensors excel at spatial breadth, sampling diverse scenes across large road networks. This paper presents a vision for Infrastructure-centric World Models (I-WM) in three phases: (I) generative scene understanding with quality-aware uncertainty propagation, (II) physics-informed predictive dynamics with multi-agent counterfactual reasoning, and (III) collaborative world models for V2X communication via latent space alignment. We propose a dual-layer architecture, annotation-free perception as a multi-modal data engine feeding end-to-end generative world models, with a phased sensor strategy from LiDAR through 4D radar and signal phase data to event cameras. We establish a taxonomy of driving world model paradigms, position I-WM relative to LeCun's JEPA, Li Fei-Fei's spatial intelligence, and VLA architectures, and introduce Infrastructure VLA (I-VLA) as a novel unification of roadside perception, language commands, and traffic control actions. Our vision builds upon existing multi-LiDAR pipelines and identifies open-source foundations for each phase, providing a path toward infrastructure that understands and anticipates traffic.

Abstract

Autonomous Driving Perception

Explore similar work

Infrastructure-Centric World Models: Bridging Temporal Depth and Spatial Breadth for Roadside Perception

Bridging Structure and Language: Graph-Based Visual Reasoning for Autonomous Road Understanding

When the City Teaches the Car: Label-Free 3D Perception from Infrastructure