Digital-Twin Losses for Lane-Compliant Trajectory Prediction at Urban Intersections

Kuo-Yi Chao; Erik Leo Haß; Melina Gegg; Jiajie Zhang; Ralph Raßhofer; Alois Christian Knoll

cs.ROMar 4, 2026

Digital-Twin Losses for Lane-Compliant Trajectory Prediction at Urban Intersections

Authors: Kuo-Yi Chao, Erik Leo Haß, Melina Gegg, Jiajie Zhang, Ralph Raßhofer, Alois Christian Knoll

Organizations: * Chair of Robotics, Artificial Intelligence and Real-Time Systems Technical University of Munich, Munich, Germany · † BMW Group, Munich, Germany

Abstract

Accurate and safety-conscious trajectory prediction is a key technology for intelligent transportation systems, especially in V2X-enabled urban environments with complex multi-agent interactions. In this paper, we created a digital twin-driven V2X trajectory prediction pipeline that jointly leverages cooperative perception from vehicles and infrastructure to forecast multi-agent motion at signalized intersections. The proposed model combines a Bi-LSTM-based generator with a structured training objective consisting of a standard mean squared error (MSE) loss and a novel twin loss. The twin loss encodes infrastructure constraints, collision avoidance, diversity across predicted modes, and rule-based priors derived from the digital twin. While the MSE term ensures point-wise accuracy, the twin loss penalizes traffic rule violations, predicted collisions, and mode collapse, guiding the model toward scene-consistent and safety-compliant predictions. We train and evaluate our approach on real-world V2X data sent from the intersection to the vehicle and collected in urban corridors. In addition to standard trajectory metrics (ADE, FDE), we introduce ITS-relevant safety indicators, including infrastructure and rule violation rates. Experimental results demonstrate that the proposed training scheme significantly reduces critical violations while maintaining comparable prediction accuracy and real-time performance, highlighting the potential of digital twin-driven multi-loss learning for V2X-enabled intelligent transportation systems.

Figures & tables

Fig. 1: TUM intersection scene: OpenDRIVE lane map (grey, 2,233 segments) overlaid with recorded agent trajectories (946 objects, coloured by class) from the first 30,000 frames. The lane geometry serves as the infrastructure prior for the proposed twin-consistency loss.

Split	Objects	1s	2s	3s	5s
Train	15 843	971 668	921 277	876 418	797 690
Val	3 395	227 450	216 105	205 976	188 287
Test	3 396	187 587	176 715	166 958	149 939
Total	22 634	1 386 705	1 314 097	1 249 352	1 135 916

TABLE I: Dataset statistics per split and prediction horizon (sliding-window samples), with a 70%/15%/15% split for training, validation, and testing.

Quantity	Range	Notes
History dx/dy (input)	$-30$ … $0$ m	2 s $\times$ 50 km/h max
Future dx/dy (1 s)	$0$ … $14$ m	cumulative position
Future dx/dy (5 s)	$0$ … $70$ m	cumulative position
$v_{x}$ , $v_{y}$	$-15$ … $+15$ m/s	not normalised
Lane distance / 100	$0$ … $0.5$
Lane heading / $\pi$	$-1$ … $+1$

TABLE II: Typical value ranges after anchor-relative normalisation.

Panel of Figure 4 — Fig. 2: LSTM encoder-decoder architecture. The dashed orange region highlights where MC-Dropout is active during inference for uncertainty estimation.

Variant	Extra losses	Description
Baseline	—	MSE only
Map_Loss	$\mathcal{L}_{\text{infra}}$	Lane proximity
Collision_Loss	$\mathcal{L}_{\text{coll}}$	Batch repulsion
Twin_All	$\mathcal{L}_{\text{infra}}+\mathcal{L}_{\text{coll}}$	Both digital-twin losses
CV	(no training)	Constant velocity
KF-CA	(no training)	Kalman–Singer

TABLE III: Model variants and their loss compositions.

Model	ADE $\downarrow$	FDE $\downarrow$	RMSE $\downarrow$	NL-ADE $\downarrow$
CV	0.931	1.957	1.111	1.098
KF-CA	1.001	2.286	1.217	1.169
Baseline	0.548	1.107	0.643	0.642
Map_Loss	0.716	1.268	0.803	0.760
Collision_Loss	0.778	1.255	0.859	0.797
Twin_All	0.936	1.484	1.021	0.917

TABLE IV: Results at 2 s horizon (

\downarrow

= lower is better).

Multi-Agent Trajectory Prediction

Multi-Agent Trajectory Prediction forecasts the possible future paths of interacting people, vehicles, robots, or other agents.

See all 79 papers

Multi-Pedestrian Safety Warning at Urban Intersections Use Case of Digital Twin
May 12, 2026Yongjie Fu, Qi Gao, Mahshid Ghasemi Dehkordi +2Intelligent Transportation Systems Digital Twins
Department of Civil Engineering and Engineering Mechanics at Columbia University, New York. · Mahshid Ghasemi Dehkordi and Gil Zussman are with the Department of Electrical Engineering at Columbia University, New York. · Yongjie Fu, Qi Gao, and Xuan Di are with the Department of Civil Engineering and Engineering Mechanics at Columbia University, New York. +1
Spatiotemporal Semantic V2X Framework for Cooperative Collision Prediction
Jan 23, 2026Murat Arda Onsu, Poonam Lohan, Burak Kantarci +3Intelligent Transportation Systems Semantic Communication
School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON, Canada. · Nokia Bell Labs, 600 March Road, Kanata, ON K2K 2E6, Canada.
Safety-oriented pedestrian trajectory prediction at urban intersections using time-to-collision and crossing-zone context
Aug 31, 2026Erel Avineri, Yftach Gil, Yehudit ApersteinPedestrian Trajectory Prediction Trajectory Prediction
Energy and Power Systems Engineering, Afeka Academic College of Engineering, Tel Aviv, Israel · Intelligent Systems, Afeka Academic College of Engineering, Tel Aviv, Israel

May 12, 2026J/K move · Enter open · S save

Multi-Pedestrian Safety Warning at Urban Intersections Use Case of Digital Twin

Yongjie Fu, Qi Gao, Mahshid Ghasemi Dehkordi, Gil Zussman+1

Department of Civil Engineering and Engineering Mechanics at Columbia University, New York. · Mahshid Ghasemi Dehkordi and Gil Zussman are with the Department of Electrical Engineering at Columbia University, New York. · Yongjie Fu, Qi Gao, and Xuan Di are with the Department of Civil Engineering and Engineering Mechanics at Columbia University, New York.+1

Intelligent Transportation Systems Digital Twins

Open paper AskPDF arXiv

Digital twins (DTs) for urban transportation systems have gained increasing attention; however, their systematic evaluation in safety-critical scenarios remains limited. This paper presents a multi-pedestrian safety warning system at urban intersections enabled by a tightly coupled physical-digital twin framework. Built upon the COSMOS city-scale wireless testbed in New York City, the proposed system integrates camera and ultra-wideband (UWB), edge-cloud computing, predictive trajectory modeling, and MQTT-based communication to deliver real-time safety alerts to vulnerable road users (VRUs). The system is evaluated through both field deployment and virtual reality (VR) experiments. Results demonstrate high warning generation accuracy, localization accuracy, efficient end-to-end latency under different model configurations, and significant reductions in user response time when warnings are issued. The proposed DT framework provides a scalable, modular, and generalizable solution for real-time multi-pedestrian safety enhancement at complex urban intersections.

Abstract

Multi-Agent Trajectory Prediction

Explore similar work

Multi-Pedestrian Safety Warning at Urban Intersections Use Case of Digital Twin

Spatiotemporal Semantic V2X Framework for Cooperative Collision Prediction

Safety-oriented pedestrian trajectory prediction at urban intersections using time-to-collision and crossing-zone context