Extracting and Analyzing Rail Crossing Behavior Signatures from Videos using Tensor Methods
Authors: Dawon Ahn, Het Patel, Aemal Khattak, Jia Chen, Evangelos E. Papalexakis
Organizations: Department of Computer Science and Engineering, UC Riverside, Riverside, USA · Department of Civil and Environmental Engineering, University of Nebraska-Lincoln, Lincoln, USA · Department of Electrical and Computer Engineering, UC Riverside, Riverside, USA
Railway crossings present complex safety challenges where driver behavior varies by location, time, and conditions. Traditional approaches analyze crossings individually, limiting the ability to identify shared behavioral patterns across locations. We propose a multi-view tensor decomposition framework that captures behavioral similarities across three temporal phases: Approach (warning activation to gate lowering), Waiting (gates down to train passage), and Clearance (train passage to gate raising). We analyze railway crossing videos from multiple locations using TimeSformer embeddings to represent each phase. By constructing phase-specific similarity matrices and applying non-negative symmetric CP decomposition, we discover latent behavioral components with distinct temporal signatures. Our tensor analysis reveals that crossing location appears to be a stronger determinant of behavior patterns than time of day, and that approach-phase behavior provides particularly discriminative signatures. Visualization of the learned component space confirms location-based clustering, with certain crossings forming distinct behavioral clusters. This automated framework enables scalable pattern discovery across multiple crossings, providing a foundation for grouping locations by behavioral similarity to inform targeted safety interventions.
Figures & tables
Fig. 1: Overview of the proposed multi-view tensor decomposition pipeline. (1) Each crossing event video is segmented into three behavioral phases (Approach, Waiting, Clearance) and processed through TimeSformer to extract 768-dimensional embeddings per phase. (2) For each phase p , we construct a 31×31 similarity matrix by computing pairwise cosine similarity between video embeddings, then stack the three matrices to form a tensor X∈R31×31×3 . (3) Symmetric non-negative CP decomposition factorizes the tensor into phase loadings ar∈RP and event loadings ur∈RN , revealing latent behavioral components across videos and phases.
Fig. 2: CORCONDIA diagnostic for rank selection. CORCONDIA measures core consistency, where values ≥ 80% (orange line) indicate acceptable CP structure and ≈ 100% (green line) indicates a perfect fit. Rank 2 shows severe structural issues (-109.9%), while ranks 1 and 3 demonstrate a valid CP structure. Only valid for R ≤ min(I,J,K) = 3.
Fig. 3: Reconstruction error (sum of squared errors) across ranks 1-10. The curve shows a clear elbow between ranks 3-5 (shaded region), indicating diminishing returns beyond rank 3. Values shown in thousands (K)
Fig. 4: Holdout validation error (RMSE) with 10% masked entries across ranks 1-10. Lower values indicate better generalization. Error generally decreases with rank, with substantial improvements up to rank 5. Proper masking prevents information leakage during training (5 random restarts per trial, averaged over 3 trials).
Fig. 5: Reading a CP decomposition component. Component magnitude ( λr ) indicates overall importance. Phase loadings ( ar ) reveal which phases (Approach, Waiting, Clearance) define the behavioral pattern. Video loadings ( ur ) identify which crossing events exhibit this pattern. The example component is Approach-dominant (0.82) and characterizes videos 5, 12, and 23.
Fig. 6: Distribution of 31 videos across crossing locations and time-of-day categories.
Fig. 7: Multi-view similarity tensor. Each 31×31 matrix shows pairwise cosine similarity between videos for one phase. Darker red indicates higher similarity.
Fig. 8: Phase loadings ( ar ) for non-negative symmetric CPD at rank 4. Each grouped bar shows the contribution of the three phases (Approach, Waiting, Clearance) to one of the four latent components.
Fig. 9: Non-negative CPD (rank 4) video loadings. Left: colored by time-of-day. Right: colored by crossing location.
Fig. 10: t-SNE projection of non-negative CPD component space. Left: colored by time-of-day. Right: colored by crossing location.
Multi-view clustering groups data by combining complementary representations while accounting for their shared structure, differences, and missing or noisy observations.
Given one or more images of a railway crossing, can we leverage visual cues that allow us to robustly estimate how safe it is? Can we improve our ability to do so by introducing structured data (such as official accident reports) about the accident history of that crossing into our models? In this work, we explore how to best answer those questions towards building an AI system that can ingest multi-modal data for railway crossings and provide safety assessment and scores that align with expert opinion and with safety scoring used by the Federal Railroad Administration (FRA). To that end, we propose a proof-of-concept pipeline that delivers on that goal, while at the same time exploring and tackling a number of critical research challenges that pertain to different parts of the pipeline, from data preparation to different learning paradigms that can allow us to realize such a system. Indicatively, our proposed system identifies HIGH-RISK and LOW-RISK crossings with a macro F1 score of 0.757 and estimates FRA-based safety scores with an RMSE of 0.078 and correlation of 0.492 using a routed fine-tuned compact VLM pipeline, while producing qualitative results that align with domain-expert assessment.
Paimon Goulart, Chansong Lim, Nícolas Roque dos Santos +4
University of California, Riverside Riverside, California, USA · Riverside County Transportation Commission Riverside, California, USA
The identification of hazardous driving behaviors from in-cabin video streams is essential for enhancing road safety and supporting the detection of traffic violations and unsafe driver actions. However, current temporal action localization techniques often struggle to balance accuracy with computational efficiency. In this work, we develop and evaluate a temporal action localization framework tailored for driver monitoring scenarios, particularly suitable for periodic inspection settings such as transportation safety checkpoints or fleet management assessment systems. Our approach follows a two-stage pipeline that combines VideoMAE-based feature extraction with an Augmented Self-Mask Attention (AMA) detector, enhanced by a Spatial Pyramid Pooling-Fast (SPPF) module to capture multi-scale temporal features. Experimental results reveal a distinct trade-off between model capacity and efficiency. At the feature extraction stage, the ViT-Giant backbone delivers higher representations with 88.09% Top-1 test accuracy, while the ViT-based variant proves to be a practical alternative, achieving 82.55% accuracy with significantly lower computational fine-tuning costs (101.85 GFLOPs/segment compared to 1584.06 GFLOPs/segment for Giant). In the downstream localization task, the integration of SPPF consistently improves performance across all configurations. Notably, the ViT-Giant + SPPF model achieves a peak mAP of 92.67%, while the lightweight ViT-based configuration maintains robust results.
Gia-Bao Doan, Nam-Khoa Huynh, Minh-Nhat-Huy Ho +3
Department of Information Technology FPT University Ho Chi Minh City 700000, Vietnam · Faculty of Electronics Technology Industrial University of Ho Chi Minh City Ho Chi Minh City 700000, Vietnam · School of Computer Science and Engineering The Saigon International University Ho Chi Minh City 700000, Vietnam
Traffic safety analysis at signalized intersections is essential for reducing vehicle and pedestrian collisions, yet traditional crash-based studies are limited by data sparsity and reporting latency. This paper presents a multi-camera computer vision framework for real-time safety assessment through Post-Encroachment Time (PET) computation, demonstrated at the intersection of H Street and Broadway in Chula Vista, California. Four synchronized cameras provide continuous visual coverage, with frames processed on NVIDIA Jetson AGX Xavier edge devices using YOLOv11 segmentation for vehicle detection. Detected vehicle polygons are transformed into a unified bird's-eye map via homography, enabling alignment across overlapping camera views. A pixel-level PET algorithm tracks temporal occupancy at each spatial location by measuring the time between successive vehicle passages, enabling fine-grained hazard visualization through dynamic heatmaps with 3.3 sq-cm spatial resolution. Timestamped vehicle trajectories and PET data are stored in an SQL database for longitudinal analysis. Results across multiple time intervals demonstrate the framework's ability to identify high-risk regions with sub-second temporal sensitivity and real-time edge throughput, generating 800 x 800 logarithmic heatmaps at an average of 2.68 FPS. This paper validates decentralized vision-based PET analysis for intelligent transportation systems and presents a scalable methodology for high-resolution, real-time intersection safety evaluation.
Shounak Ray Chaudhuri, Arash Jahangiri, Christopher Paolini
Department of Electrical and Computer Engineering San Diego State University California, USA · Department of Civil, Construction, and Environmental Engineering San Diego State University California, USA