Extracting and Analyzing Rail Crossing Behavior Signatures from Videos using Tensor Methods
Authors: Dawon Ahn, Het Patel, Aemal Khattak, Jia Chen, Evangelos E. Papalexakis
Organizations: Department of Computer Science and Engineering, UC Riverside, Riverside, USA · Department of Civil and Environmental Engineering, University of Nebraska-Lincoln, Lincoln, USA · Department of Electrical and Computer Engineering, UC Riverside, Riverside, USA
Railway crossings present complex safety challenges where driver behavior varies by location, time, and conditions. Traditional approaches analyze crossings individually, limiting the ability to identify shared behavioral patterns across locations. We propose a multi-view tensor decomposition framework that captures behavioral similarities across three temporal phases: Approach (warning activation to gate lowering), Waiting (gates down to train passage), and Clearance (train passage to gate raising). We analyze railway crossing videos from multiple locations using TimeSformer embeddings to represent each phase. By constructing phase-specific similarity matrices and applying non-negative symmetric CP decomposition, we discover latent behavioral components with distinct temporal signatures. Our tensor analysis reveals that crossing location appears to be a stronger determinant of behavior patterns than time of day, and that approach-phase behavior provides particularly discriminative signatures. Visualization of the learned component space confirms location-based clustering, with certain crossings forming distinct behavioral clusters. This automated framework enables scalable pattern discovery across multiple crossings, providing a foundation for grouping locations by behavioral similarity to inform targeted safety interventions.
Figures & tables
Fig. 1: Overview of the proposed multi-view tensor decomposition pipeline. (1) Each crossing event video is segmented into three behavioral phases (Approach, Waiting, Clearance) and processed through TimeSformer to extract 768-dimensional embeddings per phase. (2) For each phase p , we construct a 31×31 similarity matrix by computing pairwise cosine similarity between video embeddings, then stack the three matrices to form a tensor X∈R31×31×3 . (3) Symmetric non-negative CP decomposition factorizes the tensor into phase loadings ar∈RP and event loadings ur∈RN , revealing latent behavioral components across videos and phases.
Fig. 2: CORCONDIA diagnostic for rank selection. CORCONDIA measures core consistency, where values ≥ 80% (orange line) indicate acceptable CP structure and ≈ 100% (green line) indicates a perfect fit. Rank 2 shows severe structural issues (-109.9%), while ranks 1 and 3 demonstrate a valid CP structure. Only valid for R ≤ min(I,J,K) = 3.
Fig. 3: Reconstruction error (sum of squared errors) across ranks 1-10. The curve shows a clear elbow between ranks 3-5 (shaded region), indicating diminishing returns beyond rank 3. Values shown in thousands (K)
Fig. 4: Holdout validation error (RMSE) with 10% masked entries across ranks 1-10. Lower values indicate better generalization. Error generally decreases with rank, with substantial improvements up to rank 5. Proper masking prevents information leakage during training (5 random restarts per trial, averaged over 3 trials).
Fig. 5: Reading a CP decomposition component. Component magnitude ( λr ) indicates overall importance. Phase loadings ( ar ) reveal which phases (Approach, Waiting, Clearance) define the behavioral pattern. Video loadings ( ur ) identify which crossing events exhibit this pattern. The example component is Approach-dominant (0.82) and characterizes videos 5, 12, and 23.
Fig. 6: Distribution of 31 videos across crossing locations and time-of-day categories.
Fig. 7: Multi-view similarity tensor. Each 31×31 matrix shows pairwise cosine similarity between videos for one phase. Darker red indicates higher similarity.
Fig. 8: Phase loadings ( ar ) for non-negative symmetric CPD at rank 4. Each grouped bar shows the contribution of the three phases (Approach, Waiting, Clearance) to one of the four latent components.
Fig. 9: Non-negative CPD (rank 4) video loadings. Left: colored by time-of-day. Right: colored by crossing location.
Fig. 10: t-SNE projection of non-negative CPD component space. Left: colored by time-of-day. Right: colored by crossing location.
Multi-view clustering groups data by combining complementary representations while accounting for their shared structure, differences, and missing or noisy observations.
Department of Information Technology FPT University Ho Chi Minh City 700000, Vietnam · Faculty of Electronics Technology Industrial University of Ho Chi Minh City Ho Chi Minh City 700000, Vietnam · School of Computer Science and Engineering The Saigon International University Ho Chi Minh City 700000, Vietnam
Department of Electrical and Computer Engineering San Diego State University California, USA · Department of Civil, Construction, and Environmental Engineering San Diego State University California, USA