cs.CVMar 17, 2026

x2x^2-Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space

Authors: Ruishan Guo, Ciyu Ruan, Haoyang Wang, Zihang Gong, Jingao Xu, Xinlei Chen

Organizations: Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology · The University of Hong Kong

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Estimating dense 2D optical flow and 3D scene flow is essential for dynamic scene understanding. Recent work combines images, LiDAR, and event data to jointly predict 2D and 3D motion, yet most approaches operate in separate heterogeneous feature spaces. Without a shared latent space that all modalities can align to, these systems rely on multiple modality-specific blocks, leaving cross-sensor mismatches unresolved and making fusion unnecessarily complex.Event cameras naturally provide a spatiotemporal edge signal, which we can treat as an intrinsic edge field to anchor a unified latent representation, termed the Event Edge Space. Building on this idea, we introduce x2x^2-Fusion, which reframes multimodal fusion as representation unification: event-derived spatiotemporal edges define an edge-centric homogeneous space, and image and LiDAR features are explicitly aligned in this shared representation.Within this space, we perform reliability-aware adaptive fusion to estimate modality reliability and emphasize stable cues under degradation. We further employ cross-dimension contrast learning to tightly couple 2D optical flow with 3D scene flow. Extensive experiments on both synthetic and real benchmarks show that x2x^2-Fusion achieves state-of-the-art accuracy under standard conditions and delivers substantial improvements in challenging scenarios.

Figures & tables

Figure 1: Fusion paradigms for multimodal perception. (a) RPEFlow [ 8 ] : stage-wise fusion across 2D/3D spaces. (b) CMX [ 9 ] :image-like encodings with pairwise rectification/attention at each backbone stage. (c) VisMoFlow [ 10 ] : separate luminance/structure/correlation spaces with dedicated modules. (d) Ours: preserves native domains and learns an Event Edge Space , a shared edge-centric latent space guided by a frozen event teacher, that aligns image, LiDAR, and events before fusion.
Figure 2: Overview of x2x^{2} -Fusion. Given image, events and LiDAR, we first pretrain an Event Edge Encoder to distill motion-aware edge features. We then freeze the encoder and use its embeddings as edge prototypes to symmetrically regularize channel-wise representations across modalities, aligning them into a shared Event Edge Space (Sec. 3.1 ). Within this space, Reliability-aware Adaptive Fusion estimates global and local reliability and fuses modalities via a cross-attention block to produce 2D/3D features (Sec. 3.2 ). Finally, Cross-dimension Contrast Learning enforces inter-frame coherence and 2D–3D consistency, and the task heads output optical and scene flow (Sec. 3.3 ).
Figure 3: Our proposed event edge encoder pretraining learns explicit, high-fidelity motion-aware edge representations by predicting edge strength from voxelized event streams.
Figure 4: The proposed reliability-aware adaptive fusion module adaptively integrates image, LiDAR, and event features through hierarchical reliability weighting and cross-modal attention.
ModalityMethod#Params(M)EKubricDSEC
EPE2D\text{EPE}_{\text{2D}} ↓\downarrowACC1px\text{ACC}_{\text{1px}} ↑\uparrowFl ↓\downarrowEPE3D\text{EPE}_{\text{3D}} ↓\downarrowACC.05\text{ACC}_{\text{.05}} ↑\uparrowACC.10\text{ACC}_{\text{.10}} ↑\uparrowEPE2D\text{EPE}_{\text{2D}} ↓\downarrowACC1px\text{ACC}_{\text{1px}} ↑\uparrowFl ↓\downarrowEPE3D\text{EPE}_{\text{3D}} ↓\downarrowACC.05\text{ACC}_{\text{.05}} ↑\uparrowACC.10\text{ACC}_{\text{.10}} ↑\uparrow
ImgRAFT [ 29 ]5.3M0.83893.31%2.36%---0.58688.98%1.47%---
ImgFlowFormer [ 4 ]16.2M0.70292.58%2.07%---0.56789.82%1.33%---
PC (Point Cloud)PV-RAFT [ 50 ]239.2K---0.09382.42%92.60%---0.19032.74%55.62%
EVE-RAFT [ 51 ]5.3M------0.48191.75%1.31%---
Img+DepthRAFT-3D [ 52 ]7.7M0.71494.39%-0.04992.88%-0.57289.55%-0.14449.30%-
Table 1: Quantitative evaluation on both standard and degraded scenarios of EKubric and DSEC datasets. †\dagger Reproduced per original methodology (code unavailable). Extended results are provided in the supplementary material.
Figure 5: Visual comparison of optical flows on EKubric and DSEC dataset. x2x^{2} -Fusion achieves the state-of-the-art performance across various exposure degradation scenarios with clearer motion boundaries and finer details. Please zoom in for details.
Figure 6: Visual comparison of scene flows on EKubric and DSEC dataset. x2x^{2} -Fusion demonstrates achieves lower end-point errors across diverse degradation scenarios. Comprehensive results and analyses are provided in the supplementary material. Please zoom in for details.
SettingsEPE2D\text{EPE}_{\text{2D}} ↓\downarrowEPE3D\text{EPE}_{\text{3D}} ↓\downarrow
w/o Event Edge Space0.4910.146
w/o Edge-anchored Regularization0.3930.119
w/o Event Edge Encoder0.3780.114
w/ Event Edge Space0.3220.094
Table 2: Effects of the Event Edge Space.
Figure 7: t-SNE visualization of cross-modal feature alignment. The edge-anchored symmetric regularization reduces tri-model feature divergence through pyramid-level alignment.
VariantsSettingEPE2D\text{EPE}_{\text{2D}} ↓\downarrowEPE3D\text{EPE}_{\text{3D}} ↓\downarrow
Model#AIndep. 2D + 3D0.4040.119
Model#BJoint 2D & 3D0.3860.113
Model#CJoint 2D & 3D + CCL0.3250.103
Table 3: Effects of cross-dimension learning and joint task.
Figure 8: Qualitative analysis of joint task performance and contrastive learning effects. (Top) t-SNE visualization of fused 2D-3D feature distributions; (Bottom) Visualization of (left) 2D optical flow and (right) 3D scene flow error maps.
More in this topic

Scene Flow Estimation

See all 21 papers

Explore similar work

CardsList
Mar 2, 2026

LEAR: Learning Edge-Aware Representations for Event-to-LiDAR Localization

Event cameras offer high-temporal-resolution sensing that remains reliable under high-speed motion and challenging lighting, making them promising for localization from LiDAR point clouds in GPS-denied and visually degraded environments. However, aligning sparse, asynchronous events with dense LiDAR maps is fundamentally ill-posed, as direct correspondence estimation suffers from modality gaps. We propose LEAR, a dual-task learning framework that jointly estimates edge structures and dense event-depth flow fields to bridge the sensing-modality divide. Instead of treating edges as a post-hoc aid, LEAR couples them with flow estimation through a cross-modal fusion mechanism that injects modality-invariant geometric cues into the motion representation, and an iterative refinement strategy that enforces mutual consistency between the two tasks over multiple update steps. This synergy produces edge-aware, depth-aligned flow fields that enable more robust and accurate pose recovery via Perspective-n-Point (PnP) solvers. On several popular and challenging datasets, LEAR achieves superior performance over the best prior method. The source code, trained models, and demo videos are made publicly available online.

Kuangyi Chen, Jun Zhang, Yuxi Hu +2

Institute of Visual Computing, Graz University of Technology, 8010 Graz, Austria · Neuromorphic Automation and Intelligence Lab (NAIL), School of Artificial Intelligence and Robotics, Hunan University, Changsha 410012, China

LiDAR LocalizationCamera-LiDAR FusionEvent-Based VisionEvent-Based Optical Flow
AskPDF
Feb 25, 2026

SF3D-RGB: Scene Flow Estimation from Monocular Camera and Sparse LiDAR

Scene flow estimation is an extremely important task in computer vision to support the perception of dynamic changes in the scene. For robust scene flow, learning-based approaches have recently achieved impressive results using either image-based or LiDAR-based modalities. However, these methods have tended to focus on the use of a single modality. To tackle these problems, we present a deep learning architecture, SF3D-RGB, that enables sparse scene flow estimation using 2D monocular images and 3D point clouds (e.g., acquired by LiDAR) as inputs. Our architecture is an end-to-end model that first encodes information from each modality into features and fuses them together. Then, the fused features enhance a graph matching module for better and more robust mapping matrix computation to generate an initial scene flow. Finally, a residual scene flow module further refines the initial scene flow. Our model is designed to strike a balance between accuracy and efficiency. Furthermore, experiments show that our proposed method outperforms single-modality methods and achieves better scene flow accuracy on real-world datasets while using fewer parameters compared to other state-of-the-art methods with fusion.

Rajai Alhimdiat, Ramy Battrawy, René Schuster +2

Islamic University of Gaza, Gaza, Palestine · German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Scene Flow Estimation3D Scene UnderstandingCamera-LiDAR Fusion
AskPDF
Jul 17, 2026

On the Geometry of Learned Representations in Event-Based Multi-Modal Egomotion Estimation

Classical approaches to event-based egomotion estimation, including those adopted by the top-performing teams of the ELOPE challenge, rely on geometric optimization frameworks such as contrast maximization, homography estimation, or dense optical flow combined with analytic motion inversion. This work investigates the geometric structure that emerges inside a multi-modal network for egomotion estimation. Event tensors, inertial measurements, and range signals are fused through a cross-modal attention architecture and trained in a batch setting. We analyze the latent space geometry and attention dynamics, showing that (i) embeddings lie on low-dimensional manifolds aligned with motion variables, (ii) attention weights adapt with angular excitation and visual reliability, and (iii) the fused representation recovers classical observability cues. These results bridge analytical estimation theory and modern data-driven fusion.

Stefano Silvestrini, Michele Ceresoli

Politecnico di Milano, Via Giuseppe La Masa, 34, 20156, Milan, Italy

Representation GeometryEvent-Based VisionVisual OdometryMultimodal Sensor Fusion
AskPDF