Cross-Modal Hierarchical Fusion for from Multi-Sensor Ground Observation

We develop the Cloud 3-Dimensional Imager Retrieval (C3DIR), a deep learning model that estimates 3-D cloud properties for multiple passive satellite imagers trained to match retrievals from the Earth Cloud Aerosol and Radiation Explorer(EarthCARE) ACM-CAP product. This work is aimed towards moving AI/ML 3-D cloud algorithms closer towards operational use. C3DIR predicts the occurrence water content of ice, cloud liquid, and rain along the imager line-of-sight and uses a voxel-level collocation approach to account for the misaligned viewing geometries of passive imagers and active profiling instruments. This precise collocation methodology allows for constructing vertical profiles using voxels contained by multiple imager pixels to facilitate comparisons with active profiling instruments. Qualitative case studies show that C3DIR can accurately depict multiple distinct overlapping cloud layers, albeit with some smoothing. Quantitative evaluations illustrate that C3DIR overall excels at hydrometeor detection which intuitively tends to be a function of water content. However, detection of voxels classified as liquid cloud remains difficult due to the their small geometric thickness, finer horizontal scale, and frequent tendency to be obscured or embedded within ice clouds. In general, water content estimation is reasonably accurate, yielding the best results in ice clouds but uncertainties remain for liquid and rain water content. Column-integrated water paths are in tighter agreement with EarthCARE. Comparisons with the algorithms underpinning current NOAA operational products highlight several areas where C3DIR may offer improvement. Overall, these results demonstrate the potential for C3DIR to provide flexible 3-D output depicting vertically resolved cloud structure which can offer broader utility for aviation applications, numerical weather modeling, and climate research.

Frequent cloud cover severely limits the usability of Sentinel-2 (S2) optical time series for Earth surface monitoring. Sentinel-1 (S1) SAR provides all-weather complementary observations, but practical S1/S2 fusion remains difficult because acquisitions are irregular and asynchronous. Many existing approaches assume temporally aligned inputs (or require external nearest-date matching) and typically restore only observed timestamps, limiting reconstruction under long gaps and preventing on-demand synthesis. We propose AGFlow (Time Aligned Generative Flow Matching), a spatiotemporal flow-matching model for S1/S2 cloud removal and time-series reconstruction with three capabilities: (1) timestamp-conditioned internal alignment that fuses asynchronous S1 and cloudy S2 observations without preprocessing-based pairing; (2) spatiotemporal, context-aware denoising that models spatial structure jointly with temporal dynamics (rather than independent per-pixel time series); and (3) anytime querying, enabling generation of cloud-free S2 frames at both observed and user-specified timestamps within the monitoring window. We evaluate on the RESTORE-DiT benchmark protocol with quantitative metrics, qualitative comparisons, and component ablations. AGFlow notably improves fully missing-frame reconstruction (MAE and RMSE reduce by 16-19% over RESTORE-DiT) and provides reliable reconstructions under persistent gaps, while also yielding competitive cloud removal performance and flexible temporal querying for downstream tasks such as dense vegetation monitoring.

Cross-view localization classically asks: where does this ground image lie on the satellite tile? Existing methods are typically limited to 3-DoF estimates -- an $(x,y)$ position and a yaw angle -- because nadir satellite imagery provides no direct cues for roll, pitch, or altitude, forcing a reliance on planar-motion and zero-tilt assumptions. These assumptions break on real terrain with slopes, ramps, and tilted camera mounts. To overcome this, we introduce a single UAV image as an intermediate viewpoint: it reveals the 3D structure invisible from nadir, supplies the cues for roll, pitch, and altitude that the satellite alone cannot provide, and needs only spatial overlap with the ground camera -- no known relative pose is required. Building on this insight, we propose Cross3R, a flexible feed-forward model that ingests a satellite tile together with a UAV image, a ground image, or both, and, in a single forward pass, recovers a cross-view 3D point cloud, the 6-DoF poses of every input camera, and the on-tile $(x,y)$ position and yaw of each perspective camera. For training and evaluation, we also construct CrossGeo, a 278K-image tri-view dataset spanning 85 scenes across every continent except Antarctica. On CrossGeo, Cross3R consistently outperforms feed-forward 3D baselines in point-cloud reconstruction, 6-DoF camera-pose estimation, and cross-view localization. On KITTI, it outperforms dedicated cross-view methods trained on KITTI on most metrics, despite having no KITTI training itself.