astro-ph.IMApr 10, 2026
SaveLearning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics
Organizations: Massachusetts Institute of Technology · Flatiron Institute, Simons Foundation · Institute for Advanced Studies · AstroAI, CfA | Harvard & Smithsonian ·
Abstract
Data collected from the physical world is always a combination of multiple sources: an underlying signal from the physical process of interest and a signal from measurement-dependent artifacts from the sensor or instrument. This secondary signal acts as a confounding factor, limiting our ability to extract information about the physics underlying the phenomena we observe. Furthermore, it complicates the combination of observations in heterogeneous or multi-instrument settings. We propose a deep learning framework that leverages overlapping observations, a dual-encoder architecture, and a counterfactual generation objective to disentangle these factors of variation. The resulting representations explicitly separate intrinsic signals from sensor-specific distortions and noise, and can be used for counterfactual view generation, parameter inference unconfounded by measurement distortions, and instrument-independent similarity search. We demonstrate the effectiveness of our approach on astrophysical galaxy images from the DESI Legacy Imaging Survey (Legacy) and the Hyper Suprime-Cam (HSC) Survey as a representative multi-instrument setting. This framework provides a general recipe for scientific and multi-modal self-supervised pretraining: construct training pairs from overlapping observations of the same physical system, treat sensor- or modality-specific effects as augmentations, and learn invariant representations through counterfactual generation.
Figures & tables
Figure 1: Counterfactual Reconstruction. The model learns to disentangle intrinsic galaxy properties from instrument systematics by reconstructing an anchor image. Training uses data triplets : an anchor observation (signal from instrument ), an instrument-augmented observation ( same source , a different instrument ), and a physics-augmented observation (a different source , same instrument ). The physics encoder extracts latents from the instrument-augmented pair to capture invariant physical features, while the instrument encoder processes the physics-augmented observation to isolate measurement artifacts. These latent variables jointly condition the flow-matching decoder. Crucially, the anchor image is never fed into the encoders; it is used only as a target for the flow-matching loss, forcing the model to generate the reconstruction counterfactually.
Figure 2: Disentanglement. UMAP projections of the physics (left) and instrument (right) latent spaces. Orange and blue points represent HSC and Legacy images, respectively. Matched markers ( , , , ) denote cross-survey pairs. In the physics space , the encoders produce overlapping distributions where cross-survey pairs are mapped to similar coordinates. In contrast, the instrument space shows a clean separation between surveys and pairs, confirming that this space captures survey-specific features rather than intrinsic galaxy properties.
Figure 3: Outlier Detection. Top twelve outliers from each of the three latent spaces. Left: our physics latent reveals morphologically unusual sources (e.g., galaxy mergers), free of artifacts. Middle & Right: anomalies from AION-1 and our instrument latent are dominated by instrumental pathologies (saturated pixels, bands, noise patterns), confirming that our representation separates physical anomalies from instrument systematics.
Figure 4: Probing Latent Disentanglement via Downstream Regression. We report scores for the prediction of physics-related properties (left) and instrumental properties (right) on four sets of representations: our physics embeddings (blue), our instrument embeddings (red), AION-1 embeddings which treat each survey as an independent modality (green), and a randomly initialized ResNet-18 of the same architecture as our encoders (hatched black). We distinguish physics-related properties and instrument properties, which are grouped by survey. The asymmetry between the two panels demonstrates that our architecture successfully separates physics properties from instrument-specific information.
Figure 5: Instrument-Invariant Search and Rare-Object Discovery. Top (Cross-Survey Retrieval): We search the latent spaces using an HSC observation as query. Physics Space: The pair from the alternate survey is identified as the top-1 neighbor, with additional top-ranked neighbors showing physically similar galaxies from a mixture of both instruments. Notably, the four examples shown could have been found within the top nine nearest neighbors independent of which survey the query came from. Instrument Space: Neighbors in the instrument space show similar noise characteristics independent of the underlying galactic signal. Bottom (Strong Lenses): Nearest neighbor search within the physics latent space using a known strong lens as query, focusing only on the HSC images. The retrieved top neighbors show features suggestive of strong lensing, demonstrating the model’s capacity for rare-object discovery.
Figure 6: Cross-instrument Galaxy Reconstructions. Columns 1–2 (Inputs): The target galaxy observed via an alternate instrument (input to the physics encoder) and a set of up to five different galaxies imaged by the target instrument (input to the instrument encoder). Column 3 (Ground Truth): The original target image. Columns 4–5 (Posterior Samples): Independent samples generated by the decoder, demonstrating the model’s ability to capture noise and uncertainty. Column 6 (Posterior Mean): The pixel-wise mean of the generated posterior samples.
Figure 7: Data-driven Noise Model. Top (Progressive Denoising): Reconstructions of a target galaxy from its Legacy counterpart, varying the instrument conditioning across different percentiles of the HSC SNR distribution. The model successfully generates progressively cleaner images; the average SNR shown is that of the images used to condition the reconstruction. Bottom (Systematic Artifact Removal): By conditioning the physics encoder on an artifact-corrupted HSC image and the instrument encoder on a random set of HSC images, the model successfully removes severe instrumental distortions while preserving the underlying galaxy morphology.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Data Dimensions | |
| Image size | |
| Input channels | 4 |
| Conditioning channels | 4 |
| UNet Architecture | |
| Base model channels | 128 |
Table 1: Model architecture parameters for the conditional flow matching model with dual ResNet-18 encoders.
| Parameter | Value |
|---|---|
| Optimization | |
| Optimizer | AdamW |
| Learning rate | |
| LR scheduler | CosineAnnealingLR |
| Loss function | Velocity MSE |
| Flow Matching | |
Table 2: Training configuration and hyperparameters.
| Variable | Source | Description |
|---|---|---|
| Ellipticity ( ) | HSC catalog | Galaxy shape ellipticity components, measured via the SDSS-style second moments of the surface brightness distribution. |
| Redshift ( ) | DESI spectra | Spectroscopic redshift. |
| Stellar mass ( ) | PROVABGS | Total mass in stars,, derived from Bayesian spectral energy distribution (SED) fitting to DESI spectra [ 10 ] . |
| Half-light radius ( ) | HSC catalog | Circularised radius enclosing half of the galaxy’s total light. A measure of galaxy size. |
| sSFR | PROVABGS | Specific star formation rate : the rate at which new stars are forming normalised by the galaxy’s stellar mass. |
| PROVABGS | Gas-phase metallicity: the abundance of elements heavier than helium in the galaxy’s gas. Traces the galaxy’s chemical enrichment history. |
Table 3: Physical parameters used for downstream regression evaluation of the physics latent space.
| Variable | Bands | Description |
| HSC (Hyper Suprime-Cam) | ||
| psf_fwhm | Effective size of the point spread function (arcsec): how much the instrument and atmosphere spread the light from a point source. Analogous to Legacy PSFSIZE . | |
| a | Milky Way dust extinction per band (mag). Equivalent to Legacy EBV scaled by a filter-dependent coefficient: . | |
| Legacy Survey (DESI Legacy Imaging Surveys) | ||
| NOBS | Number of exposures contributing to the coadd. | |
| GALDEPTH | detection depth for a round exponential galaxy with (mag). Shallower than PSFDEPTH because galaxy flux is spread over more pixels. | |
Table 4: Instrumental and observational parameters used for downstream regression evaluation of the instrument latent space. All quantities vary spatially across each survey’s footprint.
| Property | Probe on cross-matched Legacy-HSC sample (Im) | Literature Reported Values | |||||
| Ours (Phys) | Ours (Instr) | AION-1-B | Rnd In. ResNet | X-Predict | AION-1-B (Ph+Im) | AstroCLIP (Im) | |
| Phys.-related properties | |||||||
| Ellipticity | 0.934 | 0.321 | 0.941 | 0.616 | — | — | — |
| Ellipticity | 0.918 | 0.421 | 0.939 | 0.613 | — | — | — |
| Redshift | 0.801 | 0.535 | 0.857 | 0.514 | — | 0.930 | 0.780 |
| 0.755 | 0.491 | 0.818 | 0.485 | — | 0.890 | 0.730 | |
Table 5: MLP-probe on the held-out overlap -MMU sample. Bold = best in row among the five probe methods (the two right-most columns are literature reference values and are not compared against). ‘Ours (Physics)’ and ‘Ours (Instrument)’ use our flow-matching encoder’s (flat) and (mean-pooled) latents respectively. ‘AION-1 (Base)’ is the AION-1 Base model [ 19 ] probed identically. ‘Rand. Init. ResNet’ is a frozen randomly-initialised ResNet-18 of the same architecture as our encoders. ‘Cross-predict’ is a fresh ResNet-18 trained from scratch to predict each survey’s instrument properties from the other survey’s raw image (only defined for instrument targets). Reference values: AION-1-B (Ph+Im) from [ 19 ] Table 1; AstroCLIP image-only on Legacy from [ 15 ] . Band-averaged where applicable.
Figure 8: Spatial Structure Preservation. Power spectrum (left) and autocorrelation (right) per band for ground-truth and generated images. Each plot shows the true value (blue), the posterior mean from 32 generated samples (orange dashed line), and the posterior standard deviation (orange shaded region). Strong agreement across spatial frequencies and pixel separation (lag) distances indicates that the model captures realistic morphological structure and noise properties.
Figure 9: Left: Pixel-wise Z-score distribution of generated posterior samples relative to ground-truth target images, across all pixels and held-out galaxies. Both HSC-anchored and Legacy-anchored reconstructions closely approximate a standard normal distribution, indicating that the posterior mean is approximately unbiased, the posterior variance captures the true prediction error, and that the residual uncertainty is approximately Gaussian. The standard deviations slightly above unity indicate mild overconfidence, more pronounced for HSC-anchored reconstructions, consistent with the greater ambiguity in generating higher-resolution images from lower-resolution conditioning. Right: for galaxy ellipticity from a ResNet trained on real HSC images, evaluated on real versus counterfactual HSC images. Near-identical performance shows that existing HSC pipelines can be readily applied to Legacy images through our counterfactual model, extending HSC-quality analysis across the full Legacy footprint.
| Parameter | Value |
|---|---|
| Flow Architecture | |
| Family | Neural Spline Flow (NSF, zuko ) |
| Number of transforms | 6 |
| Hidden features per transform | |
| Context dimension | 0 (unconditional) |
| Input dimension | 64 / 16 (Ours) / 768 (AION-1) |
Table 6: Normalizing flow architecture and training hyperparameters used for outlier detection.