The Lift Spectrum: How Measurement-to-Space Adaptivity Shapes Robustness in Image-Free Single-Pixel Sensing
Authors: Yuyuan Han, Jingwei Li, Xiaoxia Zhang, Long Qiu, Chong Wang, Wenxuan Hao, Jiangyu Han, Xinyu Yao, +4 more
Abstract
Single-pixel sensing encodes a scene as a short sequence of coded measurements, and image-free methods infer the task directly from that sequence. We show that removing image reconstruction relocates the central design problem to the lift: how 1D measurements become a 2D task representation. We organize this choice as a lift spectrum from a fixed-physics inverse, through a learned static projection, to content-adaptive retrieval. These are not interchangeable forms of reconstruction: the fixed-physics route reconstructs an image consumed at inference, whereas our spatiotemporal soft-fusion (STSF) network lifts measurements directly into task features, and task-prioritized loss scheduling (TPLS) uses a separate learned reconstruction branch only as scheduled training supervision. A probe-selected recurrent encoder and a parameter-matched lift ablation identify the STSF design. In simulation, STSF+TPLS exceeds the prior image-free baseline on three datasets at 3.13% sampling (+3.2 to +9.9 pp foreground mIoU) and remains competitive down to 0.39%. The strongest clean-trained reconstruct-then-segment baseline wins without measurement noise, but measurement noise reverses the ranking: the reconstructed task input carries a 20-70x larger normalized relative perturbation than the measurements themselves. Stressed to failure, the three lift regions exhibit distinct dominant signatures--collapse, imprinting, and coarsening. STSF+TPLS transfers without fine-tuning to a real single-pixel bench, where the reversal reappears as a proof of concept; inference takes about 14 ms per mask on an RTX 4090. Within the tested fixed-acquisition regime, measurement-to-space adaptivity therefore organizes both the clean-to-noisy operating envelope and the failure a system encounters. Code and pretrained weights: https://github.com/Hanyuyuan6/STSF-TPLS.
MG-SpaIR is a training-data-free framework for restoring a clean image from a single observation corrupted by a mixture of blur, downsampling, noise, and missing pixels. Building on implicit neural representations (INRs), we introduce a multi-grade coarse-to-fine residual hierarchy that progressively refines the reconstruction across resolution grades, improving representational fidelity and mitigating spectral limitations. To stabilize reconstruction optimization and suppress INR-induced artifacts, we further propose an explicit sparse proximal regularization (e.g., ℓ0-type) applied directly in the high-resolution image domain, which discourages spurious high-frequency patterns while preserving sharp structures. The resulting optimization is solved efficiently via a multi-grade proximal alternating scheme, and we establish convergence guarantees for the associated updates under standard regularity conditions. Experiments on mixed-degradation benchmarks demonstrate that MG-SpaIR consistently outperforms strong training-data-free baselines such as Deep Image Prior, providing a stable, interpretable, and data-efficient alternative to conventional learning-based restoration methods.
Many high-resolution imaging systems face the same fundamental question: when have enough measurements been collected to reconstruct an image accurately? We develop Conformalized Rate-Adaptive Sensing (CoRAS), a method that adaptively chooses an acquisition or compression rate for each image while keeping the reconstruction error below a target level with high probability. As measurements are collected, an image reconstruction model gradually recovers the true image, producing a reconstruction path over acquisition rates. CoRAS uses this path up to an early decision time to estimate the target stopping time, defined as the first time at which the reconstruction error falls below the target level. It then calibrates this estimate using images with similar early reconstruction behavior, producing an upper bound on the stopping time with marginal and approximate conditional coverage guarantees. Experiments on image datasets show that CoRAS attains the target stopping-time coverage, uses fewer measurements on average than fixed-rate stopping rules, and assigns more measurements to images that are harder to reconstruct.
Single-photon LiDAR (SPL) based on single-photon avalanche diode (SPAD) sensing enables time-resolved photon measurements with extreme sensitivity, offering unique potential for active 3D perception in photon-starved scenarios.However, real-world single photon perception remains fundamentally challenging due to unique measurement noise and complex multi-return transient phenomena, which jointly complicate geometric reconstruction and semantic scene understanding. Despite growing interest in SPAD-based sensing, existing studies are largely limited to simulated data or small-scale controlled captures. As a result, systematic evaluation of real-world single photon perception across depth estimation, multi-view reconstruction, and 3D semantic understanding remains underexplored. To bridge this gap, we introduce SP-TransientBench (STB), a real-captured multi-task benchmark for single photon perception. SP-TransientBenc comprises 10 diverse scenes and 10,297 views captured using a solid-state single-photon LiDAR at 256×192 resolution. Each view provides full time-of-flight histograms with multi-return behavior,standardized metadata, and calibrated camera poses for multi-view evaluation. We further provide 13-class 3D semantic annotations for selected scenes. By providing dedicated data splits and evaluation protocols for each task, STB enables consistent and reproducible benchmarking of real-world single photon perception across multiple 3D vision problems. The dataset and code will be released upon acceptance.