Lumosaic: Hyperspectral Video via Active Illumination and Coded-Exposure Pixels
Authors: Dhruv Verma, Andrew Qiu, Roberto Rangel, Ayandev Barman, Hao Yang, Chenjia Hu, Fengqi Zhang, Roman Genov, +3 more
Organizations: Department of Computer Science, University of Toronto, Canada · Department of Electrical & Computer Engineering, University of Toronto, Canada
We present Lumosaic, a compact active hyperspectral video system designed for real-time capture of dynamic scenes. Our approach combines a narrowband LED array with a coded-exposure-pixel (CEP) camera capable of high-speed, per-pixel exposure control, enabling joint encoding of scene information across space, time, and wavelength within each video frame. Unlike passive snapshot systems that divide light across multiple spectral channels simultaneously and assume no motion during a frame's exposure, Lumosaic actively synchronizes illumination and pixel-wise exposure, improving photon utilization and preserving spectral fidelity under motion. A learning-based reconstruction pipeline then recovers 31-channel hyperspectral (400-700 nm) video at 30 fps and VGA resolution, producing temporally coherent and spectrally accurate reconstructions. Experiments on synthetic and real data demonstrate that Lumosaic significantly improves reconstruction fidelity and temporal stability over existing snapshot hyperspectral imaging systems, enabling robust hyperspectral video across diverse materials and motion conditions.
Figures & tables
Figure 2: The operation of a single pixel in a coded-exposure-pixel (CEP) camera. Each pixel alternates between two charge storage sites (Bucket 0 and Bucket 1) according to a binary exposure code. During each sub-frame, only one bucket integrates incident light, enabling temporal multiplexing across the exposure period.
Figure 3: (A) Lumosaic features a CEP camera synchronized with a programmable array of narrowband LEDs. (B) Each sub-frame is assigned a unique LED and spatial exposure mask, producing a dense spatio-spectro-temporal scene encoding within a single video frame.
Figure 4: Reconstruction performance of QDO [ 20 ] , MST++ [ 7 ] , and Lumosaic using HAN [ 27 ] , MCAN [ 9 ] , and SRNet [ 5 ] backbones according to (top-left) PSNR, (top-right) MAE, (bottom-left) SSIM, and (bottom-right) SAM for the simulations under varying Gaussian noise levels.
Figure 5: Quantitative and qualitative evaluation of spectral reconstruction accuracy using a ColorChecker target (real data). (A) Raw coded frame captured by Lumosaic. (B) Zoomed-in region showing the mosaic coding. (C) Reconstructed hyperspectral image rendered in sRGB. (D) Spectral reflectance curves for all 24 patches, comparing ground-truth (solid blue) and reconstructed (dashed yellow) spectra.
Figure 6: An example of real-world reconstruction results obtained using Lumosaic. (A) Reconstructed hyperspectral image rendered as sRGB. (B) Reflectance spectra extracted from marked regions. (C) Reconstructed hyperspectral channels.
Figure 7: Hyperspectral video reconstruction of a dynamic scene: a colored droplet diffusing in water, captured using Lumosaic at 30 fps. Each column shows a representative frame over time rendered in sRGB, while the bottom row visualizes reconstructed spectral channels for the highlighted frame. See the Supplementary Video for the full sequence visualization.
Figure 2: The hardware schematic of Lumosaic’s active illumination module. The ESP32 microcontroller coordinates the LED driver array and issues synchronization pulses to the CEP camera.
LED Name
Relative Allocation
Time per Frame (µs)
UV
5.70%
1,350
Violet
3.16%
750
Royal Blue
3.16%
750
Blue
3.16%
750
Cyan
5.70%
1,350
Green
6.96%
1,650
Table 2: The LED exposure allocation per video frame compensates for relative intensity variations across LEDs and camera spectral sensitivity. Each LED’s sub-frame activations occur contiguously within the frame.
Figure 3: The 3×4 mosaic tile that forms the basis of Lumosaic’s spatial-spectral coding.
Figure 4: (top) LED spectral power distributions before and after calibration via optimized scaling factors αλ . (bottom) Camera spectral sensitivity S(λ) and incandescent bulb spectral power distribution (SPD).
Figure 6: A visualization of the spatial weighting kernel K used for merging overlapping patch predictions.
Figure 7: Qualitative comparisons on public datasets. Representative reconstructions from the test set comparing QDO [ 20 ] , MST++ [ 7 ] , and Lumosaic. Each column shows the reconstructed sRGB image, selected hyperspectral channels, and zoomed-in sRGB crops under varying Gaussian noise levels (annotated with the corresponding standard deviation). We also present spectral validation plots for two specific locations, marked with white stars in the first-row sRGB images, displayed at the bottom of the figure.
Figure 8: Spectral recovery test on a high-frequency rainbow scene. ( Left ) sRGB visualization of the rainbow scene: (A) ground truth and reconstructions from (B) QDO [ 20 ] , (C) MST++ [ 7 ] , and (D) Lumosaic. Rows (D-F) illustrate Lumosaic results under increasing proportions of high-frequency synthetic spectra in the training set (0%, 33%, 66%, 100%). ( Right ) Reconstructed spectral reflectance, interpolated at 1-nm intervals, and center wavelength positions at selected locations on the rainbow (marked by triangles), compared to ground truth, demonstrating Lumosaic’s improved ability to recover sharp spectral transitions.
Figure 9: Additional Lumosaic reconstruction results on diverse static scenes: optical filters, a globe, a butterfly specimen, and a figurine with printed matter. Each scene includes (A) the input coded image, (B) the reconstructed hyperspectral image rendered as sRGB, and (C) the 31-channel hyperspectral composites rendered in grayscale for visualization.
Figure 10: Metamerism analysis with Lumosaic using genuine and printed ColorChecker targets. ( A and B ) RGB visualization of the genuine ColorChecker and a printed photocopy under identical illumination. ( C ) Spectral curves illustrating the differences between genuine and printed patches.
Figure 11: Additional hyperspectral reconstruction results with Lumosaic on scenes exhibiting diverse motion characteristics. (top) A figurine with printed elements, including Ishihara patterns and text, undergoing both rotational and falling motion relative to the camera. (bottom) Dynamic hand gestures with non-rigid motion. The top row for each scene shows rendered sRGB views from non-consecutive frames of the reconstructed hyperspectral video, while the bottom row shows the corresponding full 31-channel hyperspectral images for the frames outlined in yellow.
Figure 12: Additional hyperspectral reconstruction results with Lumosaic on scenes exhibiting diverse motion characteristics. (top) A transparent plastic cup filled with carbonated tonic water, capturing high-frequency motion due to effervescence. (bottom) A complex indoor scene with colorful objects captured while freely panning the imaging system to introduce continuous viewpoint and illumination changes. The top row for each scene shows rendered sRGB views sampled from non-consecutive frames of the reconstructed hyperspectral video, while the bottom row shows the corresponding full 31-channel hyperspectral images for the frames outlined in red.
Figure 13: The effect of temporal alignment on Lumosaic’s reconstruction quality in dynamic scenes. We compare results on two motion regimes— (top) slow and (bottom) fast—showing hyperspectral reconstructions rendered in sRGB with and without temporal alignment.
Recently, Spectral Compressive Imaging (SCI) has achieved remarkable success, unlocking significant potential for dynamic spectral vision. However, existing reconstruction methods, primarily image-based, suffer from two limitations: (i) Encoding process masks spatial-spectral features, leading to uncertainty in reconstructing missing information from single compressed measurements, and (ii) The frame-by-frame reconstruction paradigm fails to ensure temporal consistency, which is crucial in the video perception. To address these challenges, this paper seeks to advance spectral reconstruction from the image level to the video level, leveraging the complementary features and temporal continuity across adjacent frames in dynamic scenes. Initially, we construct the first high-quality dynamic hyperspectral image dataset (DynaSpec), comprising 30 sequences obtained through frame-scanning acquisition. Subsequently, we propose the Propagation-Guided Spectral Video Reconstruction Transformer (PG-SVRT), which employs a spatial-then-temporal attention to effectively reconstruct spectral features from abundant video information, while using a bridged token to reduce computational complexity. Finally, we conduct simulation experiments to assess the performance of four SCI systems, and construct a DD-CASSI prototype for real-world data collection and benchmarking. Extensive experiments demonstrate that PG-SVRT achieves superior performance in reconstruction quality, spectral fidelity, and temporal consistency, while maintaining minimal FLOPs. Project page: https://github.com/nju-cite/DynaSpec
Snapshot hyperspectral imaging avoids sequential scanning, but systems that jointly achieve stable reconstruction, low cost, and compact optics remain limited. We present a snapshot hyperspectral imaging system based on angular-to-spectral diversity conversion. A tapered kaleidoscope creates replicated views with distinct incidence directions, and a directly attached birefringent filter converts them into view-channel-dependent spectral transmittances, yielding complementary measurements that better condition the inverse problem for more stable single-shot spectral reconstruction. The system preserves a simple pixel-wise linear model for fast non-learning-based reconstruction and uses only off-the-shelf components without relay optics or cascaded modules. We select the birefringent filter configuration using a condition-number-based criterion and validate the system on both synthetic and real data.
The task of capturing and rendering 3D dynamic scenes from 2D images has become increasingly popular in recent years. However, most conventional cameras are bandwidth-limited to 30-60 FPS, restricting these methods to static or slowly evolving scenes. While overcoming bandwidth limitations is difficult for general scenes, recent years have seen a flurry of computational imaging methods that yield high-speed videos using conventional cameras for specific applications (e.g., motion capture and particle image velocimetry). However, most of these methods require modifications to a camera's optics or the addition of mechanically moving components, limiting them to a single-view high-speed capture. Consequently, these methods cannot be readily used to capture a 3D representation of rapid scene motion. In this paper, we propose a novel method to capture and reconstruct a volumetric representation of a high-speed scene using only unaugmented low-speed cameras. Instead of modifying the hardware or optics of each individual camera, we encode high-speed scene dynamics by illuminating the scene with a rapid, sequential color-coded sequence. This results in simultaneous multi-view capture of the scene, where high-speed temporal information is encoded in the spatial intensity and color variations of the captured images. To construct a high-speed volumetric representation of the dynamic scene, we develop a novel dynamic Gaussian Splatting-based approach that decodes the temporal information from the images. We evaluate our approach on simulated scenes and real-world experiments using a multi-camera imaging setup, showing first-of-a-kind high-speed volumetric scene reconstructions.