eess.IVFeb 25, 2026
SaveLumosaic: Hyperspectral Video via Active Illumination and Coded-Exposure Pixels
Organizations: Department of Computer Science, University of Toronto, Canada · Department of Electrical & Computer Engineering, University of Toronto, Canada
Abstract
We present Lumosaic, a compact active hyperspectral video system designed for real-time capture of dynamic scenes. Our approach combines a narrowband LED array with a coded-exposure-pixel (CEP) camera capable of high-speed, per-pixel exposure control, enabling joint encoding of scene information across space, time, and wavelength within each video frame. Unlike passive snapshot systems that divide light across multiple spectral channels simultaneously and assume no motion during a frame's exposure, Lumosaic actively synchronizes illumination and pixel-wise exposure, improving photon utilization and preserving spectral fidelity under motion. A learning-based reconstruction pipeline then recovers 31-channel hyperspectral (400-700 nm) video at 30 fps and VGA resolution, producing temporally coherent and spectrally accurate reconstructions. Experiments on synthetic and real data demonstrate that Lumosaic significantly improves reconstruction fidelity and temporal stability over existing snapshot hyperspectral imaging systems, enabling robust hyperspectral video across diverse materials and motion conditions.
Figures & tables
Figure 2: The operation of a single pixel in a coded-exposure-pixel (CEP) camera. Each pixel alternates between two charge storage sites (Bucket 0 and Bucket 1) according to a binary exposure code. During each sub-frame, only one bucket integrates incident light, enabling temporal multiplexing across the exposure period.
Figure 3: (A) Lumosaic features a CEP camera synchronized with a programmable array of narrowband LEDs. (B) Each sub-frame is assigned a unique LED and spatial exposure mask, producing a dense spatio-spectro-temporal scene encoding within a single video frame.
Figure 4: Reconstruction performance of QDO [ 20 ] , MST++ [ 7 ] , and Lumosaic using HAN [ 27 ] , MCAN [ 9 ] , and SRNet [ 5 ] backbones according to (top-left) PSNR, (top-right) MAE, (bottom-left) SSIM, and (bottom-right) SAM for the simulations under varying Gaussian noise levels.
Figure 5: Quantitative and qualitative evaluation of spectral reconstruction accuracy using a ColorChecker target (real data). (A) Raw coded frame captured by Lumosaic. (B) Zoomed-in region showing the mosaic coding. (C) Reconstructed hyperspectral image rendered in sRGB. (D) Spectral reflectance curves for all 24 patches, comparing ground-truth (solid blue) and reconstructed (dashed yellow) spectra.
Figure 6: An example of real-world reconstruction results obtained using Lumosaic. (A) Reconstructed hyperspectral image rendered as sRGB. (B) Reflectance spectra extracted from marked regions. (C) Reconstructed hyperspectral channels.
Figure 7: Hyperspectral video reconstruction of a dynamic scene: a colored droplet diffusing in water, captured using Lumosaic at 30 fps. Each column shows a representative frame over time rendered in sRGB, while the bottom row visualizes reconstructed spectral channels for the highlighted frame. See the Supplementary Video for the full sequence visualization.
Figure 2: The hardware schematic of Lumosaic’s active illumination module. The ESP32 microcontroller coordinates the LED driver array and issues synchronization pulses to the CEP camera.
| LED Name | Relative Allocation | Time per Frame (µs) |
|---|---|---|
| UV | 5.70% | 1,350 |
| Violet | 3.16% | 750 |
| Royal Blue | 3.16% | 750 |
| Blue | 3.16% | 750 |
| Cyan | 5.70% | 1,350 |
| Green | 6.96% | 1,650 |
Table 2: The LED exposure allocation per video frame compensates for relative intensity variations across LEDs and camera spectral sensitivity. Each LED’s sub-frame activations occur contiguously within the frame.
Figure 3: The mosaic tile that forms the basis of Lumosaic’s spatial-spectral coding.
Figure 4: (top) LED spectral power distributions before and after calibration via optimized scaling factors . (bottom) Camera spectral sensitivity and incandescent bulb spectral power distribution (SPD).
Figure 6: A visualization of the spatial weighting kernel used for merging overlapping patch predictions.
Figure 7: Qualitative comparisons on public datasets. Representative reconstructions from the test set comparing QDO [ 20 ] , MST++ [ 7 ] , and Lumosaic. Each column shows the reconstructed sRGB image, selected hyperspectral channels, and zoomed-in sRGB crops under varying Gaussian noise levels (annotated with the corresponding standard deviation). We also present spectral validation plots for two specific locations, marked with white stars in the first-row sRGB images, displayed at the bottom of the figure.
Figure 8: Spectral recovery test on a high-frequency rainbow scene. ( Left ) sRGB visualization of the rainbow scene: (A) ground truth and reconstructions from (B) QDO [ 20 ] , (C) MST++ [ 7 ] , and (D) Lumosaic. Rows (D-F) illustrate Lumosaic results under increasing proportions of high-frequency synthetic spectra in the training set (0%, 33%, 66%, 100%). ( Right ) Reconstructed spectral reflectance, interpolated at 1-nm intervals, and center wavelength positions at selected locations on the rainbow (marked by triangles), compared to ground truth, demonstrating Lumosaic’s improved ability to recover sharp spectral transitions.
Figure 9: Additional Lumosaic reconstruction results on diverse static scenes: optical filters, a globe, a butterfly specimen, and a figurine with printed matter. Each scene includes (A) the input coded image, (B) the reconstructed hyperspectral image rendered as sRGB, and (C) the 31-channel hyperspectral composites rendered in grayscale for visualization.
Figure 10: Metamerism analysis with Lumosaic using genuine and printed ColorChecker targets. ( A and B ) RGB visualization of the genuine ColorChecker and a printed photocopy under identical illumination. ( C ) Spectral curves illustrating the differences between genuine and printed patches.
Figure 11: Additional hyperspectral reconstruction results with Lumosaic on scenes exhibiting diverse motion characteristics. (top) A figurine with printed elements, including Ishihara patterns and text, undergoing both rotational and falling motion relative to the camera. (bottom) Dynamic hand gestures with non-rigid motion. The top row for each scene shows rendered sRGB views from non-consecutive frames of the reconstructed hyperspectral video, while the bottom row shows the corresponding full 31-channel hyperspectral images for the frames outlined in yellow.
Figure 12: Additional hyperspectral reconstruction results with Lumosaic on scenes exhibiting diverse motion characteristics. (top) A transparent plastic cup filled with carbonated tonic water, capturing high-frequency motion due to effervescence. (bottom) A complex indoor scene with colorful objects captured while freely panning the imaging system to introduce continuous viewpoint and illumination changes. The top row for each scene shows rendered sRGB views sampled from non-consecutive frames of the reconstructed hyperspectral video, while the bottom row shows the corresponding full 31-channel hyperspectral images for the frames outlined in red.
Figure 13: The effect of temporal alignment on Lumosaic’s reconstruction quality in dynamic scenes. We compare results on two motion regimes— (top) slow and (bottom) fast—showing hyperspectral reconstructions rendered in sRGB with and without temporal alignment.