Cyclops: LiDAR as a Camera That Dreams in Color

Wei Gao; Jian Shu; Mingle Zhao; Maani Ghaffari; David Kong; Chengzhong Xu; Hui Kong

cs.ROAug 17, 2026

Cyclops: LiDAR as a Camera That Dreams in Color

Authors: Wei Gao, Jian Shu, Mingle Zhao, Maani Ghaffari, David Kong, Chengzhong Xu, Hui Kong

Organizations: State Key Laboratory of Internet of Things for Smart City, Faculty of Science and Technology, University of Macau, Macau, China · The Hong Kong University of Science and Technology (Guangzhou), China · Department of Naval Architecture and Marine Engineering and Department of Robotics, University of Michigan, Ann Arbor, MI, USA · Independent Researcher, Pittsburgh, PA, USA

Abstract

Conventionally, robotic perception relies heavily on cameras due to the rich semantic texture they provide. However, their performance degrades significantly in low-light or high-dynamic-range environments. Conversely, while Light Detection and Ranging (LiDAR) captures illumination-invariant geometric and intensity properties, the resulting data are typically single-channel and sparse, creating a significant modality gap when applying vision models pre-trained on RGB datasets. In this paper, we propose Cyclops, a framework that translates sparse Non-Repetitive Scanning LiDAR (NRS-LiDAR) intensity into RGB video, enabling camera-free inference for all-day perception tasks. Our approach first converts sparse LiDAR intensity projections into dense representations via a frozen pre-trained densification module, serving as a geometrically rich source condition. The dense intensity latent is then transported toward the target RGB distribution through Latent Bridge Matching (LBM) with a learned velocity field in a few ODE integration steps. To mitigate inter-frame flickering, we inject prior-frame context via temporal attention layers and further formulate the velocity field as a policy optimized by a differentiable terminal reward that encourages terminal fidelity through backpropagation along the ODE trajectory. Extensive experiments demonstrate that the synthesized RGB, including those generated under near-dark conditions, enable standard RGB-based perception models to substantially outperform both LiDAR baselines and conventional cameras on semantic segmentation, lane detection, and point cloud colorization across diverse lighting conditions.

Figures & tables

Figure 1: Our proposed Cyclops converts sparse NRS-LiDAR intensity into RGB imagery through a two-stage pipeline: densification (Stage I) followed by colorization (Stage II). Bottom-left: Generated images maintain consistent visual quality across diverse lighting conditions, whereas camera images degrade severely. Bottom-right: The synthesized RGB directly enables downstream tasks, including semantic segmentation, lane detection, visual navigation, and point cloud colorization.

Figure 2: Overview of our proposed framework. (a) Stage I densifies LiDAR intensity image via densification network with AFM and DCM. (b) Stage II transports the dense intensity image latent toward an RGB latent via a 4-step ODE governed by velocity policy $v_{\theta}$ , with temporal attention for inter-frame conditioning and a differentiable reward $\mathcal{R}$ for end-to-end trajectory optimization.

Figure 3: NRS-LiDAR scan accumulation and sparsity challenge. (a) Point cloud progressively accumulates. (b) Accumulated intensity map projected onto the camera viewpoint. (c) LiDAR points projected onto the image plane and colorized with corresponding camera RGB values. (d) Reference camera image. Comparing (c) and (d) reveals significant texture loss and large spatial gaps in short-duration observations.

Method	Image Translation Quality				Temporal Consistency			Time (s) $\downarrow$
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	MFD $\downarrow$	FC-MSE $\downarrow$	FV $\downarrow$	Time (s) $\downarrow$
CycleGAN [ 69 ]	14.25	0.452	0.385	138.5	38.56	0.0478	22.35	0.028
Pix2Pix [ 25 ]	16.34	0.526	0.318	102.3	31.82	0.0385	17.62	0.022
BBDM [ 27 ]	20.12	0.625	0.238	68.5	26.48	0.0312	14.56	2.75
LDM [ 39 ]	19.58	0.608	0.252	74.2	33.25	0.0415	18.92	3.28
CtrLoRA [ 60 ]	17.47	0.553	0.307	90.7	41.44	0.0521	27.01	13.6

Table 1: Quantitative comparison of image translation quality and temporal consistency. The upper block lists baseline methods; the lower block presents our ablation study.

Figure 4: Qualitative comparison. The top row shows baseline results; the bottom row presents ablation variants. Each group includes the full image at frame $N$ and zoomed-in patches from frames $N{-}2$ to $N{+}2$ to visualize temporal consistency. Our full model produces the most realistic colors with faithful geometry and strong temporal coherence.

Method	Input	Bright			Low Light			Near Dark
Method	Input	mIoU	mAcc	aAcc	mIoU	mAcc	aAcc	mIoU	mAcc	aAcc
Patchwork++ † [ 26 ]	Point Cloud	44.2	53.5	73.6	44.0	53.2	73.4	43.6	52.8	73.1
GroundGrid † [ 46 ]	Point Cloud	47.5	56.8	75.8	47.2	56.5	75.5	46.8	56.1	75.2
SAM 2 [ 38 ]	Densified Intensity	35.2	43.8	67.2	35.0	43.5	67.0	34.7	43.2	66.7
	Camera	76.8	84.5	92.8	63.5	72.2	86.8	31.2	39.8	63.2
	Cyclops Rendering	65.8	75.2	86.2	65.2	74.5	85.6	64.5	73.8	85.1

Table 2: Semantic segmentation under varying illumination. SAM 2 is evaluated in video-predictor mode using identical point prompts every five frames for all image modalities. † denotes binary ground/non-ground segmentation methods whose outputs are mapped to the evaluation labels as described in Appendix B.3 .

Figure 5: Qualitative results of downstream tasks. (a) Left: ground segmentation from Patchwork++ and GroundGrid projected onto the image plane. Right: semantic segmentation across different inputs–camera-based results degrade under poor lighting, whereas our colorized images maintain consistent quality. (b) Lane detection on Cyclops-generated RGB images under challenging lighting conditions. (c) Visual navigation using pretrained ViNT, which directly takes Cyclops-generated RGB images as input to predict waypoint trajectories. (d) Compared with grayscale point clouds, our colorized point clouds exhibit richer texture details.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative results of training-free cross-LiDAR adaptation for Stage I. From top to bottom: input sparse point clouds, sparse intensity projections after alignment with the Livox MID-360 input domain, and dense intensity images produced by the frozen densification network. The same Stage-I model is applied to multiple LiDARs with different scan patterns and resolutions without sensor-specific retraining or finetuning.

Figure 7: Data collection platform. A ground robot equipped with a Livox MID-360 LiDAR and a RealSense D435i RGB-D camera. The two sensors are rigidly mounted and extrinsically calibrated.

Condition	Sequences	Frames	Train / Val / Test
Bright	34	24,983	18,721 / 3,308 / 2,954
Low Light	4	1,504	— / — / 1,504
Near Dark	5	2,008	— / — / 2,008
Total	43	28,495	18,721 / 3,308 / 6,466

Appendix

Table 3: Dataset composition by lighting condition and split. Training and validation use only Bright-condition data; Low Light and Near Dark data appear exclusively in the test set.

Figure 8: SAM 2 mask propagation using identical prompts. The first column shows the prompted keyframe, followed by masks propagated to subsequent unprompted frames. Cyclops-rendered RGB produces more complete and temporally stable masks than densified intensity input.

Figure 9: Comparison with projection-based LiDAR segmentation baselines. (a–b) Spherical range-view and camera-frustum projections of regular spinning-LiDAR scans (top) and Livox MID-360 scans (bottom). (c–e) Semantic segmentation results from the Cyclops-based pipeline, FRNet, and RangeViT, respectively.

Condition	Input	Acc.(%) $\uparrow$	F1(%) $\uparrow$	FPR(%) $\downarrow$	FNR(%) $\downarrow$
Bright	Camera	96.5	96.2	3.1	3.5
	Densified Intensity Image	96.1	95.2	5.2	5.5
	Ours (colorized)	95.8	95.4	3.8	4.2
Near Dark	Camera	71.2	68.5	16.3	19.8
	Densified Intensity Image	93.3	92.5	6.9	6.9
	Ours (colorized)	95.2	94.7	4.2	4.8

Appendix

Table 4: Traffic lane detection results under different lighting conditions.

Figure 10: Qualitative comparison under different numbers of function evaluations (NFE). The model is trained exclusively with $M=4$ . From left to right: the ground-truth camera image and outputs with $M\in\{1,2,4\}$ .

Figure 11: An example of consecutive frames generated by our method on the GNSS-denial02 sequence from the public M3DGR dataset [ 62 ] .

Figure 12: Examples of predicted RGB generated by our method on a low-light sequence.

Figure 13: Examples demonstrating that Cyclops produces stable geometry-conditioned RGB renderings across varying illumination conditions, enabling robust all-day visual perception.

Conditional Image Generation

Conditional image generation creates images guided by inputs such as text, sketches, layouts, masks, reference images, or physical measurements.

See all 160 papers

NIDAR: NIR-Guided Intrinsic Decomposition for Scalable Scene-Agnostic LiDAR Intensity Reconstruction
Sep 29, 2026Junjie Zhang, Jie Yin, Kefei Qian +5
Chongqing University · Shanghai Jiao Tong University · Ministry of Industry and Information Technology +1
LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance
Oct 14, 2026Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang +10Point Cloud Generation Diffusion Model Guidance
University of Science and Technology of China · Shanghai Innovation Institute · Fudan University +1
Physics-Aware Diffusion for LiDAR Point Cloud Densification
Mar 23, 2026Zeping Zhang, Robert Laganière3D Diffusion Models Physics-Informed ML
VIVA Research Lab, Department of Electrical and Computer Engineering, University of Ottawa, Ottawa, Ontario, Canada

Sep 29, 2026J/K move · Enter open · S save

NIDAR: NIR-Guided Intrinsic Decomposition for Scalable Scene-Agnostic LiDAR Intensity Reconstruction

Junjie Zhang, Jie Yin, Kefei Qian, Jie Li+4

Chongqing University · Shanghai Jiao Tong University · Ministry of Industry and Information Technology+1

Open paper AskPDF arXiv

LiDAR return intensity provides complementary surface-response cues for robotic perception and state estimation, yet many simulation pipelines omit it or reproduce it using reconstruction methods that require real intensity supervision and per-scene optimization. These requirements increase data-collection and fitting costs and limit reuse across simulated scenes. We present NIDAR, a feed-forward framework that synthesizes dense intensity-like observations from RGB appearance and simulator geometry. NIDAR combines pretrained pseudo-NIR translation, hierarchical intrinsic decomposition, geometry-aware modulation, and source-domain distribution calibration to transfer reflectance-related image cues to simulated point clouds. Its learned components are trained offline using Waymo data; their weights and calibration remain fixed during evaluation on Waymo and nuScenes. Deployment therefore requires neither target-scene intensity labels nor target-scene gradient-based fitting. The reported comparisons show competitive pixel-wise accuracy and favorable structural and perceptual fidelity against the evaluated reconstruction baselines. A controlled pseudo-NIR-versus-RGB diagnostic further shows that the pseudo-NIR prior is most beneficial when used through the paper-aligned reflectance-and-remapping route, rather than as a simple direct intensity regressor. We further integrate NIDAR with Unreal Engine 5, Isaac Sim, and a generative LiDAR pipeline. Two intensity-aware SLAM systems evaluated in two simulated indoor scenes suggest potential downstream utility, but do not constitute real-robot validation. NIDAR therefore offers a scalable intensity-synthesis interface for the evaluated settings; cross-wavelength, camera-configuration, embedded, and real-sensor validation remain future work.

Abstract

Conditional Image Generation

Explore similar work

NIDAR: NIR-Guided Intrinsic Decomposition for Scalable Scene-Agnostic LiDAR Intensity Reconstruction

LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance

Physics-Aware Diffusion for LiDAR Point Cloud Densification