Paleoinspired Vision: From Exploring Colour Vision Evolution to Inspiring Camera Design
Authors: Yijie Lu, Zhimin Zong, Junjie Zhang, Shenghan Su, Zeyun Zhao, Ziteng Cui, Yan Pu, Jing Lu, Daisuke Kojima, Lin Gu, Ruogu Fang
Organizations: The University of Florida · The University of Tokyo · Institute of Vertebrate Paleontology and Paleoanthropology · RIKEN AIP, The University of Tokyo
The evolution of colour vision is captivating, as it reveals the adaptive strategies of extinct species while simultaneously inspiring innovations in modern imaging technology. In this study, we present a simplified model of visual transduction in the retina, introducing a novel opsin layer. We quantify evolutionary pressures by measuring machine vision recognition accuracy on colour images shaped by specific opsins. Building on this, we develop an evolutionary conservation optimisation algorithm to reconstruct the spectral sensitivity of opsins, enabling mutation-driven adaptations to more effectively spot fruits or predators. This model condenses millions of years of evolution into seconds using GPU, providing an experimental framework to test long-standing hypotheses in evolutionary biology, such as vision of early mammals, primate trichromacy from gene duplication, retention of colour blindness, blue-shift of fish rod and multiple rod opsins with bioluminescence. Beyond validating existing paleontological hypotheses, our model further enables speculative explorations of hypothetical species, such as organisms with visual systems adapted to Martian conditions. These constitute our contributions to AI for life science and neuroscience. Besides, the principles derived from these biological studies can translate into practical engineering solutions. In practice, our findings suggest a minimalist yet effective approach to task-specific camera filter design, optimising the spectral response function to meet application-driven demands. Code repository: https://github.com/luyijie-research/PaleVision.
Figures & tables
Figure 1: Our framework reconstructs the 518-million-year evolutionary trajectory of color vision across the vertebrate-to-primate lineage and evaluates corresponding explanatory hypotheses. This includes the transition from ancestral tetrachromacy through mammalian gene loss (dichromatic regression) to primate gene duplication (trichromatic progression). Furthermore, the framework quantifies the dim-light advantage hypothesis, deep-sea opsin blueshifts, and predictive visual adaptations and sensor designs for novel environments, such as the Martian landscape.
Figure 2: Computational framework for Evolutionary Conservation Optimization. The pipeline translates ecological pressures into semantic segmentation tasks, utilizing an opsin layer to bridge low-level spectral perception with high-level decision-making. By incorporating conservative regularization—parameterizing kernels as Gaussian functions with constrained λmax shifts, the framework simulates the evolution and derive task-adaptive spectral response functions for specialized camera designs in Martian and medical applications.
Figure 3Table 4
Lighting
Normal
G.-blind
R.-blind
G.-weak
R.-weak
Bright
0.208
0.316
0.333
0.254
0.237
Dark (0.10)
0.309
0.391
0.365
0.293
0.278
Dark (0.05)
0.355
0.420
0.397
0.324
0.324
Table 3: Fruit recognition ( SR ) across lighting and colour vision types. Lower SR indicates better camouflage-breaking ability.
Figure 5: Depth-dependent blue-shift in squirrelfish rod opsins.
Depth
λmax
mIoU
(nm)
(%)
0 m
496.19
30.38
10 m
488.73
25.88
50 m
484.69
16.97
70 m
481.10
14.98
Table 4: Blue-shift of λmax vs. depth.
Dataset
Methods
Param (M)
λ1 (nm)
λ2 (nm)
λ3 (nm)
mIoU (%)
Baselines (MHSI / MODID)
MHSI/MODID
UNet (2015) Ronneberger et al. (2015)
31.07
–
–
–
51.04 / 54.02
SwinUNet (2022) Cao et al. (2022)
41.48
–
–
–
47.84 / 51.12
ResUNet (2020) Diakogiannis et al. (2020)
13.11
–
–
–
49.52 / 52.76
Segformer (2021) Xie et al. (2021a)
64.47
–
–
–
50.27 / 52.98
EMCAD (2024) Rahman et al. (2024)
84.53
–
–
–
50.07 / 52.67
Table 5: Camera design for bile duct and oral cancer detection (MHSI and MODID datasets).
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Architecture of the network.
Figure 7: Training stage with Opsin Layer, Transformer Encoder and Segmentation Decoder.
Figure 8: Environment simulation with and without bioluminescence.
Table 12
Epoch
λ1max
λ2max
λ3max
λ4max
λ5max
mIoU %
0
481.00
481.00
481.00
481.00
481.00
-
50
475.41
475.41
475.26
475.56
475.29
8.44
100
472.89
472.89
472.74
473.04
472.76
12.16
150
470.40
470.38
470.24
470.56
470.26
13.02
200
467.76
467.73
467.59
467.92
467.62
13.77
Appendix
Table 9: 5-kernel opsin layer without bioluminescence.
Epoch
λ1max
λ2max
λ3max
λ4max
λ5max
mIoU %
0
481.00
481.00
481.00
481.00
481.00
-
50
481.78
482.40
480.63
481.46
482.12
10.20
100
481.60
482.53
479.68
481.12
481.76
13.17
150
481.37
482.48
478.96
480.74
481.40
17.67
200
481.73
482.96
478.98
480.98
481.69
21.26
Appendix
Table 10: 5-kernel opsin layer with bioluminescence.
Table 15
Epoch
λ1max
λ2max
λ3max
λ4max
mIoU
0
647.87
647.87
582.86
435.80
-
100
646.60
644.15
577.73
434.19
18.62
200
647.90
643.76
576.71
434.22
26.45
300
648.76
643.28
575.91
434.24
29.72
400
649.90
643.35
575.27
434.32
30.44
500
651.01
643.49
574.80
434.37
34.62
Appendix
Table 13: Tetrachromatic Vision on Mars.
Filters
λ1max
λ2max
λ3max
λ4max
mIoU %
R
613.31
-
-
-
36.26
R/B
624.22
459.67
-
-
40.78
R/G
622.91
517.07
-
-
40.61
R/R/B
617.27
615.00
457.84
-
40.37
R/R/B/B
629.36
623.09
457.93
454.86
41.60
Appendix
Table 14: Camera Specialised for Martian Environment.
Filters
λ1max
λ2max
λ3max
mIoU %
R
612.95
-
-
54.22
R/B
620.43
477.91
-
56.97
R/G
623.44
538.26
-
56.64
R/R/B
621.11
619.56
472.21
54.53
Appendix
Table 15: Camera Specialised for Cancer Detection.
Camera
λ1max
λ2max
λ3max
Iou for cancer %
Standard
600.00
540.00
460.00
52.33
Design
610.39
555.09
456.02
54.97
Appendix
Table 16: Train With U-Net Model.
Encoder
Opsin
λ3max
λ2max
λ1max
mIoU (%)
SR
(nm)
MiT
General
425.00
560.00
560.00
47.29
0.276
Design
440.65
547.34
582.10
47.99
0.239
PVT-v2
General
425.00
560.00
560.00
49.21
0.265
Design
430.33
542.15
578.22
50.67
0.222
ConvNeXt
General
425.00
560.00
560.00
46.89
0.270
Appendix
Table 17: Dichromacy-to-trichromacy reconstruction across segmentation encoders on MinneApple. Lower SR is better.
Task
Opsin
λ3max
λ2max
λ1max
Task-specific performance
(nm)
Segmentation
General
425.00
560.00
560.00
mIoU =47.29% , SR=0.276
Design
440.65
547.34
582.10
mIoU =47.99% , SR=0.239
Detection
General
425.00
560.00
560.00
AP@50/75 =0.592/0.411 , mAP =0.398 , F1 =0.6805
Design
439.22
545.24
581.45
AP@50/75 =0.617/0.435 , mAP =0.409 , F1 =0.7021
Counting
General
425.00
560.00
560.00
MAE/RMSE =13.1/18.3
Appendix
Table 18: Dichromacy-to-trichromacy reconstruction across downstream tasks on MinneApple.
Encoder
Opsin
MODID mIoU
MHSI mIoU
MiT
General
0.5642
0.5435
Design
0.5867
0.5636
PVT-v2
General
0.5924
0.5670
Design
0.6057
0.5767
ConvNeXt
General
0.5350
0.5186
Design
0.5403
0.5250
Appendix
Table 19: Camera-design robustness across segmentation encoders.
Color constancy is a fundamental ability of many biological visual systems and a crucial step in computer imaging systems. Bio-inspired modeling offers a promising way to elucidate the computational principles underlying color constancy and to develop efficient computational methods. However, bio-inspired methods for color constancy remain underexplored and lack a comprehensive analysis. This paper presents a comprehensive technical framework that integrates biological mechanisms, computational theory, and algorithmic implementation for bio-inspired color constancy. Specifically, we systematically revisit the computational theory of biological color constancy, which shows that illuminant estimation can be reduced to the task of gray-anchor (pixel or surface) detection in early vision. Subsequently, typical gray-pixel detection methods, including Gray-Pixel and Grayness-Index, are reinterpreted within a unified theoretical framework with the Lambertian reflection model and biological color-opponent mechanisms. Finally, we propose a simple learning-based method that couples reflection-model constraints with feature learning to explore the potential of bio-inspired color constancy based on gray-pixel detection. Extensive experiments confirm the effectiveness of gray-pixel detection for color constancy and demonstrate the potential of bio-inspired methods.
Kai-Fu Yang, Fu-Ya Luo, Yong-Jie Li
MOE Key Laboratory for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 610054, China · School of Electronic Engineering and Automation, Guilin University of Electronic Technology, Guilin, Guangxi 541004, China
Standard dynamic vision sensors approximate retinal processing by detecting temporal contrast changes, offering high speed and high dynamic range. In this work, we explore whether incorporating additional biologically inspired processing stages - specifically spatial filtering and gain control - can offer advantages for certain downstream tasks such as saliency prediction. We present the first implementation of a multi-stage Silicon Retina model on the SCAMP-5 Pixel Processor Array, along with a GPU-based simulation framework. We evaluate the performance of our model on Video Intensity Reconstruction and Video Saliency Prediction. While the bio-inspired model is less effective at reconstructing absolute intensity frames, it achieves a 13% reduction in saliency prediction loss in comparison to standard DVS event representation, while reducing the event rate by approximately 47%. These experiments are obtained using a lightweight ≈100k-parameter FireNet-style network, adapted from event-based reconstruction to saliency prediction. These results suggest that the silicon retina's "information distillation" mechanism can achieve a more efficient representation for downstream neural networks, particularly in bandwidth-constrained edge applications.
Maciej Lewandowski, Prince Philip, Alexandre Marcireau +3
Department of Electrical and Electronic Engineering University of Manchester, UK · Department of Electronic Systems Engineering, Indian Institute of Science, Bangalore, India · Department of Computer Science University of Manchester, UK
Small tabular datasets with expert-designed spectral features are the norm in operational Earth observation, and the networks applied to them are typically hand-designed. We revisit one such published model -- a Sentinel-2 algal bloom classifier -- and ask what architecture search adds, holding the task, the features and the lake-level train/test split of the original study fixed. Searching an extended multilayer-perceptron space with regularized evolution, and selecting on inner-cross-validation AUC only, we find networks that improve held-out AUC from 0.790 to 0.820 and accuracy from 0.733 to 0.748 while using 409 trainable parameters, 26 times fewer than the strongest hand-designed reference. The search converges on a consistent recipe -- a single narrow layer, RMS normalisation, tanh activation, step-decayed RMSprop and weight averaging -- that a practitioner would be unlikely to reach by default. At 1.6,kB the resulting model is small enough to serve as an onboard screening trigger, which is the setting that motivates the work. Code: https://github.com/VU-AIML/automl4eo-bloom-nas.
Kursat Komurcu, Linas Petkevicius
Vilnius University, Institute of Computer Science, Artificial Intelligence Methods Lab · IRISA, Universite Bretagne Sud · European Commission Joint Research Center