Turning spectra into images improves plant trait retrieval with 2D-CNNs

Javier Lopatin; Teja Kattenborn; Eya Cherif; Sebastián Moreno

cs.CVAug 17, 2026

Turning spectra into images improves plant trait retrieval with 2D-CNNs

Authors: Javier Lopatin, Teja Kattenborn, Eya Cherif, Sebastián Moreno

Organizations: Data Observatory Foundation, ANID Technology Center No. DO210001, Santiago, Chile, 7510277 · Center for Climate Resilience Research (CR)2, University of Chile, Santiago, Chile, 8370449 · Sensor-based Geoinformatics (geosense), University of Freiburg, Freiburg, Breisgau, Germany · Institute for Earth System Science and Remote Sensing, Leipzig University, Germany · Faculty of Engineering and Science, Adolfo Ibáñez University, Av. Diagonal Las Torres 2460, Santiago, Chile, 7941169

Abstract

Hyperspectral reflectance spectroscopy enables non-destructive estimation of plant functional traits, yet current deep learning approaches process spectra as one-dimensional sequences, which limits how they capture long-range inter-band dependencies. We asked whether transforming 1D spectra into 2D image representations improves multi-trait prediction with convolutional neural networks (CNN). We compared nine transformations using EfficientNet-B0 on the GreenHyperSpectra dataset (7,897 labeled spectra, eight traits, 400-2450 nm), benchmarked against published 1D CNN results on the same split. Trained from scratch, the simplest transformation, a direct Reshape of the spectrum into a 2D grid, performed best ( $R^2 = 0.684 \pm 0.001$ ) and improved on the state-of-the-art 1D baseline ( $R^2 = 0.587$ , $+0.097$ ). We then pretrained a 2D masked autoencoder (MAE-2D) on 139,000 unlabeled spectral images. Linear probing, which freezes the encoder and trains only a multilayer perceptron head, reached $R^2 = 0.646$ and exceeded every 1D self-supervised counterpart, including the fine-tuned MAE-1D ( $R^2 = 0.641$ ). Under cross-dataset evaluation all models lost most of their accuracy and none beat the 1D baseline significantly. To identify which wavelengths drive each prediction, we applied Integrated Gradients and Grad-CAM and unfolded band importance back to the spectral axis. Protein ( $r = 0.45$ ) and leaf water ( $r = 0.33$ ) agreed with sensitivities simulated by the PROSAIL radiative-transfer model, while carotenoids ( $r = 0.06$ ) and leaf area index ( $r = -0.11$ ) did not, showing that the model reads established leaf chemistry for traits with sharp absorption features. The representational advantage of 2D spectral images, rather than architectural complexity or ImageNet pretraining, drives the gain over 1D approaches.

Figures & tables

Figure 1: Illustration of seven of the nine 1D-to-2D spectral transformations applied to three representative spectra with contrasting trait profiles. The Serpentine and Hilbert-curve variants are re-layouts of the same Reshape values and are not shown separately. (a) Original reflectance spectra: Spectrum 1 exhibits high chlorophyll content and high leaf area index (LAI); strong visible absorption and a near-infrared plateau. Spectrum 2 shows low chlorophyll and high leaf mass per area (reduced visible absorption, prominent dry matter features), and Spectrum 3 displays high anthocyanin content (elevated green-edge reflectance). (b) Corresponding 2D image representations produced by each transformation method. Each transformation encodes different aspects of the spectral information: Reshape preserves wavelength adjacency as spatial texture; Continuous wavelet transform (CWT) reveals multi-scale absorption features; Spectrogram captures localized frequency content through three complementary windows; 2D-COS highlights inter-band correlations; Gramian Angular Fields (GAF) encodes angular relationships between spectral values; Markov Transition Fields (MTF) represent transition dynamics between discretized intensity bins; and the all-band normalized-difference matrix (NDI) encodes every pair of bands as a normalized-difference index. Reflectance is not available between 1351–1430 and 1801–2050 nm, where the water absorption regions were excluded from the dataset. The curves are drawn continuously across those two intervals, so the straight segments there are interpolation, not measured signal.

Figure 2: Spectral importance pipeline, illustrated for chlorophyll (C ab ). A 1D spectrum is folded into a 2D Reshape image and fed to the EfficientNet-B0 model. Grad-CAM and Integrated Gradients then attribute the prediction on that image independently of each other, so neither method takes the other as input. Grad-CAM stays in two dimensions, because it operates on the $7\times 7$ convolutional embedding and localizes regions rather than individual bands. The Integrated Gradients map is unfolded back to the 1721-band wavelength axis using the band-order-preserving property of the Reshape transform. In panel (d) the green shading marks the PROSAIL theoretical sensitivity and the grey bands mark the removed water-absorption ranges.

Transform	Channels	Mean $R^{2}$ $\pm$ std	$\Delta$ vs. 1D
Reshape	1	0.684 $\pm$ 0.001	+0.097 ∗
Serpentine	1	0.675 $\pm$ 0.004	+0.088 ∗
Hilbert	1	0.674 $\pm$ 0.005	+0.087 ∗
CWT	1	0.641 $\pm$ 0.004	+0.054 ∗
Spectrogram	3	0.636 $\pm$ 0.025	+0.049 ∗
NDI	3	0.630 $\pm$ 0.029	+0.043

Table 1: Comparison of 1D-to-2D spectral transformations for multi-trait retrieval using EfficientNet-B0 on the GreenHyperSpectra dataset. Mean

R^{2}

and standard deviation are reported across three random seeds (155, 240, 318). All methods are compared against the supervised EfficientNet-1D baseline of Cherif et al. (2025) (

R^{2}=0.587

). Bold and underlined fonts show the best- and second-best-fitted models, respectively. CWT=Continuous Wavelet Transform, NDI=all-band Normalized-Difference matrix, GAF=Gramian Angular Fields, 2D-COS=Two-Dimensional Correlation Spectroscopy, MTF=Markov Transition Field. An asterisk ( ∗ ) marks a

\Delta

significantly different from the 1D baseline (Wilcoxon signed-rank test across the eight traits,

p<0.05

Trait	Multi-trait 1D a	Reshape 2D	$\Delta$
C ab (Chlorophyll)	0.517	0.597	+0.080
C ar (Carotenoids)	0.544	0.691	+0.147
C anth (Anthocyanins)	0.454	0.628	+0.174
C w (EWT)	0.621	0.703	+0.082
C m (LMA)	0.667	0.773	+0.106
LAI	0.561	0.604	+0.043

Table 2: Per-trait

R^{2}

comparison between the best 2D method (Reshape + EfficientNet-B0) and the supervised 1D baseline ( Cherif et al., 2025 ) . The best value per trait is shown in bold. The asterisk ( ∗ ) on the mean marks a significant overall difference (Wilcoxon signed-rank test across the eight traits,

p=0.008

). a Cherif et al. (2025)

Figure 3: Observed versus predicted trait values for the supervised 2D model (Reshape + EfficientNet-B0) on the GreenHyperSpectra test set ( $n=1{,}127$ ). Each point is the prediction averaged over the three training seeds, and the grey whisker spans one standard deviation across those seeds. Points are colored by kernel density estimation. The dashed line represents the 1:1 relationship. $R^{2}$ and nRMSE are given as the mean and standard deviation of the per-seed scores, with nRMSE normalized by the 1st–99th percentile range of the observed values following Cherif et al. (2025) . Not all test samples have measurements for every trait.

Training regime	Multi-trait 1D a	Ours 2D	$\Delta$
Supervised	0.587	0.684 $\pm$ 0.001	+0.097 ∗
MAE fine-tuning	0.641	0.667 $\pm$ 0.006	+0.026 ∗
MAE linear probing	0.466	0.646 $\pm$ 0.020	+0.180 ∗

Table 3: Comparison of our 2D methods against the corresponding 1D method of Cherif et al. (2025) for each training regime. All 2D methods use the Reshape transformation. We report the mean

R^{2}

across three random seeds for the 2D methods.

\Delta

is the 2D minus 1D difference for the same regime. Linear probing trains only a regression head on the frozen encoder, while fine-tuning updates the whole encoder. An asterisk ( ∗ ) marks a

\Delta

significantly different from the 1D counterpart (Wilcoxon signed-rank test across the eight traits,

p<0.05

). Bold font indicates the best method for each regime. a Cherif et al. (2025)

Transform	Channels	$R^{2}$	$\Delta$ vs. 1D	Wins
Reshape + CWT + NDI	5	0.333	+0.090	7/8
Spectrogram	3	0.327	+0.085	5/8
Serpentine	1	0.326	+0.084	7/8
Reshape	1	0.305	+0.062	5/8
CWT	1	0.286	+0.043	4/8
2D-COS	2	0.268	+0.025	5/8

Table 4: Out-of-distribution cross-dataset

R^{2}

of every 1D-to-2D transformation under the leave-datasets-out protocol of Cherif et al. (2025) , averaged over the eight traits and ordered from best to worst.

\Delta

is the difference from the supervised 1D baseline of Cherif et al. (2025) (

R^{2}=0.243

), except in the self-supervised block, where it is the difference from their MAE-FR-FT 1D model (

R^{2}=0.311

). Wins counts the traits on which the 2D model exceeds the corresponding 1D reference. Bold and underlined fonts show the best- and second-best-fitted models, respectively. An asterisk ( ∗ ) marks a

\Delta

significantly different from the 1D reference (Wilcoxon signed-rank test across the eight traits,

p<0.05

). All values come from a single run. Per-trait nRMSE and

R^{2}

are given in Tables S4 to S6 of the supplementary material.

Panel of Figure 4 — Figure 4: Spectral importance of the supervised 2D model (Reshape + EfficientNet-B0). (a) Unfolded Integrated Gradients importance profiles for the eight traits, each normalized to its own peak. The blue line is the mean across the three training seeds and the light blue band spans one standard deviation across them. Overlaid are the PROSAIL theoretical spectral sensitivity (green gradient, darker where the constituent is more diagnostic and breaking at the removed water bands), the importance peak, and a per-trait agreement badge. Agreement is the Pearson correlation between the IG importance and the theoretical sensitivity across all 1721 bands. (b) Linking 2D and 1D attribution for chlorophyll (C ab ). Left, the Grad-CAM relevance over the Reshape image. Right, the Integrated Gradients profile unfolded to the 1721-band wavelength axis. The two panels are independent attributions of the same image, so the right one is not the unfolding of the left.

Transform	Mean nRMSE (%)	$\Delta$ vs. 1D
Reshape	12.00 $\pm$ 0.02	$-$ 1.70
Serpentine	12.15 $\pm$ 0.09	$-$ 1.55
Hilbert	12.16 $\pm$ 0.09	$-$ 1.54
CWT	12.76 $\pm$ 0.06	$-$ 0.94
Spectrogram	12.82 $\pm$ 0.43	$-$ 0.88
NDI	12.96 $\pm$ 0.50	$-$ 0.74

Table S1 : Mean nRMSE (%) by 1D-to-2D transformation on the in-distribution GreenHyperSpectra test set (mean

\pm

standard deviation over three seeds), complementing the

R^{2}

ranking of the main-text transform comparison. nRMSE is normalized by the 1st–99th percentile range of the observed values, following Cherif et al. (2025), so that lower is better.

\Delta

is the difference from their supervised 1D baseline (nRMSE 13.70), where a negative value means a lower error than the 1D baseline. The best value is in bold and the composite is listed below the rule.

Trait	Cherif 1D	Reshape 2D	$\Delta$
C ab (Chlorophyll)	17.34	15.83	$-$ 1.51
C ar (Carotenoids)	13.65	11.23	$-$ 2.42
C anth (Anthocyanins)	16.46	13.58	$-$ 2.89
C w (EWT)	13.10	11.59	$-$ 1.52
C m (LMA)	10.67	8.83	$-$ 1.84
LAI	17.49	16.50	$-$ 0.98

Table S2 : Per-trait in-distribution nRMSE (%) of the best model (Reshape plus EfficientNet-B0, mean over three seeds) against the supervised 1D baseline of Cherif et al. (2025), complementing the per-trait

R^{2}

comparison of the main text. Lower is better and the lower value per trait is in bold.

\Delta

is the 2D minus 1D difference.

Training regime	Cherif 1D	Ours 2D	$\Delta$
Supervised	13.70	12.00 $\pm$ 0.02	$-$ 1.70
MAE fine-tuning	12.78	12.30 $\pm$ 0.13	$-$ 0.48
MAE linear probing	15.50	12.62 $\pm$ 0.27	$-$ 2.87

Table S3 : In-distribution nRMSE (%) by training regime, complementing the

R^{2}

comparison of the main-text pretraining table. All 2D methods use the Reshape transformation (mean

\pm

standard deviation over three seeds). The 1D values are the corresponding nRMSE of Cherif et al. (2025) for the same regime. Lower is better and the lower value per regime is in bold.

\Delta

is the 2D minus 1D difference.

Trait	Cherif 1D	Reshape 2D	$\Delta$
Supervised
C ab (Chlorophyll)	19.17	20.87	+1.70
C ar (Carotenoids)	19.18	20.56	+1.38
C anth (Anthocyanins)	23.16	22.02	$-$ 1.14
C w (EWT)	25.22	17.76	$-$ 7.46
C m (LMA)	14.24	12.95	$-$ 1.29

Table S4 : Out-of-distribution cross-dataset nRMSE (%) under the leave-datasets-out protocol (Cherif et al., 2025), split by training regime, complementing the

R^{2}

values reported for the same experiment in the main text. The 1D values are as reported by Cherif et al. (2025); the 2D values use the Reshape transformation from a single run.

\Delta

is the 2D minus 1D difference, and the lower (better) value per trait is shown in bold.

Trait	Cherif 1D	Reshape+CWT+NDI	Spectrogram	Serpentine	Reshape
C ab (Chlorophyll)	0.362	0.379	0.336	0.378	0.340
C ar (Carotenoids)	0.181	0.065	0.070	0.084	0.052
C anth (Anthocyanins)	0.055	0.098	0.161	0.143	0.177
C w (EWT)	0.193	0.413	0.450	0.420	0.382
C m (LMA)	0.446	0.610	0.616	0.607	0.585
LAI	0.074	0.134	0.073	0.076	$-$ 0.005

Table S5 : Out-of-distribution cross-dataset

R^{2}

per trait for the four best-performing 1D-to-2D transformations under the leave-datasets-out protocol (Cherif et al., 2025), complementing the trait-averaged values reported in the main text. The 1D column holds the supervised EfficientNet-1D values reported by Cherif et al. (2025); the 2D values come from a single run each. The highest (better) value per trait is shown in bold. Wins counts the traits on which each 2D model exceeds the 1D baseline.

Trait	Cherif MAE-1D	MAE-2D	$\Delta$
C ab (Chlorophyll)	0.271	0.284	+0.013
C ar (Carotenoids)	0.165	0.214	+0.049
C anth (Anthocyanins)	0.112	$-$ 0.031	$-$ 0.143
C w (EWT)	0.280	0.192	$-$ 0.088
C m (LMA)	0.575	0.223	$-$ 0.352
LAI	0.229	0.046	$-$ 0.183

Table S6 : Out-of-distribution cross-dataset

R^{2}

per trait under self-supervised pretraining, comparing the MAE-FR fine-tuned 1D model of Cherif et al. (2025) with our MAE-2D encoder fine-tuned on the Reshape representation. Values come from a single run.

\Delta

is the 2D minus 1D difference, and the higher (better) value per trait is shown in bold.

Trait	Per-image	Global	$\Delta$
C ab (Chlorophyll)	0.597 $\pm$ 0.033	0.566 $\pm$ 0.032	$-$ 0.031
C ar (Carotenoids)	0.691 $\pm$ 0.028	0.619 $\pm$ 0.041	$-$ 0.072
C anth (Anthocyanins)	0.628 $\pm$ 0.036	0.585 $\pm$ 0.084	$-$ 0.044
C w (EWT)	0.703 $\pm$ 0.014	0.660 $\pm$ 0.037	$-$ 0.043
C m (LMA)	0.773 $\pm$ 0.009	0.742 $\pm$ 0.028	$-$ 0.031
LAI	0.604 $\pm$ 0.021	0.565 $\pm$ 0.042	$-$ 0.039

Table S7 : Effect of the min-max scaling scope on in-distribution accuracy. Per-image scaling normalizes each image by its own minimum and maximum, while global scaling applies a single minimum and maximum fitted on the training set. Values are the mean

R^{2}

and the standard deviation across three random seeds on the GreenHyperSpectra test set, using the Reshape transform with EfficientNet-B0.

\Delta

is global minus per-image, and the higher (better) value per trait is shown in bold.

Figure S1 : Pairwise differences in mean $R^{2}$ between the ten 1D-to-2D encodings (the nine transforms and the composite) on the GreenHyperSpectra test set, ordered from best to worst. Each cell gives $\Delta R^{2}$ = row $-$ column, the difference in mean $R^{2}$ across the eight traits and three seeds. Blue indicates the row encoding outperforms the column and red the opposite. An asterisk ( ∗ ) marks a pair whose difference is significant (Wilcoxon signed-rank test across the eight traits, $p<0.05$ ).

Figure S2 : Per-trait agreement between the Integrated Gradients importance and the PROSAIL theoretical spectral sensitivity, given as the Pearson correlation

r

and the Spearman correlation

\rho

across all 1721 bands. Positive values indicate that the model concentrates importance where radiative transfer predicts the constituent acts. Seven of the eight traits are positively aligned.

Figure S3 : Grad-CAM relevance maps on the Reshape image for the eight traits. Warmer colors indicate higher relevance for the predicted trait.

Figure S4 : Linking 2D and 1D attribution for carotenoids (C ar ).

Figure S5 : Linking 2D and 1D attribution for anthocyanins (C anth ).

Figure S6 : Linking 2D and 1D attribution for equivalent water thickness (C w ).

Figure S7 : Linking 2D and 1D attribution for leaf mass per area (C m ).

Figure S8 : Linking 2D and 1D attribution for leaf area index (LAI).

Figure S9 : Linking 2D and 1D attribution for protein content (C p ).

Figure S10 : Linking 2D and 1D attribution for carbon-based constituents (C bc ).

Explore similar work

May 31, 2026

Leaf Spectral Reflectance Prediction Using Multi-Head Attention Neural Networks

Accurate modeling of leaf spectral reflectance from physiological and biochemical traits is essential for advancing remote sensing applications in plant science and precision agriculture. Widely used radiative transfer models, such as PROSPECT-PRO, rely on generalized trait-reflectance relationships developed from a wide range of species, which may not fully capture the spectral behavior of specific crops like grapevines. In this study, we developed a trait-to-spectra prediction model using a multi-head attention neural network trained on a grapevine-specific dataset that includes 16 leaf traits measured across multiple varieties, growth stages, and years. The model was evaluated using stratified 5-fold cross-validation and achieved an average coefficient of determination (R^2) of 0.84 and normalized root mean squared error (NRMSE) of 1.52 percent, demonstrating high accuracy and generalizability. When compared to PROSPECT-PRO in forward mode, the neural network exhibited lower mean absolute error (MAE), especially in the near-infrared (NIR) and shortwave-infrared (SWIR) regions. These results emphasize the importance of species-specific modeling approaches and show that integrating biochemical and structural traits into data-driven architectures can significantly improve spectral prediction. The proposed model provides a robust framework for generating accurate leaf-level reflectance data, with potential applications in canopy trait retrieval, vineyard monitoring, and remote sensing-driven crop management.

Parastoo Farajpoor, Alireza Pourreza, Mohammadreza Narimani +2

Digital Agriculture Laboratory, Department of Biological and Agricultural Engineering, University of California, Davis, CA, USA · Department of Botany and Plant Sciences, University of California, Riverside, CA, USA · Department of Viticulture and Enology, University of California, Davis, CA, USA

Agricultural Remote Sensing Remote Sensing Multi-Head Attention

Ask PDF

Oct 5, 2026

FrontVeg V2: A Training-Free Software Framework for Foreground-Aware Zero-Shot Plant Trait Segmentation in High-Resolution Images of Trellised Crops

FrontVeg V2 is an open-source, training-free software framework for foregroundaware zero-shot segmentation of plant traits in high-resolution images of trellised crops. The pipeline combines monocular depth estimation, automatic foreground extraction using Valley-Aware Depth Thresholding, tiled zero-shot segmentation, Graph-Based Mask Assembly, and geometry-aware fusion. This design enables plant organs and disease symptoms to be segmented while reducing detections arising from neighboring vegetation rows. The current implementation integrates Depth Anything V2 (DAV2) and SAM3 and can be used through both command-line batch processing and a Napari graphical interface. FrontVeg V2 provides a reusable framework for multi-crop, multi-trait digital phenotyping without task-specific model retraining.

Abdoul Djalil Ousseini Hamza, Herearii Metuarea, Corentin Lothod{é} +6

IRHS-IMHORPHEN,DPT SPE · GAFL · SVQV,BAP +2

Image Segmentation Plant Phenotyping Semantic Segmentation

Ask PDF

Sep 30, 2026

A PyTorch Library for Hyperspectral Image Models: Technical Report

Hyperspectral remote sensing has advanced across diverse deep learning paradigms, including spectral spatial CNNs, Vision Transformers, Mamba, graph neural networks, Kolmogorov Arnold networks, and self supervised masked autoencoding. Yet progress remains hindered by fragmented repositories, incompatible tensor conventions, and non standardized evaluation. Hyperspectral Image Models addresses these challenges through a modular framework unifying 55 representative models across six paradigms with a common registry, automatic 4D/5D tensor adaptation, and standardized constructors. It integrates 24 benchmark scenes from Airborne, Spaceborne, UAV, and Mars CRISM sensors, with caching, label remapping, PCA, explicit band selection or raw spectra, optional spatial max pooling, and arbitrary PxP patch extraction. To prevent inflated accuracy from overlapping windows, it supports class balanced random partitioning and spatially disjoint regional blocking with Chebyshev guard bands that eliminate train test pixel overlap. Experiments use a single config with deterministic seeds and complete provenance, generating LaTeX benchmark tables and classification maps. Across 1,320 model scene evaluations and 6,600 seeded runs, scene difficulty dominates architecture, with mean accuracy ranging from 96.40% on Botswana to 56.70% on Houston 2018, versus a 15 point spread across paradigm means. No paradigm universally dominates, while sub 1 M parameter models can match architectures two orders of magnitude larger. Code is publicly available at https://github.com/Tanishq251/Hyperspectral-Image-Models.

Tanishq Rachamalla, Aryan Das, Srishti Kaushik +1

Department of Information Technology Siddhartha Academy of Higher Education Vijayawada, Andhra Pradesh 521108, India · Department of Computer Science and Engineering Vellore Institute of Technology Bhopal, Madhya Pradesh 466114, India · Department of Computer and Information Sciences Indira Gandhi National Open University New Delhi 110068, India +1

Remote Sensing Benchmarks Hyperspectral Image Classification

Ask PDF