Fusing Multimodal CNN Features with Structurally-Masked Transformers for Hierarchical Sub-Pixel Tree Species Mapping
Organizations: Department of Information Engineering and Computer Science, University of Trento, 38123 Trento, Italy
Abstract
Precise forest species mapping using spaceborne hyperspectral data is challenging due to high spectral similarity and the inherent sub-pixel heterogeneity of natural forest stands. While traditional deep learning models exploit spatial context through convolutional architectures, these approaches can introduce noise in fragmented or mixed-species forests. In this paper, we propose a hierarchy-aware, multimodal deep learning framework to resolve sub-pixel forest composition by integrating PRISMA (PRecursore IperSpettrale della Missione Applicativa) hyperspectral imagery, high-resolution panchromatic data, and digital elevation models. The architecture features a multi-branch encoder combining an EfficientNetV2-based one-dimensional convolutional neural network (CNN) for spectral signatures and twin two-dimensional CNNs for spatial and topographic features, coupled with a novel structurally-masked Transformer. By using a binary transition tensor to condition the Transformer slot embeddings and constrain inference, the model enforces taxonomic consistency between broad forest categories and fine-grained species distributions. The framework is validated across three diverse Italian biogeographical regions (the Alps and the Central and Southern Apennines) using field-surveyed plots. The proposed method consistently outperforms state-of-the-art multi-label and ranking-based paradigms, particularly in identifying subordinate species and their relative dominance. The bidirectional learning mechanism allows sub-pixel compositional features to refine macroscopic forest type predictions, while the integration of topographic priors effectively resolves spectral ambiguities in complex terrains. This knowledge-driven method supports regional-scale biodiversity monitoring and forest characterization within the evaluated PRISMA scenarios.
Figures & tables
| Forest Types | Central-East. Alps | Central Apenn. | South. Apenn. |
| Larch and Swiss stone pine | 1,083 | – | – |
| Norway spruce | 1,124 | – | – |
| Fir | 1,088 | – | – |
| Scots / Mountain pine | 1,072 | – | – |
| Black pines | 409 | 671 | 94 |
| Mediterranean pines | – | 465 | – |
| No. of Classes | Model | OA (%) | F1-score (%) |
| 11 | Proposed | 83.42 4.63 | 84.15 4.55 |
| 11 | SpectralFormer [ 25 ] | ||
| 11 | Mamba [ 36 ] |
| No. of Classes | Model | OA (%) | F1-score (%) |
| 9 | Proposed | 88.10 4.66 | 88.57 4.37 |
| 9 | SpectralFormer [ 25 ] | ||
| 9 | Mamba [ 36 ] |
| No. of Classes | Model | OA (%) | F1-score (%) |
| 5 | Proposed | 98.16 1.15 | 97.85 1.47 |
| 5 | SpectralFormer [ 25 ] | ||
| 5 | Mamba [ 36 ] |
| Method | FT Acc. | Comp. Acc. | Acc. | Acc. | Acc. | NDCG | mAP | IoU |
| Binary Relevance | 91.02 2.75 | 70.70 4.19 | 83.01 4.34 | 59.65 2.33 | 49.12 13.6 | 0.614 0.012 | 97.43 1.67 | 57.83 1.12 |
| Approx. NDCG | 91.02 2.75 | 72.17 3.80 | 89.21 0.96 | 62.86 4.94 | 45.58 9.14 | 0.640 0.008 | 96.93 0.89 | 55.36 1.48 |
| ListNet | 91.02 2.75 | 68.51 1.72 | 90.09 1.60 | 58.08 5.80 | 34.87 3.84 | 0.639 0.010 | 97.79 1.89 | 54.01 0.91 |
| Label Powerset | 91.02 2.75 | 70.82 1.97 | 89.94 3.01 | 63.01 2.71 | 39.34 2.23 | 0.611 0.010 | 96.56 1.68 | 77.78 2.68 |
| Hierarchical | 91.02 2.75 | 64.78 4.17 | 89.37 2.70 | 35.20 5.74 | 43.53 6.91 | 0.599 0.013 | 94.90 1.47 | 47.53 1.67 |
| Switcher-HNet | – | – | 80.13 1.74 | 35.76 1.78 | 21.71 6.85 | 0.576 0.008 | 90.15 2.93 | 43.13 2.02 |
| Method | FT Acc. | Comp. Acc. | Acc. | Acc. | Acc. | NDCG | mAP | IoU |
| Binary Relevance | 90.97 0.87 | 72.72 2.10 | 89.70 3.48 | 61.82 2.99 | 48.37 5.79 | 0.806 0.021 | 98.47 0.89 | 65.80 0.74 |
| Approx. NDCG | 90.97 0.87 | 71.44 2.02 | 93.33 2.57 | 57.87 3.88 | 43.57 5.67 | 0.793 0.022 | 97.78 0.83 | 59.71 0.80 |
| ListNet | 90.97 0.87 | 72.99 2.90 | 94.02 3.14 | 59.85 1.31 | 47.11 6.48 | 0.804 0.022 | 98.12 0.57 | 61.85 1.12 |
| Label Powerset | 90.97 0.87 | 70.59 2.23 | 95.23 1.40 | 54.72 3.20 | 41.43 7.53 | 0.802 0.016 | 98.53 0.33 | 72.38 1.89 |
| Hierarchical | 90.97 0.87 | 53.90 3.70 | 92.23 0.88 | 9.98 5.65 | 22.42 5.64 | 0.696 0.024 | 96.08 2.09 | 32.65 2.97 |
| Switcher-HNet | – | – | 90.04 1.32 | 39.54 2.95 | 14.48 6.09 | 0.756 0.027 | 98.06 0.29 | 45.83 2.11 |
| Method | FT Acc. | Comp. Acc. | Acc. | Acc. | Acc. | NDCG | mAP | IoU |
| Binary Relevance | 98.05 0.20 | 78.29 3.19 | 97.19 1.37 | 72.22 9.57 | 45.71 4.40 | 0.469 0.057 | 98.93 0.69 | 48.16 1.41 |
| Approx. NDCG | 98.05 0.20 | 73.66 11.79 | 97.62 0.63 | 51.82 17.60 | 47.14 37.17 | 0.469 0.057 | 99.09 0.87 | 45.18 1.66 |
| ListNet | 98.05 0.20 | 77.69 2.52 | 98.40 1.86 | 67.77 5.38 | 46.55 9.76 | 0.474 0.063 | 99.85 0.15 | 47.98 1.48 |
| Label Powerset | 98.05 0.20 | 65.70 6.83 | 97.54 1.71 | 49.73 9.03 | 17.50 20.46 | 0.460 0.064 | 100.00 0.00 | 79.61 3.85 |
| Hierarchical | 98.05 0.20 | 67.03 8.25 | 98.02 1.32 | 31.36 13.67 | 40.71 12.32 | 0.461 0.057 | 98.66 1.21 | 39.73 2.66 |
| Proposed | 97.57 2.44 | 79.81 3.27 | 98.11 1.24 | 72.83 14.45 | 50.71 6.10 | 0.467 0.051 | 99.85 0.15 | 47.25 0.87 |
| Study Area | Data Sources | OA | Comp. Acc. | ||
| HYP | PAN | DEM | |||
| Central-Eastern Alps | ✓ | ||||
| ✓ | ✓ | ||||
| ✓ | ✓ | ✓ | 83.42 4.63 | 80.26 2.67 | |
| Central Apennines | ✓ | ||||
| ✓ | ✓ | 88.56 4.90 |
| Study Area | Ecol. Know. | Transf. | Cascade Cl. | Comp. Acc. |
| Central-Eastern Alps | ✓ | ✓ | ✓ | 80.26 2.67 |
| ✓ | ✓ | |||
| ✓ | ✓ | |||
| ✓ | ||||
| Central Apennines | ✓ | ✓ | ✓ | 74.59 0.98 |
| ✓ | ✓ |
Tree Species Classification
Tree species classification identifies tree taxa from images, spectral measurements, 3D structure, and satellite or aerial observations.
See all 23 papers