The role of spatial context and multitask learning in the detection of organic and conventional farming systems based on Sentinel-2 time series
Authors: Jan Hemmerling, Marcel Schwieder, Philippe Rufin, Leon-Friedrich Thomas, Mirela Tulbure, Patrick Hostert, Stefan Erasmi
Organizations: Thünen Earth Observation (ThEO), Thünen Institute of Farm Economics, Braunschweig, Germany. · Earth Observation Lab, Geography Department, Humboldt-Universität zu Berlin, Unter den Linden 6, 10099 Berlin, Germany. · Earth and Life Institute, UCLouvain, 1348 Louvain-la-Neuve, Belgium. · Department of Agricultural Sciences, University of Helsinki, P.O. Box 28, FI-00014 Helsinki, Finland. · Center for Geospatial Analytics, North Carolina State University, NC 27695 Raleigh, USA.
Organic farming is a key element in achieving more sustainable agriculture. For a better understanding of the development and impact of organic farming, comprehensive, spatially explicit information is needed. This study presents an approach for the discrimination of organic and conventional farming systems using intra-annual Sentinel-2 time series. In addition, it examines two factors influencing this discrimination: the joint learning of crop type information in a concurrent task and the role of spatial context. A Vision Transformer model based on the Temporo-Spatial Vision Transformer (TSViT) architecture was used to construct a classification model for the two farming systems. The model was extended for simultaneous learning of the crop type, creating a multitask learning setting. By varying the patch size presented to the model, we tested the influence of spatial context on the classification accuracy of both tasks. We show that discrimination between organic and conventional farming systems using multispectral remote sensing data is feasible. However, classification performance varies substantially across crop types. For several crops, such as winter rye, winter wheat, and winter oat, F1 scores of 0.8 or higher can be achieved. In contrast, other agricultural land use classes, such as permanent grassland, orchards, grapevines, and hops, cannot be reliably distinguished, with F1 scores for the organic management class of 0.4 or lower. Joint learning of farming system and crop type provides only limited additional benefits over single-task learning. In contrast, incorporating wider spatial context improves the performance of both farming system and crop type classification. Overall, we demonstrate that a classification of agricultural farming systems is possible in a diverse agricultural region using multispectral remote sensing data.
Figures & tables
Figure 1: Location of our study area in Germany, Zoom-in (1): grid system with 30 × 30 km tiles, indicated are training, validation and test tiles; Zoom-in (2): Subdivision of tiles into 300 × 300 m patches (30 × 30 pixel); outlined are patches with more than 15% agricultural land as used for training and validation, further subdivision of the patches into 100 × 100 m (10 × 10 pixel) and 20× 20 m (2 × 2 pixel), Background: Google Earth
Crop type
Area[ha]
Share[%]
Org,[%]
Winter wheat
34336.24
19.05
7.98
Winter barley
17152.44
9.52
1.71
Winter rye
3442.15
1.91
18.48
Triticale
6453.90
3.58
9.35
Spring barley
6110.98
3.39
11.70
Spring oat
1785.68
0.99
46.92
Table 1: Total area, share of agricultural area and area under organic management per crop type in the study area
Tiles [n]
Patches [n]
Training Data
37
218815
Validation Data
4
26079
Test Data
8
80000
Table 2: Data Distribution of Data Sets
Figure 2: Overview of the adapted Temporo-Spatial Vision Transformer (TSViT), Pixel are represented by grey grid lines, colored boxes represent sub-patches (1) Input patches are arranged into n sub-patches (PP) of size S. For each of the 36 time steps (T) 10 bands (B) are embedded into tokens of length d (2) In the temporal transformer stage, tokens interact across T and the concatenated class tokens (3) In the spatial transformer stage, class tokens interact only with their corresponding spatial class token. Final predictions for crop type and management system are produced via separate argmax layers.
Figure 3: Mean F1-score derived from validation data set during model training after each epoch.
30 x 30
10 x 10
2 x 2
1 x 1
Multitask
Singletask
Multitask
Singletask
Multitask
Singletask
RF
Crop Types
F1
Re.
Pr .
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
Winter wheat
0.94
0.94
0.95
0.95
0.94
0.95
0.94
0.94
0.94
0.94
0.94
0.94
0.93
0.93
0.94
0.92
0.92
0.93
0.80
0.72
0.90
Winter barley
0.95
0.95
0.96
0.96
0.95
0.96
0.95
0.95
0.95
0.95
0.95
0.95
0.95
0.94
0.95
0.94
0.93
0.94
0.79
0.75
0.84
Winter rye
0.83
0.86
0.81
0.84
0.85
0.83
0.81
0.81
0.81
0.82
0.84
0.80
0.80
0.84
0.77
0.78
0.79
0.76
0.20
0.40
0.13
Triticale
0.83
0.82
0.83
0.84
0.83
0.86
0.81
0.82
0.81
0.81
0.83
0.79
0.79
0.80
0.79
0.77
0.79
0.75
0.40
0.58
0.31
Table 3: Classification results, F1-scores, Recall and Precision values for single crop types
30 x 30
10 x 10
2 x 2
1 x 1
Multitask
Singletask
Multitask
Singletask
Multitask
Singletask
RF
Management
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
Conventional
0.93
0.91
0.95
0.93
0.91
0.95
0.93
0.92
0.95
0.93
0.92
0.94
0.93
0.91
0.95
0.92
0.91
0.94
0.79
0.93
0.69
Organic
0.61
0.69
0.55
0.59
0.64
0.54
0.59
0.67
0.52
0.58
0.65
0.52
0.55
0.70
0.46
0.53
0.63
0.45
0.25
0.15
0.74
Background
0.96
0.97
0.95
0.96
0.97
0.95
0.96
0.97
0.96
0.96
0.97
0.96
0.96
0.97
0.95
0.96
0.96
0.95
0.92
0.96
0.89
Mean F1
0.83
0.86
0.81
0.82
0.84
0.81
0.83
0.85
0.81
0.82
0.85
0.81
0.81
0.86
0.79
0.80
0.83
0.78
0.65
0.68
0.77
Table 4: Accuracy measures for management system classification
Figure 4: Crop type and management system predictions from model runs with different spatial input dimensions (30 x 30, 10 x 10, 2 x 2 pixel, as described in 2.3, IACS data as reference, Random Forest model predictions as described in 2.8)
Figure 5: Crop type and management form predictions from model runs with different spatial input dimensions (30 x 30, 10 x 10, 2 x 2 pixel, as described in 2.3, IACS data as reference, Random Forest model predictions as described in 2.8)
Training Data
Validation Data
Test Data
Crop type
Area[ha]
Share[%]
Org.[%]
Area[ha]
Share[%]
Org.[%]
Area[ha]
Share[%]
Org.[%]
Winter wheat
26274.5
13.42
8.32
2159.2
8.78
6.86
5902.5
8.2
6.86
Winter barley
12470.7
6.37
1.85
1476.8
6
1.01
3204.9
4.45
1.51
Winter rye
2198.8
1.12
21.27
466.7
1.9
10.59
776.7
1.08
15.32
Triticale
4168.2
2.13
10.82
859.7
3.49
5.84
1426
1.98
7.2
Spring barley
4851.5
2.48
11.85
494.6
2.01
10.23
764.9
1.06
11.73
Table Appendix A1: Data distribution of Train, Validation and Test data set, share relative to total area in respective data set
Crop Type
Samples [n]
Winter wheat
97533
Winter barley
53350
Winter rye
11223
Triticale
20706
Spring barley
24089
Spring oat
8325
Table Appendix A2: Sample Distribution used for Random Forest Crop Type Model
Managemnt Form
Samples [n]
Conventional
617042
Organic
66792
Background
19721
Table Appendix A3: Sample Distribution used for Random Forest Management Form Model
30 x 30
10 x 10
2 x 2
1 x 1
Multitask
Singletask
Multitask
Singletask
Multitask
Singletask
RF
Crop Types
F1
Re.
Pr .
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
F1
Re.
Pr.
Winter wheat
0.94
0.94
0.95
0.95
0.94
0.95
0.94
0.94
0.94
0.94
0.94
0.94
0.93
0.93
0.94
0.92
0.92
0.93
0.80
0.72
0.90
Winter barley
0.95
0.95
0.96
0.96
0.95
0.96
0.95
0.95
0.95
0.95
0.95
0.95
0.95
0.94
0.95
0.94
0.93
0.94
0.79
0.75
0.84
Winter rye
0.83
0.86
0.81
0.84
0.85
0.83
0.81
0.81
0.81
0.82
0.84
0.80
0.80
0.84
0.77
0.78
0.79
0.76
0.20
0.40
0.13
Triticale
0.83
0.82
0.83
0.84
0.83
0.86
0.81
0.82
0.81
0.81
0.83
0.79
0.79
0.80
0.79
0.77
0.79
0.75
0.40
0.58
0.31
Table Appendix A4: Classification results, F1-scores, Recall and Precision values for single crop types