cs.CVOct 23, 2026

ReLEAP: Upcycling Vision Datasets through Cross-Task Unification and Semi-Supervised Annotation

Authors: Julia Simon, Daniel Steininger, Andreas Trondl

Organizations: AIT Austrian Institute of Technology, Giefinggasse 4, 1210 Vienna, Austria Center for Vision, Automation & Control

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Developing robust vision models requires large amounts of annotated data, especially for tasks involving fine-grained class distinctions under variable environmental conditions. Even in domains with numerous existing datasets, differences in annotation formats, label definitions and targeted learning tasks hinder data reuse and integration. To address this challenge, we introduce ReLEAP, a scalable Python framework for unifying heterogeneous image datasets through data-source abstraction, systematic reuse of annotations across learning tasks, and mapping of labels to a shared hierarchical taxonomy. We demonstrate its applicability in precision agriculture by integrating fifteen diverse datasets covering a broad range of plant species and annotation types. Beyond unification, ReLEAP enables systematic recycling of existing data and provides a robust foundation for learning-based annotation upcycling. Leveraging this foundation, we conduct extensive cross-dataset and fine-tuning experiments to investigate how domain-specific models trained on unified data and foundation models can be used to automatically generate higher-level annotations under unsupervised, weakly supervised and semi-supervised settings, and how dataset characteristics influence their effectiveness. Our results show that unified data access and annotation upcycling substantially expand the pool of reusable training data and annotations while improving data transparency, enabling more sustainable and resource-efficient use of existing datasets. We release the framework and trained models for academic use to support accessible and collaborative research, particularly in data-scarce and heterogeneous domains.

Figures & tables

Figure 1: Hierarchy of common learning tasks in computer vision (label colors indicate object categories and hues of the same color individual objects). While all annotations can potentially be upcycled to higher-level tasks using specialized models, only those connected by black lines are suitable for direct recycling to the next lower level.
Figure 2: ReLEAP framework structure and main modules, along with an example for label mapping using a unified taxonomy. Each taxonomy node defines synonymic names and parent classes for a category. Automatically assigning both source and target class names to these nodes allows aggregation across data sources at arbitrary granularity.
DatasetClsDetSegISegStemLeafImgInstCatWidthHeightArea
CottonWeedID15 [ 17 ]\CIRCLE\CIRCLE↑\uparrow↑\uparrow↑\uparrow↑\uparrow↑\uparrow5.2152,7892,8007.1
EarlyCropWeed [ 35 ]\CIRCLE\CIRCLE↑\uparrow↑\uparrow↑\uparrow↑\uparrow↑\uparrow0.544,2562,83215.4
PlantSeedlings [ 43 ]\CIRCLE\CIRCLE-↑\uparrow↑\uparrow↑\uparrow↑\uparrow5.5122662660.2
SugarbeetsAndWeeds [ 113 ]↓\downarrow\CIRCLE\CIRCLE↑\uparrow↑\uparrow↑\uparrow↑\uparrow0.11.025125120.3
LincolnBeet [ 94 ]↓\downarrow\CIRCLE\CIRCLE↑\uparrow↑\uparrow↑\uparrow↑\uparrow4.439.221,9201,0802.1
PakistaniWeed [ 48 ]↓\downarrow\CIRCLE\CIRCLE↑\uparrow↑\uparrow↑\uparrow↑\uparrow9.242.871,2807200.9
Table 1: Statistical overview of datasets integrated in ReLEAP and their suitability for the tasks of Cl a s sification, Object Det ection, Semantic Seg mentation, I nstance Seg mentation, Stem -Point Localization and Leaf Segmentation ( \CIRCLE\CIRCLE : explicitly annotated in the original dataset, ↓\downarrow : available through automatic recycling, ↑\uparrow : available through learning-based upcycling approaches), along with numbers of Im a g es, annotated plant Inst ances (both in thousands) and Cat egories, as well as median image Width and Height in pixels and average image Area in megapixels.
Figure 3: Evaluated annotation-upcycling methods inferring instance-segmentation masks from unannotated image data and varying degrees of prior information (rounded boxes). ISEG and CLS refer to models trained on instance-segmentation and classification data aggregated by ReLEAP . SAM denotes a SAM 3 foundation model prompted with the priors in subscript. Spec and Agn identify models trained on all available classes individually or mapped to a single superclass.
MethodPBWGSWLMCWOverallPRt [ms]
Baseline Overall27.217.712.957.321.527.369.134.934.6
Baseline Best45.420.518.282.429.639.283.150.431.4
SAM Agn45.322.916.685.638.841.976.468.8682.9
ISEG Agn54.918.923.386.834.343.679.060.948.2
Table 2: Instance-mask accuracy (% mAP 50-95 ) of class-agnostic methods by target set ( P heno B ench, W eeds G alore, S orghum W eedSeg, L ettuce M OTS, C ropAnd W eed) as well as P recision, R ecall and inference t ime per image, including average and maximum class-agnostic results of baseline models trained on single datasets.
Figure 4: Precision and recall of upcycling approaches applied to each target dataset and their averages. ISEG and CLS refer to models trained on aggregated ReLEAP data for instance segmentation and classification, SAM to SAM 3 prompted with priors in subscript. Spec and Agn denote multi-class and single-superclass models, respectively.
MethodSupervisionPBWGSWLMCWOverallt [ms]
Baseline Overall 9.94.00.80.00.23.034.6
Baseline Best 34.815.62.30.00.510.631.4
ISEG SpecUnsupervised31.910.62.70.00.99.233.9
SAM Agn + CLS 35.77.91.20.00.19.0727.9
ISEG Agn + CLS 42.47.13.10.00.110.594.2
SAM LabelsWeakly supervised4.50.72.457.60.113.11,367.4
Table 3: Instance-mask accuracy (average per-class % mAP 50-95 ) and average inference t ime per image of upcycling methods on each target dataset ( P heno B ench, W eeds G alore, S orghum W eedSeg, L ettuce M OTS, C ropAnd W eed). ISEG and CLS refer to models trained on aggregated ReLEAP data for instance segmentation and classification, SAM to SAM 3 prompted with priors in subscript. Spec and Agn denote multi-class and single-superclass models. The baseline represents average and best results of individual models trained on the remaining four datasets for each target.
Figure 5: Representative unsupervised/weakly supervised upcycling results on images from P heno B ench [ 114 ] , W eeds G alore [ 15 ] , S orghum W eedSeg [ 60 ] , L ettuce M OTS [ 51 ] and C ropAnd W eed [ 100 ] . Left to right: Input images, manual annotations, results by method (green: correctly detected mask areas, yellow: correctly located areas assigned to wrong classes, red: incorrectly detected areas, blue: undetected instances).
Target dataset02550100200
CropAndWeed [ 100 ]1.51.5 ± 0.1\pm~0.12.6 ± 0.2\pm~0.22.7 ± 0.0\pm~0.03.2 ± 0.1\pm~0.1
LettuceMOTS [ 51 ]0.048.0 ± 0.9\pm~0.990.4 ± 1.9\pm~1.991.8 ± 0.1\pm~0.192.2 ± 0.1\pm~0.1
Table 4: Accuracy (average per-class % mAP 50-95 ) of few-shot domain-adaptation using class-specific models fine-tuned on different numbers of randomly selected image samples from two target datasets. Each value represents the mean of three experiment runs, along with standard deviation. Original results with no additional samples are included for reference, as the configuration slightly differs from previous evaluations.
Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

DatasetYearLocationInstancesSpecPA
AIWeeds [ 33 ]2021USA/China10,00015✕
BeanDiseases [ 67 ]2023Tanzania59,0691✓
CornWeedCls [ 121 ]2019China5,9985✓
CottonLeafDisease [ 12 ]2024Bangladesh2,1371✓
CottonWeedID15 [ 17 ]2022USA5,18715✓
CWD30 [ 53 ]2023South Korea219,77830✓
Appendix
Table 5: List of surveyed datasets annotated for classification as their highest-level task, which were considered for integration in ReLEAP , along with recording years and locations, numbers of instances and species (Spec) as well as public availability (PA). Datasets selected for integration are marked in bold.
DatasetYearLocationImagesSpecL/SPA
BelgiumBeet [ 39 ]2020Belgium6522 ✕
BroadleafWeeds [ 22 ]2022Australia781 ✓
CanolaSeedling [ 7 ]2024Canada4311 ✓
Chicory [ 38 ]2023Italy3,3731 ✓
Chilli [ 91 ]2023Unknown7072 ✓
CobbityWheat [ 19 ]2021Australia1403 ✓
Appendix
Table 6: List of surveyed datasets annotated for object detection as their highest-level task, which were considered for integration in ReLEAP , along with recording years and locations, numbers of images and species (Spec), integration of leaf-segmentation (L) or stem-point (S) annotations as well as public availability (PA). Datasets selected for integration are marked in bold.
DatasetYearLocationImagesSpecL/SPA
AgricultureVision [ 18 ]2020USA94,9862 ✓
Airphen [ 107 ]2020France2752L✓
Carrots2017 [ 13 ]2019UK202 ✓
Carrot-Weed [ 70 ]2017North Macedonia392 ✓
CoFly-WeedDB [ 65 ]2022Greece2013 ✓
CWFID [ 49 ]2014Germany602 ✓
Appendix
Table 7: List of surveyed datasets annotated for semantic segmentation as their highest-level task, which were considered for integration in ReLEAP , along with recording years and locations, numbers of images and species (Spec), integration of leaf-segmentation (L) or stem-point (S) annotations as well as public availability (PA). Datasets selected for integration are marked in bold.
DatasetYearLocationImagesSpecL/SPA
Aberystwyth [ 9 ]2016UK561L✓
ACRECropWeed [ 11 ]2023Italy1,0006 ✓
CN20 [ 3 ]2022Germany2839 ✓
CropAndWeed [ 100 ]2023Austria8,03474S✓
Crops3D [ 125 ]2024China1,2308 ✓
GrowliFlower [ 63 ]2023Germany2,1981LS✓
Appendix
Table 8: List of surveyed datasets annotated for instance segmentation as their highest-level task, which were considered for integration in ReLEAP , along with recording years and locations, numbers of images and species (Spec), integration of leaf-segmentation (L) or stem-point (S) annotations as well as public availability (PA). Datasets selected for integration are marked in bold.
Figure 6: Distribution of surveyed datasets regarding their highest-level explicitly annotated learning tasks (CLS: classification, DET: object detection, SEG: semantic segmentation, ISEG: instance segmentation), as well as years and locations of recording.
FeatureReLEAPRoboflowFiftyOneDatumaroMSeg
Open-source core functionality✓partial✓✓✓
Dataset management✓✓✓✓partial
Dataset visualization✓✓✓partial–
Annotation format conversion✓✓✓✓–
Label remapping✓✓✓✓✓
Domain-level taxonomy reuse✓–––✓
Appendix
Table 9: Qualitative comparison of ReLEAP with related dataset-management and label-harmonization frameworks (Roboflow [ 34 ] , FiftyOne [ 108 ] , Datumaro [ 81 ] and MSeg [ 69 ] ). Marks denote whether a capability is available as a documented part of the respective workflow, not its maturity or scope. The label partial indicates support for selected aspects, but not as a primary objective or through an automated end-to-end workflow. ReLEAP combines cross-dataset integration, taxonomy-based label harmonization and task-driven annotation recycling in a unified open-source framework.
Class nameBotanical nameEPPODatasetsInstances
AnnualMeadowGrassPoa annuaPOAANCW359
AnnualMercuryMercurialis annuaMERANCW894
BarnyardGrassEchinochloa1ECHGWG, CW3,429
BlackBindweedFallopia convolvulusPOLCOCW200
BroadBeanVicia fabaVICFXCW917
CharlockMustardSinapis arvensisSINARCW330
Appendix
Table 10: Class definition for multi-class instance-segmentation models, along with botanical names and EPPO [ 36 ] codes from our hierarchical taxonomy, total numbers of instances across the five selected datasets and their presence in each of them ( PhenoBench (PB), WeedsGalore (WG), SorghumWeedSeg (SW), LettuceMOTS (LM) and CropAndWeed (CW)).
Class nameBotanical nameEPPOInstances
AngledLuffaLuffa acutangulaLUFAC2,768
AnnualMeadowGrassPoa annuaPOAAN415
AnnualMercuryMercurialis annuaMERAN974
BarnyardGrassEchinochloa1ECHG2,262
BitterMelonMomordica charantiaMOMCH2,116
BlackBeanPhaseolus vulgaris ssp. nanusPHSVN1,970
Appendix
Table 11: Class definition for classification models, along with botanical names and EPPO [ 36 ] codes from our hierarchical taxonomy and total numbers of instances across the thirteen classification datasets (Part 1).
Class nameBotanical nameEPPOInstances
IndianGoosegrassEleusine indicaELEIN216
IvyLeavedSpeedwellVeronica hederifoliaVERHE219
LentilLens culinarisLENCU1,706
LettuceLactuca sativaLACSA17,151
LooseSilkyBentApera spica-ventiAPESV762
MaizeZea maysZEAMX11,710
Appendix
Table 12: Class definition for classification models, along with botanical names and EPPO [ 36 ] codes from our hierarchical taxonomy and total numbers of instances across the thirteen classification datasets (Part 2).
 LabelsInstances
DatasetOriginalTargetOriginalAgnosticSpecific
CropAndWeed [ 100 ]7551109,79474,00270,113
LettuceMOTS [ 51 ]1118,37017,58817,588
PhenoBench [ 114 ]3237,42632,76632,766
SorghumWeedSeg [ 60 ]335,0514,8884,888
WeedsGalore [ 15 ]5512,7799,9579,957
Appendix
Table 13: Numbers of mapped labels and included instances in target datasets for instance-segmentation experiments. Numbers of instances are provided for the class-agnostic ISEG Agn variants filtered by size and for the multi-class ISEG Spec variants additionally filtered by image and instance counts. Reductions in labels result both from mapping to superclasses and exclusion of underrepresented classes. Original values are provided for reference. As target labels overlap across datasets, their total is given as the number of distinct classes in the common label space ( Tab. 10 ).
 LabelsInstances
DatasetOriginalTargetOriginalSpecific
CottonWeedID15 [ 17 ]15135,1875,054
CropAndWeed [ 100 ]7556109,79470,811
EarlyCropWeed [ 35 ]43508454
LettuceMOTS [ 51 ]1118,37017,151
LincolnBeet [ 94 ]2139,24616,342
Appendix
Table 14: Numbers of mapped labels and included instances in training datasets for the CLS classification models after filtering by instance size. Reductions in labels result both from mapping to superclasses and exclusion of underrepresented classes. Original values are provided for reference. As target labels overlap across datasets, their total is given as the number of distinct classes in the common label space ( Tabs. 11 and 12 ).
 PrecisionRecallt [ms]
 PBWGSWLMCWPBWGSWLMCWPBWGSWLMCW
Bl Overall68.479.053.487.856.922.411.042.560.837.938.445.535.731.022.4
Bl Best91.285.766.095.077.434.319.248.590.160.137.540.429.328.621.0
ISEG Agn89.871.372.097.064.761.122.770.393.057.246.178.451.038.626.8
SAM Agn78.267.961.198.076.661.251.766.396.068.7633.7885.1685.1618.0592.4
Appendix
Table 15: Precision and recall of instance segmentation (at threshold with best F1 score), along with average inference t ime per image, on each target dataset ( P heno B ench, W eeds G alore, S orghum W eedSeg, L ettuce M OTS, C ropAnd W eed) for agnostic methods. ISEG Agn is a single-class instance-segmentation model trained on the remaining four datasets, SAM Agn denotes results of SAM 3 prompted with generic superclass names. The baseline represents average and best results of individual models trained on the remaining four datasets for each target.
Figure 7: Representative results of unsupervised and weakly supervised upcycling methods on PhenoBench [ 114 ] images. Left to right: input images, ground truth, upcycling results per method (green: correctly detected mask areas, yellow: correctly located areas assigned to wrong classes, red: incorrectly detected areas, blue: undetected instances).
Figure 8: Representative results of unsupervised and weakly supervised upcycling methods on WeedsGalore [ 15 ] images. Left to right: input images, ground truth, upcycling results per method (green: correctly detected mask areas, yellow: correctly located areas assigned to wrong classes, red: incorrectly detected areas, blue: undetected instances).
Figure 9: Representative results of unsupervised and weakly supervised upcycling methods on SorghumWeedSeg [ 60 ] images. Left to right: input images, ground truth, upcycling results per method (green: correctly detected mask areas, yellow: correctly located areas assigned to wrong classes, red: incorrectly detected areas, blue: undetected instances).
Figure 10: Representative results of unsupervised and weakly supervised upcycling methods on LettuceMOTS [ 51 ] images. Left to right: input images, ground truth, upcycling results per method (green: correctly detected mask areas, yellow: correctly located areas assigned to wrong classes, red: incorrectly detected areas, blue: undetected instances).
Figure 11: Representative results of unsupervised and weakly supervised upcycling methods on CropAndWeed [ 100 ] images. Left to right: input images, ground truth, upcycling results per method (green: correctly detected mask areas, yellow: correctly located areas assigned to wrong classes, red: incorrectly detected areas, blue: undetected instances).
PhenoBenchISEG SpecISEG AgnSAM AgnSAM LabelsISEG AgnSAM AgnSAM BoxSAM Pt
+ CLS+ CLS+ Box+ Box
SugarBeet47.554.746.30.062.152.177.931.7
Weed16.330.025.19.040.035.756.61.6
Appendix
Table 16: Instance-mask accuracy (% mAP 50-95 ) of unsupervised, weakly supervised and semi-supervised methods for each target class in the PhenoBench [ 114 ] dataset.
WeedsGaloreISEG SpecISEG AgnSAM AgnSAM LabelsISEG AgnSAM AgnSAM BoxSAM Pt
+ CLS+ CLS+ Box+ Box
BarnyardGrass0.00.00.00.66.010.018.87.9
Maize26.016.917.00.022.125.235.71.2
PotatoWeed0.01.00.00.824.336.156.82.1
RedRootAmaranth26.216.020.80.021.533.551.617.1
Weed1.01.61.81.916.620.740.08.3
Appendix
Table 17: Instance-mask accuracy (% mAP 50-95 ) of unsupervised, weakly supervised and semi-supervised methods for each target class in the WeedsGalore [ 15 ] dataset.
SorghumWeedSegISEG SpecISEG AgnSAM AgnSAM LabelsISEG AgnSAM AgnSAM BoxSAM Pt
+ CLS+ CLS+ Box+ Box
Dicots0.00.00.00.029.322.439.79.0
Grasses8.15.43.57.121.315.721.70.3
Sorghum0.03.80.00.023.118.636.30.3
Appendix
Table 18: Instance-mask accuracy (% mAP 50-95 ) of unsupervised, weakly supervised and semi-supervised methods for each target class in the SorghumWeedSeg [ 60 ] dataset.
LettuceMOTSISEG SpecISEG AgnSAM AgnSAM LabelsISEG AgnSAM AgnSAM BoxSAM Pt
+ CLS+ CLS+ Box+ Box
Lettuce0.00.00.057.687.886.889.484.9
Appendix
Table 19: Instance-mask accuracy (% mAP 50-95 ) of unsupervised, weakly supervised and semi-supervised methods for each target class in the LettuceMOTS [ 51 ] dataset.
CropAndWeedISEG SpecISEG AgnSAM AgnSAM LabelsISEG AgnSAM AgnSAM BoxSAM Pt
+ CLS+ CLS+ Box+ Box
AnnualMeadowGrass0.00.00.00.04.46.816.35.3
AnnualMercury0.00.00.00.042.650.050.51.1
BarnyardGrass0.20.00.02.320.423.036.30.4
BlackBindweed0.00.00.00.035.943.553.611.4
BroadBean0.00.00.00.066.466.171.939.8
Appendix
Table 20: Instance-mask accuracy (% mAP 50-95 ) of unsupervised, weakly supervised and semi-supervised methods for each target class in the CropAndWeed [ 100 ] dataset.
More in this topic

Cross-Dataset Generalization

Cross-dataset generalization studies whether AI models retain accuracy, calibration, and usefulness on data from sources beyond their training set.

See all 142 papers

Explore similar work

CardsList

  1. Unified Video Dense Prediction from Disjoint Data

    Jul 23, 2026Yihong Sun, Seoung Wug Oh, Jiahui Huang +2Multi-Task LearningDense Prediction

    Adobe Research · Cornell University

  2. AgrI Challenge: A Data-Centric AI Competition for Cross-Team Validation in Agricultural Vision

    Mar 7, 2026Mohammed Brahimi, Karim Laabassi, Mohamed Seghir Hadj Ameur +7Cross-Dataset GeneralizationDomain Generalization

    The National School of Artificial Intelligence (ENSIA), Algiers, 16046, Algeria · National Higher School of Agronomy (ENSA), Algiers, 16051, Algeria · National Higher School of Computer Science (ESI), Algiers, 16000, Algeria +1

  3. ReLeaf: Benchmarking Leaf Segmentation across Domains and Species

    May 5, 2026Robert Martinko, Daniel Steininger, Julia Simon +2Agricultural Image AnalysisInstance Segmentation

    1AIT Austrian Institute of Technology, Center for Vision, Automation & Control · University of Applied Sciences Technikum Wien, Computer Science & Applied Mathematics