BeeWhere: Segmenting Bumble Bee Colonies to Quantify Behavioral Effects
Authors: Roberta Hunt, August Easton-Calabria, James Crall
Organizations: University of Copenhagen, 2100 Copenhagen, Denmark · Lund University, 223 62 Lund, Sweden · University of Wisconsin-Madison, Madison, WI 53706, USA
Social bees are important pollinators that support biodiversity and crop pollination globally and serve as important model systems for collective behavior, but scalable measurement of individual- and colony-level behavior remains difficult in dense, occluded nest environments. Existing monitoring workflows use fiducial tags (e.g., ArUco) to preserve individual identity, yet tag-based tracking can fail when markers are obscured and provide limited information about body extent, spatial context, and untagged individuals. We present BeeWhere, an AI-assisted annotation and analysis workflow that combines ArUco detections with deep-learnt instance segmentations to quantify bumble bee behavior from high-resolution colony images and videos. Using bumble bee (Bombus impatiens) microcolonies as a test case, we annotate 483 frames containing 8,443 bee instances. We additionally annotate pollen balls, nest structures, and chamber boundaries, and train YOLO instance segmentation models for downstream behavioral analysis. Instance segmentations enable quantification of important behavioral metrics based on body contours, including nearest-neighbor distance, proximity to nest structures, spatial occupancy within the nest, and detection counts over time. We apply the BeeWhere models to tag-based tracking in an exploratory validation study assessing the behavioral impacts of neonicotinoid pesticide exposure. BeeWhere increased detection rates compared to tag-based tracking, particularly when bees were partially obscured or under challenging imaging conditions, and also captured treatment-associated changes in bee spatial organization not captured using tag-based tracking alone. These results suggest that instance segmentation can complement fiducial-marker tracking by recovering behaviorally meaningful signals under challenging colony conditions.
Figures & tables
Figure 1: Overview of BeeWhere pipeline. (a) Bumblebox microcolonies of Bombus impatiens are imaged under control and imidacloprid treatment conditions. (b) An AI-assisted annotation workflow produces training data for bees, nest structures, pollen balls, and chambers using SAM2 mask proposals and iterative YOLO retraining. (c) Trained models generate instance segmentations that can be linked across time with tracking and ArUco identity information. (d) Segmentation and tracking outputs are converted into behavioral proxies to compare treatments.
Class
No. Videos
No. Frames
No. Bbox
No. Seg
Train
Val
Train
Val
Train
Val
Train
Val
Bee
151
73
321
162
5,237
3,206
4,252
692
Pollen Ball
69
12
-
-
256
38
256
38
Nest Structure
89
28
-
-
290
115
286
115
Chamber
55
17
-
-
98
33
97
33
Table 1: Training and Validation dataset statistics. Note that some frames have all class annotations and others do not. Pollen Balls, Nest Structures and Chambers are annotated per video since they are mostly static over the course of the 10s video.
Figure 2: Number of YOLO detections per frame for each trial, shown with a rolling average. Control R1 showed an increase in population over the experiment.
Figure 3: Distributions of estimated median distance to nearest bee per frame using (a) ArUco markers and (b) instance segmentations. n_frames is the number of frames with detections included in the calculation. It should be noted that the R1 trial does not have any ArUco detections as the videos were too blurry to identify any, but bees were detected with the instance segmentation model. The mismatch in frame numbers between the two approaches for control is due to skipped frames which encountered errors during segmentation inference.
Model
Val AP0.5
Val AP0.5:0.95
No. Epochs
YOLO-BBox-Bee
0.9532
0.75067
323
YOLO-BBox-Bee + SAM2 1 1 1 These authors contributed equally to this work.
0.9343
0.4428
–
YOLO-BBox-Bee + YOLO-Bee-Fine
0.9099
0.6466
490
YOLO-Seg-Bee
0.97739
0.75688
240
YOLO-Seg-PollenBall
0.93417
0.7432
221
YOLO-Seg-Nest
0.79717
0.55338
360
Table 2: Average precision (AP) results per model, and number of epochs run until early stopping. The results of the bounding box and fine grained bee model are shown for completeness but used only during annotation. The segmentation (‘Seg’) models are used for inference and further results are based on these models.
Figure 4: Spatial bee detections for each trial from ArUco and YOLO-Seg-Bee. Average chamber, nest structure, and pollen ball contours from 100 random frames are shown. Note that chambers moved slightly over time. This can bee seen in the ’Avg’ column. Round 1’s imidacloprid treatment is missing ArUco detections completely because the images were blurred. We can see from the ’Last’ column that often nest structures and pollen balls are not entirely captured, partly due to pollen ball movement.
Prediction Source
Tracking Method
MOTA (%)
IDF1 (%)
ID Switches
Ground Truth
ByteTrack
89.7
89.7
322
SimpleIOU
93.7
93.7
196
Centroid - 200
99.0
99.0
34
YOLO-BBox-Bee
ByteTrack
84.7
88.3
233
SimpleIOU
87.9
91.5
138
Centroid - 200
90.9
91.5
44
Table 3: Tracking performance for segmentation and bounding box models measured using Multiple Object Tracking Accuracy (MOTA) and Identity F1 Score (IDF1). Best results are bolded.
Method
Detections
ArUco Identified Detections
FPs
FNs
ArUco
915
915
-
-
Yolo-Seg-Bee
2378
0
-
-
ArUco + Yolo-Seg-Bee
2378
1110
4
32
Ground truth
2441
1142
-
-
Table 4: Comparison of the number of bees detected using the Segmentation model only; the identified ArUco tags only; and their combination on the validation tracking dataset. FPs (False Positives) are wrong identifications due to ID switches. FNs (False Negatives) are identifications we failed to pick up, but could have with perfect tracking and identification. Results generated using Centroid - 200 tracking algorithm.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Overview of the staged annotation pipeline. Arrows going both ways represent an iterative process of train-predict-correct-repeat. Icons next to each step represent which classes were included in the process. Since the bees are the hardest to predict and also have small structures (antennae and legs) which are important for social behavior, and therefore the most time was spent annotating them, therefore they have extra steps (training Yolo-Bee-Fine) to reduce the human effort involved in the annotation process.
No. Videos
No. Frames
Bounding Boxes
Segmentations
Data Source
Year
Train
Val
Train
Val
Train
Val
A2019
2019
33
4
54
4
54
4
B2021
2021
42
7
57
10
36
4
C2022
2022
12
5
32
4
24
1
D2024
2024
37
9
111
14
108
5
Appendix
Table 5: Object Detection: Data sources for training and validation data. Each row is a different experiment run on a bumblebee microcolony. Note that the extents of the segmentations also count as bounding boxes, hence the number of bounding boxes always includes the segmentations.
No. Videos
No. Frames
Bounding Boxes
Segmentations
Data Source
Year
Val
Val
Val
A2019
2019
2
22
22
B2021
2021
1
10
0
C2022
2022
0
0
0
D2024
2024
1
11
11
Appendix
Table 6: Object Tracking: Data sources for training and validation data.
Figure 6: Example frames from training and validation data sources showing a variety of chamber and lighting arrangements.
Figure 7: Distributions of estimated distance to the nest across treatments, predicted using segmentations
IoT Engineering Savonia University of Applied Sciences Finland · Electrical and Information Engineering Ostbayerische Technische Hochschule Regensburg Germany