Training-free motion statistics match self-supervised behavior encoders
Organizations: University of Illinois Urbana-Champaign
Abstract
Self-supervised encoders for animal behavior are often trained on, or fed, hand-built movement features. We ask whether they beat those ingredients when compared fairly. A training-free representation, made of generic per-frame measurements summarized over fixed time windows and reduced with PCA, is compared with retrained and released encoders (BAMS, hBehaveMAE) under a preregistered protocol: frozen representations of equal size and one shared classifier. On CalMS21 it reaches 85.8 macro F1 against 82.4 for the best encoder, ties with four labelled videos, and leads when the annotator changes or new behaviors must be learned from a few videos; the reading holds for encoders retrained with three seeds. On the MABe22 mouse triplets it scores 83.7 against 80.3 for the released hBehaveMAE (gap +3.4, 95% interval +2.1 to +4.7). On a second species, the MABe22 fly groups, the same recipe scores 51.6 mean F1 over 50 tasks against 48.2 for BAMS, the best published method. On Shot7M2, the synthetic basketball benchmark built by hBehaveMAE's authors, it scores 68.8 against 64.7 for hBehaveMAE, the best method in that paper, though hBehaveMAE remains better on the shortest movements. Short time windows carry most of the result. Raw keypoints in each body's own frame, summarized over the same windows, already match or beat the encoders on both mouse datasets and on Shot7M2; on the fly groups they fall short, and the measurements add 5.3 points. On these benchmarks, current behavior encoders have yet to beat the hand-built features they learn from, and such baselines should be reported alongside them.
Figures & tables
| Group | Measurements |
|---|---|
| Distance | distance between the two body centres; one minus that distance; contact (below one body length) |
| Speeds | each mouse’s centre speed |
| Approach | closing speed; each mouse’s speed toward the other |
| Activity | per mouse, the share of still frames (below 0.25 body lengths per second) and the rate of switching between still and moving, over 0.5 s; the correlation of the two speeds over 0.5 s |
| Nose | each mouse’s nose to the nearest keypoint of the other |
| Heading | whether each mouse’s head points at the other (cosine); how fast each head turns |
| Method | All labels | 5% of labels |
|---|---|---|
| Training-free features (ours) | 85.8 [82.4, 88.5] | 61.1 [57.0, 64.1] |
| hBehaveMAE | 82.4 [78.2, 85.3] | 45.5 [41.1, 49.1] |
| BAMS | 78.3 [73.9, 81.3] | 63.4 [58.4, 64.9] |
| BAMS, authors’ mouse recipe | 54.6 [47.5, 56.9] | 28.4 [25.3, 29.3] |
| hBehaveMAE, mean of 3 seeds | 82.4 | 45.4 |
| BAMS, mean of 3 seeds | 77.8 | 63.1 |
| Task 2: new annotators | Task 3: new behaviors | |||||
| Encoder seed | 0 | 1 | 2 | 0 | 1 | 2 |
| Training-free features (ours) | 77.9 | 77.9 | 77.9 | 37.6 | 37.6 | 37.6 |
| hBehaveMAE | 71.7 | 72.6 | 70.2 | 28.2 | 29.1 | 30.6 |
| BAMS | 66.1 | 66.4 | 66.8 | 33.9 | 34.3 | 33.9 |
| Ours minus hBehaveMAE | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] |
| Ours minus BAMS | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] | [ , ] |
| Method | Clip-level F1 | Frame-level F1 |
|---|---|---|
| Training-free features (ours) | 83.7 [81.9, 85.3] | 24.5 |
| hBehaveMAE, released, best level (our run) | 80.3 [78.7, 81.8] | 22.3 |
| hBehaveMAE, published, transductive | 80.3 | 22.3 |
| BehaveMAE, published, transductive | 81.0 | 21.5 |
| BAMS, published, transductive | 80.1 | 19.8 |
| hBehaveMAE, published, inductive | 78.5 | 20.1 |
| Method | Mean F1 |
|---|---|
| Training-free features (ours) | 51.6 |
| BAMS | 48.2 |
| T-GPT | 45.8 |
| T-Perceiver | 44.8 |
| PCA | 42.5 |
| TVAE | 37.0 |
| Method | All F1 | Activities | Actions | Movemes |
| Training-free features (ours) | 68.8 | 92.6 | 55.2 | 58.7 |
| hBehaveMAE, released, best level (our run) | 64.7 | 76.4 | 54.8 | 62.9 |
| hBehaveMAE, best level for each scale (our run) | 80.9 | 58.5 | 67.3 | |
| hBehaveMAE, published | 64.7 | 80.9 | 58.5 | 67.3 |
| BehaveMAE, published | 64.3 | 76.7 | 55.1 | 61.0 |
| PCA-30, published | 61.9 | 68.3 | 50.8 | 66.7 |
| Raw keypoints | Best | |||
|---|---|---|---|---|
| Benchmark | Ours | body | world | encoder |
| CalMS21, all labels | 85.8 | 84.8 | 77.7 | 82.4 |
| CalMS21, 5% of labels | 61.1 | 64.7 | 38.4 | 63.4 |
| MABe22 mouse triplets | 83.7 | 82.5 | 76.0 | 80.3 |
| Shot7M2 | 68.8 | 66.0 | 65.7 | 64.7 |
| MABe22 fly groups | 51.6 | 46.3 | 41.8 | 48.2 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Test | What was locked | Locked (UTC) |
|---|---|---|
| CalMS21 Task 1 | design, opponents and reading rule | 15 Nov, 01:10 |
| remaining details; running the parts side by side | 15 Nov, 03:30–06:35 | |
| repeat with two further encoder seeds | 16 Nov, 18:15 | |
| BAMS with its authors’ mouse recipe | 16 Nov, 19:10 | |
| CalMS21 Tasks 2 and 3 | design and reading rule | 16 Nov, 19:30 |
| What carries the result | stripped-down versions (Section 6 ) | 16 Nov, 21:00 |
| All labels | 5% of labels | |||
|---|---|---|---|---|
| Encoder seed | hBehaveMAE | Features minus it | BAMS | Features minus it |
| 0 | 82.4 | [ , ] | 63.4 | [ , ] |
| 1 | 83.1 | [ , ] | 61.9 | [ , ] |
| 2 | 81.6 | [ , ] | 63.9 | [ , ] |
| Method | Labels | Attack | Investigation | Mount |
|---|---|---|---|---|
| Training-free features (ours) | all | 79.9 | 84.6 | 93.0 |
| hBehaveMAE | all | 76.4 | 79.6 | 91.2 |
| BAMS | all | 60.7 | 82.8 | 91.4 |
| Training-free features (ours) | 5% | 38.2 | 70.9 | 74.3 |
| hBehaveMAE | 5% | 30.8 | 56.4 | 49.2 |
| BAMS | 5% | 39.3 | 73.7 | 77.2 |
| Category | Tasks | Ours, pooled | Ours, flies apart | Random sample submission |
|---|---|---|---|---|
| Fly type (genotype) | 11 | 50.6 | 41.2 | 14.5 |
| Stimulation, control lines | 10 | 52.8 | 47.7 | 48.4 |
| Stimulation, aggression lines | 15 | 55.0 | 56.0 | 49.6 |
| Line category | 7 | 68.7 | 60.5 | 28.4 |
| Female vs male | 1 | 88.0 | 97.6 | 72.1 |
| Manual behaviors (per fly) | 6 | 17.0 | 24.8 | 21.0 |
Animal Behavior Analysis
Animal behavior analysis uses computational tools to measure how animals move, interact, communicate, and respond to their environments.
See all 48 papers