MICE: Multi-animal Interaction Context Encoder - A Hierarchical Foundation Model for Mouse Behavior
Organizations: Georgia Institute of Technology, Atlanta, GA, USA
Abstract
Quantifying mouse behavior from pose is a foundational tool in neuroscience, ethology, drug discovery, and animal welfare. Self-supervised foundation models are a natural fit, but a foundation model for mouse behavior must handle social behavior - many behaviors that matter (aggression, courtship, mounting, allogrooming) are inherently multi-animal. Existing self-supervised pose models fall short: they encode each animal independently and discard interaction signal, hard-code the animal count, and pretrain one dataset at a time. We introduce MICE (Multi-animal Interaction Context Encoder), a hierarchical foundation model for mouse behavior that pretrains jointly on six mouse-pose corpora spanning one to four mice and 7 to 27 keypoints. MICE combines a hierarchical individual encoder capturing per-mouse kinematics at multiple temporal scales with a Perceiver-style social encoder whose learned latent codebook decouples parameter count from animal count, so a single checkpoint serves any group size without retraining. Across four evaluation protocols spanning six datasets, MICE matches or exceeds prior keypoint-based baselines, and layer-wise probing shows the social stage incrementally improves discriminability beyond the individual encoder. Code, weights, and data are available at https://github.com/eric0708/MICE.
Figures & tables
| Model | Regime | MABe22 | CalMS21 | CRIM13 | AdaptSnail | Goldfinch | Lockbox |
|---|---|---|---|---|---|---|---|
| PointNet ( Qi et al., 2017 ) | single | ||||||
| combined | |||||||
| TBERT ( Sun et al., 2023 ) | single | ||||||
| combined | |||||||
| BAMS ( Azabou et al., 2023b ) | single | ||||||
| combined |
| Model | ARI (%) | NMI (%) | Boundary-F1 (%) |
|---|---|---|---|
| PointNet ( Qi et al., 2017 ) | |||
| TBERT ( Sun et al., 2023 ) | |||
| BAMS ( Azabou et al., 2023b ) | |||
| hBehaveMAE ( Stoffl et al., 2024 ) | |||
| Progressive (ours, ablation) | |||
| Multiscale (ours, ablation) |
| Probe dataset | ||||||
|---|---|---|---|---|---|---|
| Pretrain dataset | MABe22 | CalMS21 | CRIM13 | AdaptSnail | Goldfinch | Lockbox |
| MABe22 | ||||||
| CalMS21 | ||||||
| CRIM13 | ||||||
| AdaptSnail | ||||||
| Goldfinch | ||||||
| Partners (train and probe) | MABe22 | CalMS21 | CRIM13 | AdaptSnail | Goldfinch |
|---|---|---|---|---|---|
| correct (default) | |||||
| mis-paired (shuffled) | |||||
| removed | |||||
| ind_cat only |
| Pretraining | MABe22 | CalMS21 | CRIM13 | AdaptSnail | Goldfinch | Lockbox |
|---|---|---|---|---|---|---|
| all six datasets | ||||||
| target left out (unseen) | ||||||
| target dataset only |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Training frames | |
|---|---|---|
| AdaptableSnail | 641,842 | 3–4 |
| CalMS21 | 507,738 | 2 |
| MouseLockbox | 427,380 | 1 |
| NiftyGoldfinch | 284,590 | 2 |
| CRIM13 | 223,362 | 2 |
| MABe22 | 118,560 | 3 |
| Readout | Source | Pooling over tokens | Dim. per animal |
|---|---|---|---|
| ind_mean / ind_cat | mean-pool / flatten | / | |
| soc_mean / soc_cat | mean-pool / flatten | / | |
| ind+soc_mean / ind+soc_cat | mean-pool / flatten | / |
| Model | CalMS21 | CRIM13 | AdaptSnail | Goldfinch | Avg. |
|---|---|---|---|---|---|
| ARI | |||||
| PointNet | 57.90 | 51.49 | 10.74 | 14.77 | 33.73 |
| TBERT | 42.18 | 39.19 | 9.51 | 7.48 | 24.59 |
| BAMS | 16.54 | 13.81 | 5.04 | 1.71 | 9.28 |
| hBehaveMAE | 51.19 | 50.15 | 12.21 | 11.72 | 31.32 |
| Progressive (ours, ablation) | 11.78 | 13.76 | 8.85 | 2.92 | 9.33 |
| Dataset | KPMS (train-fit) | KPMS (per test recording) | MICE |
|---|---|---|---|
| CalMS21 | 16.50 / 16.34 / 32.82 | 17.14 / 17.11 / 28.16 | 72.31 / 58.64 / 40.69 |
| CRIM13 | 0.75 / 0.91 / 0.27 | 0.74 / 1.45 / 1.95 | 58.59 / 49.45 / 34.97 |
| AdaptableSnail | 3.21 / 3.17 / 11.58 | 3.25 / 3.23 / 13.15 | 19.82 / 14.92 / 14.20 |
| Macro-F1 (%) at dataset | Overall | ||||
| Model | avg | ||||
| TBERT ( Sun et al., 2023 ) | |||||
| PointNet ( Qi et al., 2017 ) | |||||
| BAMS ( Azabou et al., 2023b ) | |||||
| hBehaveMAE ( Stoffl et al., 2024 ) | |||||
| Progressive (ours, ablation) | |||||
| Single-dataset | Combined | ||
|---|---|---|---|
| 1 | 15.93 | 13.68 | |
| 2 | 52.99 | 54.28 | |
| 3 | 56.77 | 61.87 | |
| 4 | 20.38 | 28.50 |
| Variant | ind_mean | ind_cat | soc_mean | soc_cat | ind+soc_mean | ind+soc_cat |
|---|---|---|---|---|---|---|
| F1 (%) |
| Social stage | MABe22 | CalMS21 | CRIM13 | AdaptSnail | Goldfinch |
|---|---|---|---|---|---|
| Perceiver latents (MICE) | |||||
| direct animal-to-animal attention |
| CalMS21 | CRIM13 | ||
|---|---|---|---|
| 1 | 1.00 | 81.22 | 44.71 |
| 1 | 0.30 | 78.49 | 37.97 |
| 1 | 0.10 | 76.46 | 36.01 |
| 1 | 0.03 | 73.78 | 33.29 |
| 1 | 0.00 | 74.10 | 33.92 |
| 0 | 1.00 | 78.84 | 34.02 |
| Measurement | individual | social | difference |
| Distance between the mice | 0.358 | 0.975 | |
| Relative body orientation | 0.145 | 0.644 | |
| Approach speed | 0.018 | ||
| Partner speed | |||
| Focal mouse speed | 0.702 | 0.638 | |
| Focal mouse body spread | 0.975 | 0.953 |
| CalMS21 | ||||
|---|---|---|---|---|
| Latent | Class | Lift | ||
| 3 | attack | 10.86 | 54% | 34% |
| 12 | attack | 7.04 | 35% | 49% |
| 14 | mount | 3.05 | 37% | 24% |
| 4 | investigation | 1.75 | 41% | 59% |
| Dataset | Vanilla transformer | MICE | Gap |
|---|---|---|---|
| MABe22 | |||
| CalMS21 | |||
| CRIM13 | |||
| AdaptableSnail | |||
| NiftyGoldfinch | |||
| MouseLockbox |
| Dataset | Supervised (end-to-end) | MICE | Gap |
|---|---|---|---|
| MABe22 | |||
| CalMS21 | |||
| CRIM13 | |||
| AdaptableSnail | |||
| NiftyGoldfinch | |||
| MouseLockbox |
| Model | Parameters | vs. MICE |
|---|---|---|
| PointNet | 28K | |
| MICE | 583K | |
| Multiscale (ours, ablation) | 620K | |
| Progressive (ours, ablation) | 687K | |
| TBERT | 2.132M | |
| BAMS | 2.240M |
| Animals | Individual encoder | Full model | Social stage | Full-model throughput |
|---|---|---|---|---|
| (ms/batch) | (ms/batch) | (ms/batch) | (animal-sequences/s) | |
| 1 | 7.07 | 12.49 | 5.43 | 2,562 |
| 2 | 7.13 | 14.12 | 6.98 | 4,534 |
| 3 | 7.12 | 15.72 | 8.60 | 6,109 |
| 4 | 7.19 | 17.39 | 10.20 | 7,360 |
Animal Behavior Analysis
Animal behavior analysis uses computational tools to measure how animals move, interact, communicate, and respond to their environments.
See all 48 papers