Recognition of rodent behavior is important for understanding neural and behavioral mechanisms. Traditional manual scoring is time-consuming and prone to human error. We propose MSGL-Transformer, a Multi-Scale Global-Local Transformer for recognizing rodent social behaviors from pose-based temporal sequences. The model employs a lightweight transformer encoder with multi-scale attention to capture motion dynamics across different temporal scales. The architecture integrates parallel short-range, medium-range, and global attention branches to explicitly capture behavior dynamics at multiple temporal scales. We also introduce a Behavior-Aware Modulation (BAM) block, inspired by SE-Networks, which modulates temporal embeddings to emphasize behavior-relevant features prior to attention. We evaluate on two datasets: RatSI (5 behavior classes, 12D pose inputs) and CalMS21 (4 behavior classes, 28D pose inputs). On RatSI, MSGL-Transformer achieves 75.4% mean accuracy and F1-score of 0.745 across nine cross-validation splits, outperforming TCN, LSTM, and Bi-LSTM. On CalMS21, it achieves 87.1% accuracy and F1-score of 0.8745, a +10.7% improvement over HSTWFormer, and outperforms ST-GCN, MS-G3D, CTR-GCN, and STGAT. The same architecture generalizes across both datasets with only input dimensionality and number of classes adjusted.
Figures & tables
Figure 1: Four consecutive frames for each of the five behaviors in the RatSI [ 29 ] dataset, showing how each behavior develops over time. Behavior instance durations vary across videos, with Solitary and Following typically spanning longer periods and Moving Away occurring over shorter spans.
Video
Solitary
Approaching
Following
Moving Away
Social Nose Contact
Allogrooming
Nape Attacking
Pinning
Other
Uncertain
Total Frames
Observation01
12,281
1,405
2,150
985
1,799
1,778
221
16
792
88
21,515
Observation02
14,709
1,066
593
592
1,754
2,240
95
0
231
55
21,335
Observation03
15,428
1,083
2,522
482
1,106
587
104
1,036
472
21
22,841
Observation04
14,776
1,528
444
931
2,621
2,003
141
0
300
0
22,744
Observation05
14,297
1,310
2,955
880
1,067
111
434
127
1,678
0
22,859
Observation06
13,152
2,443
2,141
1,503
1,980
486
192
0
856
0
22,753
Table 1: Frame distribution of social behaviors across all nine RatSI observation videos. The last row indicates the overall percentage of each behavior across all videos (total 202,550 frames).
Behavior
Description / Social Context
Solitary
The rat is alone , showing no social engagement (isolation phase).
Approaching
The rat initiates interaction by moving toward another (beginning of engagement).
Following
Indicates sustained interaction , maintaining social attention.
Moving Away
Shows withdrawal or avoidance , marking disengagement.
Social Nose Contact
Reflects direct physical interaction , the highest level of social engagement.
Table 2: Summary of the selected RatSI behaviors that are considered to represent the core range of rodent social interactions.
Figure 2: Sample frames from the RatSI dataset showing the five social behaviors studied in this work: Solitary, Approaching, Following, Moving Away, and Social Nose Contact, spanning from no social engagement to direct physical contact.
Behavior
Description / Social Context
Attack
Aggressive contact initiated by the resident mouse toward the intruder.
Investigation
The resident mouse closely sniffs or investigates the intruder mouse.
Mount
The resident mouse mounts the intruder, reflecting dominance behavior.
Other
All remaining frames not belonging to the three active social behaviors.
Table 3: Summary of CalMS21 behaviors representing the core range of mouse dyadic social interactions.
Figure 3: Representative video frames from the CalMS21 dataset illustrating the four annotated behavior classes. The resident (black) mouse initiates all active behaviors toward the intruder (white) mouse in the resident-intruder assay [ 43 ] .
Split
Attack
Investigation
Mount
Other
Total Frames
Train (70 videos)
14,039
146,615
28,615
318,469
507,738
Test (19 videos)
12,630
61,275
31,848
156,354
262,107
All Videos
26,669
207,890
60,463
474,823
769,845
Train (%)
2.76
28.88
5.64
62.73
—
Table 4: Frame distribution of behavior classes across training and test splits of the CalMS21 Task 1 dataset (total 769,845 frames across 89 videos).
Aspect
RatSI
CalMS21
Species
Rats
Mice
Number of Videos
9
89
Total Labeled Frames
~202,550
~769,845
Keypoints per Animal
3
7
Input Dimension ( D )
12
28
Behavior Classes ( C )
5
4
Table 5: Comparison of the two evaluation datasets.
Figure 4: Overview of the proposed MSGL-Transformer architecture. The model combines global token generation, behavior-aware modulation, and multi-scale attention within a lightweight transformer encoder to jointly capture global context and fine-grained temporal dynamics in rodent social interactions.
Figure 5: Structure of the multi-scale attention module. The modulated input Z^0 is passed through three parallel branches: short-range causal attention on the first ⌊T/2⌋ frames (this branch focuses on rapid motion cues occurring within short temporal contexts), medium-range causal attention on all T frames, and global bidirectional attention on all T+1 tokens. The short-range and medium-range outputs are averaged on overlapping positions to produce Alocal , which is combined with Aglobal through residual addition and layer normalization.
Hyperparameter
Symbol / Setting
Value
Description
Sequence length
T
35
Number of time steps per sequence
Input dimension
D
12 (RatSI) / 28 (CalMS21)
Coordinate features per frame
Output classes
C
5 (RatSI) / 4 (CalMS21)
Number of behavior classes
Model dimension
dmodel
64
Embedding size for each token
Feed-forward dimension
dff
128
Hidden size of the FFN
Attention heads
h
4
Number of heads
Table 6: Summary of model and training hyperparameters used in the proposed MSGL-Transformer.
Window Size
Parameters
Accuracy
F1-Score
Investigation F1
T=20
223,492
0.8706
0.8751
0.7728
T=35
285,892
0.8709
0.8745
0.7881
T=50
348,292
0.8711
0.8744
0.7896
Table 7: Effect of window size on model performance on CalMS21 Task 1.
Validation
Test
Precision
Recall
F1
Accuracy
2
8
0.716
0.724
0.714
0.724
3
9
0.667
0.670
0.663
0.670
4
1
0.746
0.709
0.722
0.709
5
6
0.747
0.773
0.745
0.773
6
4
0.773
0.795
0.778
0.795
7
3
0.796
0.799
0.797
0.799
Table 8: Cross-validation results of the MSGL-Transformer on the RatSI dataset.
Figure 6: Accuracy, precision, recall, and F1-score across three representative splits in RatSI: Valid-7–Test-3 (best-performing), Valid-2–Test-8 (near the mean), and Valid-3–Test-9 (most challenging).
Class
Valid-2-Test-8
Valid-3-Test-9
Valid-7-Test-3
Approaching
0.557
0.595
0.453
Following
0.566
0.520
0.592
Moving away
0.142
0.194
0.073
Social Nose Contact
0.624
0.510
0.348
Solitary
0.863
0.798
0.909
Table 9: Per-class F1-scores across three representative splits in RatSI dataset
Figure 7: ROC curves across the three representative splits of the RatSI dataset.
Figure 8: Confusion matrices obtained for the Valid-2–Test-8, Valid-3–Test-9, and Valid-7–Test-3 splits.
Model
Accuracy
Precision
Recall
F1-score
TCN
0.7519
0.7457
0.7519
0.7393
LSTM
0.6929
0.8361
0.6929
0.7392
Bi-LSTM
0.6865
0.8414
0.6865
0.7377
MSGL-Transformer (Ours)
0.8148
0.7976
0.8148
0.7967
Table 10: Comparison with different baseline models on the RatSI dataset (Valid-8–Test-2 split).
Model Variant
BAM
MSA
Accuracy
Base
×
×
0.7567
Base + Multi-Scale Attention (MSA)
×
✓
0.7798
Base + BAM Module
✓
×
0.7859
Full Model (MSGL-Transformer)
✓
✓
0.8148
Table 11: Ablation study results on the Valid-8–Test-2 split of the RatSI dataset.
Class
Precision
Recall
F1-Score
Support
Attack
0.5942
0.5720
0.5829
12,586
Investigation
0.7157
0.8768
0.7881
60,964
Mount
0.8646
0.8798
0.8722
31,813
Other
0.9774
0.8909
0.9321
156,079
Table 12: Per-class results of the MSGL-Transformer on CalMS21 Task 1.
Figure 9: ROC curves of the MSGL-Transformer on the CalMS21 test set, showing per-class discrimination performance.
Figure 10: Confusion matrix of the MSGL-Transformer on the CalMS21 test set. We can see the model does best on the dominant Other class, while Attack is the hardest class. This reflects the severe class imbalance in the dataset.
Model
Accuracy
Precision
Recall
F1-Score
TCN
0.8486
0.8825
0.8486
0.8588
LSTM
0.8683
0.8714
0.8683
0.8696
Bi-LSTM
0.8606
0.8682
0.8606
0.8638
MSGL-Transformer (Ours)
0.8709
0.8842
0.8709
0.8745
Table 13: Baseline comparison on CalMS21 Task 1 using the official test set provided by Sun et al. [ 43 ] .
Model Variant
BAM
MSA
Accuracy
Attack F1
Base (no MSA, no BAM)
×
×
0.8771
0.5635
Base + MSA only
×
✓
0.8677
0.5518
Base + BAM only
✓
×
0.8786
0.5221
Full (MSA + BAM)
✓
✓
0.8709
0.5829
Table 14: Ablation study on CalMS21. The full model achieves the highest F1 for Attack, the most class-imbalanced behavior (2.76% of training frames).
Figure 11: Boundary error analysis results on the CalMS21 dataset. The left plot illustrates the variation in prediction accuracy with respect to the distance from the nearest behavior transition, while the right plot presents the distribution of samples observed at each distance.
Method
Input Type
Avg. Per-Class Acc (%)
Year
ST-GCN †
Keypoints
74.5
2018
MS-G3D †
Keypoints
73.5
2020
CTR-GCN †
Keypoints
71.6
2021
STGAT †
Keypoints
73.9
2021
HSTWFormer † [ 38 ]
Keypoints
76.4
2024
MSGL-Transformer (Ours)
Keypoints
87.1
N/A
Table 15: Comparison with published keypoint-based methods on CalMS21 Task 1. Results for methods marked with † are reported from [ 38 ] .
Model
Parameters
Time/Epoch (s)
TCN
30,788
∼ 97
LSTM
57,604
∼ 87
Bi-LSTM
147,972
∼ 104
MSGL-Transformer (Ours)
285,892
∼ 295
Table 16: Computational complexity of models on CalMS21 using an NVIDIA RTX A4000 GPU.
Model
RatSI Acc
RatSI F1
CalMS21 Acc
CalMS21 F1
TCN
0.7519
0.7393
0.8486
0.8588
LSTM
0.6929
0.7392
0.8683
0.8696
Bi-LSTM
0.6865
0.7377
0.8606
0.8638
MSGL-Transformer (Ours)
0.8148
0.7967
0.8709
0.8745
Table 17: Performance results of all evaluated models on the RatSI and CalMS21 datasets. For the RatSI dataset, the Valid-8–Test-2 split was used for evaluation, while for the CalMS21 dataset the Task 1 test set provided in Sun et al. [ 43 ] was used.
We present Rodent-Bench, a novel benchmark designed to evaluate the ability of Multimodal Large Language Models (MLLMs) to annotate rodent behaviour footage. We evaluate state-of-the-art MLLMs, including Gemini-2.5-Pro, Gemini-2.5-Flash and Qwen-VL-Max, using this benchmark and find that none of these models perform strongly enough to be used as an assistant for this task. Our benchmark encompasses diverse datasets spanning multiple behavioral paradigms including social interactions, grooming, scratching, and freezing behaviors, with videos ranging from 10 minutes to 35 minutes in length. We provide two benchmark versions to accommodate varying model capabilities and establish standardized evaluation metrics including second-wise accuracy, macro F1, mean average precision, mutual information, and Matthew's correlation coefficient. While some models show modest performance on certain datasets (notably grooming detection), overall results reveal significant challenges in temporal segmentation, handling extended video sequences, and distinguishing subtle behavioral states. Our analysis identifies key limitations in current MLLMs for scientific video annotation and provides insights for future model development. Rodent-Bench serves as a foundation for tracking progress toward reliable automated behavioral annotation in neuroscience research.
Animal models, particularly rats, play a critical role in seizure research for studying epileptogenesis and treatment response. However, progress is limited by the lack of datasets with precise temporal annotations and standardized evaluation protocols. Existing animal behavior datasets often have limited accessibility, coarse labeling, and insufficient temporal localization of clinically meaningful events. To address these limitations, we introduce RatSeizure, the first publicly benchmark for fine-grained seizure behavior analysis. The dataset consists of recorded clips annotated with seizure-related action units and temporal boundaries, enabling both behavior classification and temporal localization. We further propose RaSeformer, a saliency-context Transformer for temporal action localization that highlights behavior-relevant context while suppressing redundant cues. Experiments on RatSeizure show that RaSeformer achieves strong performance and provides a competitive reference model for this challenging task. We also establish standardized dataset splits and evaluation protocols to support reproducible benchmarking.
Ting Yu Tsai, An Yu, Lucy Lee +5
University at Albany, SUNY · University of Louisville · GE HealthCare
Scaling Transformers has driven large gains in language modeling, but transplanting this to behavior-sequence modeling in production ranking is challenging: recommendation differs in signal quality, where behavior sequences are noisy, temporally irregular, and sparsely supervised, and in computation asymmetry, where each request scores many candidates against one shared user history under tight latency budgets. We propose ReST, a recommendation-native Transformer scaling framework. For signal quality, it introduces a sequence encoder with dual-gated attention, rotary positional and temporal embedding, stabilized residual normalization, and training-only auxiliary objectives. For computation asymmetry, it factorizes ranking into a heavy reusable encoder and a lightweight cross decoder with projection-free KV attention and token-specific parameterization, coupling user-level shared-prefix training with shared-prefix serving for compute-once, decode-many-times ranking. Across industrial and public benchmarks, ReST achieves higher accuracy and scales more consistently along sequence length, depth, and width, where LLM-style Transformer blocks saturate. A one-week online A/B test on a production advertising platform improves online AUC by 1.31% and lifts a core revenue metric by 11.93% within a 50 ms P99 budget; ReST has since been fully deployed in production, showing that behavior-sequence scaling remains a promising, under-exploited axis for production ranking.