cs.CVMar 13, 2025
SaveMouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis
Organizations: School of Information Science and Technology, ShanghaiTech University, 201210, Shanghai, China. · LumiAni Technology, 201210, Shanghai, China. · School of Life Science and Technology, ShanghaiTech University, 201210, Shanghai, China. · Laboratory of Chemical Biology, Changchun Institute of Applied Chemistry, Chinese Academy of Sciences, Changchun, 130022, Jilin, China. ·
Abstract
Analyzing animal behavior is crucial in neuroscience, yet deciphering its intricate dynamics remains challenging. Traditional machine vision approaches detect spontaneous behaviors but are constrained by limited interpretability and reliance on manual labeling, restricting exploration of the full behavioral spectrum. Here, we introduce MouseGPT, an integrated vision-language model and analytical framework that translates raw video of animal activity into natural language to automate open-vocabulary behavior analysis. Built upon a comprehensive dataset of over 42 million frames spanning diverse psychiatric conditions, MouseGPT maps multi-view or single-camera recordings directly into computable semantic representations. An agentic framework leverages these representations for detailed behavior profiling, cross-view stable clustering, and zero-shot novel behavior discovery without requiring labor-intensive annotation. Surpassing conventional kinematics-based pipelines and generalist vision-language models in behavioral precision and adaptability, MouseGPT establishes continuous semantic representations as a powerful tool for deciphering complex ethological dynamics in animal models.
Figures & tables
Fig. 1: MouseGPT, a comprehensive mouse behavior understanding model and analysis framework. By integrating single-camera or multi-view video data, pose estimation, and kinematic embeddings, MouseGPT generates open-vocabulary ethological descriptions of mouse behaviors. It leverages a visual-guided prompt engine and a pioneering open-vocabulary behavioral dataset, featuring disease-specific behavior phenotypes (e.g., hallucination, depression, schizophrenia). The system supports behavior classification, temporal analysis, fine-grained behavior profiling, and novel behavior discovery, delivering transformative capabilities for studying and analyzing complex animal behaviors.
Fig. 2: MouseGPT Behavior Understanding Model. a , Data Preparation: A multi-view 3D capture system generates keypoints and limb velocities for each frame, with training data produced by GPT-4o and refined through automated selection processes. b , Vision-Language Model Training and Inference Workflow: MouseGPT-Large is optimized for detailed behavior analysis, while MouseGPT-Lite provides a lightweight alternative for streamlined tasks. c , Performance Evaluation: A blind panel of 15 independent senior researchers evaluated model outputs for factual correctness. The assessment confirms the rigorous quality of the GPT-4o training annotations (78.6% baseline true rate) and highlights MouseGPT’s superior performance against both open-source models of comparable size and GPT-4o. d , Comparison with Other VLMs: Experts performed blinded, paired comparisons of descriptions generated by MouseGPT and other vision-language models for the same input. The plot reports the proportions of judgments in four categories: Win, Both Good, Both Bad and Lose. e , Dataset Visualization: UMAP projection of the MouseGPT dataset in 2D, with color-coded clusters representing distinct drug treatments.
Fig. 3: MouseGPT Behavior Analysis Framework. a , Data Flow: MouseGPT converts open-vocabulary behavior descriptions into two computable features: high-dimensional embeddings for semantic analysis and keyword-based summaries for interpretability, powering various downstream applications. b , Major Behavior Profiling: A two-stage LLM-enhanced clustering groups mouse actions into distinct behavioral categories, with colors representing clusters, enabling robust profiling. c , Fine-Grained Analysis: MouseGPT extracts and quantifies detailed behavior patterns via keyword analysis and frequency calculations, offering precise insights. d , Novel Behavior Discovery: Using Isolation Forest in embedding space, MouseGPT identifies anomalous behaviors, visualized in 2D UMAP to highlight rare actions. e , Phenotype Prediction: MouseGPT combines behavioral analysis with expert knowledge to predict drug treatments, aiding in phenotype understanding and experimental outcomes.
Fig. 4: MouseGPT Analyzes Distinct Phenotypes of Spontaneous Behaviors Induced by psychoactive compounds.
Fig. 5: MouseGPT Natural Language User Interface. a , Interactive Interface: Users engage with the MouseGPT agent through natural language to perform clustering, behavior search, and automated code generation and execution, enabling intuitive interaction. b , Agent Workflow: The agent integrates data access, MouseGPT model invocation, and analysis tasks via a unified workflow, streamlining complex behavior analyses. c , Toolbox Overview: A comprehensive suite of tools supports major behavior profiling, fine-grained analysis, novel behavior detection, temporal analysis, and phenotype prediction. d , Python REPL Integration: The agent dynamically generates and executes Python scripts to process data, exemplifying seamless automation. e , Behavior Search: Embedding-based retrieval matches user-specified behaviors to the database, ranking results by similarity for precise behavior identification.
Extended Fig. 1: Data samples and structured behavioral descriptions in MouseGPT. An example of a single sample from the MouseGPT dataset, illustrating how multiview images, three-dimensional joint keypoints and velocity data are combined to produce structured, open-vocabulary behavioral descriptions. The figure demonstrates the dataset’s ability to capture detailed annotations for overall behavior, head orientation, limb positioning, torso posture and additional traits, summarized through concise keywords.
Extended Fig. 6: Semantic embedding distribution and examples of major behaviors. a , Distribution of major behaviors in MouseGPT description embeddings: UMAP visualization of behavioral description embeddings from the hallucinogen experiment dataset, colored by the major behavior categories summarized by MouseGPT. b , Representative images of major behavior categories: Representative examples of each behavior category.
Extended Fig. 7: Temporal profiles of fine-grained postures across treatment groups. a , Temporal profiles of head postures across treatment groups: Ethograms showing head posture features at each time point throughout the 20-minute recording for all mice in the Saline, LSD, MK-801 and Psilocybin groups. b , Temporal profiles of forepaw states across treatment groups: Ethograms showing limb posture features at each time point throughout the 20-minute recording for all mice in the four groups. c , Temporal profiles of torso postures across treatment groups: Ethograms showing torso posture features at each time point throughout the 20-minute recording for all mice in the four groups.
Extended Fig. 8: Example interactions with the MouseGPT agent. An example of a natural-language interaction demonstrating the capabilities of the MouseGPT agent. The dialogue shows user requests for behavior-specific frame retrieval, clustering analysis and code generation for analyzing behavioral transitions. The agent effectively combines tool invocation, data processing and execution to provide results that support subsequent operations, illustrating its versatility and ease of use for mouse behavior analysis tasks.
Extended Fig. 4: Evaluation of cross-view stability in MouseGPT behavior classification. a , Frame proportions of Keypoint-MoSeq behavioral syllables. Frame proportions of 20 Keypoint-MoSeq syllables derived from three-dimensional keypoint trajectories, ranked by occupancy. A few syllables account for most frames. b , Semantic behavior composition within individual syllables. MouseGPT semantic labels within major syllables; bubble area denotes the proportion of each behavior category. c , Examples from the highest-occupancy syllable: randomly sampled frames showing different behaviors (left) and subsets distinguished by MouseGPT semantic labels (right). d , Cumulative syllable occupancy for Keypoint-MoSeq applied with identical settings to the same video using fused three-dimensional, top-view two-dimensional or side-view 1/2 two-dimensional keypoints. Syllables are ranked within each condition; distributions differ across viewpoints. e , Behavioral syllable correlations across viewpoints. Pearson correlations of syllable time series for 3D–top, 3D–side 1, 3D–side 2 and side 1–side 2 comparisons. Following the vertical-axis order, each syllable is matched to the highest-correlated unmatched syllable in the other condition, reordering the horizontal axis. Sparse high correlations indicate limited cross-view correspondence. f , MouseGPT behavior proportions for View 2+4 and View 3+5 show similar category distributions. g , Semantic behavior correlations across viewpoint pairs. Pearson correlations of semantic label sequences between the viewpoint pairs. Explicit semantic categories align the axes directly; the diagonal indicates agreement for corresponding behaviors. h , Semantic behavior time series across viewpoint pairs: Ethograms show preserved behavior onset, duration and transitions across viewpoint pairs.
Extended Fig. 3: Benchmarking visual descriptions of mouse behavior. a , Benchmark overview: 435 manually selected frames from 25 recordings with expert-verified, AI-assisted references and automated grading. Eight vision–language models were evaluated across 14 settings using the same camera views and velocity-bearing prompts. A fixed GPT-6.1 Sol judge configuration graded all 6,090 descriptions. b , Examples illustrating the benchmark’s behavioral diversity: synchronized views of less-frequent, challenging lying belly-down and hopping postures. c , Four rubric dimensions: factual correctness ( ), required-fact coverage ( ), structure ( ) and instruction adherence ( ). Semantic-field-weighted correctness and coverage combine as , with when . The MouseGPT Benchmark Index is . Components range from 0 to 1; indices are calculated per frame before averaging. Structure and instruction contributions depend on factual quality. d , Rubric scores ( ) for the conditions with the lowest (GPT-4o) and highest (GPT-6 Astra, max) mean indices. e , Mean index across all 435 frames for each condition. Parentheses indicate requested reasoning effort; output-token limits also varied by setting. f , Mean index for all 435 frames and a shared subset of 44 challenging frames (lowest 10%, rounded up). Frames were ranked by factual , first averaging settings within each model and then averaging the eight models equally. The same subset was used for every condition. Selection used these same outputs post hoc; the overlapping-group comparison is descriptive, not independent validation. Dots and numerical labels show means. Whiskers in d,e show 95% percentile confidence intervals from 10,000 recording-cluster bootstrap resamples: 25 recordings were sampled with replacement, retaining all their frames and calculating frame-weighted means.
Extended Fig. 9: Evaluation of task execution performance in the MouseGPT agent. Schematic of the standardized agent benchmark. Left, five independent evaluation datasets, with dataset identifiers shown. Middle, 20 benchmark tasks spanning 10 functional categories; full prompts are provided in Supplementary Table 3. Right, dataset-level success rates and failed task identifiers. A trial was judged successful upon manual verification that the agent correctly understood the request, invoked appropriate tools and produced a valid final output. Twenty tasks were performed on each of the five datasets, yielding 100 trials and an overall success rate of 87% (87/100); dataset-level success rates were 90%, 80%, 95%, 90% and 80%, respectively.
Extended Fig. 5: Evaluation of MouseGPT language-based representations on public social behavior datasets. a , Example single-camera CalMS21 images and MouseGPT-Lite descriptions of single-animal activity (left) and social interaction (right). b , Expert evaluation of description quality on CalMS21. Five independent experts rated descriptions of 390 frames sampled using existing labels (Attack, Investigation, Mount and Other). Evaluation dimensions were Overall, Head, Torso, Interaction and Others, with Interaction added for this social setting. Heatmap values are mean 1–10 ratings normalized to [0,1]; the overall mean was 7.93/10. c , Supervised classification evaluation of semantic representations on CalMS21. The CalMS21 supervised classification protocol compares keypoint features with semantic embeddings of MouseGPT descriptions. Left, test-set macro F1; right, behavior-specific F1 for MouseGPT (Methods). Semantic embeddings yield a higher macro F1. d , Example video frames from MABe 2022 showing interactions among multiple mice, recorded from above at lower resolution, providing less visual detail. e , Mean F1 scores across methods on MABe 2022. The MABe 2022 evaluation compares semantic representations generated within MouseGPT using different general-purpose vision-language models with published results for AmadeusGPT (language-based analysis) and BAMS/PCA (behavior representations). Mean positive-class F1 across nine behaviors is shown; settings are detailed in Methods and Supplementary Table 2. f , F1 scores for individual behavior classes on MABe 2022. Behavior-specific F1 for the methods in e (rows, methods; columns, behaviors). GPT-6 Astra and GPT-5.1 settings score higher on Close, Contact, Approaches and Watching, and lower on Oral-genital, Oral-oral and Oral-ear.
Extended Fig. 2: Evaluation of cross-tool compatibility and the role of kinematic input in MouseGPT. a , Pearson correlations between instantaneous keypoint speeds derived from DeepLabCut (DLC, blue) or SLEAP (orange) and those from MouseGPT’s built-in pose estimator. Black line segments indicate the difference between the two tools’ correlation coefficients for each keypoint. The keypoints used to calculate MouseGPT’s kinematic inputs—the snout tip, mid-tail, left forepaw tip and both hindpaw tips—are marked by open inverted triangles (see Supplementary Table 1 for keypoint definitions). b , Root mean square error (RMSE, m/s) of DLC- and SLEAP-derived keypoint speeds relative to those from the built-in pose estimator. Colors and symbols are as in a. c , Kinematic inputs derived from the built-in pose estimator were replaced with those derived from DLC or SLEAP. A large language model (LLM) judge evaluated differences between behavioral descriptions generated for the same samples before and after replacement (N = 5,992 and 5,988, respectively). Differences were classified as L0 (semantically equivalent), L1 (minor differences without changing the core behavior), L2 (speed or intensity differences) or L3 (action differences). Descriptions remained semantically equivalent or differed only slightly in 90.39% and 87.49% of samples following replacement with DLC and SLEAP inputs, respectively (L0+L1, shown together). d , Behavioral descriptions generated by MouseGPT-Lite with and without kinematic information in the prompt were compared on the same test set (N = 5,994). The LLM judge classified differences using the L0–L3 categories defined in c. Removing kinematic information changed speed or action descriptions in 41.99% of samples (L2+L3), indicating that kinematic input influences how the model describes movement speed and actions. e , Human evaluation of 109 samples randomly selected from the L2/L3 cases in d. Raters viewed the original images and judged whether descriptions with speed information were better (pink), descriptions without speed information were better (purple), both were good (olive green) or both were poor (dark gray).
Supplementary Fig. 1: Expert evaluation interface for behavioral descriptions. An interface designed for expert evaluation of MouseGPT’s performance. a , Interface for human evaluation of description correctness: The interface allows experts to assess the correctness of model-generated behavioral descriptions. b , Interface for human comparison of descriptions from two models: Expert evaluation interface for paired comparisons between MouseGPT and other models.
Supplementary Fig. 2: Interactive workflow for drug treatment prediction from behavioral phenotypes. An example of the step-by-step process used by MouseGPT to predict the drug treatment administered to a mouse from behavioral phenotype data. The procedure involves standardizing behavioral data, calculating Euclidean distances between feature vectors and identifying the closest match among known treatment groups. The workflow combines computational tools and statistical analysis to generate a prediction of drug treatment.
Supplementary Fig. 3: Overview of processing pipelines in MouseGPT and conventional kinematics-based unsupervised methods. Schematic comparison of input requirements, intermediate representations and primary outputs. Keypoint-MoSeq operates on kinematic trajectories and produces behavioral syllables that have not yet been assigned semantic names and require subsequent semantic interpretation. MouseGPT operates directly on raw videos (multiview input is optional) and generates open-vocabulary semantic descriptions or labels, supporting flexible, language-configurable behavioral categorization.
Supplementary Fig. 4: Comparison of MouseGPT Behavior Understanding Model and Other Vision-Language Models. A comparative result of behavior descriptions generated by MouseGPT-Large, MouseGPT-Lite, GPT-4o, InternVL2-Llama3-76B, and MiniCPM-2.6. The figure demonstrates the differences in descriptive detail, accuracy, and interpretability across models for key behavioral components such as overall activity, head orientation, limb positioning, and torso posture.
Supplementary Fig. 5: Zero-shot rare-event example: MouseGPT surfaces an unusual frame as a semantic outlier. The model successfully flags a rare event (wall-climbing) in the continuous embedding space and generates a structured open-vocabulary, body-part-resolved description without relying on predefined labels (verbatim output shown).
Supplementary Fig. 6: Single-camera inference example of MouseGPT-Lite. Representative single-camera frames paired with verbatim MouseGPT-Lite structured descriptions, including head orientation, paw contact state and torso posture. For this qualitative demonstration, one side-view image was supplied as visual input together with the corresponding kinematic summary used by the standard inference prompt. No additional fine-tuning, retraining or architecture modification was introduced.
| Index | Label | Detailed Description |
|---|---|---|
| 1 | EarL | Tip of the Left Ear |
| 2 | EarR | Tip of the Right Ear |
| 3 | Snout | Tip of the Snout |
| 4 | SpineF | Beginning of the Spine |
| 5 | SpineG | First Tripartite Point of the Spine |
| 6 | SpineH | Second Tripartite Point of the Spine |
Supplementary Table 1: Mouse body keypoint labels and anatomical definitions.
| Metric / Setup | Our framework w/ GPT-6 Astra | Our framework w/ GPT-5.6-Luna | Our framework w/ GPT-5.1 | Our framework w/ Doubao-1.6-lite | AmadeusGPT | BAMS | PCA baseline |
|---|---|---|---|---|---|---|---|
| Model type | Generalist VLM perception module | Generalist VLM perception module | Generalist VLM perception module | Generalist VLM perception module | Language-guided, rule-based task programming | Learned behavioral representation | PCA representation baseline |
| Data coverage | Same one-fifth subset of MABe 2022 | Full local evaluation condition | Same one-fifth subset of MABe 2022 | Full local evaluation condition | Published MABe Round 1 evaluation split | Published evaluation split | Published evaluation split |
| Train / test split | Custom source-video-level train/test split | Shared custom 80/20 video-level split | Custom source-video-level train/test split | Shared custom 80/20 video-level split | As reported in the reference | As reported in the reference | As reported in the reference |
| Input / clip setting | Video clips; archived segment lengths: 15–1,800 frames | Video clips; archived segment lengths: 15–1,800 frames | Video clips; archived segment lengths: 15–1,800 frames | Video clips; archived segment lengths: 15–1,800 frames | Pre-computed keypoints; natural-language behavior definitions | Keypoint trajectories | Keypoint trajectories |
| Representation for scoring | Description JSON text-embedding-3-large PCA (128D) | Description JSON text-embedding-3-large PCA (128D) | Description JSON text-embedding-3-large PCA (128D) | Description JSON text-embedding-3-large PCA (128D) | Rule-based behavior events generated by task programs | Learned representations, as reported | PCA features, as reported |
| Downstream evaluation | Per-behavior Ridge classification; archived frame-label scoring | Per-behavior Ridge classification; archived frame-label scoring | Per-behavior Ridge classification; archived frame-label scoring | Per-behavior Ridge classification; archived frame-label scoring | Rule-based detection; F1 reported in Table 1 | Benchmark evaluation reported in Table 1 | Benchmark evaluation reported in Table 1 |
Supplementary Table 2: Experimental settings and results for the MABE 2022 benchmark.
| Category | Task ID | Prompt (verbatim) |
|---|---|---|
| Basic Behavior Query | Task 1 | “What behavior is the mouse performing in the current frame? Please explain the reasoning.” |
| Basic Behavior Query | Task 2 | “Show me the five frames that are most similar to the current frame’s posture.” |
| Behavior Classification | Task 3 | “Please classify the mouse behavior across the entire video and summarize the percentage of each class.” |
| Behavior Classification | Task 4 | “Generate a detailed ethogram for the full recording and save it as an output file.” |
| Behavior Clustering | Task 5 | “Cluster all mouse behaviors.” |
| Behavior Clustering | Task 6 | “Compare cluster 2 with cluster 5 and describe how their postures differ.” |
Supplementary Table 3: Tasks and full prompts for the MouseGPT agent benchmark.
More in this topic
Animal Behavior Analysis
Animal behavior analysis uses computational tools to measure how animals move, interact, communicate, and respond to their environments.
See all 37 papers