Animal Behavior Analysis

Understanding freely moving animal behavior is central to neuroscience, where pose estimation and behavioral understanding form the foundation for linking neural activity to natural actions. Yet both tasks still depend heavily on human annotation or unstable unsupervised pipelines, limiting scalability and reproducibility. We present BehaviorVLM, a unified vision-language framework for pose estimation and behavioral understanding that requires no task-specific finetuning and minimal human labeling by guiding pretrained Vision-Language Models (VLMs) through detailed, explicit, and verifiable reasoning steps. For pose estimation, we leverage quantum-dot-grounded behavioral data and propose a multi-stage pipeline that integrates temporal, spatial, and cross-view reasoning. This design greatly reduces human annotation effort, exposes low-confidence labels through geometric checks such as reprojection error, and produces labels that can later be filtered, corrected, or used to fine-tune downstream pose models. For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery, VLM-based per-clip video captioning, and LLM-based reasoning to merge and semantically label behavioral segments. The behavioral pipeline can operate directly from visual information and does not require keypoints to segment behavior. Together, these components enable scalable, interpretable, and label-light analysis of multi-animal behavior.

Monitoring the health conditions of fish is essential, as it enables the early detection of disease, safeguards animal welfare, and contributes to sustainable aquaculture practices. Physiological and pathological conditions of cultivated fish can be inferred by analyzing locomotion activities. In this paper, we present a system that estimates the locomotion activities from videos using multi object tracking. The core of our approach is a YOLOv11 detector embedded in a tracking-by-detection framework. We investigate various configurations of the YOLOv11-architecture as well as extensions that incorporate multiple frames to improve detection accuracy. Our system is evaluated on a manually annotated dataset of Sulawesi ricefish recorded in a home-aquarium-like setup, demonstrating its ability to reliably measure swimming direction and speed for fish health monitoring. The dataset will be made publicly available upon publication.

Despite significant analysis of bird flight, generative physics models for flight dynamics do not currently exist. Yet the underlying mechanisms responsible for various flight manoeuvres are important for understanding how agile flight can be accomplished. Even in a simple flight, multiple objectives are at play, complicating analysis of the overall flight mechanism. Using the data-driven method of dynamic mode decomposition (DMD) on motion capture recordings of hawks, we show that multiple behavioral states such as flapping, turning, landing, and gliding, can be modeled by simple and interpretable modal structures (i.e. the underlying wing-tail shape) which can be linearly combined to reproduce the experimental flight observations. Moreover, the DMD model can be used to extrapolate naturalistic flapping. Flight is highly individual, with differences in style across the hawks, but we find they share a common set of dynamic modes. The DMD model is a direct fit to data, unlike traditional models constructed from physics principles which can rarely be tested on real data and whose assumptions are typically invalid in real flight. The DMD approach gives a highly accurate reconstruction of the flight dynamics with only three parameters needed to characterize flapping, and a fourth to integrate turning manoeuvres. The DMD analysis further shows that the underlying mechanism of flight, much like simplest walking models, displays a parametric coupling between dominant modes suggesting efficiency for locomotion.

We present Rodent-Bench, a novel benchmark designed to evaluate the ability of Multimodal Large Language Models (MLLMs) to annotate rodent behaviour footage. We evaluate state-of-the-art MLLMs, including Gemini-2.5-Pro, Gemini-2.5-Flash and Qwen-VL-Max, using this benchmark and find that none of these models perform strongly enough to be used as an assistant for this task. Our benchmark encompasses diverse datasets spanning multiple behavioral paradigms including social interactions, grooming, scratching, and freezing behaviors, with videos ranging from 10 minutes to 35 minutes in length. We provide two benchmark versions to accommodate varying model capabilities and establish standardized evaluation metrics including second-wise accuracy, macro F1, mean average precision, mutual information, and Matthew's correlation coefficient. While some models show modest performance on certain datasets (notably grooming detection), overall results reveal significant challenges in temporal segmentation, handling extended video sequences, and distinguishing subtle behavioral states. Our analysis identifies key limitations in current MLLMs for scientific video annotation and provides insights for future model development. Rodent-Bench serves as a foundation for tracking progress toward reliable automated behavioral annotation in neuroscience research.

Computer vision methods are increasingly used for the automated analysis of large volumes of video data collected through camera traps, drones, or direct observations of animals in the wild. While recent advances have focused primarily on detecting individual actions, much less work has addressed the detection and annotation of interactions -- a crucial aspect for understanding social and individualized animal behavior. Existing open-source annotation tools support either behavioral labeling without localization of individuals, or localization without the capacity to capture interactions. To bridge this gap, we present SiLVi, an open-source labeling software that integrates both functionalities. SiLVi enables researchers to annotate behaviors and interactions directly within video data, generating structured outputs suitable for training and validating computer vision models. By linking behavioral ecology with computer vision, SiLVi facilitates the development of automated approaches for fine-grained behavioral analyses. Although developed primarily in the context of animal behavior, SiLVi could be useful more broadly to annotate human interactions in other videos that require extracting dynamic scene graphs. The software, along with documentation and download instructions, is available at: https://silvi.ozankanbertay.com.

Monitoring cattle behaviour provides valuable insights into animal health, productivity and welfare. However, robust behaviour recognition in real-world farm environments remains challenging because well-annotated livestock video datasets are scarce and agricultural surveillance footage differs substantially from large-scale pretraining data. To address these challenges, we propose Cattle-CLIP, a domain-adaptive vision-language framework based on Contrastive Language-Image Pretraining (CLIP), which reformulates cattle behaviour recognition as a cross-modal semantic alignment task. Cattle-CLIP incorporates a temporal integration module that extends image-level contrastive pretraining to video-based behaviour understanding and promotes consistent semantic alignment across time. To reduce domain discrepancy, customised data augmentation and behaviour-oriented prompts are employed. We also construct CattleBehaviours6, a curated dataset containing 1,905 annotated clips from six indoor behaviours: feeding, drinking, standing-self-grooming, standing-ruminating, lying-self-grooming and lying-ruminating. The dataset provides standardised ethogram definitions for model training, evaluation and future livestock behaviour research. Cattle-CLIP was evaluated under supervised and few-shot learning settings, with particular emphasis on data-constrained behaviour recognition. In supervised experiments, the model achieved 96.1% accuracy across six behaviours and showed high precision and recall for feeding, drinking and standing-ruminating. It also maintained promising few-shot performance with limited labelled data, demonstrating the potential of multimodal learning for cattle behaviour recognition in data-constrained farm environments.

Analyzing animal behavior is crucial in neuroscience, yet deciphering its intricate dynamics remains challenging. Traditional machine vision approaches detect spontaneous behaviors but are constrained by limited interpretability and reliance on manual labeling, restricting exploration of the full behavioral spectrum. Here, we introduce MouseGPT, an integrated vision-language model and analytical framework that translates raw video of animal activity into natural language to automate open-vocabulary behavior analysis. Built upon a comprehensive dataset of over 42 million frames spanning diverse psychiatric conditions, MouseGPT maps multi-view or single-camera recordings directly into computable semantic representations. An agentic framework leverages these representations for detailed behavior profiling, cross-view stable clustering, and zero-shot novel behavior discovery without requiring labor-intensive annotation. Surpassing conventional kinematics-based pipelines and generalist vision-language models in behavioral precision and adaptability, MouseGPT establishes continuous semantic representations as a powerful tool for deciphering complex ethological dynamics in animal models.