Understanding freely moving animal behavior is central to neuroscience, where pose estimation and behavioral understanding form the foundation for linking neural activity to natural actions. Yet both tasks still depend heavily on human annotation or unstable unsupervised pipelines, limiting scalability and reproducibility. We present BehaviorVLM, a unified vision-language framework for pose estimation and behavioral understanding that requires no task-specific finetuning and minimal human labeling by guiding pretrained Vision-Language Models (VLMs) through detailed, explicit, and verifiable reasoning steps. For pose estimation, we leverage quantum-dot-grounded behavioral data and propose a multi-stage pipeline that integrates temporal, spatial, and cross-view reasoning. This design greatly reduces human annotation effort, exposes low-confidence labels through geometric checks such as reprojection error, and produces labels that can later be filtered, corrected, or used to fine-tune downstream pose models. For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery, VLM-based per-clip video captioning, and LLM-based reasoning to merge and semantically label behavioral segments. The behavioral pipeline can operate directly from visual information and does not require keypoints to segment behavior. Together, these components enable scalable, interpretable, and label-light analysis of multi-animal behavior.
Figures & tables
Figure 1: Overview of BehaviorVLM. This VLM & LLM-based framework addresses pose estimation and behavioral understanding with minimum manual labeling and no finetuning.
Figure 2: Pose estimation experimental setup. (A) Data collection: a mouse injected with near-infrared quantum dots (QDs) at 12 body keypoints is recorded by six synchronized NIR-optimized cameras (B) Example six-view frames with QD fluorescence centroids detected and overlaid as numbered candidates on the reflectance images. Centroid indices are local to each view; the goal is to assign anatomical identities to these candidates across all cameras and timepoints.
Figure 3: BehaviorVLM pose estimation pipeline and results. (A) pipeline overview. (B) Detailed example for one frame from camera 0: the VLM first localizes four body regions (ears, back, paws, tail) via bounding boxes, then assigns centroids to keypoints within each cropped region, merges assignments, and resolves conflicts. Six-view predictions are triangulated into 3D and refined via RANSAC consensus. (C) Ablation study showing mean 3D keypoint error (mm) averaged over 12 keypoints across 500 frames. The full BehaviorVLM pipeline (6.59 mm) outperforms variants without 3D cross-view refinement (9.16 mm) and without both body region detection and 3D refinement (14.29 mm), demonstrating the contribution of each component. (D) Representative 3D keypoint trajectories for four body keypoints: back_top , tail_tip , ear_R , and hindpaw_R . Ground truth is shown in orange, BehaviorVLM predictions in blue.
Figure 4: Overview of the BehaviorVLM pipeline for semantic behavioral understanding. Behavioral features are first over-segmented into fine-grained candidate clips for each animal. A vision-language model (VLM) then generates natural-language labels and descriptions for each clip. The mouse A0 segments shown here are the direct VLM-stage segments and are therefore more fine-grained than the final LLM-merged mouse A0 segments shown in Figure 5 .
Figure 5: Behavioral understanding results for video 3ZOUFPHJ7JOHFBE8RHY6 in the MABe2022 Mouse Triplets dataset. BehaviorVLM produces temporally coherent behavioral segmentation for each mouse. For every candidate segment, a vision-language model (VLM) first generates natural-language descriptions of the observed actions and interactions (Figure 4 , mouse A0 segments). A large language model (LLM) then refines and merges these descriptions into the final behavioral events shown here.
Vision-language models (VLMs) rely on storing and transferring appropriate information across their sub-components. Verifying that the VLMs exhibit desired behaviours, while avoiding harmful ones, is central to their reliable deployment. Yet, methods that identify VLM behaviours remain scarce. We present DiaVLo, a diagnostic framework that leverages human curation and VLMs' generation capabilities to construct specifications of desired and observed VLM behaviours, surfacing potential misalignments. Beyond this, DiaVLo also provides causal estimates to identify the most influential concepts steering VLM behaviours. We evaluate DiaVLo on several open-source VLMs under both classification and generation conditions. Our experiments show that DiaVLo produces behaviour labels that correlate with model performance and provide context for measured performance. DiaVLo surfaced behaviours that are clearly aligned and misaligned, alongside patterns in how VLMs perceive, organise, and prioritise concepts.
Animal pose estimation and tracking is important for wildlife monitoring and conservation research, and with limited expert time for labelling automated approaches are imperative. While human pose estimation and tracking has seen rapid progress thanks to large annotated datasets, animal pose remain challenging, due to large morphological and behavioural differences between species and limited annotated data. Existing approaches either optimise generic keypoint localisation from annotated datasets (such as APTv2) with poor generalisation, or track custom keypoints using visual tracking, at the cost of performance. In this paper, we demonstrate that vision foundation models trained on large datasets can be used effectively to track animal pose with limited labelled data. We propose two models, one unsupervised and the other supervised, to track user-selected keypoints in videos. The supervised approach delivers superior tracking accuracy by employing a keypoint prompt encoder to explicitly inject structural priors from a reference frame into feature matching. In parallel, the unsupervised route provides strong cross-species robustness by leveraging diverse foundation-model features for training-free correspondence matching. Extensive evaluation on challenging animal video benchmarks APTv2 and TigDog demonstrates that our framework achieves strong performance while maintaining an effective balance between accuracy and generalisation, offering a practical solution for real-world animal behaviour analysis and conservation applications.
Vision-Language-Action (VLA) models often suffer from performance degradation under distribution shifts, as they struggle to learn generalized behavior representations across varying environments. While existing approaches attempt to construct behavior representations through action-centric latent variables, they are often limited by short-horizon temporal fragmentation and static execution-alignment, leading to inconsistent behaviors in complex scenarios. To address these limitations, we propose BehaviorVLA, a framework that facilitates robust manipulation through the learning of a temporally coherent behavioral representations. Our approach features two symmetric components: (1) the Visuomotor Behavior Encoder (VBE), which utilizes a causal Mamba-based architecture to aggregate long-horizon trajectory information into a unified behavior representation; and (2) the Phase-conditioned Behavior Decoder (PBD), which decodes this representation into precise actions by dynamically aligning task-level priors with real-time execution progress. Experiments on RoboTwin 2.0, LIBERO, and CALVIN demonstrate state-of-the-art success rates of 58%, 98%, and 4.36 (Avg.Len), respectively. Notably, in real-world sim-to-real transfer, BehaviorVLA matches the performance of OpenVLA-OFT using only 50% of the demonstration data, showcasing its superior data efficiency and generalization.
Bing Hu, Zaijing Li, Rui Shao +4
Harbin Institute of Technology, Shenzhen · PengCheng Laboratory · Shenzhen Loop Area Institute +2