Natural behavior evolves continuously and often reuses movement structure across contexts, but existing behavioral segmentation methods typically assign a single discrete syllable at each time point or impose restrictive generative assumptions. We introduce Motif-based Continuous Dynamics (MCD), a computational framework that models behavior as time-varying combinations of reusable behavioral motifs. MCD learns motifs as latent behavior-level features from transition dynamics, through spectral representation learning within a reinforcement learning framework. MCD then infers policies that combine these motifs across time or context, through offline imitation learning. This formulation avoids hard behavioral boundaries while capturing continuity and compositionality. Across simulated gridworld and continuous-control tasks, mouse maze navigation and pose data, MCD identifies reusable motifs that explain task and temporal variation. MCD therefore provides a flexible and interpretable framework for computational behavior analysis. Code is available in https://github.com/G1NO3/MCD.
Figures & tables
Figure 1: MCD models behavior as continuous compositions of reusable motor motifs. For example, “side grooming” may emerge from the combination of grooming and turning-side motifs.
Figure 2: A. State-action map: each grid contains four action values (Up, Down, Left, Right). In task i , rewards are assigned to actions moving toward the i th location. B. Explained variance of the top 15 principal components; variance drops near zero after PC7. C. Left: ground-truth rewards for all 9 tasks (yellow indicates high reward and green indicates low reward). Right: recovered rewards. D. Task-specific reward weights w and the top 8 principal-component motifs derived from ϕ .
Figure 3: The maze contains 63 T-junctions and 64 terminal nodes. The central root is connected to the home cage. The water port is placed at one terminal node. At each T-junction, an action denotes a transition to one of the adjacent nodes (the parent or the left/right child). The house and droplet icons indicate the home cage and the water port, respectively. A. Ground-truth and recovered rewards for the three behavioral modes. Darker colors denote higher reward. B. Dominant motifs for the water-seeking and home-seeking behaviors. Orange arrows denote high-value motifs. Darker colors denote higher state values v(s) defined in the main text. C. Task-specific coefficients w for all motifs (blue, orange, and green indicate water seeking, home seeking, and exploration, following A ). Positive coefficients w indicate a motif contributes highly to that recovered reward. The two blue stars highlight the two water-seeking motifs in B , and orange stars highlight home-seeking motifs in B .
Figure 4: A. Boxplot for human-annotated label prediction accuracy (top) and trajectory discrimination AUC (bottom). AUC is evaluated on 10 sequences (length=100) with 1000 negative samples per sequence. B. Example motif weights u(t) and representative motifs inferred by MCD. The middle "clip" row shows real animal behavior clips. C-D. Latent representations and segmentation results for SemiSeg and OPAL. E-F. Segmentation results and representative syllables from Keypoint-MoSeq ( Weinreb et al., 2024 ) .
Figure 5: A. The strokes can be categorized into different primitives, and the primitives are reused in different characters. B. Primitive HQ coverage for the test set. Bars show HQ rates. Red error bars show 95% bootstrap intervals. All methods use the same evaluation strokes.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: A. State-action maps for all 64 motifs. B. Reward weight w for all 64 motifs.
Figure 7: A. Left. The value for each (s,a) pair in motif 0 before PCA. Right. The most important (s,a) pairs. B. Left. The value for each (s,a) pair in motif 0 after PCA. Right. The most important (s,a) pairs. C. Boxplot for the effective dimensions before and after PCA.
Figure 8: A. Policy weight u(t) for all 64 motifs. B. The motion field for all 64 motifs learned from the video, computed by averaging the states and actions that most strongly activate each motif.
Subset
Contents
Purpose
Isolated primitive basis
413 isolated strokes from 19 predefined single-shape types
MCD training and K-means template construction
Practised characters
9,981 strokes from 1,071 character identities
MCD training
Novel characters
2,040 strokes from 665 character identities
Held-out evaluation
Primitive controls
3,577 strokes from separate isolated and grid trials
Similarity-threshold calibration
Appendix
Table 1: Data subsets constructed from Diego’s drawing trajectories. Calibration and test strokes are excluded from MCD training.
Method
Full test set
Shared subset
Label-based DTW
85.0% (83.5–86.6%)
90.3% (87.0–93.2%)
MCD
80.2% (78.5–81.8%)
86.9% (83.6–90.1%)
Untrained MCD
65.1% (63.1–67.1%)
70.9% (66.2–75.5%)
Appendix
Table 2: Primitive-matching HQ rates on Diego’s novel drawings, with 95% bootstrap intervals. All methods use the same evaluation strokes within each subset. MCD and untrained MCD use K-means templates without primitive-type labels. Their results are averaged over ten clustering seeds.
Figure 9: A. Reward maps inferred by different models. The maps from DIRL are ranked based on the time-varying weights (right at the corresponding row). "Water" weight increases and then decreases; "home" weight remains low and then increases; and "exploration" weight remains low. B. Correlations across the reward maps.
Figure 10: Motif weights for a clip from the lunar lander dataset.
Figure 11: For each PC motif, we collect the data entries that most strongly activate it, and show the distribution of positions, velocities, angle, and angle velocities.
Representation
Boundary entropy
Interior entropy
Interior top-1 mass
Continuous MCD u(t)
0.7915
0.7877
0.1126
Smoothed discrete labels
0.2640
0.0062
0.9978
Appendix
Table 3: Frame-weighted mixing statistics near behavior changes and within stable bouts. Boundary frames use a ±5 -frame window.
Representation
Decoding accuracy
Continuous MCD
0.9615±0.0018
Hard 64 -state K-means
0.5761±0.0099
Smoothed 64 -state K-means
0.6216±0.0056
Majority-class baseline
0.5566
Appendix
Table 4: Human-label decoding accuracy. Model results are mean ± standard deviation over ten seeds.
Figure 12: Reward maps generated by models of different motif dimension d in the gridworld dataset.
Figure 13: Reward maps generated by models of different motif dimension d in the labyrinth navigation dataset.
Figure 14: AUC generated by models of different motif dimension d in the animal free-moving dataset.
Figure 15: AUC generated by models trained under different noise distribution in the animal free-moving dataset.
Figure 16: AUC generated by models trained using different numbers of negative samples for motif contrastive learning in the animal free-moving dataset. nn=k means the n umber of n egative samples per positive sample is k .
Figure 17: AUC generated by models trained using different train-test split methods. For MCD(random), the dataset is randomly split. For MCD(heldout), the behavior of one mouse is specially held out as the test set while the rest is taken as the training set.
Figure 18: AUC generated by models tested using different Gaussian random walk prior coefficients in the animal free-moving dataset.
Figure 19: Decoding accuracy ( A ) and AUC ( B ) generated by models tested using different downsampling rates in the animal free-moving dataset. C. CKA under different downsampling rates for MCD in the same dataset.
Figure 20: Motif-specific rollout trajectories generated by HMC sampling from the trained energy model. Indexes are consistent with Fig. 8 .
Figure 21: Rollout trajectories based on inferred u(t) generated by HMC sampling from the trained energy model.
Understanding animal behavior at an algorithmic level -- what animals attend to, how they form internal representations, and how this maps to action -- remains a central challenge in neuroethology. We explore how well large generative models capture properties of fly social behavior. We combine transformers with agent-centric representations. Models input egocentric sensory observations and output egocentric movements, mirroring biological constraints. Social behavior emerges from agents independently sensing and responding to one another. We demonstrate that this simple model captures many properties of fly behavior, outperforming the world-centric representation and achieving SOTA on MABe 2022-fly. We systematically evaluate properties of the representation important for capturing the behavior distribution through a series of lesioning experiments. These comparisons were enabled by a new, general-purpose library for animal-centric pose forecasting.
Understanding how animals move between behaviors is a central question in neuroethology, yet most segmentation methods first partition time into discrete states, leaving transitions as the unmodeled residue between them. We take the opposite view: in a suitable feature space, sustained behaviors are dense regions of the trajectory's density landscape, and transitions are the sparse corridors and loops that connect them. Because this structure is a property of the landscape's shape rather than of any particular coordinates, it can be read off directly with topology and should persist across feature spaces. We build this landscape from frame-by-frame trajectories, extract its discrete-Morse graph, and keep two kinds of structure: paths, the corridors a trajectory follows between dense behavioral regions, and cycles, closed routes whose entry and exit take different ways. Clustering these structures yields transitions directly, with no upstream state partition. On a rat behavior dataset across three feature spaces, including the latent space of a neural-network controller that imitates the animal in a physics simulator, the approach recovers transitions at coverage comparable to state-space and topological-clustering baselines, while carrying substantially more information about the transition identity, forming clusters at least as tight as the baselines', and matching annotated transitions with less reliance on time-warping. More broadly, the results show that behavioral transitions are a topological feature of a representation: recoverable from the shape of its trajectories, and consistent across distinct feature spaces.
Kaiwen Bian, Eric J. Leonardis, Yuanjia Yang +3
University of California San Diego · Salk Institute for Biological Studies · Harvard University
Behavior policies are often formulated as continuous generative models, whose iterative denoising processes are expressive but difficult to interpret and prone to producing implausible actions. We propose the Large Discrete Policy (LDiP), a fully discrete behavior modeling framework that selects actions from a large vocabulary of physically plausible candidates. Rather than perturbing actions, LDiP improves expressivity through stochastic iterative scoring: it progressively re-scores and prunes candidates with score-space stochasticity, enabling fine-grained ranking and exploration among plausible actions while preserving an explicit decision process. Across end-to-end planning, closed-loop driving, robotic manipulation, and vision-language-action settings, LDiP consistently outperforms strong discrete and continuous baselines in autonomous driving, and exceeds or matches continuous generative policies in robotic manipulation. These results show that discrete policies, when equipped with effective scoring mechanisms, offer an expressive, plausible, and interpretable alternative for behavior modeling. Project website: https://zhenxinli.net/LargeDiscretePolicy/.