Latent Dynamics Modeling

Ask about this topic

Overview

Latent dynamics modeling studies how to represent a changing system with hidden states and learn how those states evolve from observations such as video, sensor streams, or sparse measurements. Common approaches include autoencoders with state-space models, neural ODEs and SDEs for continuous-time or stochastic change, Koopman operators for approximately linear evolution, and JEPA-style world models for action-conditioned prediction. These models support forecasting and control in robotics and physical simulation, as well as analysis of biological, clinical, and neural time series. A central challenge is that good one-step prediction or reconstruction does not guarantee reliable long-horizon rollouts, useful control, or recovery of meaningful physical mechanisms; work therefore explores stability, identifiability, and physics-based structure while balancing these constraints against model flexibility.

Latest papers

CardsList

254 papers48 papers in the last four weeks, up 92% on the four weeks before. 0.3% of all new papers.

Apr 2, 2026

Koopman-Based Nonlinear Identification and Model Predictive Control of a Turbofan Engine

This paper investigates Koopman operator-based approaches for multivariable control of a two-spool turbofan engine. A physics-based component-level model is developed to generate training data and validate the controllers. A meta-heuristic extended dynamic mode decomposition is adapted, with a cost function designed to accurately capture both spool-speed dynamics and the engine pressure ratio (EPR), enabling the construction of a single Koopman model that can be reused across multiple control strategies. Using the identified time-varying Koopman model, an adaptive Koopman-based model predictive controller (AKMPC) with a disturbance observer is developed and compared with a Koopman-based feedback linearization controller (K-FBLC) and its integrator-augmented version (K-FBLC-I). The Koopman representation further enables nonlinear GTE output limiters, such as rotor-acceleration and turbine-inlet-temperature limits, to be expressed as linear constraints in the AKMPC. The controllers are evaluated for two control configurations of spool speeds and EPR, under both sea-level and varying flight conditions. The results demonstrate that the proposed identification approach enables accurate predictions of both spool speeds and EPR, allowing the Koopman model to be reused flexibly across different control formulations. While all strategies achieve comparable performance in sea-level conditions, the AKMPC demonstrates improved performance under varying flight conditions due to its ability to capture nonlinear dynamics, handle constraints, and compensate for model mismatch. Moreover, the EPR control strategy improves the thrust response. The study highlights the applicability of Koopman-based control and the advantages of the AKMPC framework for robust turbofan engine control.

David Grasev

Department of Aviation Technology, University of Defence, Brno, Czech Republic.

Adaptive MPCExtended Dynamic Mode DecompositionFeedback ControlModel Predictive ControlNonlinear MPCNonlinear System Identification+2
AskPDF
Apr 1, 2026

LAtent Phase Inference from Short time sequences using SHallow REcurrent Decoders (LAPIS-SHRED)

Reconstructing full spatio-temporal dynamics from sparse observations in both space and time remains a central challenge in complex systems, as measurements can be spatially incomplete and can be also limited to narrow temporal windows. Yet approximating the complete spatio-temporal trajectory is essential for mechanistic insight and understanding, model calibration, and operational decision-making. We introduce LAPIS-SHRED (LAtent Phase Inference from Short time sequence using SHallow REcurrent Decoders), a modular architecture that reconstructs and/or forecasts complete spatiotemporal dynamics from sparse sensor observations confined to short temporal windows. LAPIS-SHRED operates through a three-stage pipeline: (i) a SHRED model is pre-trained entirely on simulation data to map sensor time-histories into a structured latent space, (ii) a temporal sequence model, trained on simulation-derived latent trajectories, learns to propagate latent states forward or backward in time to span unobserved temporal regions from short observational time windows, and (iii) at deployment, only a short observation window of hyper-sparse sensor measurements from the true system is provided, from which the frozen SHRED model and the temporal model jointly reconstruct or forecast the complete spatiotemporal trajectory. The framework supports bidirectional inference, inherits data assimilation and multiscale reconstruction capabilities from its modular structure, and accommodates extreme observational constraints including single-frame terminal inputs. We evaluate LAPIS-SHRED on six experiments spanning complex spatio-temporal physics: turbulent flows, multiscale propulsion physics, volatile combustion transients, and satellite-derived environmental fields, highlighting a lightweight, modular architecture suited for operational settings where observation is constrained by physical or logistical limitations.

Yuxuan Bao, Xingyue Zhang, J. Nathan Kutz

Department of Applied Mathematics, University of Washington. · School of Environmental and Forest Sciences, University of Washington. · Autodesk Research, London, UK.

Spatiotemporal ForecastingLatent Dynamics ModelingRecurrent Neural Networks
AskPDF
Apr 1, 2026

Embedded Variational Neural Stochastic Differential Equations for Learning Heterogeneous Dynamics

This study examines the challenges of modeling complex and noisy data related to socioeconomic factors over time, with a focus on data from various districts in Odisha, India. Traditional time-series models struggle to capture both trends and variations together in this type of data. To tackle this, a Variational Neural Stochastic Differential Equation (V-NSDE) model is designed that combines the expressive dynamics of Neural SDEs with the generative capabilities of Variational Autoencoders (VAEs). This model uses an encoder and a decoder. The encoder takes the initial observations and district embeddings and translates them into a Gaussian distribution, which determines the mean and log-variance of the first latent state. Then the obtained latent state initiates the Neural SDE, which utilize neural networks to determine the drift and diffusion functions that govern continuous-time latent dynamics. These governing functions depend on the time index, latent state, and district embedding, which help the model learn the unique characteristics specific to each district. After that, using a probabilistic decoder, the observations are reconstructed from the latent trajectory. The decoder outputs a mean and log-variance for each time step, which follows the Gaussian likelihood. The Evidence Lower Bound (ELBO) training loss improves by adding a KL-divergence regularization term to the negative log-likelihood (nll). The obtained results demonstrate the effective learning of V-NSDE in recognizing complex patterns over time, yielding realistic outcomes that include clear trends and random fluctuations across different areas.

Sandeep Kumar Samota, Reema Gupta, Snehashish Chakraverty

Department of Mathematics, National Institute of Technology, Rourkela, India · Poverty Alleviation Research Centre, National Institute of Technology, Rourkela, India

Latent Dynamics ModelingTime Series ForecastingStochastic Differential EquationsVariational Autoencoders
AskPDF
Mar 27, 2026

Koopman Operator Identification of Model Parameter Trajectories for Temporal Domain Generalization (KOMET)

Parametric models deployed in non-stationary environments degrade as the underlying data distribution evolves over time (a phenomenon known as temporal domain drift). In the current work, we present KOMET (Koopman Operator identification of Model parameter Evolution under Temporal drift), a model-agnostic, data-driven framework that treats the sequence of trained parameter vectors as the trajectory of a nonlinear dynamical system and identifies its governing linear operator via Extended Dynamic Mode Decomposition (EDMD). A warm-start sequential training protocol enforces parameter-trajectory smoothness, and a Fourier-augmented observable dictionary exploits the periodic structure inherent in many real-world distribution drifts. Once identified, KOMET's Koopman operator predicts future parameter trajectories autonomously, without access to future labeled data, enabling zero-retraining adaptation at deployment. Evaluated on six datasets spanning rotating, oscillating, and expanding distribution geometries, KOMET achieves mean autonomous-rollout accuracies between 0.981 and 1.000 over 100 held-out time steps. Spectral and coupling analyses further reveal interpretable dynamical structure consistent with the geometry of the drifting decision boundary.

Randy C. Hoover, Jacob James, Paul May +1

Department of Electrical Engineering and Computer Science (EECS), South Dakota Mines, Rapid City, SD 57702, USA · Data Science and Engineering Program within EECS, South Dakota Mines, Rapid City, SD 57702, USA · Department of Mathematics, South Dakota Mines, Rapid City, SD 57702, USA

Extended Dynamic Mode DecompositionDrifting ModelsDomain GeneralizationLatent Dynamics ModelingKoopman Operator Learning
AskPDF
Mar 24, 2026

Neural ODE and SDE Models for Adaptation and Planning in Model-Based Reinforcement Learning

We investigate neural ordinary and stochastic differential equations (neural ODEs and SDEs) to model stochastic dynamics in fully and partially observed environments within a model-based reinforcement learning (RL) framework. Through a sequence of simulations, we show that neural SDEs more effectively capture the inherent stochasticity of transition dynamics, enabling high-performing policies with improved sample efficiency in challenging scenarios. We leverage neural ODEs and SDEs for efficient policy adaptation to changes in environment dynamics via inverse models, requiring only limited interactions with the new environment. To address partial observability, we introduce a latent SDE model that combines an ODE with a GAN-trained stochastic component in latent space. Policies derived from this model provide a strong baseline, outperforming or matching general model-based and model-free approaches across stochastic continuous-control benchmarks. This work demonstrates the applicability of action-conditional latent SDEs for RL planning in environments with stochastic transitions. Our code is available at: https://github.com/ChaoHan-UoS/NeuralRL

Chao Han, Stefanos Ioannou, Luca Manneschi +4

School of Computer Science, The University of Sheffield, UK · Cancer Research UK National Biomarker Centre, The University of Manchester, UK · School of Chemical, Materials and Biological Engineering, The University of Sheffield, UK +2

Model-Based RLLatent Dynamics ModelingPartially Observable RLNeural ODEsStochastic Differential EquationsNon-Stationary RL
AskPDF
Mar 24, 2026

Conditionally Identifiable Latent Representation for Multivariate Time Series with Structural Dynamics

We propose the Identifiable Variational Dynamic Factor Model (iVDFM) for multivariate time series. The model combines variational inference with partial identifiability guarantees up to a known ambiguity class. Innovations follow a conditional exponential-family prior over observed auxiliary context and regime embeddings. Linear diagonal dynamics map innovations to factors and preserve that class, so factors are partially identifiable up to permutation and component-wise affine maps. We train by maximizing the ELBO and estimate uncertainty in both latent trajectories and forecasts. Under explicit assumptions, we prove partial identifiability up to the stated class and provide the full proof in the appendix. We evaluate factor recovery on synthetic DGPs, intervention behavior on synthetic SCMs, and forecasting on real benchmarks with CRPS and MSE.

Minkey Chang, Jae-Young Kim

Latent Variable ModelsLatent Dynamics ModelingRepresentation Learning
AskPDF
Mar 24, 2026

Behavioral Heterogeneity as Quantum-Inspired Representation

Driver heterogeneity is often reduced to labels or discrete regimes, compressing what is inherently dynamic into static categories. We introduce quantum-inspired representation that models each driver as an evolving latent state, presented as a density matrix with structured mathematical properties. Behavioral observations are embedded via non-linear Random Fourier Features, while state evolution blends temporal persistence of behavior with context-dependent profile activation. We evaluate our approach on empirical driving data, Third Generation Simulation Data (TGSIM), showing how driving profiles are extracted and analyzed.

Mohammad Elayan, Wissam Kontar

University of Nebraska–Lincoln, Lincoln, NE, USA

State Representation LearningLatent Dynamics ModelingQuantum-Inspired ML
AskPDF
Mar 24, 2026

Generalizing Dynamics Modeling More Easily from Representation Perspective

Learning system dynamics from observations is a critical problem in many applications over various real-world complex systems, e.g., climate, ecology, and fluid systems. Recently, neural dynamics modeling method have become a prevalent solution that embeds the object's observations into a latent space before learning dynamics using neural methods such as neural Ordinary Differential Equations (ODE). Existing dynamics modeling methods induce a specific model for each observation of different complex systems, resulting in poor generalization across systems. Inspired by the great success of pre-trained models, we conduct a generalized Pre-trained Dynamics EncoDER (PDEDER) which can embed the original state observations into a latent space where the dynamics can be captured more easily. To conduct the generalized PDEDER, we pre-train any Pre-trained Language Model (PLM) by minimizing the Lyapunov exponent objective, which constrains the chaotic behavior of governing dynamics learned in the latent space. By penalizing the divergence of embedded observations, our PDEDER promotes locally stable and well-structured latent dynamics, thereby facilitating more effective dynamics modeling than in the original observation space. In addition, we incorporate reconstruction and forecasting objectives to mitigate the risk of obtaining an over-smoothed latent space. Specifically, we collect 152 sets of real-world and synthetic observations from 23 complex systems as pre-training corpora and employ them to pre-train PDEDER. Given any future dynamic observation, we can fine-tune PDEDER with any specific dynamics modeling method. We evaluate PDEDER on 12 dynamic systems by short/long-term forecasting under both in-domain and cross-domain settings, and the empirical results indicate the effectiveness and generalizability of PDEDER.

Yiming Wang, Zhengnan Zhang, Genghe Zhang +7

College of Computer Science and Technology, and Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China · College of Software and Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China · School of Computing, Belfast, Northern Ireland, UK +1

Dynamical SystemsLatent Dynamics ModelingState Representation LearningRepresentation Learning
AskPDF
Mar 22, 2026

Evaluating Factor-Wise Auxiliary Dynamics Supervision for Latent Structure and Robustness in Simulated Humanoid Locomotion

We evaluate whether factor-wise auxiliary dynamics supervision produces useful latent structure or improved robustness in simulated humanoid locomotion. DynaMITE -- a transformer encoder with a factored 24-d latent trained by per-factor auxiliary losses during proximal policy optimization (PPO) -- is compared against Long Short-Term Memory (LSTM), plain Transformer, and Multilayer Perceptron (MLP) baselines on a Unitree G1 humanoid across four Isaac Lab tasks. The supervised latent shows no evidence of decodable or functionally separable factor structure: probe R^2 ~ 0 for all five dynamics factors, clamping any subspace changes reward by < 0.05, and standard disentanglement metrics (MIG, DCI, SAP) are near zero. An unsupervised LSTM hidden state achieves higher probe R^2 (up to 0.10). A 2x2 factorial ablation (n = 10 seeds) isolates the contributions of the tanh bottleneck and auxiliary losses: the auxiliary losses show no measurable effect on either in-distribution (ID) reward (+0.03, p = 0.732) or severe out-of-distribution (OOD) reward (+0.03, p = 0.669), while the bottleneck shows a small, consistent advantage in both regimes (ID: +0.16, p = 0.207; OOD: +0.10, p = 0.208). The bottleneck advantage persists under severe combined perturbation but does not amplify, indicating a training-time representation benefit rather than a robustness mechanism. LSTM achieves the best nominal reward on all four tasks (p < 0.03); DynaMITE degrades less under combined-shift stress (2.3% vs. 16.7%), but this difference is attributable to the bottleneck compression, not the auxiliary supervision. For locomotion practitioners: auxiliary dynamics supervision does not produce an interpretable estimator and does not measurably improve reward or robustness beyond what the bottleneck alone provides; recurrent baselines remain the stronger choice for nominal performance.

Chayanin Chamachot

Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok 10330, Thailand

Latent Dynamics ModelingDisentangled Representation LearningRL for Legged LocomotionRobust RL
AskPDF
Mar 20, 2026

Structured Latent Dynamics in Wireless CSI via Homomorphic World Models

We introduce a self-supervised framework for learning predictive and structured representations of wireless channels by modeling the temporal evolution of channel state information (CSI) in a compact latent space. Our method casts the problem as a world modeling task and leverages the Joint Embedding Predictive Architecture (JEPA) to learn action-conditioned latent dynamics from CSI trajectories. To promote geometric consistency and compositionality, we parameterize transitions using homomorphic updates derived from Lie algebra, yielding a structured latent space that reflects spatial layout and user motion. Evaluations on the DICHASUS dataset show that our approach outperforms strong baselines in preserving topology and forecasting future embeddings across unseen environments. The resulting latent space enables metrically faithful channel charts, offering a scalable foundation for downstream applications such as mobility-aware scheduling, localization, and wireless scene understanding.

Salmane Naoumi, Mehdi Bennis, Marwa Chafii

NYU WIRELESS, NYU Tandon School of Engineering, New York, USA. · Center for Wireless Communications, University of Oulu, Finland. · Engineering Division, New York University (NYU), Abu Dhabi, UAE.

Latent Dynamics ModelingChannel State PredictionWorld Model LearningTopological Representation Learning
AskPDF
Mar 20, 2026

Accurate Open-Loop Control of a Soft Continuum Robot Using Visually Learned Latent Dynamics

This work addresses open-loop control of a soft continuum robot (SCR) from video-learned latent dynamics. Visual Oscillator Networks (VONs) from previous work are used, which provide mechanically interpretable 2D oscillator latents through an attention broadcast decoder (ABCD). Open-loop, single-shooting optimal control is performed in latent space to track image-specified waypoints without camera feedback. An interactive SCR live simulator enables design of static, dynamic, and extrapolated targets and maps them to model-specific latent waypoints. On a two-segment pneumatic SCR, Koopman, MLP, and oscillator dynamics, each with and without ABCD, are evaluated on setpoint and dynamic trajectories. ABCD-based models consistently reduce image-space tracking error. The VON and ABCD-based Koopman models attain the lowest mean squared errors (MSEs). Ablation and simulation stress tests confirm that training and architecture choices support open-loop performance, including static holding, stable extrapolated equilibria, and relaxation to rest. To the best of our knowledge, this is the first demonstration of reliable long-horizon open-loop control of a physical soft robot using interpretable latent dynamics models learned solely from video.

Henrik Krauss, Johann Licher, Naoya Takeishi +2

Department of Advanced Interdisciplinary Studies, The University of Tokyo, Tokyo, 153-8904, Japan. · Institute of Assembly Technology and Robotics, Leibniz University Hannover, 30823 Germany. · Research Center for Advanced Science and Technology, The University of Tokyo, Tokyo, 153-8904, Japan.

Robotic ControlContinuum Robot ControlSoft RoboticsLatent Dynamics ModelingRobot Dynamics Learning
AskPDF
Mar 18, 2026

Identifying Latent Actions and Dynamics from Offline Data via Demonstrator Diversity

Can latent actions and environment dynamics be recovered from offline trajectories when actions are never observed? We study this question in a setting where trajectories are action-free but tagged with demonstrator identity. We assume that each demonstrator follows a distinct policy, while the environment dynamics are shared across demonstrators and identity affects the next observation only through the chosen action. Under these assumptions, the conditional next-observation distribution p(ot+1∣ot,e)p(o_{t+1}\mid o_t,e) is a mixture of latent action-conditioned transition kernels with demonstrator-specific mixing weights. We show that this induces, for each state, a column-stochastic nonnegative matrix factorization of the observable conditional distribution. Using sufficiently scattered policy diversity and rank conditions, we prove that the latent transitions and demonstrator policies are identifiable up to permutation of the latent action labels. We extend the result to continuous observation spaces via a Gram-determinant minimum-volume criterion, and show that continuity of the transition map over a connected state space upgrades local permutation ambiguities to a single global permutation. A small amount of labeled action data then suffices to fix this final ambiguity. These results establish demonstrator diversity as a principled source of identifiability for learning latent actions and dynamics from offline RL data.

Felix Schur

Department of Mathematics, ETH Zürich

Latent Action LearningLatent Dynamics ModelingOffline RLReinforcement Learning
AskPDF
Mar 16, 2026

Regularized Latent Dynamics Prediction is a Strong Baseline For Behavioral Foundation Models

Behavioral Foundation Models (BFMs) produce agents with the capability to adapt to any unknown reward or task. These methods, however, are only able to produce near-optimal policies for the reward functions that are in the span of some pre-existing state features, making the choice of state features crucial to the expressivity of the BFM. As a result, BFMs are trained using a variety of complex objectives and require sufficient dataset coverage, to train task-useful spanning features. In this work, we examine the question: are these complex representation learning objectives necessary for zero-shot RL? Specifically, we revisit the objective of self-supervised next-state prediction in latent space for state feature learning, but observe that such an objective alone is prone to increasing state-feature similarity, and subsequently reducing span. We propose an approach, Regularized Latent Dynamics Prediction (RLDP), that adds a simple orthogonality regularization to maintain feature diversity and can match or surpass state-of-the-art complex representation learning methods for zero-shot RL. Furthermore, we empirically show that prior approaches perform poorly in low-coverage scenarios where RLDP still succeeds.

Pranaya Jajoo, Harshit Sikchi, Siddhant Agarwal +3

Department of Computing Science, University of Alberta, Canada · Alberta Machine Intelligence Institute (Amii) · The University of Texas at Austin +2

Behavioral Foundation ModelsZero-Shot RLReinforcement LearningPredictive Representation LearningState Representation LearningLatent Dynamics Modeling
AskPDF
Mar 13, 2026

Parameterized Temporal Extrapolation of PDEs via Disentangled Latent Dynamics Manifold Fusion

Generalizing neural surrogate models across different PDE parameters remains difficult because changes in PDE coefficients often make learning harder and optimization less stable. The problem becomes even more severe when the model must also predict beyond the training time range. Existing methods usually cannot handle parameter generalization and temporal extrapolation at the same time. Standard parameterized models treat time as just another input and therefore fail to capture intrinsic dynamics, while recent continuous-time latent methods often rely on expensive test-time auto-decoding for each instance, which is inefficient and can disrupt continuity across the parameterized solution space. To address this, we propose Disentangled Latent Dynamics Manifold Fusion (DLDMF), a physics-informed framework that explicitly separates space, time, and parameters. Instead of unstable auto-decoding, DLDMF maps PDE parameters directly to a continuous latent embedding through a feed-forward network. This embedding initializes and conditions a latent state whose evolution is governed by a parameter-conditioned Neural ODE. We further introduce a dynamic manifold fusion mechanism that uses a shared decoder to combine spatial coordinates, parameter embeddings, and time-evolving latent states to reconstruct the corresponding spatiotemporal solution. By modeling prediction as latent dynamic evolution rather than static coordinate fitting, DLDMF reduces interference between parameter variation and temporal evolution while preserving a smooth and coherent solution manifold. As a result, it performs well on unseen parameter settings and in long-term temporal extrapolation. Experiments on several benchmark problems show that DLDMF consistently outperforms state-of-the-art baselines in accuracy, parameter generalization, and extrapolation robustness.

Zhangyong Liang, Jihong Zhu, Huanhuan Gao

National Center for Applied Mathematics, Tianjin University, Tianjin, 300072, PR China · School of Mechanical Engineering, Northwestern Polytechnical University, Xi’an, 710072, China · School of Mechanical and Aerospace Engineering, Jilin University, Changchun, 130025, China

Latent Dynamics ModelingPDE SolvingPDE Surrogate ModelingPhysics-Informed ML
AskPDF
Mar 12, 2026

Anomaly detection in time-series via inductive biases in the latent space of conditional normalizing flows

Deep generative models for anomaly detection in multivariate time-series are typically trained by maximizing observed data likelihood. However, likelihood in observation space measures marginal density rather than conformity to structured temporal dynamics, and therefore can assign high probability to anomalous or out-of-distribution samples. We address this structural limitation by relocating the notion of anomaly to a prescribed latent space. We introduce explicit inductive biases in conditional normalizing flows, modeling time-series observations within a discrete-time state-space framework that constrains latent representations to evolve according to prescribed temporal dynamics. Under this formulation, expected behavior corresponds to compliance with a specified distribution over latent trajectories, while anomalies are defined as violations of these dynamics. Anomaly detection is consequently reformulated as a statistically grounded compliance test, such that observations are mapped to latent space and evaluated via goodness-of-fit tests against the prescribed latent evolution. This yields a principled decision rule that remains effective even in regions of high observation likelihood. Experiments on synthetic and real-world time-series demonstrate reliable detection of anomalies in frequency, amplitude, and observation noise, while providing interpretable diagnostics of model compliance.

David Baumgartner, Eliezer de Souza da Silva, Iñigo Urteaga

Department of Computer Science, Norwegian University of Science and Technology, Norway · BCAM — Basque Center for Applied Mathematics, Spain · Department of Informatics Engineering, University of Coimbra, Portugal +2

Normalizing FlowsLatent Dynamics ModelingTime-Series Anomaly Detection
AskPDF
Mar 12, 2026

Learn Structure, Adapt on the Fly: Multi-Scale Residual Learning and Online Adaptation for Aerial Manipulators

Autonomous Aerial Manipulators (AAMs) are inherently coupled, nonlinear systems that exhibit nonstationary and multiscale residual dynamics, particularly during manipulator reconfiguration and abrupt payload variations. Conventional analytical dynamic models rely on fixed parametric structures, while static data-driven model assume stationary dynamics and degrade under configuration changes and payload variations. Moreover, existing learning architectures do not explicitly factorize cross-variable coupling and multi-scale temporal effects, conflating instantaneous inertial dynamics with long-horizon regime evolution. We propose a predictive-adaptive framework for real-time residual modeling and compensation in AAMs. The core of this framework is the Factorized Dynamics Transformer (FDT), which treats physical variables as independent tokens. This design enables explicit cross-variable attention while structurally separating short-horizon inertial dependencies from long-horizon aerodynamic effects. To address deployment-time distribution shifts, a Latent Residual Adapter (LRA) performs rapid linear adaptation in the latent space via Recursive Least Squares, preserving the offline nonlinear representation without prohibitive computational overhead. The adapted residual forecast is directly integrated into a residual-compensated adaptive controller. Real-world experiments on an aerial manipulator subjected to unseen payloads demonstrate higher prediction fidelity, accelerated disturbance attenuation, and superior closed-loop tracking precision compared to state-of-the-art learning baselines, all while maintaining strict real-time feasibility.

Samaksh Ujjawal, Naveen Sudheer Nair, Shivansh Pratap Singh +3

1Robotics Research Center, IIIT Hyderabad, India. · Department of Computer Science, University of Manchester, UK. · 3Newcastle University, UK.

Aerial RoboticsTest-Time AdaptationLatent Dynamics ModelingResidual LearningAdaptive ControlMultiscale Time-Series Modeling
AskPDF
Mar 11, 2026

World Model for Battery Degradation Prediction Under Non-Stationary Aging

Degradation prognosis for lithium-ion cells requires forecasting the state-of-health (SOH) trajectory over future cycles. Existing data-driven approaches can produce trajectory outputs through direct regression, but lack a mechanism to propagate degradation dynamics forward in time. This paper formulates battery degradation prognosis as a world model problem, encoding raw voltage, current, and temperature time-series from each cycle into a latent state and propagating it forward via a learned dynamics transition to produce a future trajectory spanning 80 cycles. To investigate whether electrochemical knowledge improves the learned dynamics, a Single Particle Model (SPM) constraint is incorporated into the training loss. Three configurations are evaluated on the Severson LiFePO4 (LFP) dataset of 138 cells. Iterative rollout halves the trajectory forecast error compared to direct regression from the same encoder. The SPM constraint improves prediction at the degradation knee where the resistance to SOH relationship is most applicable, without changing aggregate accuracy.

Kai Chin Lim, Khay Wai See

Independent Researcher, Australia

Lithium-Ion BatteriesRUL EstimationLatent Dynamics ModelingLatent World Models
AskPDF
Mar 11, 2026

JEDI: Jointly Embedded Inference of Neural Dynamics

Animal brains flexibly and efficiently achieve many behavioral tasks with a single neural network. A core goal in modern neuroscience is to map the mechanisms of the brain's flexibility onto the dynamics underlying neural populations. However, identifying task-specific dynamical rules from limited, noisy, and high-dimensional experimental neural recordings remains a major challenge, as experimental data often provide only partial access to brain states and dynamical mechanisms. While recurrent neural networks (RNNs) directly constrained neural data have been effective in inferring underlying dynamical mechanisms, they are typically limited to single-task domains and struggle to generalize across behavioral conditions. Here, we introduce JEDI, a hierarchical model that captures neural dynamics across tasks and contexts by learning a shared embedding space over RNN weights. This model recapitulates individual samples of neural dynamics while scaling to arbitrarily large and complex datasets, uncovering shared structure across conditions in a single, unified model. Using simulated RNN datasets, we demonstrate that JEDI accurately learns robust, generalizable, condition-specific embeddings. By reverse-engineering the weights learned by JEDI, we show that it recovers ground truth fixed point structures and unveils key features of the underlying neural dynamics in the eigenspectra. Finally, we apply JEDI to motor cortex recordings during monkey reaching to extract mechanistic insight into the neural dynamics of motor control. Our work shows that joint learning of contextual embeddings and recurrent weights provides scalable and generalizable inference of brain dynamics from recordings alone.

Anirudh Jamkhandi, Ali Korojy, Olivier Codol +2

Mila - Quebec AI Institute · Université de Montréal

Latent Dynamics ModelingDynamical SystemsRecurrent Neural NetworksSystem Identification
AskPDF
Mar 8, 2026

Generative prediction of laser-induced rocket ignition with dynamic latent space representations

Accurate and predictive scale-resolving simulations of laser-ignited rocket engines are highly time-consuming because the problem includes turbulent fuel-oxidizer mixing dynamics, laser-induced energy deposition, and high-speed flame growth. This is conflated with the large design space primarily corresponding to the laser operating conditions and target location. To enable rapid exploration and uncertainty quantification, we propose a data-driven surrogate modeling approach that combines convolutional autoencoders (cAEs) with neural ordinary differential equations (neural ODEs). The present target application of an ML-based surrogate model to leading-edge multi-physics turbulence simulation is part of a paradigm shift in the deployment of surrogate models towards increasing real-world complexity. Sequentially, the cAE spatially compresses high-dimensional flow fields into a low-dimensional latent space, wherein the system's temporal dynamics are learned via neural ODEs. Once trained, the model generates fast spatiotemporal predictions from initial conditions and specified operating inputs. By learning a surrogate to replace the entirety of the time-evolving simulation, the cost of predicting an ignition trial is reduced by several orders of magnitude, allowing efficient exploration of the input parameter space. Further, as the current framework yields a spatiotemporal field prediction, appraisal of the model output's physical grounding is more tractable. This approach marks a significant step toward real-time digital twins for laser-ignited rocket combustors and represents surrogate modeling in a complex system context.

Tony Zahtila, Ettore Saetta, Murray Cutforth +3

Center for Turbulence Research, Stanford University, USA · University of Naples Federico II, Italy

Neural Surrogate ModelingPDE Surrogate ModelingComputational Fluid DynamicsLatent Dynamics Modeling
AskPDF
Mar 8, 2026

Cost-Driven Representation Learning for Linear Quadratic Gaussian Control: Part II

We study the problem of state representation learning for control from partial and potentially high-dimensional observations. We approach this problem via cost-driven state representation learning, in which we learn a dynamical model in a latent state space by predicting cumulative costs. In particular, we establish finite-sample guarantees on finding a near-optimal representation function and a near-optimal controller using the learned latent model for infinite-horizon time-invariant Linear Quadratic Gaussian (LQG) control. We study two approaches to cost-driven representation learning, which differ in whether the transition function of the latent state is learned explicitly or implicitly. The first approach has also been investigated in Part I of this work, for finite-horizon time-varying LQG control. The second approach closely resembles MuZero, a recent breakthrough in empirical reinforcement learning, in that it learns latent dynamics implicitly by predicting cumulative costs. A key technical contribution of this Part II is to prove persistency of excitation for a new stochastic process that arises from the analysis of quadratic regression in our approach, and may be of independent interest.

Yi Tian, Kaiqing Zhang, Russ Tedrake +1

Massachusetts Institute of Technology · University of Maryland, College Park · Technical University Munich

Linear Quadratic RegulatorState Representation LearningLatent Dynamics ModelingStochastic Optimal ControlModel-Based RL
AskPDF
Mar 7, 2026

LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models

Recent generative video world models aim to simulate visual environment evolution, allowing an observer to interactively explore the scene via camera control. However, they implicitly assume that the world only evolves within the observer's field of view. Once an object leaves the observer's view, its state is "frozen" in memory, and revisiting the same region later often fails to reflect events that should have occurred in the meantime. In this work, we identify and formalize this overlooked limitation as the "out-of-sight dynamics" problem, which impedes video world models from representing a continuously evolving world. To address this issue, we propose LiveWorld, a novel framework that extends video world models to support persistent world evolution. Instead of treating the world as static observational memory, LiveWorld models a persistent global state composed of a static 3D background and dynamic entities that continue evolving even when unobserved. To maintain these unseen dynamics, LiveWorld introduces a monitor-based mechanism that autonomously simulates the temporal progression of active entities and synchronizes their evolved states upon revisiting, ensuring spatially coherent rendering. For evaluation, we further introduce LiveBench, a dedicated benchmark for the task of maintaining out-of-sight dynamics. Extensive experiments show that LiveWorld enables persistent event evolution and long-term scene consistency, bridging the gap between existing 2D observation-based memory and true 4D dynamic world simulation. The baseline and benchmark will be publicly available at https://zichengduan.github.io/LiveWorld/index.html.

Zicheng Duan, Jiatong Xia, Zeyu Zhang +7

Adelaide University · The Australian National University · Monash University +2

Video World ModelsWorld ModelsInteractive World ModelsLatent Dynamics ModelingLong-Horizon Video Generation
AskPDF
Mar 6, 2026

Latent Autoencoder Ensemble Kalman Filter for Nonlinear Data assimilation

The ensemble Kalman filter (EnKF) is widely used for data assimilation in high-dimensional systems, but its performance often deteriorates for strongly nonlinear dynamics due to the structural mismatch between the Kalman update and the underlying system behavior. In this work, we propose a latent autoencoder ensemble Kalman filter (LAE-EnKF) that addresses this limitation by reformulating the assimilation problem in a learned latent space with linear and stable dynamics. The proposed method learns a nonlinear encoder--decoder together with a stable linear latent evolution operator and a consistent latent observation mapping, yielding a closed linear state-space model in the latent coordinates. This construction restores compatibility with the Kalman filtering framework and allows both forecast and analysis steps to be carried out entirely in the latent space. Compared with existing autoencoder-based and latent assimilation approaches that rely on unconstrained nonlinear latent dynamics, the proposed formulation emphasizes structural consistency, stability, and interpretability. We provide a theoretical analysis of learning linear dynamics on low-dimensional manifolds and establish generalization error bounds for the proposed latent model. Numerical experiments on representative nonlinear and chaotic systems demonstrate that the LAE-EnKF yields more accurate and stable assimilation than the standard EnKF and related latent-space methods, while maintaining comparable computational cost and data-driven.

Xin T. Tong, Yanyan Wang, Liang Yan

Department of Mathematics, National University of Singapore, 119076, Singapore · School of Mathematics, Southeast University, Nanjing 210096, China

Data AssimilationEnsemble Kalman FilteringLatent Dynamics Modeling
AskPDF
Mar 5, 2026

Random Dot Product Graphs as Dynamical Systems: Limitations and Opportunities

Can we learn the differential equations governing the evolution of a temporal network? We investigate this within Random Dot Product Graphs (RDPGs), where each network snapshot is generated from latent positions evolving under unknown dynamics. We identify three fundamental obstructions: gauge freedom from rotational ambiguity in latent positions, realizability constraints from the manifold structure of the probability matrix, and trajectory recovery artifacts from spectral embedding. We develop a geometric framework based on principal fiber bundles that formalizes these obstructions. We characterize invisible dynamics as exactly the skew-symmetric generators, and show the realizable tangent space has dimension nd−d(d−1)/2nd - d(d-1)/2. An holonomy dichotomy emerges: polynomial dynamics have commuting generators, stationary eigenvectors, and trivial holonomy, making gauge alignment purely statistical; Laplacian dynamics satisfy a non-commutativity criterion producing nontrivial holonomy, with curvature weighted by 1/(λι+λγ)1/(λ_ι+ λ_γ) linking gauge sensitivity to the spectral gap. In d=2d=2 this yields full restricted holonomy SO(2)\mathrm{SO}(2); for d≥3d \ge 3 generic full SO(d)\mathrm{SO}(d) remains conjectural. Cram'er--Rao lower bounds reveal that the same spectral gap controlling curvature and injectivity simultaneously controls Fisher information, so geometric and statistical difficulty are inextricable. We prove an identifiability principle: symmetric dynamics cannot absorb skew-symmetric gauge contamination, so dynamics structure can resolve gauge ambiguity. We demonstrate this constructively with anchor-based alignment and a UDE pipeline recovering vector fields from noisy graph sequences. Yet finite-sample interactions between noise, gauge, and dynamics expressiveness remain beyond the asymptotic theory. We frame this gap as an open challenge.

Giulio Valentino Dalla Riva

Latent Dynamics ModelingDynamic Graph LearningSystem IdentificationLatent Variable Models
AskPDF
Mar 5, 2026

Behavior-dLDS: A decomposed linear dynamical systems model for neural activity partially constrained by behavior

Brain-wide recordings of large-scale networks of neurons now provide an unprecedented view into how the brain drives behavior. However, brain activity contains both information directly related to behavior as well as the potential for many internal computations. Moreover, observable behavior is executed not only by the brain, but also by the spinal cord and peripheral nervous system. Behavior is a coarse-grained product of neural activity, and we thus take the view that it can be best represented by lower-dimensional latent neural dynamics. Capturing this indirect relationship while disambiguating behavior-generating networks from internal computations running in parallel requires new modeling approaches that can embody the parallel and distributed nature of large-scale neural populations. We thus present behavior-decomposed linear dynamical systems (b-dLDS) to disentangle simultaneously recorded subsystems and identify how the latent neural subsystems relate to behavior. We demonstrate the ability of b-dLDS to decouple behavioral vs. internal computations on controlled, simulated data, showing improvements over a state-of-the-art model that uses behavior to supervise all dynamics based on behavior. We also demonstrate b-dLDS's interpretability benefits on a task-driven RNN dataset featuring a nonlinear relationship between behavior and activations. We then show that b-dLDS can further scale up to tens of thousands of neurons by applying our model to a large-scale recording of a zebrafish hindbrain during the complex positional homeostasis behavior, wherein b-dLDS highlights asymmetry in behavior-related dynamic connectivity networks.

Eva Yezerets, En Yang, Misha B. Ahrens +1

Department of Biomedical Engineering Center for Imaging Science Johns Hopkins University · Department of Biology University of North Carolina · Janelia Research Campus Howard Hughes Medical Institute

Latent Dynamics ModelingDynamical SystemsBrain Network Analysis
AskPDF
Mar 5, 2026

BLINK: Behavioral Latent Modeling of NK Cell Cytotoxicity

Machine learning models of cellular interaction dynamics hold promise for understanding cell behavior. Natural killer (NK) cell cytotoxicity is a prominent example of such interaction dynamics and is commonly studied using time-resolved multi-channel fluorescence microscopy. Although tumor cell death events can be annotated at single frames, NK cytotoxic outcome emerges over time from cellular interactions and cannot be reliably inferred from frame-wise classification alone. We introduce BLINK, a trajectory-based recurrent state-space model that serves as a cell world model for NK-tumor interactions. BLINK learns latent interaction dynamics from partially observed NK-tumor interaction sequences and predicts apoptosis increments that accumulate into cytotoxic outcomes. Experiments on long-term time-lapse NK-tumor recordings show improved cytotoxic outcome detection and enable forecasting of future outcomes, together with an interpretable latent representation that organizes NK trajectories into coherent behavioral modes and temporally structured interaction phases. BLINK provides a unified framework for quantitative evaluation and structured modeling of NK cytotoxic behavior at the single-cell level.

Iman Nematollahi, Jose Francisco Villena-Ossa, Alina Moter +6

Department of Computer Science, University of Freiburg, Freiburg, Germany · Institute for Transfusion Medicine and Gene Therapy, University Medical Center Freiburg, Freiburg, Germany · Goethe University, Department of Pediatrics, Experimental Immunology and Cell Therapy, Frankfurt am Main, Germany +2

Toxicity DetectionLatent Dynamics ModelingState Space Models
AskPDF
Mar 3, 2026

On Geometry Regularization in Autoencoder Reduced-Order Models with Latent Neural ODE Dynamics

We investigate geometric regularization strategies for learned latent representations in encoder--decoder reduced-order models. In a fixed experimental setting for the advection--diffusion--reaction (ADR) equation, we model latent dynamics using a neural ODE and evaluate four regularization approaches applied during autoencoder pre-training: (a) near-isometry regularization of the decoder Jacobian, (b) a stochastic decoder gain penalty based on random directional gains, (c) a second-order directional curvature penalty, and (d) Stiefel projection of the first decoder layer. Across multiple seeds, we find that (a)--(c) often produce latent representations that make subsequent latent-dynamics training with a frozen autoencoder more difficult, especially for long-horizon rollouts, even when they improve local decoder smoothness or related sensitivity proxies. In contrast, (d) consistently improves conditioning-related diagnostics of the learned latent dynamics and tends to yield better rollout performance. We discuss the hypothesis that, in this setting, the downstream impact of latent-geometry mismatch outweighs the benefits of improved decoder smoothness.

Mikhail Osipov

Independent Researcher, Italy

Reduced-Order ModelingAutoencodersGeometric Representation LearningJacobian RegularizationLatent Dynamics Modeling
AskPDF
Feb 27, 2026

Accelerating Masked Image Generation by Learning Controlled Latent Dynamics

Masked Image Generation Models (MIGMs) have achieved great success, yet their efficiency is hampered by the multiple steps of bi-directional attention. In fact, there exists notable redundancy in their computation: when sampling discrete tokens, the rich semantics contained in the continuous features are lost. Some existing works attempt to cache the features to approximate future features. However, they exhibit considerable approximation error under aggressive acceleration settings. We attribute this to their limited expressivity and the failure to account for sampling information. To fill this gap, we propose learning a lightweight model that incorporates both previous features and sampled tokens, and regresses the average velocity field of feature evolution. The model has moderate complexity that suffices to capture the subtle dynamics while keeping lightweight compared to the original base model. We apply our method to two representative MIGMs and tasks. In particular, on the state-of-the-art Lumina-DiMOO, it achieves over 4x acceleration of text-to-image generation while maintaining quality, significantly pushing the Pareto frontier of masked image generation. The code and model weights are available at https://github.com/Kaiwen-Zhu/MIGM-Shortcut.

Kaiwen Zhu, Quansheng Zeng, Yuandong Pu +9

Shanghai Jiao Tong University Shanghai, China · Shanghai AI Laboratory Shanghai, China · Shanghai Innovation Institute Shanghai, China +6

Latent Dynamics ModelingT2I GenerationMasked Image Modeling
AskPDF
Feb 26, 2026

ODEBrain: Continuous-Time EEG Graph for Modeling Dynamic Brain Networks

Modeling neural population dynamics is crucial for foundational neuroscientific research and various clinical applications. Conventional latent variable methods typically model continuous brain dynamics through discretizing time with recurrent architecture, which necessarily results in compounded cumulative prediction errors and failure of capturing instantaneous, nonlinear characteristics of EEGs. We propose ODEBRAIN, a Neural ODE latent dynamic forecasting framework to overcome these challenges by integrating spatio-temporal-frequency features into spectral graph nodes, followed by a Neural ODE modeling the continuous latent dynamics. Our design ensures that latent representations can capture stochastic variations of complex brain states at any given time point. Extensive experiments verify that ODEBRAIN can improve significantly over existing methods in forecasting EEG dynamics with enhanced robustness and generalization capabilities.

Haohui Jia, Zheng Chen, Lingwei Zhu +6

Information Science and Techinology, Hokkaido University, Japan · SANKEN, The University of Osaka, Japan · Great Bay University, China +1

Latent Dynamics ModelingElectroencephalographyTime Series ForecastingBrain Network Analysis
AskPDF
Feb 26, 2026

Latent Matters: Learning Deep State-Space Models

Deep state-space models (DSSMs) enable temporal predictions by learning the underlying dynamics of observed sequence data. They are often trained by maximising the evidence lower bound. However, as we show, this does not ensure the model actually learns the underlying dynamics. We therefore propose a constrained optimisation framework as a general approach for training DSSMs. Building upon this, we introduce the extended Kalman VAE (EKVAE), which combines amortised variational inference with classic Bayesian filtering/smoothing to model dynamics more accurately than RNN-based DSSMs. Our results show that the constrained optimisation framework significantly improves system identification and prediction accuracy on the example of established state-of-the-art DSSMs. The EKVAE outperforms previous models w.r.t. prediction accuracy, achieves remarkable results in identifying dynamical systems, and can furthermore successfully learn state-space representations where static and dynamic features are disentangled.

Alexej Klushyn, Richard Kurle, Maximilian Soelch +2

Technical University of Munich · AWS AI Labs · Machine Learning Research Lab, Volkswagen Group, Munich +1

State Space ModelsLatent Dynamics ModelingVariational AutoencodersSystem IdentificationState Representation Learning
AskPDF
Feb 26, 2026

Advancing accelerator virtual beam diagnostics through latent evolution modeling: an integrated solution to forward, inverse, tuning, and UQ problems

Virtual beam diagnostics relies on computationally intensive beam dynamics simulations where high-dimensional charged particle beams evolve through the accelerator. We propose Latent Evolution Model (LEM), a hybrid machine learning framework with an autoencoder that projects high-dimensional phase spaces into lower-dimensional representations, coupled with transformers to learn temporal dynamics in the latent space. This approach provides a common foundational framework addressing multiple interconnected challenges in beam diagnostics. For forward modeling, a Conditional Variational Autoencoder (CVAE) encodes 15 unique projections of the 6D phase space into a latent representation, while a transformer predicts downstream latent states from upstream inputs. For inverse problems, we address two distinct challenges: (a) predicting upstream phase spaces from downstream observations by utilizing the same CVAE architecture with transformers trained on reversed temporal sequences along with aleatoric uncertainty quantification, and (b) estimating RF settings from the latent space of the trained LEM using a dedicated dense neural network that maps latent representations to RF parameters. For tuning problems, we leverage the trained LEM and RF estimator within a Bayesian optimization framework to determine optimal RF settings that minimize beam loss. This paper summarizes our recent efforts and demonstrates how this unified approach effectively addresses these traditionally separate challenges.

Mahindra Rautela, Alexander Scheinker

Los Alamos National Laboratory, Los Alamos, NM, US

Latent Dynamics ModelingInverse Problems
AskPDF