Markov Models

Follow this topic Ask about this topic

Markov models represent sequences and decisions as states linked by probabilistic transitions, making them useful when uncertainty and temporal dependence matter. Hidden Markov models use forward-backward inference and Baum–Welch estimation for unseen states, while Markov decision processes support planning and reinforcement learning; continuous-time chains also underpin discrete diffusion models. Researchers study how to learn transition kernels from limited or dependent data, build useful state abstractions, and scale inference and planning to large systems, with applications ranging from language generation to robotics and molecular dynamics. A central tension is that simple Markov states enable efficient analysis, but real systems may require richer memory to capture non-stationarity, partial observability, or slow mixing.

Latest papers

93 papers3 papers in the last four weeks, down 70% on the four weeks before. 0.0% of all new papers.

Deep Bayesian Approaches for Interpretable Nonlinear Structural System Identification
Oct 27, 2026Wei Liu, Konstantinos Vlachas, Eleni ChatziSystem Identification Nonlinear System Identification
National University of Singapore Singapore · ETH Zurich Zurich, Switzerland
Minimax estimation of the pseudo-spectral gap of continuous Markov chains, with applications to empirical concentration bounds
Oct 14, 2026Vahe Karagulyan, Mikołaj Kasprzak, Pierre AlquierMarkovian Bandits Minimax Estimation
ESSEC Business School
Learning Transition Kernels of Jump-Diffusion Processes with Conditional Diffusion Models
Oct 6, 2026Yuzhen Zhao, Yating Liu, Quentin GuibertStochastic Differential Equations Conditional Distribution Estimation
CEREMADE, CNRS, Universite Paris-Dauphine, PSL, 75016 Paris, France
Conditional Flow Matching for Transport Between Markov Processes
Oct 5, 2026Syamantak Kumar, Dheeraj Nagaraj, Saptarshi Roy +1Conditional Flow Matching Domain Adaptation
The University of Texas at Austin, Department of Computer Science · Google Deepmind · Mohamed bin Zayed University of Artificial Intelligence, CMS Division +1
Convergence-Aware Pareto Selection of Covariate Scaling Transformations for Markov Deterioration Hazard Models: Evidence from Bridge Inspection Data
Sep 24, 2026Takato Yasuno, Keita Kobayashi, Ryuta SakaguchiMarkov Models Survival Analysis
Digital Business Promotion Department, Yachiyo Engineering Co., Ltd.
Marginal Log-Likelihood Increments under Dirichlet-Smoothed Markov Estimation
Sep 22, 2026Levin David SchwabMarkov Models Process Mining
Machine learning kinetics from molecular dynamics data
Sep 15, 2026Jonathan Weare, Aaron R. DinnerMarkov Models Molecular Dynamics Simulation
Courant Institute School of Mathematics, Computing, and Data Science, New York University, New York, New York 10012, United States · Department of Chemistry and James Franck Institute, University of Chicago, Chicago, Illinois 60637, United States
High-Performance Tensor Formulation of the Viterbi Algorithm for Hidden Semi-Markov Models
Sep 15, 2026Lorenzo Piarulli, Elia Belli, Daniele De SensiMarkov Models High-Performance Computing
Department of Computer Science Sapienza University of Rome
Strong and Compact Policies for Submodular Markov Decision Processes via LP-Based Submodular Orienteering
Sep 14, 2026Lars Rohwedder, Rico ZenklusenSubmodular Optimization Markov Decision Processes
University of Southern Denmark, Odense, Denmark · ETH Zurich, Zurich, Switzerland
Windowed A-K-MDP
Sep 12, 2026Xiangwen Yang, Frankie Cho, Iadine ChadesMarkov Decision Processes Markov Models
Monash University Melbourne, Australia
Supermartingale Certificates for Parametric MDPs
Sep 11, 2026Kaushik Mallik, Ðorđe ŽikelićRobust Markov Decision Processes Markov Decision Processes
IMDEA Software Institute, Madrid, Spain
Artificial Rosetta Stone: Constrained Maximum A Posteriori (MAP) Reconstruction of Symbolic Raga Sequences via Order-k Markov Models
Sep 1, 2026Saanvi Raghavendran, Abhishek BhattacharjeeMarkov Models Sequence Modeling
Abstract Math Institute
When Metropolis and Hastings Meet Bradley and Terry: Exact MCMC From Preference Voting
Sep 1, 2026Ariel Smogorghevski, Nir Rosenfeld, Yaniv RomanoMarkov Chain Monte Carlo Markov Models
Department of Data and Decision Sciences, Technion – Israel Institute of Technology · Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology
Exact Global MCMC with Denoising Diffusion
Aug 31, 2026Mitch HillIndependent Metropolis-Hastings Markov Chain Monte Carlo
Propensity Straight-Through Gradients for Discrete Stochastic Systems
Aug 26, 2026Jose M. G. Vilar, Leonor SaizStraight-Through Estimator Markov Chains
Biofisika Institute (CSIC, UPV/EHU), University of the Basque Country, P.O. Box 644, 48080 Bilbao, Spain · IKERBASQUE, Basque Foundation for Science, 48011 Bilbao, Spain · Department of Biomedical Engineering, University of California, 451 East Health Sciences Drive, Davis, CA 95616, USA
From local kernels to global form: modeling the emergence of musical content
Aug 25, 2026Francesco Vitucci, Michele Lorusso, Francesco ScagliolaMarkov Models Music Information Retrieval
Conservatorio di Musica “N. Piccinni” di Bari
A Data-Driven Approach to State Construction in Markov Models
Aug 21, 2026Linde Van Gestel, Marie-Anne Guerry, Evy RombautMarkov Models State Abstraction
Department of Business Technology and Operations, Data Analytics Laboratory, Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium · Mobility and Automotive Technology, Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium
Spectral partitioning for $k$ -block averaging kernels of finite Markov chains
Aug 21, 2026Michael C. H. Choi, Youjia WangMarkov Models Spectral Methods
Department of Statistics and Data Science, National University of Singapore, Level 7, 6 Science Drive 2, 117546, Singapore
On the Pseudo-Mixing of Kac's Walk
Aug 18, 2026Natesh S. Pillai, Aaron Smith, Vinod VaikuntanathanMarkov Models
Harvard · uOttawa and TIMC · MIT
Spectral Gaps of Hit-and-Run and Coordinate Hit-and-Run
Aug 17, 2026Yunbum Kook, Santosh S. VempalaMarkov Chain Monte Carlo Markov Models
University of Michigan · Georgia Tech
From Optimal Actions to World Models: Identifiability of Transition Kernels in Discounted MDPs
Aug 7, 2026Neal BatraMarkov Decision Processes Markov Models
Dovetail Research · University of Warwick
Newton-Schulz Retraction-Based Inference Enables Hidden Quantum Markov Models to Outperform Classical HMMs
Aug 6, 2026Ning NingNewton-Schulz Iteration Markov Models
Department of Statistics, Texas A&M University College Station, Texas, USA
Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions
Aug 6, 2026Yuepeng Yang, Yuxin Chen, Yuejie ChiRobust Markov Decision Processes Average-Reward RL
*Department of Statistics and Data Science, Yale University. · †Department of Statistics and Data Science, the Wharton School, University of Pennsylvania.
MACRO: Markov Chain Routing of Transformer Layers
Aug 6, 2026Paweł Batorski, Abtin Pourhadi, Akylgali Aitaza +2Markov Models LLM Inference Acceleration
Heinrich Heine University Düsseldorf · Jagiellonian University · IDEAS Research Institute
PAC Approximation and DIRECT Optimization for Parametric Markov Models
Aug 3, 2026Zhiming Chi, Ying Liu, Andrea Turrini +2Robust Markov Decision Processes Markov Decision Processes
Key Laboratory of System Software (Chinese Academy of Sciences), Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +2
Tensor Probabilistic Model Checking of Finite-Horizon Markov Chains (Extended Version)
Aug 1, 2026Jianlin Li, Nick Guo, Peter Ye +1Reachability Analysis Markov Models
David R. Cheriton School of Computer Science University of Waterloo, Canada
Minimal Markovization via Stable Quotients in Holonomy-Cover Decision Processes
Jul 29, 2026Zuyuan Zhang, Yongshan Chen, Mahdi Imani +1Markov Decision Processes Partially Observable RL
Department of Electrical and Computer Engineering, The George Washington University · Department of Electrical and Computer Engineering, Northeastern University
Property-driven Causal Abstractions for Markov Decision Processes
Jul 29, 2026Jule Schmidt, Maximilian Weininger, Clemens Dubslaff +2Markov Decision Processes Markov Models
Ruhr-University Bochum, Germany · Eindhoven University of Technology, The Netherlands · Oxford University, UK +1
dtControl2+ $\varepsilon$ : Trading Optimality for Explainability in MDPs via Decision Trees
Jul 28, 2026Tereza Kinská, Jan Křetínský, Tobias Meggendorfer +2Markov Decision Processes Markov Models
Masaryk University, Brno, Czech Republic · Technical University of Munich, Munich, Germany · Lancaster University Leipzig, Leipzig, Germany +1
A Characterization of the Orthocomplement of the Tangent Space of Semiparametric Markov Models
Jul 26, 2026Trung Phung, Ilya ShpitserSemiparametric Inference Markov Models
Computer Science Department, Johns Hopkins University, Baltimore, Maryland, USA

Oct 27, 2026J/K move · Enter open · S save

Deep Bayesian Approaches for Interpretable Nonlinear Structural System Identification

Wei Liu, Konstantinos Vlachas, Eleni Chatzi

National University of Singapore Singapore · ETH Zurich Zurich, Switzerland

System Identification Nonlinear System Identification Bayesian Filtering Physics-Informed ML Markov Models Extended Kalman Filtering

Open paper AskPDF arXiv

System identification, namely the inference of mathematical representations of dynamical systems from measured data, is fundamental to engineering diagnostics, prognostics, and decision-making. Bayesian filtering offers a principled framework for probabilistic state estimation but traditionally relies on prescribed transition and observation models. Recent advances in deep Bayesian filtering address this limitation by embedding learnable representations within probabilistic state-space formulations, enabling data-driven identification of unknown dynamics while retaining physical structure and uncertainty-aware inference. This tutorial provides a systematic introduction to deep Bayesian filtering for nonlinear structural system identification, focusing on the interplay between learned dynamics, physical inductive biases, and posterior inference. The Deep Markov Model serves as a data-driven reference, while the Physics-guided Deep Markov Model (PgDMM) and Neural Extended Kalman Filter (Neural EKF) illustrate complementary approaches based on amortized variational inference and differentiable Kalman filtering, respectively. Their capabilities are examined through dynamic state estimation and joint dynamic state and parameter estimation on a three-dimensional shear-frame benchmark with hysteretic Bouc-Wen connections and sparse acceleration measurements. The results highlight important distinctions between measurement-conditioned reconstruction and open-loop forecasting, demonstrate how physical structure can improve latent-state interpretability without necessarily ensuring predictive fidelity, and reveal differences in the robustness of response and parameter estimation. Practical guidelines for model construction, training, evaluation, and deployment are provided, together with an outlook on remaining challenges in uncertainty calibration, identifiability, and real-world validation.