A Data-Driven Approach to State Construction in Markov Models

Linde Van Gestel; Marie-Anne Guerry; Evy Rombaut

stat.MLAug 21, 2026

A Data-Driven Approach to State Construction in Markov Models

Authors: Linde Van Gestel, Marie-Anne Guerry, Evy Rombaut

Organizations: Department of Business Technology and Operations, Data Analytics Laboratory, Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium · Mobility and Automotive Technology, Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium

Abstract

A Markov chain is a widely used stochastic process modelling random events over time. These models are built on subsets of the entire dataset, referred to as states, which are considered to be homogeneous regarding transition probabilities. However, the creation of these states is often disregarded or based on prior assumption, potentially violating the homogeneity requirement and thus decreasing the validity and predictive power of the model. In order to fill this gap, this paper combines supervised feature selection with unsupervised learning techniques for data-driven state construction. Density-based clustering, spectral clustering, and Kohonen self-organizing maps are examined for their ability to identify latent groups without prior assumptions. The contribution of this study is twofold. First, the paper presents a methodological framework for state construction incorporating suitable unsupervised learning techniques, with appropriate measures both for classification performance and Markov model accuracy. Secondly, the framework is tested on an application, resulting in a comparative analysis showing that spectral clustering and Kohonen self-organizing maps are best at capturing inherent structure. These results serve as a cornerstone in providing theoretical and methodological guidance for improving state definition in applied Markov modelling.

Figures & tables

Panel of Figure 1 — Figure 1: Flowchart of the proposed clustering and validation procedure.

Figure 2: UMAP embedding of Grade 5 employees coloured according to their HDBSCAN cluster assignment.

Figure 3: Eigengap plot for the Grade 5 transition dataset.

Panel of Figure 4 — Figure 4: Diagnostic plots of the self-organizing map (SOM) for Grade 3 employees.

Figure 5: Topology-constrained SOM merge plot for the Grade 2 transition dataset.

		HDBSCAN		Spectral		SOM
Grade	Tree	Before	After	Before	After	Before	After
$G_{1}$	4	9	3	2	2	3	2
$G_{2}$	1	3	2	1	1	4	3
$G_{3}$	3	3	2	2	2	4	2
$G_{4}$	2	4	2	3	2	4	2
$G_{5}$	2	5	3	3	2	3	2

Table 1: Number of subgroups before and resulting states after transition-based merging.

Panel of Figure 6 — Figure 6: Transition probabilities for the class model and alternative state construction methods.

Grade	Model	$\chi^{2}$	df	Critical value	Rejected folds	AIC
$G_{1}$	Spectral	2.31	6	12.59	0	14.31
	SOM	2.45	10	18.31	0	22.45
	HDBSCAN	7.04	15	25.00	0	37.04
	Tree	11.46	20	31.41	0	51.46
	Class	54.02	2	5.99	1	58.02
$G_{2}$	Spectral	2.54	3	7.81	0	8.54

Table 2: Cross-validated goodness-of-fit measures by grade and model. Values for

\chi^{2}

and AIC represent averages across the three folds.

Model	Total $\chi^{2}$	df	Critical value	Rejected folds	Total AIC
Spectral	38.33	96	119.87	0	230.33
Class	261.99	33	47.40	6	327.99
SOM	82.33	144	173.00	1	370.33
HDBSCAN	70.29	162	192.70	0	394.29
Tree	74.55	162	192.70	0	398.55

Table 3: Overall cross-validated goodness-of-fit measures across all grades.

Model	Mean AIC	$\Delta$ AIC	Weight	Evidence ratio
Spectral	12.80	0.00	0.90	1.00
Class	18.22	5.43	0.06	15.07
SOM	20.57	7.78	0.02	48.86
HDBSCAN	21.90	9.11	0.01	95.04
Tree	22.14	9.35	0.01	107.00

Table 4: Overall Akaike weights across all grades.

Comparison	Mean $\Delta$ log-likelihood	$z$	$p$ -value	Preferred model
Spectral vs Tree	0.214	41.801	0.00e+00	Spectral
Spectral vs SOM	0.030	7.739	9.99e-15	Spectral
Spectral vs HDBSCAN	0.369	20.106	0.00e+00	Spectral
Tree vs SOM	-0.184	-31.900	0.00e+00	SOM
Tree vs HDBSCAN	0.155	8.192	2.22e-16	Tree
SOM vs HDBSCAN	0.339	18.347	0.00e+00	SOM

Table 5: Pairwise Vuong tests based on casewise Markov transition log-likelihoods.

Model	Total $\chi^{2}$	df	Critical value	Rejected folds	Total AIC
Hybrid	31.66	96	119.87	0	223.66

Table 6: Overall cross-validated goodness-of-fit measures for the hybrid model.

Comparison	Mean $\Delta$ log-likelihood	$z$	$p$ -value	Preferred model
Spectral vs Hybrid	-0.016	-6.87	6.44e-12	Hybrid

Table 7: Overall Vuong test comparing the spectral and hybrid Markov models.

Grade	Model	$\chi^{2}$	df	Critical value	Rejected folds	AIC
$G_{1}$	Spectral	2.31	6	12.59	0	14.31
	SOM	2.45	10	18.31	0	22.45
	Tree	11.46	20	31.41	0	51.46
	Class	54.02	2	5.99	1	58.02
	HDBSCAN (no UMAP)	13.21	28	41.34	0	69.21
$G_{2}$	Spectral	2.54	3	7.81	0	8.54

Table 8: Cross-validated goodness-of-fit measures by grade and model (without UMAP). Values for

\chi^{2}

and AIC represent averages across the three folds.

Model	Total $\chi^{2}$	df	Critical value	Rejected folds	Total AIC
Spectral	38.33	96	119.87	0	230.33
Class	261.99	33	47.40	6	327.99
SOM	82.33	144	173.00	1	370.33
Tree	74.55	162	192.70	0	398.55
HDBSCAN (no UMAP)	99.90	198	231.83	0	495.90

Table 9: Overall cross-validated goodness-of-fit measures across all grades without UMAP.

Grade	Model	Group	$n$	Stay probability	Promotion probability
G1	Class	1	12300	0.830	0.025
	HDBSCAN	1	10029	0.827	0.027
		2	793	0.758	0.010
		3	1478	0.888	0.003
	Hybrid	1	9732	0.830	0.028
		2	2568	0.828	0.006

Table 10: Observed group sizes and stay/promotion probabilities by grade and model.

Markov Models

Markov models describe systems that evolve probabilistically, with future behavior determined by the current state and transition rules.

See all 93 papers

Robust State-Conditional Feature-Weighted Jump Models for Temporal Clustering
Jun 11, 2026Federico P. Cortese, Alessio FarcomeniClustering Unsupervised Clustering
University of Milan, Department of Economics, Management, and Quantitative Methods · National Research Council of Italy, Institute for Applied Mathematics and Information Technologies, Milan · University of Rome “Tor Vergata”, Department of Enterprise Engineering
Constrained latent state modeling: A unifying perspective on representation learning under competing constraints
May 15, 2026Gwenolé QuellecRepresentation Learning Predictive Representation Learning
LaTIM UMR 1101, Inserm, Brest, 29200, France.
Diffusion-Driven State Space Models
Jun 19, 2026Jack Ruder, Michael WojnowiczLatent Dynamics Modeling Latent Diffusion Models
1Montana State University

Jun 11, 2026J/K move · Enter open · S save

Robust State-Conditional Feature-Weighted Jump Models for Temporal Clustering

Federico P. Cortese, Alessio Farcomeni

University of Milan, Department of Economics, Management, and Quantitative Methods · National Research Council of Italy, Institute for Applied Mathematics and Information Technologies, Milan · University of Rome “Tor Vergata”, Department of Enterprise Engineering

Clustering Unsupervised Clustering Feature Selection Time Series Clustering

Open paper AskPDF arXiv

We propose a robust feature-weighted jump model for time-dependent clustering. A penalty is used to encourage smoothness of transitions over time, while robustness is achieved through the use of a Tukey's biweight loss function. An additional parameter controls the variability of feature weights across states, allowing the model to assign state-specific relevance to each feature. We illustrate in simulation how the method accurately recovers the true cluster sequence and reliably identifies relevant features, outperforming competing approaches, particularly in the presence of outliers. We conclude with two empirical applications, one on the number of conflict-related homicides in Kosovo in the period 1998-2000, and another on macroeconomic performance of twelve European countries in the period 1949-2024.

Abstract

Markov Models

Explore similar work

Robust State-Conditional Feature-Weighted Jump Models for Temporal Clustering

Constrained latent state modeling: A unifying perspective on representation learning under competing constraints

Diffusion-Driven State Space Models