A Behavior-first approach to Multi-Intention Inverse Reinforcement Learning

Antonio Mone; Frans A. Oliehoek; Luciano Cavalcante Siebert

cs.LGFeb 7, 2026

A Behavior-first approach to Multi-Intention Inverse Reinforcement Learning

Authors: Antonio Mone, Frans A. Oliehoek, Luciano Cavalcante Siebert

Organizations: Delft University of Technology, the Netherlands

Abstract

Inverse Reinforcement Learning (IRL) seeks to infer reward functions from expert demonstrations. When demonstrations originate from multiple experts with different intentions, the problem is known as Multi-Intention IRL (MI-IRL). Recent deep generative MI-IRL approaches couple behavior clustering and reward learning, but typically require prior knowledge of the number of true behavioral modes $K^*$ . This reliance on expert knowledge limits their adaptability to new behaviors, and only enables analysis related to the learned rewards, and not across the behavior modes used to train them. We propose Contrastive Multi-Intention IRL (CoMI-IRL), a transformer-based unsupervised framework that decouples behavior representation and clustering from downstream reward learning. Our experiments show that CoMI-IRL outperforms existing approaches without a priori knowledge of $K^*$ or labels, while allowing for visual interpretation of behavior relationships and adaptation to unseen behavior without full retraining.

Figures & tables

Figure 1: CoMI-IRL pipeline. States and actions of each quantile normalized input trajectory $\tau_{i}$ pass through a Random Fourier Feature ( RFF ) encoding with Gaussian mapping, a shallow MLP and a 1D CNN layer. After adding timestep and modality embeddings (distinguishing states and actions), we interleave the two sequences and prepend a $CLS$ token summarizing the trajectory. On the resulting embeddings, we apply graph-clustering to get cluster labels and apply IRL independently on each cluster.

Figure 6: Median

\pm

IQR and Mean

\pm

Std for HalfCheetah.

Metric	Model	Reacher	Pusher	Walker	Hopper	HalfCheetah
NMI $\uparrow$	(K=3)-GAIL/AIRL	$0.16\pm 0.02$	$0.72\pm 0.00$	$0.53\pm 0.02$	$\mathbf{1.00\pm 0.00}$	$0.71\pm 0.06$
	(K=6)-GAIL/AIRL	$0.43\pm 0.04$	$\mathbf{1.00\pm 0.00}$	$0.45\pm 0.03$	$0.76\pm 0.00$	$0.73\pm 0.00$
	(K=12)-GAIL/AIRL	$0.41\pm 0.01$	$0.87\pm 0.01$	$0.45\pm 0.02$	$0.66\pm 0.01$	$0.68\pm 0.02$
	G-GAIL/AIRL	$0.01\pm 0.00$	$\mathbf{1.00\pm 0.00}$	$0.50\pm 0.00$	$\mathbf{1.00\pm 0.00}$	$0.90\pm 0.00$
	(K=3) Ess-InfoGAIL	$0.53\pm 0.03$	$0.74\pm 0.06$	$0.45\pm 0.07$	$\mathbf{1.00\pm 0.00}$	$0.76\pm 0.06$
	(K=6) Ess-InfoGAIL	$0.68\pm 0.03$	$0.90\pm 0.04$	$0.36\pm 0.08$	$0.58\pm 0.01$	$0.62\pm 0.06$

Table 1: Clustering performance of all model categories across the considered environments. Results are reported as mean

\pm

standard deviation over five seeds. Higher values are better for all metrics, with best results bolded. For Reacher and Pusher,

K^{*}=6

, while for Walker, Hopper and HalfCheetah

K^{*}=3

Figure 7: 2D UMAP visualizations of the original trajectory space at the top of each environment (from left to right: Reacher, Pusher, Walker2D, Hopper, HalfCheetah), and the respective embedding space resulting from the BE at the bottom.

Table 7

Method	Pusher-v4	Hopper-v4	HalfCheetah-v4	Walker2d-v4
CoMI-IRL (AIRL)	$\mathbf{0.3360\pm 0.1081}$	$\mathbf{0.5639\pm 0.1691}$	$0.4409\pm 0.0478$	$0.6134\pm 0.0895$
	(Reference)	(Reference)	( $p=0.0059$ )	( $p=0.2676$ [n.s.])
CoMI-IRL (GAIL)	$0.4938\pm 0.0884$	$0.7159\pm 0.1445$	$\mathbf{0.3746\pm 0.0673}$	$\mathbf{0.5759\pm 0.0919}$
	( $p=1.12\times 10^{-7}$ )	( $p=0.0164$ )	(Reference)	(Reference)
Ess-InfoGAIL (K=12)	$0.6528\pm 0.0888$	$0.7037\pm 0.0848$	$0.7928\pm 0.0950$	$0.7338\pm 0.0773$
	( $p=2.60\times 10^{-18}$ )	( $p=0.0082$ )	( $p=1.20\times 10^{-15}$ )	( $p=4.79\times 10^{-6}$ )

Table 4: EPIC distance (Mean

\pm

Std) to the true reward function. Comparisons and exact

p

-values are computed against the best-performing CoMI-IRL configuration (in bold) for each environment. Statistically non-significant differences (

p\geq 0.05

) are marked with [n.s.].

Method	Type	EPIC Distance	$p$ -value vs. Ref
CoMI-IRL (AIRL)	Single	$\mathbf{0.7113\pm 0.0274}$	(Reference)
CoMI-IRL (AIRL) (Combined)	Combined	$\mathbf{0.5373\pm 0.0595}$	(Reference)
CoMI-IRL (GAIL)	Single	$0.7762\pm 0.0317$	$8.03\times 10^{-22}$
CoMI-IRL (GAIL) (Combined)	Combined	$0.6165\pm 0.0994$	$0.0006$
Ess-InfoGAIL (K=12)	Single	$0.7060\pm 0.1167$	$0.7341$ [n.s.]
Ess-InfoGAIL (K=12) (Combined)	Combined	$0.7109\pm 0.1217$	$2.00\times 10^{-8}$

Table 5: EPIC distance (Mean

\pm

Std) for Reacher-v4. Single models are compared against CoMI-IRL (AIRL) , and combined (max-reduced) models are compared against CoMI-IRL (AIRL) (Combined) .

Reacher-v4 ( $K^{*}=6$ )
Stage	NMI	ARI	Sil	ATR	$\mathbf{\hat{K}}$
$K_{\text{base}}$	0.76 $\pm$ 0.00	0.57 $\pm$ 0.00	0.85 $\pm$ 0.00	0.93 $\pm$ 0.11	6 $\pm$ 0.00
$K_{\text{finetuned}}$	0.84 $\pm$ 0.00	0.63 $\pm$ 0.00	0.81 $\pm$ 0.00	0.93 $\pm$ 0.10	12 $\pm$ 0.00
Pusher-v4 ( $K^{*}=6$ )
$K_{\text{base}}$	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.76 $\pm$ 0.00	1.10 $\pm$ 0.14	3 $\pm$ 0.00
$K_{\text{finetuned}}$	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.76 $\pm$ 0.00	1.09 $\pm$ 0.12	6 $\pm$ 0.00

Table 6: Adaptation Performance. NMI, ARI, Silhouette, and ATR scores before (

K_{\text{base}}

) and after (

K_{\text{finetuned}}

) adapting to unseen behaviors.

K^{*}

denotes the total number of modes in the final dataset and

\hat{K}

the number of inferred ones.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Left to Right: Reacher, Pusher, Walker2d, HalfCheetah, Hopper.

AIRL Hyperparameters
Buffer: batch=256, capacity=2048
Disc: updates=32, reward_hid=(32,), pot_hid=(32,32)
PPO: lr=3e-4, batch=64, ent=0.01, $\gamma$ =0.99
clip=0.2, epochs=10
Training: 250k timesteps/cluster (1.5M for Walker2d-v4)

Appendix

Table 7: AIRL configuration. Disc=discriminator, pot_hid=potential hidden sizes.

Hyperparameter	Value	Hyperparameter	Value
use_obs_norm	True	lr_actor	3e-3
auto_lr	True	lr_critic	3e-3
rollout_length	5000	lr_prior	3e-3
batch_size	1000	lr_disc	5e-3
num_steps	2000000	lr_q	1e-2
eval_interval	200000	epoch_ppo	20

Appendix

Table 8: Hyperparameters used for Ess-InfoGAIL.

Shared: emb=32, heads=4, layers=2, hid=1024
Param	Reach	Push	Walk	Hopp	HC
drop=0.1, $m_{s}$ =64, $m_{a}$ =32
Loss: $\alpha/\beta/\gamma/\delta$ =0.5/1.0/0.5/1.0; epochs=100
lr	1e-4	1e-4	5e-4	1e-4	1e-4
inp_ch	6	20	17	11	17
max_len	50	100	100	100	100

Appendix

Table 9: BE hyperparameters. Reach=Reacher-v4, Push=Pusher-v4, Walk=Walker2d-v4, Hopp=Hopper-v4, HC=HalfCheetah-v4. Shared parameters use abbreviated names: emb=embedding dim, heads=attention heads, hid=hidden dim, drop=dropout,

m_{s}/m_{a}

=Gaussian means (state/action),

\sigma_{s}/\sigma_{a}

=Gaussian sigmas.

	Description	Value
$k_{\min}$	Min neighbors for graph	$\max(5,N/50)$
$k_{\max}$	Max neighbors for graph	$\min(100,N/3)$
$\sigma$	RBF kernel bandwidth	1.0
$m$	Min cluster size	$\max(5,0.02\cdot N)$
$\alpha_{\text{behav}}$	Jacobian weight	0.3
$\theta_{\text{novelty}}$	Novelty threshold	0.05–0.1

Appendix

Table 10: Graph clustering hyperparameters

Environment	Active Components ( $K$ )	NMI	ARI	ATR
Reacher-v4	1.0 $\pm$ 0.0	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.16 $\pm$ 0.06
Pusher-v4	1.2 $\pm$ 0.4	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.50 $\pm$ 0.40
Walker2d-v4	3.8 $\pm$ 0.7	0.07 $\pm$ 0.05	0.04 $\pm$ 0.06	-0.86 $\pm$ 1.87
Hopper-v4	1.0 $\pm$ 0.0	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.18 $\pm$ 0.22
HalfCheetah-v4	2.4 $\pm$ 1.0	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.37 $\pm$ 0.33

Appendix

Table 11: Clustering Metrics for the Subgoal Baseline (Averaged across 5 seeds)

Method	Pusher-v4	Hopper-v4	HalfCheetah-v4	Walker2d-v4
CoMI-IRL (AIRL)	$\mathbf{0.3360\pm 0.1081}$	$\mathbf{0.5639\pm 0.1691}$	$0.4409\pm 0.0478$	$0.6134\pm 0.0895$
	(Reference)	(Reference)	( $p=0.0059$ )	( $p=0.2676$ [n.s.])
CoMI-IRL (GAIL)	$0.4938\pm 0.0884$	$0.7159\pm 0.1445$	$\mathbf{0.3746\pm 0.0673}$	$\mathbf{0.5759\pm 0.0919}$
	( $p=1.12\times 10^{-7}$ )	( $p=0.0164$ )	(Reference)	(Reference)
G-AIRL	$0.3470\pm 0.1135$	$0.6343\pm 0.2171$	$0.4356\pm 0.0386$	$0.6356\pm 0.0778$
	( $p=0.7061$ [n.s.])	( $p=0.3470$ [n.s.])	( $p=0.0074$ )	( $p=0.1040$ [n.s.])

Appendix

Table 12: EPIC distance (Mean

\pm

Std) to the true reward function. Comparisons and exact

p

-values are computed against the best-performing CoMI-IRL configuration (in bold) for each environment. Statistically non-significant differences (

p\geq 0.05

) are marked with [n.s.].

Method	Type	EPIC Distance	$p$ -value vs. Ref
CoMI-IRL (AIRL)	Single	$\mathbf{0.7113\pm 0.0274}$	(Reference)
CoMI-IRL (AIRL) (Combined)	Combined	$\mathbf{0.5373\pm 0.0595}$	(Reference)
CoMI-IRL (GAIL)	Single	$0.7762\pm 0.0317$	$8.03\times 10^{-22}$
CoMI-IRL (GAIL) (Combined)	Combined	$0.6165\pm 0.0994$	$0.0006$
Ess-InfoGAIL (K=12)	Single	$0.7060\pm 0.1167$	$0.7341$ [n.s.]
Ess-InfoGAIL (K=12) (Combined)	Combined	$0.7109\pm 0.1217$	$2.00\times 10^{-8}$

Appendix

Table 13: EPIC distance (Mean

\pm

Std) for Reacher-v4 . Single models are compared against CoMI-IRL (AIRL) , and combined (max-reduced) models are compared against CoMI-IRL (AIRL) (Combined) .

Configuration	NMI	ARI	Sil
Reacher-v4
Architecture
BE (Full)	0.84 $\pm$ 0.00	0.63 $\pm$ 0.00	0.87 $\pm$ 0.06
BE $\setminus$ (preBlock)	0.36 $\pm$ 0.32	0.21 $\pm$ 0.23	0.29 $\pm$ 0.25
Loss Design
BE + C	0.44 $\pm$ 0.10	0.26 $\pm$ 0.08	0.39 $\pm$ 0.07

Appendix

Table 14: Ablation Study. Impact of architecture (removing the RFF+MLP+CNN preprocessing) over

3

seeds and loss components on clustering quality over

10

seeds. Loss keys: C =Global Contrastive, D =DIM, S =Segment/Pairwise.

Figure 9: Jacobian Edge Reweighting on Pusher.

Figure 10: Jacobian Edge Reweighting on Reacher.

Figure 11: Jacobian Edge Reweighting on Walker.

CoMI-IRL
	Reacher	Pusher	Walker	Hopper	HalfCheetah
BE	$\sim 16$	$\sim 25$	$\sim 17$	$\sim 18$	$\sim 18$
Clustering (s)	$\sim 0$	$\sim 0$	$\sim 20$	$\sim 0$	$\sim 0$
IRL	$\sim 14$	$\sim 14$	$\sim 48$	$\sim 20$	$\sim 48$
Total	$\sim 30$	$\sim 39$	$\sim 65$	$\sim 38$	$\sim 66$
Ess-InfoGAIL

Appendix

Table 15: Wall-clock time for the different components of CoMI-IRL and total, and the Ess-InfoGAIL experiments with different values of

K

. While all values are reported in minutes, the clustering values are reported in seconds. In CoMI-IRL’s IRL phase, the reported time is the time of a single cluster, as it’s fully parallelizable.

Table 24

Table 25

Method 1	Method 2	$p_{\text{FDR}}$	$d_{z}$
CoMI-IRL (GAIL)	Ess-InfoGAIL(12)	$4.14e{-18}$	$2.90$
Ess-InfoGAIL(6)	G-GAIL	$4.14e{-18}$	$-3.08$
CoMI-IRL (AIRL)	Ess-InfoGAIL(12)	$4.14e{-18}$	$3.59$
CoMI-IRL (AIRL)	Ess-InfoGAIL(6)	$4.14e{-18}$	$3.49$
Ess-InfoGAIL(3)	G-GAIL	$4.14e{-18}$	$-2.93$
CoMI-IRL (AIRL)	Ess-InfoGAIL(3)	$4.14e{-18}$	$3.35$

Appendix

Table 20: Pairwise statistical comparisons for HalfCheetah-v4 (Benjamini–Hochberg FDR correction,

\alpha=0.05

). Only significant pairs shown (

p_{\text{FDR}}<0.05

d_{z}

= Cohen’s

d_{z}

(paired effect size).

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL
Jun 2, 2026Yikang Gui, Bikramjit Banerjee, Prashant DoshiDisentangled Representation Learning Reinforcement Learning
School of Computing, University of Georgia · School of Computing Sciences & Computer Engineering, The University of Southern Mississippi
Probabilistic Recurrent Intention Switching Model
May 26, 2026Wenyuan Sheng, Hao Zhu, Joschka BoedeckerReinforcement Learning
Department of Computer Science, University of Freiburg · IMBIT//BrainLinks-BrainTools
Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning
Jul 20, 2026Ziyi Liu, Grace ZhangReinforcement Learning Learning from Demonstration
University of Southern California

Jun 2, 2026J/K move · Enter open · S save

ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL

Yikang Gui, Bikramjit Banerjee, Prashant Doshi

School of Computing, University of Georgia · School of Computing Sciences & Computer Engineering, The University of Southern Mississippi

Disentangled Representation Learning Reinforcement Learning Representation Learning for RL Transfer Learning in RL

Open paper AskPDF arXiv

Reward transfer in Inverse Reinforcement Learning (IRL) is unreliable when policies must generalize to unseen combinations of environment dynamics and task goals. We propose Factorized Contrastive Abstractions for Transferable IRL (ConTraIRL), a framework that enables compositional reward transfer by learning decoupled latent representations of these two factors. ConTraIRL uses a dual-encoder architecture that maps observations into separate dynamics and goal latent spaces, trained with a dual contrastive objective. Temporal alignment encourages the dynamics encoder to learn goal-invariant structure, while the goal encoder captures dynamics-invariant features. This factorization supports reward inference under recombined dynamics-goal settings. Experiments on continuous control benchmarks demonstrate effective few-shot transfer to unseen dynamics-goal pairings, improving sample efficiency and reward recovery over transfer IRL baselines.

Abstract

Reinforcement Learning

Explore similar work

ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL

Probabilistic Recurrent Intention Switching Model

Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning