A Behavior-first approach to Multi-Intention Inverse Reinforcement Learning

Antonio Mone; Frans A. Oliehoek; Luciano Cavalcante Siebert

cs.LGFeb 7, 2026

A Behavior-first approach to Multi-Intention Inverse Reinforcement Learning

Authors: Antonio Mone, Frans A. Oliehoek, Luciano Cavalcante Siebert

Organizations: Delft University of Technology, the Netherlands

Abstract

Inverse Reinforcement Learning (IRL) seeks to infer reward functions from expert demonstrations. When demonstrations originate from multiple experts with different intentions, the problem is known as Multi-Intention IRL (MI-IRL). Recent deep generative MI-IRL approaches couple behavior clustering and reward learning, but typically require prior knowledge of the number of true behavioral modes $K^*$ . This reliance on expert knowledge limits their adaptability to new behaviors, and only enables analysis related to the learned rewards, and not across the behavior modes used to train them. We propose Contrastive Multi-Intention IRL (CoMI-IRL), a transformer-based unsupervised framework that decouples behavior representation and clustering from downstream reward learning. Our experiments show that CoMI-IRL outperforms existing approaches without a priori knowledge of $K^*$ or labels, while allowing for visual interpretation of behavior relationships and adaptation to unseen behavior without full retraining.

Figures & tables

Figure 1: CoMI-IRL pipeline. States and actions of each quantile normalized input trajectory $\tau_{i}$ pass through a Random Fourier Feature ( RFF ) encoding with Gaussian mapping, a shallow MLP and a 1D CNN layer. After adding timestep and modality embeddings (distinguishing states and actions), we interleave the two sequences and prepend a $CLS$ token summarizing the trajectory. On the resulting embeddings, we apply graph-clustering to get cluster labels and apply IRL independently on each cluster.

Figure 6: Median

\pm

IQR and Mean

\pm

Std for HalfCheetah.

Metric	Model	Reacher	Pusher	Walker	Hopper	HalfCheetah
NMI $\uparrow$	(K=3)-GAIL/AIRL	$0.16\pm 0.02$	$0.72\pm 0.00$	$0.53\pm 0.02$	$\mathbf{1.00\pm 0.00}$	$0.71\pm 0.06$
	(K=6)-GAIL/AIRL	$0.43\pm 0.04$	$\mathbf{1.00\pm 0.00}$	$0.45\pm 0.03$	$0.76\pm 0.00$	$0.73\pm 0.00$
	(K=12)-GAIL/AIRL	$0.41\pm 0.01$	$0.87\pm 0.01$	$0.45\pm 0.02$	$0.66\pm 0.01$	$0.68\pm 0.02$
	G-GAIL/AIRL	$0.01\pm 0.00$	$\mathbf{1.00\pm 0.00}$	$0.50\pm 0.00$	$\mathbf{1.00\pm 0.00}$	$0.90\pm 0.00$
	(K=3) Ess-InfoGAIL	$0.53\pm 0.03$	$0.74\pm 0.06$	$0.45\pm 0.07$	$\mathbf{1.00\pm 0.00}$	$0.76\pm 0.06$
	(K=6) Ess-InfoGAIL	$0.68\pm 0.03$	$0.90\pm 0.04$	$0.36\pm 0.08$	$0.58\pm 0.01$	$0.62\pm 0.06$

Table 1: Clustering performance of all model categories across the considered environments. Results are reported as mean

\pm

standard deviation over five seeds. Higher values are better for all metrics, with best results bolded. For Reacher and Pusher,

K^{*}=6

, while for Walker, Hopper and HalfCheetah

K^{*}=3

Figure 7: 2D UMAP visualizations of the original trajectory space at the top of each environment (from left to right: Reacher, Pusher, Walker2D, Hopper, HalfCheetah), and the respective embedding space resulting from the BE at the bottom.

Table 7

Method	Pusher-v4	Hopper-v4	HalfCheetah-v4	Walker2d-v4
CoMI-IRL (AIRL)	$\mathbf{0.3360\pm 0.1081}$	$\mathbf{0.5639\pm 0.1691}$	$0.4409\pm 0.0478$	$0.6134\pm 0.0895$
	(Reference)	(Reference)	( $p=0.0059$ )	( $p=0.2676$ [n.s.])
CoMI-IRL (GAIL)	$0.4938\pm 0.0884$	$0.7159\pm 0.1445$	$\mathbf{0.3746\pm 0.0673}$	$\mathbf{0.5759\pm 0.0919}$
	( $p=1.12\times 10^{-7}$ )	( $p=0.0164$ )	(Reference)	(Reference)
Ess-InfoGAIL (K=12)	$0.6528\pm 0.0888$	$0.7037\pm 0.0848$	$0.7928\pm 0.0950$	$0.7338\pm 0.0773$
	( $p=2.60\times 10^{-18}$ )	( $p=0.0082$ )	( $p=1.20\times 10^{-15}$ )	( $p=4.79\times 10^{-6}$ )

Table 4: EPIC distance (Mean

\pm

Std) to the true reward function. Comparisons and exact

p

-values are computed against the best-performing CoMI-IRL configuration (in bold) for each environment. Statistically non-significant differences (

p\geq 0.05

) are marked with [n.s.].

Method	Type	EPIC Distance	$p$ -value vs. Ref
CoMI-IRL (AIRL)	Single	$\mathbf{0.7113\pm 0.0274}$	(Reference)
CoMI-IRL (AIRL) (Combined)	Combined	$\mathbf{0.5373\pm 0.0595}$	(Reference)
CoMI-IRL (GAIL)	Single	$0.7762\pm 0.0317$	$8.03\times 10^{-22}$
CoMI-IRL (GAIL) (Combined)	Combined	$0.6165\pm 0.0994$	$0.0006$
Ess-InfoGAIL (K=12)	Single	$0.7060\pm 0.1167$	$0.7341$ [n.s.]
Ess-InfoGAIL (K=12) (Combined)	Combined	$0.7109\pm 0.1217$	$2.00\times 10^{-8}$

Table 5: EPIC distance (Mean

\pm

Std) for Reacher-v4. Single models are compared against CoMI-IRL (AIRL) , and combined (max-reduced) models are compared against CoMI-IRL (AIRL) (Combined) .

Reacher-v4 ( $K^{*}=6$ )
Stage	NMI	ARI	Sil	ATR	$\mathbf{\hat{K}}$
$K_{\text{base}}$	0.76 $\pm$ 0.00	0.57 $\pm$ 0.00	0.85 $\pm$ 0.00	0.93 $\pm$ 0.11	6 $\pm$ 0.00
$K_{\text{finetuned}}$	0.84 $\pm$ 0.00	0.63 $\pm$ 0.00	0.81 $\pm$ 0.00	0.93 $\pm$ 0.10	12 $\pm$ 0.00
Pusher-v4 ( $K^{*}=6$ )
$K_{\text{base}}$	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.76 $\pm$ 0.00	1.10 $\pm$ 0.14	3 $\pm$ 0.00
$K_{\text{finetuned}}$	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0.76 $\pm$ 0.00	1.09 $\pm$ 0.12	6 $\pm$ 0.00

Table 6: Adaptation Performance. NMI, ARI, Silhouette, and ATR scores before (

K_{\text{base}}

) and after (

K_{\text{finetuned}}

) adapting to unseen behaviors.

K^{*}

denotes the total number of modes in the final dataset and

\hat{K}

the number of inferred ones.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Left to Right: Reacher, Pusher, Walker2d, HalfCheetah, Hopper.

AIRL Hyperparameters
Buffer: batch=256, capacity=2048
Disc: updates=32, reward_hid=(32,), pot_hid=(32,32)
PPO: lr=3e-4, batch=64, ent=0.01, $\gamma$ =0.99
clip=0.2, epochs=10
Training: 250k timesteps/cluster (1.5M for Walker2d-v4)

Appendix

Table 7: AIRL configuration. Disc=discriminator, pot_hid=potential hidden sizes.

Hyperparameter	Value	Hyperparameter	Value
use_obs_norm	True	lr_actor	3e-3
auto_lr	True	lr_critic	3e-3
rollout_length	5000	lr_prior	3e-3
batch_size	1000	lr_disc	5e-3
num_steps	2000000	lr_q	1e-2
eval_interval	200000	epoch_ppo	20

Appendix

Table 8: Hyperparameters used for Ess-InfoGAIL.

Shared: emb=32, heads=4, layers=2, hid=1024
Param	Reach	Push	Walk	Hopp	HC
drop=0.1, $m_{s}$ =64, $m_{a}$ =32
Loss: $\alpha/\beta/\gamma/\delta$ =0.5/1.0/0.5/1.0; epochs=100
lr	1e-4	1e-4	5e-4	1e-4	1e-4
inp_ch	6	20	17	11	17
max_len	50	100	100	100	100

Appendix

Table 9: BE hyperparameters. Reach=Reacher-v4, Push=Pusher-v4, Walk=Walker2d-v4, Hopp=Hopper-v4, HC=HalfCheetah-v4. Shared parameters use abbreviated names: emb=embedding dim, heads=attention heads, hid=hidden dim, drop=dropout,

m_{s}/m_{a}

=Gaussian means (state/action),

\sigma_{s}/\sigma_{a}

=Gaussian sigmas.

	Description	Value
$k_{\min}$	Min neighbors for graph	$\max(5,N/50)$
$k_{\max}$	Max neighbors for graph	$\min(100,N/3)$
$\sigma$	RBF kernel bandwidth	1.0
$m$	Min cluster size	$\max(5,0.02\cdot N)$
$\alpha_{\text{behav}}$	Jacobian weight	0.3
$\theta_{\text{novelty}}$	Novelty threshold	0.05–0.1

Appendix

Table 10: Graph clustering hyperparameters

Environment	Active Components ( $K$ )	NMI	ARI	ATR
Reacher-v4	1.0 $\pm$ 0.0	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.16 $\pm$ 0.06
Pusher-v4	1.2 $\pm$ 0.4	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.50 $\pm$ 0.40
Walker2d-v4	3.8 $\pm$ 0.7	0.07 $\pm$ 0.05	0.04 $\pm$ 0.06	-0.86 $\pm$ 1.87
Hopper-v4	1.0 $\pm$ 0.0	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.18 $\pm$ 0.22
HalfCheetah-v4	2.4 $\pm$ 1.0	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.37 $\pm$ 0.33

Appendix

Table 11: Clustering Metrics for the Subgoal Baseline (Averaged across 5 seeds)

Method	Pusher-v4	Hopper-v4	HalfCheetah-v4	Walker2d-v4
CoMI-IRL (AIRL)	$\mathbf{0.3360\pm 0.1081}$	$\mathbf{0.5639\pm 0.1691}$	$0.4409\pm 0.0478$	$0.6134\pm 0.0895$
	(Reference)	(Reference)	( $p=0.0059$ )	( $p=0.2676$ [n.s.])
CoMI-IRL (GAIL)	$0.4938\pm 0.0884$	$0.7159\pm 0.1445$	$\mathbf{0.3746\pm 0.0673}$	$\mathbf{0.5759\pm 0.0919}$
	( $p=1.12\times 10^{-7}$ )	( $p=0.0164$ )	(Reference)	(Reference)
G-AIRL	$0.3470\pm 0.1135$	$0.6343\pm 0.2171$	$0.4356\pm 0.0386$	$0.6356\pm 0.0778$
	( $p=0.7061$ [n.s.])	( $p=0.3470$ [n.s.])	( $p=0.0074$ )	( $p=0.1040$ [n.s.])

Appendix

Table 12: EPIC distance (Mean

\pm

Std) to the true reward function. Comparisons and exact

p

-values are computed against the best-performing CoMI-IRL configuration (in bold) for each environment. Statistically non-significant differences (

p\geq 0.05

) are marked with [n.s.].

Method	Type	EPIC Distance	$p$ -value vs. Ref
CoMI-IRL (AIRL)	Single	$\mathbf{0.7113\pm 0.0274}$	(Reference)
CoMI-IRL (AIRL) (Combined)	Combined	$\mathbf{0.5373\pm 0.0595}$	(Reference)
CoMI-IRL (GAIL)	Single	$0.7762\pm 0.0317$	$8.03\times 10^{-22}$
CoMI-IRL (GAIL) (Combined)	Combined	$0.6165\pm 0.0994$	$0.0006$
Ess-InfoGAIL (K=12)	Single	$0.7060\pm 0.1167$	$0.7341$ [n.s.]
Ess-InfoGAIL (K=12) (Combined)	Combined	$0.7109\pm 0.1217$	$2.00\times 10^{-8}$

Appendix

Table 13: EPIC distance (Mean

\pm

Std) for Reacher-v4 . Single models are compared against CoMI-IRL (AIRL) , and combined (max-reduced) models are compared against CoMI-IRL (AIRL) (Combined) .

Configuration	NMI	ARI	Sil
Reacher-v4
Architecture
BE (Full)	0.84 $\pm$ 0.00	0.63 $\pm$ 0.00	0.87 $\pm$ 0.06
BE $\setminus$ (preBlock)	0.36 $\pm$ 0.32	0.21 $\pm$ 0.23	0.29 $\pm$ 0.25
Loss Design
BE + C	0.44 $\pm$ 0.10	0.26 $\pm$ 0.08	0.39 $\pm$ 0.07

Appendix

Table 14: Ablation Study. Impact of architecture (removing the RFF+MLP+CNN preprocessing) over

3

seeds and loss components on clustering quality over

10

seeds. Loss keys: C =Global Contrastive, D =DIM, S =Segment/Pairwise.

Figure 9: Jacobian Edge Reweighting on Pusher.

Figure 10: Jacobian Edge Reweighting on Reacher.

Figure 11: Jacobian Edge Reweighting on Walker.

CoMI-IRL
	Reacher	Pusher	Walker	Hopper	HalfCheetah
BE	$\sim 16$	$\sim 25$	$\sim 17$	$\sim 18$	$\sim 18$
Clustering (s)	$\sim 0$	$\sim 0$	$\sim 20$	$\sim 0$	$\sim 0$
IRL	$\sim 14$	$\sim 14$	$\sim 48$	$\sim 20$	$\sim 48$
Total	$\sim 30$	$\sim 39$	$\sim 65$	$\sim 38$	$\sim 66$
Ess-InfoGAIL

Appendix

Table 15: Wall-clock time for the different components of CoMI-IRL and total, and the Ess-InfoGAIL experiments with different values of

K

. While all values are reported in minutes, the clustering values are reported in seconds. In CoMI-IRL’s IRL phase, the reported time is the time of a single cluster, as it’s fully parallelizable.

Table 24

Table 25

Method 1	Method 2	$p_{\text{FDR}}$	$d_{z}$
CoMI-IRL (GAIL)	Ess-InfoGAIL(12)	$4.14e{-18}$	$2.90$
Ess-InfoGAIL(6)	G-GAIL	$4.14e{-18}$	$-3.08$
CoMI-IRL (AIRL)	Ess-InfoGAIL(12)	$4.14e{-18}$	$3.59$
CoMI-IRL (AIRL)	Ess-InfoGAIL(6)	$4.14e{-18}$	$3.49$
Ess-InfoGAIL(3)	G-GAIL	$4.14e{-18}$	$-2.93$
CoMI-IRL (AIRL)	Ess-InfoGAIL(3)	$4.14e{-18}$	$3.35$

Appendix

Table 20: Pairwise statistical comparisons for HalfCheetah-v4 (Benjamini–Hochberg FDR correction,

\alpha=0.05

). Only significant pairs shown (

p_{\text{FDR}}<0.05

d_{z}

= Cohen’s

d_{z}

(paired effect size).

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Jun 2, 2026

ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL

Reward transfer in Inverse Reinforcement Learning (IRL) is unreliable when policies must generalize to unseen combinations of environment dynamics and task goals. We propose Factorized Contrastive Abstractions for Transferable IRL (ConTraIRL), a framework that enables compositional reward transfer by learning decoupled latent representations of these two factors. ConTraIRL uses a dual-encoder architecture that maps observations into separate dynamics and goal latent spaces, trained with a dual contrastive objective. Temporal alignment encourages the dynamics encoder to learn goal-invariant structure, while the goal encoder captures dynamics-invariant features. This factorization supports reward inference under recombined dynamics-goal settings. Experiments on continuous control benchmarks demonstrate effective few-shot transfer to unseen dynamics-goal pairings, improving sample efficiency and reward recovery over transfer IRL baselines.

Yikang Gui, Bikramjit Banerjee, Prashant Doshi

School of Computing, University of Georgia · School of Computing Sciences & Computer Engineering, The University of Southern Mississippi

Disentangled Representation Learning Reinforcement Learning Representation Learning for RL Transfer Learning in RL

Ask PDF

May 26, 2026

Probabilistic Recurrent Intention Switching Model

Inverse reinforcement learning (IRL) recovers reward functions from observed behavior, yet traditional methods assume a single stationary reward that cannot capture goal switching within an episode. Recent multi-intention IRL methods address this by segmenting trajectories, but model intention transitions as either a memoryless Markov chain or via manual state augmentation with a fixed history window. We propose the Probabilistic Recurrent Intention Switching Model (PRISM), which replaces both mechanisms with a lightweight recurrent network that maps observation history to a per-step intention distribution. We prove that the resulting EM objective decomposes exactly into independent per-intention reward subproblems, each solvable in closed form, yielding an

\mathcal{O}(nK)

E-step with no variational approximation. We evaluate PRISM on a non-Markovian gridworld, a mouse labyrinth, and BridgeData~V2 robotic manipulation, the first large-scale robotic application of multi-intention IRL. Across all settings PRISM achieves the highest held-out log-likelihood while recovering nameable, temporally coherent intentions from unlabeled demonstrations, suggesting that discrete goal switching is present in both biological and artificial agents.

Wenyuan Sheng, Hao Zhu, Joschka Boedecker

Department of Computer Science, University of Freiburg · IMBIT//BrainLinks-BrainTools

Reinforcement Learning

Ask PDF

Jul 20, 2026

Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning

Inverse reinforcement learning (IRL) provides a powerful framework for learning from demonstrations. However, real-world tasks often exhibit substantial natural variations (e.g., picking up mugs with varying shapes), making it impractical to collect demonstrations that fully specify a new task under every possible scenario. In practice, while demonstrations for the target task are limited, it is often easier to obtain datasets of heterogeneous but related behaviors. This motivates the problem of few-shot IRL with multi-task demonstrations (FM-IRL), where an agent must learn a new task with substantial variations from only a limited number of target-task demonstrations, together with sufficient demonstrations of related tasks and online agent experience. To do so, we must both recover the expert distribution of the new task and provide guidance when the agent deviates from it. We introduce Multitask discriminator Proximity-Guided IRL (MPG), which learns two complementary reward components: (1) a generalizable discriminator that transfers shared structure across related tasks to identify expert behavior in a new task, and (2) a proximity function that measures how far a state deviates from expert behavior and provides corrective guidance during exploration. We demonstrate the effectiveness of our method on multiple challenging navigation and manipulation tasks under significant variations (e.g., object configurations, table layouts, and initial robot poses), achieving an average success rate of 81.2%, outperforming the strongest per-task baseline by an average of 24.7 percentage points.

Ziyi Liu, Grace Zhang

University of Southern California

Reinforcement Learning Learning from Demonstration Robot Imitation Learning

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL

Probabilistic Recurrent Intention Switching Model

Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning