Temporal Logic Guided Universal Task Representations for Reinforcement Learning

Hao Zhang; Zhangli Zhou; Zhen Kan

cs.ROAug 16, 2026

Temporal Logic Guided Universal Task Representations for Reinforcement Learning

Authors: Hao Zhang, Zhangli Zhou, Zhen Kan

Organizations: Department of Automation at the University of Science and Technology of China, Hefei, Anhui, China, 230026 · Anhui Provincial Key Laboratory of Humanoid Robots, Hefei, China, 230000

Abstract

Task guided agents demonstrate strong performance in a wide range of complex tasks. However, most existing task representation algorithms are tailored to specific contexts and struggle to generalize across diverse scenarios. Moreover, they typically depend on gradient signals from reinforcement learning controllers to update their weights, which can degrade both representation quality and learning efficiency. To overcome these limitations, we propose LOTUS, a temporal logic inspired universal task representation framework that can be seamlessly integrated into any RL algorithm to enhance agent performance across diverse task settings. Specifically, we design a novel task representation architecture capable of modeling relationships and extracting task semantics from LTL formulas. We further introduce a more effective update mechanism that treats the LTL encoder as a policy, thereby improving representation capacity. To enhance stability and robustness, LOTUS leverages the bisimulation metric, which provides theoretical guarantees for LTL representation, including behavioral equivalence, optimality fidelity, and trajectory robustness. Experimental results show that LOTUS outperforms most existing methods in learning efficiency, generalization capability, and representation quality. Specifically, LOTUS accelerates convergence over 20% in single-task scenarios, achieves a 15%-45% higher success rate in unseen manipulation tasks, and improves generalization performance over 25% in complex multi-task environments with increased sub-goal depth or conjunctions. The corresponding code, videos, and appendix are available at: https://lotus-website.github.io/.

Figures & tables

Figure 1: Example of a robot completing different LTL tasks $\varphi_{even}$ , $\varphi_{avoid}$ , $\varphi_{cleanup}$ , and $\varphi_{toilet}$ across diverse scenarios. The different colored trajectories present possible solutions for the robot in completing the tasks.

Task Scenarios	LTL Formulas
Partially-Ordered Tasks	$\varphi_{even}=\lozenge(\mathrm{Yellow}\wedge\lozenge(\mathrm{Black}\wedge\lozenge(\mathrm{White}\wedge\lozenge\mathrm{Red})))$
Avoidance Tasks	$\varphi_{avoid}=\varphi_{dang}\cup(\mathrm{White}\wedge(\varphi_{dang}\cup\mathrm{Yellow}))$
Cleanup	$\varphi_{cleanup}=\lozenge(\varphi_{push\text{\_}jellobox}\wedge\lozenge\varphi_{pnp\text{\_}spamcan})$
Toilet	$\varphi_{toilet}=\lozenge(\mathrm{lid\_approached}\wedge\lozenge(\mathrm{lid\_grasped}\wedge\lozenge\mathrm{lid\_opened}))$

Table I: Task Scenarios and corresponding LTL formulas

Figure 2: The outline of LOTUS. (a) The task scenarios in our simulation for evaluation. (b) The architecture of LOTUS. (c) The snapshots of task execution. (d) The real-world experimental platform constructed for $\varphi_{stack}$ and $\varphi_{cleanup}$ .

Figure 3: The examples about how to build the directed graph from LTL specifications.

Figure 5: The comparison of different update methods for the LTL encoder. (a) Optimize the LTL representations by the back- from the RL controller. (b) Regard the LTL encoder as a policy and further optimize it by leveraging the rewards from the environment.

Figure 6: Plots of discounted return curves for four kinds of multi-task learning from different algorithms in two scenarios.

Figure 7: Plots of normalized reward curves from different algorithms in four manipulation tasks.

Task	LetterEnv
Type	Avoidance Tasks
Split	I.I.D	Depth ( $\uparrow$ )	Conjuncts ( $\uparrow$ )
No LTL	-0.054 $\pm$ 0.228	-0.054 $\pm$ 0.229	-0.072 $\pm$ 0.248
LSTM	0.883 $\pm$ 0.246	0.464 $\pm$ 0.551	0.164 $\pm$ 0.406
T1TL	-0.071 $\pm$ 0.344	-0.086 $\pm$ 0.278	-0.223 $\pm$ 0.422
LTL2Action	0.965 $\pm$ 0.233	0.899 $\pm$ 0.314	0.536 + 0.557

Table II: Total return of different methods in two scenarios (LetterWorld and ZoneEnv) with different task types.

Task	Toilet		Trashcan		Faucet		Dispenser
Split	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen
DFA	0.815 $\pm$ 0.062	0.584 $\pm$ 0.069	0.643 $\pm$ 0.065	0.494 $\pm$ 0.008	0.687 $\pm$ 0.101	0.509 $\pm$ 0.112	0.746 $\pm$ 0.057	0.220 $\pm$ 0.031
LSTM	0.811 $\pm$ 0.033	0.566 $\pm$ 0.056	0.733 $\pm$ 0.061	0.566 $\pm$ 0.147	0.799 $\pm$ 0.066	0.609 $\pm$ 0.097	0.656 $\pm$ 0.052	0.258 $\pm$ 0.025
T2TL	0.844 $\pm$ 0.028	0.695 $\pm$ 0.116	0.570 $\pm$ 0.135	0.444 $\pm$ 0.034	0.842 $\pm$ 0.062	0.657 $\pm$ 0.104	0.733 $\pm$ 0.037	0.249 $\pm$ 0.071
LTL2Action	0.786 $\pm$ 0.074	0.603 $\pm$ 0.067	0.513 $\pm$ 0.120	0.477 $\pm$ 0.156	0.763 $\pm$ 0.064	0.595 $\pm$ 0.007	0.647 $\pm$ 0.105	0.171 $\pm$ 0.052
HGT	0.831 $\pm$ 0.030	0.651 $\pm$ 0.038	0.646 $\pm$ 0.057	0.555 $\pm$ 0.150	0.775 $\pm$ 0.075	0.585 $\pm$ 0.102	0.746 $\pm$ 0.024	0.271 $\pm$ 0.033

Table III: Comparison of success rate on four manipulation scenarios (Toilet, Trashcan, Faucet, and Dispenser tasks) for both seen and unseen objects.

Task	LetterWorld
Type	Avoidance Tasks			Partially-Ordered Tasks
p-value ( $\downarrow$ )	I.I.D	Depth ( $\uparrow$ )	( $\uparrow$ )	I.I.D	Depth ( $\uparrow$ )	( $\uparrow$ )
No LTL	<0.001	<0.001	<0.001	0.001	<0.001	<0.001
LSTM	0.164	0.001	0.012	0.008	0.004	<0.001
T1TL	<0.001	<0.001	<0.001	0.005	<0.001	<0.001
LTL2Action	0.869	0.956	0.784	/	0.245	0.001

Table IV: P-values obtained for other algorithms relative to LOTUS via the Wilcoxon signed-rank test in LetterWorld and ZoneEnv tasks. (/ indicates the same value as LOTUS, so no Wilcoxon test is performed.)

Task	Toilet		Trashcan		Faucet		Dispenser
Split	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen
DFA	0.002	0.002	<0.001	<0.001	<0.001	<0.001	<0.001	<0.001
LSTM	0.002	0.002	0.202	0.017	<0.001	<0.001	<0.001	<0.001
T2TL	0.003	0.084	<0.001	<0.001	<0.001	<0.001	<0.001	<0.001
LTL2Action	0.002	0.002	<0.001	<0.001	<0.001	<0.001	<0.001	<0.001
HGT	0.002	0.002	<0.001	0.021	<0.001	<0.001	<0.001	<0.001

Table V: P-values obtained for other algorithms relative to LOTUS via the Wilcoxon signed-rank test in four manipulation scenarios (Toilet, Trashcan, Faucet, and Dispenser tasks) for Both Seen and Unseen Objects.

Figure 8: The trajectories of different algorithms when tackling the increasingly partially-ordered tasks $\varphi_{even}^{1}$ , $\varphi_{even}^{2}$ , $\varphi_{even}^{3}$ , and $\varphi_{even}^{4}$ in ZoneEnv.

Task	Toilet		Faucet
Split	Seen	Unseen	Seen	Unseen
No LTL	0.850 $\pm$ 0.01	0.550 $\pm$ 0.010	0.790 ± 0.020	0.580 ± 0.070
RGTN	0.880 $\pm$ 0.020	0.590 $\pm$ 0.007	0.880 $\pm$ 0.012	0.747 $\pm$ 0.029
TLPG	0.774 $\pm$ 0.031	0.569 $\pm$ 0.049	0.909 $\pm$ 0.007	0.749 $\pm$ 0.026
LOTUS	0.888 $\pm$ 0.004	0.728 $\pm$ 0.093	0.920 $\pm$ 0.022	0.794 $\pm$ 0.013

Table VI: Success Rate of Different Methods in Toilet and Faucet tasks when ablating different component designs.

Task	Toilet		Faucet
Split	Seen	Unseen	Seen	Unseen
$\mathrm{RGTN_{w/o}^{MRA}}$	0.754 $\pm$ 0.085	0.585 $\pm$ 0.144	0.754 $\pm$ 0.085	0.604 $\pm$ 0.144
$\mathrm{RGTN_{w/o}^{MHA}}$	0.768 $\pm$ 0.072	0.581 $\pm$ 0.104	0.787 $\pm$ 0.059	0.594 $\pm$ 0.077
$\mathrm{RGTN_{w/o}^{Residual}}$	0.819 $\pm$ 0.026	0.548 $\pm$ 0.011	0.856 $\pm$ 0.008	0.728 $\pm$ 0.030
RGTN	0.880 $\pm$ 0.020	0.590 $\pm$ 0.007	0.880 $\pm$ 0.012	0.747 $\pm$ 0.029

Table VII: Success Rate of Different Methods in Toilet and Faucet tasks when ablating different architecture designs.

Figure 9: Visualizations of (a) the embedding spaces generated by R-GCN and RGTN, (b) cosine similarity heatmap generated by RGTN, and (c) Euclidean distance heatmap generated by RGTN.

Figure 10: Snapshots of $\varphi_{stack}$ and $\varphi_{cleanup}$ .

Task	Toilet		Faucet
Split	Seen	Unseen	Seen	Unseen
$\mathrm{RGTN_{w/o}^{MRA}}$	0.754 $\pm$ 0.085	0.585 $\pm$ 0.144	0.754 $\pm$ 0.085	0.604 $\pm$ 0.144
$\mathrm{RGTN_{fixed}^{MRA}}$	0.844 $\pm$ 0.048	0.644 $\pm$ 0.044	0.729 $\pm$ 0.103	0.537 $\pm$ 0.140
$\mathrm{RGTN_{msg}^{MRA}}$	0.840 $\pm$ 0.054	0.590 $\pm$ 0.083	0.766 $\pm$ 0.035	0.609 $\pm$ 0.105
RGTN	0.880 $\pm$ 0.020	0.590 $\pm$ 0.007	0.880 $\pm$ 0.012	0.747 $\pm$ 0.029

Table VIII: Success Rate of Different Methods in Toilet and Faucet tasks when ablating different designs of MRA architecture

Task	Toilet		Faucet
Split	Seen	Unseen	Seen	Unseen
$\mathrm{RGTN_{w/o}^{MHA}}$	0.768 $\pm$ 0.072	0.581 $\pm$ 0.104	0.787 $\pm$ 0.059	0.594 $\pm$ 0.077
$\mathrm{RGTN_{stand}^{MHA}}$	0.817 $\pm$ 0.049	0.605 $\pm$ 0.047	0.781 $\pm$ 0.032	0.652 $\pm$ 0.052
$\mathrm{RGTN_{all}^{MHA}}$	0.846 $\pm$ 0.039	0.581 $\pm$ 0.066	0.788 $\pm$ 0.042	0.628 $\pm$ 0.020
RGTN	0.880 $\pm$ 0.020	0.590 $\pm$ 0.007	0.880 $\pm$ 0.012	0.747 $\pm$ 0.029

Table IX: Success Rate of Different Methods in Toilet and Faucet tasks when ablating different designs of MHA architecture

Task	Toilet		Faucet
Split	Seen	Unseen	Seen	Unseen
$\mathrm{RGTN_{add}^{fuse}}$	0.773 $\pm$ 0.040	0.557 $\pm$ 0.004	0.699 $\pm$ 0.038	0.475 $\pm$ 0.067
$\mathrm{RGTN_{weighted}^{fuse}}$	0.782 $\pm$ 0.048	0.590 $\pm$ 0.077	0.781 $\pm$ 0.071	0.547 $\pm$ 0.075
$\mathrm{RGTN_{concat}^{fuse}}$ (Ours)	0.880 $\pm$ 0.020	0.590 $\pm$ 0.007	0.880 $\pm$ 0.012	0.747 $\pm$ 0.029

Table X: Success Rate of Different Methods in Toilet and Faucet tasks when ablating different designs of the fusion information module

Task	LetterWorld (second)		ZoneEnv (second)
Task Type	Avoidance	Partially-Ordered	Avoidance	Partially-Ordered
No LTL	10.416 + 1.255	8.583 + 2.625	112.583 + 12.552	179.833 + 5.907
LSTM	11.667 + 0.425	10.583 + 0.816	210.167 + 21.602	288.917 + 10.274
T2TL	10.333 + 0.471	11.667 + 0.471	263.667 + 2.160	273.083 + 39.802
LTL2Action	12.750 + 2.867	13.667 + 3.091	258.583 + 10.198	308.000 + 10.677
HGT	15.435 + 3.658	14.333 + 4.784	292.917 + 21.297	243.417 + 8.340

Table XI: The computational cost (second) of different methods consuming for each training epoch in LetterWorld and ZoneEnv tasks

Task	LetterWorld		ZoneEnv
Type	Avoidance	Partially-Ordered	Avoidance	Partially-Ordered
No LTL	0.770	0.770	0.620	0.620
LSTM	0.660	0.602	0.645	0.529
T2TL	0.732	0.992	0.732	0.918
LTL2Action	0.488	0.490	0.990	0.949
HGT	0.649	0.672	1.215	1.066

Table XII: The GPU memory usage (GB) of different methods for each training epoch in LetterWorld and ZoneEnv tasks

Partially-Ordered Tasks	LetterWorld
Number of Sub-goals	cDFAs	LTL2Action	RGTN	LOTUS
25	0.113 $\pm$ 0.316	0.806 $\pm$ 0.394	0.963 $\pm$ 0.187	0.976 $\pm$ 0.103
26	0.080 $\pm$ 0.271	0.660 $\pm$ 0.473	0.930 $\pm$ 0.255	0.938 $\pm$ 0.170
27	0.043 $\pm$ 0.203	0.556 $\pm$ 0.496	0.826 $\pm$ 0.378	0.852 $\pm$ 0. 307
28	0.023 $\pm$ 0.150	0.416 $\pm$ 0.493	0.726 $\pm$ 0.445	0.740 $\pm$ 0.438
29	0.010 $\pm$ 0.099	0.303 $\pm$ 0.459	0.683 $\pm$ 0.465	0.716 $\pm$ 0.479

Table XIII: Total Return performance of Different Methods in deepen Partially-Ordered Tasks of LetterWorld.

Representation Learning for RL

Representation learning for reinforcement learning builds state, task, and dynamics features that help agents learn, explore, and transfer.

See all 69 papers

PlatoLTL: Scaling LTL-Guided Multi-Task RL
Jan 30, 2026Jacques Cloete, Mathias Jackermeier, Ioannis Havoutis +1RL for Robotics Constrained RL
Oxford Robotics Institute University of Oxford Oxford, United Kingdom · Department of Computer Science University of Oxford Oxford, United Kingdom
Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
Jun 4, 2026Johan Obando-Ceron, Lu Li, Scott Fujimoto +3Reinforcement Learning Representation Learning for RL
Mila – Québec AI Institute · Université de Montréal · McGill University +2
Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RL
Sep 29, 2026Mathias Jackermeier, Jacques Cloete, Alessandro AbateRL Benchmarks Reinforcement Learning
Department of Computer Science, University of Oxford · Oxford Robotics Institute, University of Oxford

Jan 30, 2026J/K move · Enter open · S save

PlatoLTL: Scaling LTL-Guided Multi-Task RL

Jacques Cloete, Mathias Jackermeier, Ioannis Havoutis, Alessandro Abate

Oxford Robotics Institute University of Oxford Oxford, United Kingdom · Department of Computer Science University of Oxford Oxford, United Kingdom

RL for Robotics Constrained RL Multi-Task RL Generalization in RL Goal-Conditioned RL

Open paper AskPDF arXiv

Linear temporal logic (LTL) has emerged as a powerful formalism for specifying structured, temporally extended tasks in multi-task reinforcement learning (RL). However, while existing approaches in LTL-guided multi-task RL demonstrate success in simple environments, they suffer from challenges in representation learning and exploration efficiency when applied to high-dimensional environments with parameterized specifications. We present PlatoLTL, which elevates state-of-the-art methods to address both challenges. We model atomic propositions as instances of atomic predicates and inject task parameters directly into the goal embedding to enable efficient generalization. We also leverage simple priors on closeness to predicate satisfaction to enable and accelerate learning of complex tasks without biasing the optimal policy. We validate our approach on challenging environments including robotic manipulation and multi-drone navigation.

Abstract

Representation Learning for RL

Explore similar work

PlatoLTL: Scaling LTL-Guided Multi-Task RL

Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RL