Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner

Andrei Polubarov; Lyubaykin Nikita; Alexander Derevyagin; Artyom Grishin; Igor Saprygin; Aleksandr Serkov; Mark Averchenko; Daniil Tikhonov

cs.LGApr 6, 2026

Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner

Authors: Andrei Polubarov, Lyubaykin Nikita, Alexander Derevyagin, Artyom Grishin, Igor Saprygin, Aleksandr Serkov, Mark Averchenko, Daniil Tikhonov,

Organizations: AXXX, Applied AI Institute · AXXX, Innopolis University · AXXX, HSE · Innopolis University ·

Abstract

Recent progress in in-context reinforcement learning (ICRL) has demonstrated its potential for training generalist agents that can acquire new tasks directly at inference. Algorithm Distillation (AD) pioneered this paradigm and was subsequently scaled to multi-domain settings, although its ability to generalize to unseen tasks remained limited. The Decision Pre-Trained Transformer (DPT) was introduced as an alternative, showing stronger in-context reinforcement learning abilities in simplified domains, but its scalability had not been established. In this work, we extend DPT to diverse multi-domain environments, applying Flow Matching as a natural training choice that preserves its interpretation as Bayesian posterior sampling. As a result, we obtain an agent trained across hundreds of diverse tasks that achieves clear gains in generalization to the held-out test set. This agent improves upon prior AD scaling and demonstrates stronger performance in both online and offline inference, reinforcing ICRL as a viable alternative to expert distillation for training generalist agents.

Figures & tables

Figure 1: Approach Overview. Stage 1: A dataset collected using the noise-distillation technique ( Zisman et al., 2024 ) , which covers suboptimal regions of the state space, is relabeled with the demonstrator’s optimal actions. Stage 2: During training, the flow-matching (OT) head is conditioned on hidden representations from the Decision Pretrained Transformer ( Lee et al., 2023 ) . Stage 3: At inference, actions are decoded using Heun’s method.

Domain	Tasks	Episodes	Timesteps	Sample Weight
Industrial-Benchmark	16	288k	72M	10.1%
Bi-DexHands	15	216.2k	31.7M	4.5%
Meta-World	45	670k	67M	9.4%
Kinetix	42	1.1M	62.8M	8.9%
CityLearn	20	146.4k	106.7M	15.0%
ControlGym	9	230k	100M	14.1%

Table 1: Training Dataset Summary. Training dataset covers 209 distinct tasks across 10 domains ranging from robotic manipulation to HVAC control and autonomous driving

Figure 2: Online inference on training tasks Although the model starts with no context, it infers relevant task-specific information for self-correction. All runs are conducted with four different random seeds, and the IQM of normalized scores is reported.

Figure 3: Domain-level normalized scores (Left) Evaluation results for 46 tasks unseen during training. (Right) Evaluation results for 209 training tasks. Offline runs of our model were conducted with a prompt size of 2500 transitions, compared to 5000 transitions for Vintix , and 25 and 100 episodes for REGENT on testing and training tasks, respectively. Domain-level aggregation is performed using IQM.

Figure 4: Action beliefs over context sizes Action distributions are shown for different prompt sizes during offline evaluation, ranging from no context to 10, 100, and 500 transitions of task-specific demonstrator data. Projections into 2D space are obtained using Truncated SVD. The gradual decrease in distributions’ entropy indicates that our model exhibits a behavior consistent with posterior sampling.

Figure 5: Normalized returns vs. number of demonstrations Offline evaluation is conducted on 46 held-out tasks with task-specific prompts of varying size. Results for Vintix and REGENT are reported on the corresponding domains. IQM aggregation is applied across 4 random seeds.

Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Action Decoder Steps	32
Learning Rate	0.00005
Gradient Clipping Norm	2.5
Optimizer	Adam
Beta 1	0.9
Beta 2	0.99

Appendix

Table 2: Hyperparameter configuration

Figure 6: Action beliefs over context sizes for other tasks in Meta-World

Figure 7: Action beliefs over context sizes for other tasks in SinerGym

Figure 8: Action beliefs over context sizes for other tasks in MetaDrive

Figure 9: Action beliefs over context sizes for other tasks in CityLearn

Figure 10: Mean normalized noise-distilled trajectories for Meta-World domain

Figure 11: Mean normalized noise-distilled trajectories for Kinetix domain

Figure 12: Mean normalized noise-distilled trajectories for Bi-DexHands domain

Figure 13: Mean normalized noise-distilled trajectories for MetaDrive domain

Figure 14: Mean normalized noise-distilled trajectories for Industrial-Benchmark domain

Figure 15: Mean normalized noise-distilled trajectories for ControlGym domain

Figure 16: Mean normalized noise-distilled trajectories for MuJoCo domain

Figure 17: Mean normalized noise-distilled trajectories for SinerGym domain

Figure 18: Mean normalized noise-distilled trajectories for CityLearn domain

Figure 19: Mean normalized noise-distilled trajectories for HumEnv domain

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
bin-picking-v2	test	1.9	410.0 ± 64.44	0.0 ± 0.0
box-close-v2	test	82.0	583.9 ± 13.3	-0.0 ± 0.0
door-lock-v2	test	119.3	831.9 ± 9.22	0.6 ± 0.05
door-unlock-v2	test	94.7	813.0 ± 2.07	0.2 ± 0.03
hand-insert-v2	test	2.5	775.5 ± 5.46	0.1 ± 0.04
assembly-v2	train	44.7	283.3 ± 27.63	1.0 ± 0.05

Appendix

Table 3: Online Evaluation Results for Meta-World

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
l-hard-lunar-lander-v1	test	-0.4	1.0 ± 0.0	0.3 ± 0.24
l-hard-pinball-v1	test	-0.6	1.0 ± 0.0	0.1 ± 0.28
l-mjc-walker-v1	test	-0.3	1.0 ± 0.0	0.2 ± 0.0
m-arm-hard-v1	test	0.0	1.0 ± 0.0	1.0 ± 0.0
m-h14-thrustblock-v1	test	-1.0	1.0 ± 0.0	0.0 ± 0.0
s-h8-unicycle-balance-v1	test	0.4	1.0 ± 0.0	0.4 ± 0.5

Appendix

Table 4: Online Evaluation Results for Kinetix

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
Ant-v4	param_shift	-102.9	4989.7 ± 2058.04	1.0 ± 0.01
HalfCheetah-v4	param_shift	-264.2	7331.8 ± 420.5	1.0 ± 0.01
Hopper-v4	param_shift	18.2	2264.3 ± 229.33	1.0 ± 0.47
Humanoid-v4	param_shift	121.9	7373.8 ± 132.4	1.0 ± 0.0
HumanoidStandup-v4	param_shift	33027.7	303552.9 ± 19279.02	1.0 ± 0.03
InvertedDoublePendulum-v4	param_shift	58.5	9256.8 ± 293.62	1.0 ± 0.0

Appendix

Table 5: Online Evaluation Results for MuJoCo

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
shadowhanddoorcloseoutward	test	930.0	1380.0 ± 0.0	-1.3 ± 0.01
shadowhanddooropeninward	test	-15.0	410.0 ± 0.0	0.3 ± 0.03
shadowhanddooropenoutward	test	5.0	610.0 ± 0.0	0.1 ± 0.0
shadowhandkettle	test	-200.0	53.0 ± 0.0	-1.0 ± 0.01
shadowhandblockstack	train	90.0	285.0 ± 0.0	1.0 ± 0.09
shadowhandbottlecap	train	100.0	400.0 ± 0.0	1.1 ± 0.07

Appendix

Table 6: Online Evaluation Results for Bi-DexHands

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
industrial-benchmark-100-v1	test	-998.6	-561.5 ± 0.0	1.0 ± 0.02
industrial-benchmark-80-v1	test	-779.9	-485.4 ± 0.0	1.0 ± 0.0
industrial-benchmark-85-v1	test	-813.8	-507.7 ± 0.0	1.0 ± 0.0
industrial-benchmark-90-v1	test	-883.4	-521.9 ± 0.0	1.0 ± 0.0
industrial-benchmark-95-v1	test	-926.8	-546.0 ± 0.0	1.0 ± 0.0
industrial-benchmark-0-v1	train	-357.4	-180.8 ± 0.0	1.0 ± 0.0

Appendix

Table 7: Online Evaluation Results for Industrial-Benchmark

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
GOAL-t_pose	test	-3200.0	-605.0 ± 0.0	0.2 ± 0.01
REWARD-rotate-x-5-0_8	test	8.0	211.0 ± 0.0	0.1 ± 0.01
REWARD-split-0_5	test	9.0	258.0 ± 0.0	0.0 ± 0.02
GOAL-crouch_medium	train	-3120.0	-341.0 ± 0.0	0.3 ± 0.04
GOAL-lie_front	train	-3200.0	-570.0 ± 0.0	1.0 ± 0.01
GOAL-sit_hand_behind	train	-3200.0	-535.0 ± 0.0	0.9 ± 0.01

Appendix

Table 8: Online Evaluation Results for HumEnv

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
5zone-mixed_dec_jan_feb	test	-320.0	-173.0 ± 0.0	0.4 ± 0.16
5zone-mixed_jun_jul_aug	test	-240.0	-110.0 ± 0.0	1.0 ± 0.02
5zone-mixed_mar_apr_may	test	-330.0	-90.0 ± 0.0	1.0 ± 0.13
5zone-mixed_oct_nov	test	-300.0	-80.0 ± 0.0	0.9 ± 0.14
5zone-mixed_sep	test	-250.0	-104.0 ± 0.0	-0.1 ± 0.04
warehouse-mixed_apr_may	test	-290.0	-145.0 ± 0.0	1.2 ± 0.21

Appendix

Table 9: Online Evaluation Results for SinerGym

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
cones_C_69	test	9.1	535.0 ± 0.0	0.9 ± 0.22
def_C_574	test	9.4	611.4 ± 0.0	1.0 ± 0.01
def_O_785	test	8.8	589.0 ± 0.0	1.0 ± 0.03
def_T_971	test	6.7	454.7 ± 0.0	1.0 ± 0.0
cones_C_520	train	10.0	813.4 ± 0.0	0.7 ± 0.15
cones_C_574	train	11.4	616.2 ± 0.0	0.8 ± 0.14

Appendix

Table 10: Online Evaluation Results for MetaDrive

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
2022_phase_1_task_10	test	-1833.1	-702.2 ± 0.0	0.8 ± 0.0
2022_phase_1_task_11	test	-2386.1	-1262.1 ± 0.0	0.8 ± 0.0
2022_phase_2_task_10	test	-1724.5	-611.3 ± 0.0	0.8 ± 0.0
2022_phase_2_task_11	test	-2143.7	-967.4 ± 0.0	0.8 ± 0.0
2022_phase_1_task_00	train	-2212.6	-1048.3 ± 0.0	1.0 ± 0.0
2022_phase_1_task_01	train	-2017.0	-831.4 ± 0.0	1.0 ± 0.0

Appendix

Table 11: Online Evaluation Results for CityLearn

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
aircraft_1	test	-67000.0	-0.5 ± 0.0	1.0 ± 0.0
cable_5	test	-56000.0	-9.7 ± 0.0	1.0 ± 0.0
pde_schrodinger	test	-6300000.0	-2146.7 ± 0.0	-4.5 ± 1.49
aircraft_2	train	-67000.0	-0.9 ± 0.0	1.0 ± 0.0
aircraft_4	train	-10000.0	-0.3 ± 0.0	1.0 ± 0.0
aircraft_5	train	-12000.0	-1.5 ± 0.0	0.9 ± 0.05

Appendix

Table 12: Online Evaluation Results for ControlGym

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
bin-picking-v2	test	1.9	410.0 ± 64.37	0.0 ± 0.0
box-close-v2	test	82.0	583.9 ± 13.28	-0.0 ± 0.0
door-lock-v2	test	119.3	831.9 ± 9.21	1.0 ± 0.0
door-unlock-v2	test	94.7	813.0 ± 2.07	0.9 ± 0.01
hand-insert-v2	test	2.5	775.5 ± 5.46	1.0 ± 0.0
assembly-v2	train	44.7	283.3 ± 27.63	1.0 ± 0.0

Appendix

Table 13: Offline Evaluation Results for Meta-World

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
l-hard-lunar-lander-v1	test	-0.4	1.0 ± 0.0	-0.1 ± 0.36
l-hard-pinball-v1	test	-0.6	1.0 ± 0.0	0.4 ± 0.22
l-mjc-walker-v1	test	-0.3	1.0 ± 0.0	0.2 ± 0.0
m-arm-hard-v1	test	0.0	1.0 ± 0.0	1.0 ± 0.0
m-h14-thrustblock-v1	test	-1.0	1.0 ± 0.0	0.0 ± 0.0
s-h8-unicycle-balance-v1	test	0.4	1.0 ± 0.0	0.4 ± 0.2

Appendix

Table 14: Offline Evaluation Results for Kinetix

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
Ant-v4	param_shift	-103.1	4988.8 ± 2057.47	1.0 ± 0.01
HalfCheetah-v4	param_shift	-264.2	7331.8 ± 420.5	1.0 ± 0.01
Hopper-v4	param_shift	18.9	2139.9 ± 286.52	0.8 ± 0.26
Humanoid-v4	param_shift	121.9	7373.8 ± 132.4	1.0 ± 0.0
HumanoidStandup-v4	param_shift	33027.7	303552.9 ± 19279.02	1.0 ± 0.03
InvertedDoublePendulum-v4	param_shift	58.5	9256.8 ± 293.62	1.0 ± 0.0

Appendix

Table 15: Offline Evaluation Results for MuJoCo

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
shadowhanddoorcloseoutward	test	930.0	1380.0 ± 0.0	-1.2 ± 0.01
shadowhanddooropeninward	test	-15.0	410.0 ± 0.0	0.3 ± 0.03
shadowhanddooropenoutward	test	5.0	610.0 ± 0.0	0.1 ± 0.0
shadowhandkettle	test	-200.0	53.0 ± 0.0	-0.0 ± 0.03
shadowhandblockstack	train	90.0	285.0 ± 0.0	1.0 ± 0.08
shadowhandbottlecap	train	100.0	400.0 ± 0.0	1.1 ± 0.05

Appendix

Table 16: Offline Evaluation Results for Bi-DexHands

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
industrial-benchmark-100-v1	test	-998.6	-561.5 ± 0.0	1.0 ± 0.0
industrial-benchmark-80-v1	test	-779.9	-485.4 ± 0.0	0.8 ± 0.0
industrial-benchmark-85-v1	test	-813.8	-507.7 ± 0.0	1.0 ± 0.01
industrial-benchmark-90-v1	test	-883.4	-521.9 ± 0.0	1.0 ± 0.0
industrial-benchmark-95-v1	test	-926.8	-546.0 ± 0.0	0.9 ± 0.0
industrial-benchmark-0-v1	train	-357.4	-180.8 ± 0.0	1.0 ± 0.0

Appendix

Table 17: Offline Evaluation Results for Industrial-Benchmark

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
GOAL-t_pose	test	-3200.0	-605.0 ± 0.0	0.2 ± 0.01
REWARD-rotate-x-5-0_8	test	8.0	211.0 ± 0.0	0.0 ± 0.02
REWARD-split-0_5	test	9.0	258.0 ± 0.0	0.1 ± 0.03
GOAL-crouch_medium	train	-3120.0	-341.0 ± 0.0	0.4 ± 0.15
GOAL-lie_front	train	-3200.0	-570.0 ± 0.0	1.0 ± 0.01
GOAL-sit_hand_behind	train	-3200.0	-535.0 ± 0.0	0.9 ± 0.05

Appendix

Table 18: Offline Evaluation Results for HumEnv

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
5zone-mixed_dec_jan_feb	test	-320.0	-173.0 ± 0.0	0.9 ± 0.09
5zone-mixed_jun_jul_aug	test	-240.0	-110.0 ± 0.0	1.0 ± 0.03
5zone-mixed_mar_apr_may	test	-330.0	-90.0 ± 0.0	1.2 ± 0.08
5zone-mixed_oct_nov	test	-300.0	-80.0 ± 0.0	1.1 ± 0.07
5zone-mixed_sep	test	-250.0	-104.0 ± 0.0	0.7 ± 0.02
warehouse-mixed_apr_may	test	-290.0	-145.0 ± 0.0	1.0 ± 0.15

Appendix

Table 19: Offline Evaluation Results for SinerGym

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
cones_C_69	test	9.1	535.0 ± 0.0	0.7 ± 0.31
def_C_574	test	9.4	611.4 ± 0.0	1.0 ± 0.01
def_O_785	test	8.8	589.0 ± 0.0	1.1 ± 0.03
def_T_971	test	6.7	454.7 ± 0.0	1.0 ± 0.01
cones_C_520	train	10.0	813.4 ± 0.0	1.0 ± 0.06
cones_C_574	train	11.4	616.2 ± 0.0	0.8 ± 0.17

Appendix

Table 20: Offline Evaluation Results for MetaDrive

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
2022_phase_1_task_10	test	-1833.1	-702.2 ± 0.0	0.8 ± 0.0
2022_phase_1_task_11	test	-2386.1	-1262.1 ± 0.0	0.8 ± 0.0
2022_phase_2_task_10	test	-1724.5	-611.3 ± 0.0	0.8 ± 0.0
2022_phase_2_task_11	test	-2143.7	-967.4 ± 0.0	0.8 ± 0.0
2022_phase_1_task_00	train	-2212.6	-1048.3 ± 0.0	1.0 ± 0.0
2022_phase_1_task_01	train	-2017.0	-831.4 ± 0.0	1.0 ± 0.0

Appendix

Table 21: Offline Evaluation Results for CityLearn

Task Name	Split	Random Score	Expert Score	Normalized Return IQM
aircraft_1	test	-67000.0	-0.5 ± 0.0	1.0 ± 0.0
cable_5	test	-56000.0	-9.7 ± 0.0	1.0 ± 0.0
pde_schrodinger	test	-6300000.0	-2146.7 ± 0.0	1.0 ± 0.0
aircraft_2	train	-67000.0	-0.9 ± 0.0	1.0 ± 0.0
aircraft_4	train	-10000.0	-0.3 ± 0.0	1.0 ± 0.0
aircraft_5	train	-12000.0	-1.5 ± 0.0	1.0 ± 0.0

Appendix

Table 22: Offline Evaluation Results for ControlGym

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
InvertedPendulum-v4	1000488	10	68.2	(4,)	(1,)	0.1	train
InvertedDoublePendulum-v4	1000480	10	55.7	(11,)	(1,)	0.1	train
Humanoid-v4	1000488	15	207.8	(376,)	(17,)	0.1	train
Walker2d-v4	1000586	10	350.4	(17,)	(6,)	0.1	train
Hopper-v4	1000462	10	227.7	(11,)	(3,)	0.1	train
Ant-v4	1000653	10	524.7	(27,)	(8,)	0.1	train

Appendix

Table 23: Dataset Metadata for MuJoCo

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
shadowhandblockstack	250140	10	248.6	(428,)	(52,)	1	train
shadowhandbottlecap	135065	10	123.7	(420,)	(52,)	1	train
shadowhandcatchabreast	165078	10	98.5	(422,)	(52,)	1	train
shadowhandcatchover2underarm	100034	10	55.6	(422,)	(52,)	1	train
shadowhandcatchunderarm	100029	10	64.9	(422,)	(52,)	1	train
shadowhanddoorcloseinward	100014	10	23.8	(417,)	(52,)	1	train

Appendix

Table 24: Dataset Metadata for Bi-DexHands

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
assembly-v2	100000	15	100.0	(39,)	(4,)	1	train
basketball-v2	100000	15	100.0	(39,)	(4,)	1	train
button-press-topdown-v2	100000	15	100.0	(39,)	(4,)	1	train
button-press-topdown-wall-v2	100000	15	100.0	(39,)	(4,)	1	train
button-press-v2	100000	15	100.0	(39,)	(4,)	1	train
button-press-wall-v2	100000	15	100.0	(39,)	(4,)	1	train

Appendix

Table 25: Dataset Metadata for Meta-World

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
def_S	300184	10	232.9	(275,)	(2,)	1	train
def_C_69	400192	10	301.1	(275,)	(2,)	1	train
def_C_859	450235	10	283.1	(275,)	(2,)	1	train
def_C_574	500242	10	263.3	(275,)	(2,)	1	test
def_C_85	500302	10	286.1	(275,)	(2,)	1	train
def_C_785	500188	10	293.8	(275,)	(2,)	1	train

Appendix

Table 26: Dataset Metadata for MetaDrive

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
2022_phase_1_task_00	533628	10	729.0	(44,)	(5,)	0.25	train
2022_phase_1_task_01	533628	10	729.0	(44,)	(5,)	0.25	train
2022_phase_1_task_02	533628	10	729.0	(44,)	(5,)	0.35	train
2022_phase_1_task_03	533628	10	729.0	(44,)	(5,)	0.35	train
2022_phase_1_task_04	533628	10	729.0	(44,)	(5,)	0.3	train
2022_phase_1_task_05	533628	10	729.0	(44,)	(5,)	0.25	train

Appendix

Table 27: Dataset Metadata for CityLearn

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
s-h0-weak-thrust-v1	100012	15	25.0	(140,)	(6,)	1	train
s-h1-thrust-over-ball-v1	100014	15	22.3	(140,)	(6,)	1	train
s-h2-one-wheel-car-v1	100104	15	209.3	(140,)	(6,)	1	train
s-h3-point-the-thruster-v1	100019	15	90.1	(140,)	(6,)	1	train
s-h4-thrust-aim-v1	100012	15	42.5	(140,)	(6,)	1	train
s-h5-rotate-fall-v1	100013	15	27.0	(140,)	(6,)	1	train

Appendix

Table 28: Dataset Metadata for Kinetix

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
industrial-benchmark-0-v1	300000	15	250.0	(6,)	(3,)	1	train
industrial-benchmark-5-v1	300000	15	250.0	(6,)	(3,)	1	train
industrial-benchmark-10-v1	300000	15	250.0	(6,)	(3,)	1	train
industrial-benchmark-15-v1	300000	15	250.0	(6,)	(3,)	1	train
industrial-benchmark-20-v1	300000	15	250.0	(6,)	(3,)	1	train
industrial-benchmark-25-v1	300000	15	250.0	(6,)	(3,)	1	train

Appendix

Table 29: Dataset Metadata for Industrial-Benchmark

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
REWARD-move-ego-0-0	300000	10	300.0	(358,)	(69,)	1	train
REWARD-move-ego-0-2	300000	10	300.0	(358,)	(69,)	1	train
REWARD-raisearms-h-h	300000	10	300.0	(358,)	(69,)	1	train
REWARD-crawl-0_5-0-d	300000	10	300.0	(358,)	(69,)	1	train
REWARD-crouch-0	300000	10	300.0	(358,)	(69,)	1	train
REWARD-rotate-x–5-0_8	300000	10	300.0	(358,)	(69,)	1	train

Appendix

Table 30: Dataset Metadata for HumEnv

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
aircraft_1	1000000	10	1000.0	(3,)	(6,)	0.005	test
aircraft_2	1000000	10	1000.0	(3,)	(6,)	0.005	train
aircraft_4	1000000	10	1000.0	(2,)	(5,)	0.02	train
aircraft_5	1000000	10	1000.0	(5,)	(11,)	0.02	train
aircraft_6	1000000	10	1000.0	(5,)	(14,)	0.0005	train
cable_1	1000000	10	1000.0	(2,)	(2,)	0.01	train

Appendix

Table 31: Dataset Metadata for ControlGym

Task Name	Trajectory Length	Number of Trajectories	Mean Eplen.	Obs. Shape	Action Shape	Reward Scale	Split
5zone-cool_dec_jan_feb	140340	10	721.2	(17,)	(2,)	1	train
5zone-cool_jun_jul_aug	140380	10	736.1	(17,)	(2,)	1	train
5zone-cool_mar_apr_may	140438	10	736.4	(17,)	(2,)	1	train
5zone-cool_oct_nov	140515	10	731.8	(17,)	(2,)	1	train
5zone-cool_sep	140400	10	720.0	(17,)	(2,)	1	train
5zone-hot_dec_jan_feb	140136	10	720.5	(17,)	(2,)	1	train

Appendix

Table 32: Dataset Metadata for SinerGym

Domain	Online Single	Online Full	Offline Single	Offline Full
Industrial Benchmark (Test)	$0.950\pm 0.004$	$0.980\pm 0.007$	$0.950\pm 0.007$	$0.960\pm 0.009$
Meta-World (Test)	$-0.000\pm 0.001$	$0.140\pm 0.023$	$0.460\pm 0.029$	$0.690\pm 0.036$
SinerGym (Test)	$0.900\pm 0.026$	$0.910\pm 0.020$	$0.970\pm 0.010$	$0.960\pm 0.017$
Industrial Benchmark (Train)	$0.940\pm 0.009$	$0.990\pm 0.006$	$1.010\pm 0.001$	$1.020\pm 0.001$
Meta-World (Train)	$0.650\pm 0.015$	$0.850\pm 0.080$	$0.990\pm 0.001$	$1.000\pm 0.000$
SinerGym (Train)	$0.960\pm 0.031$	$1.000\pm 0.026$	$1.010\pm 0.004$	$0.990\pm 0.017$

Appendix

Table 33: Single-domain and multi-domain models comparison

Domain	Online GH	Online FM	Offline GH	Offline FM
HumEnv (Test)	$\bm{0.070\pm 0.023}$	$\bm{0.070\pm 0.030}$	$0.070\pm 0.020$	$\bm{0.130\pm 0.020}$
Industrial Benchmark (Test)	$\bm{0.990\pm 0.003}$	$0.950\pm 0.004$	$\bm{0.990\pm 0.007}$	$0.950\pm 0.007$
Meta-World (Test)	$0.000\pm 0.001$	$-0.000\pm 0.001$	$0.390\pm 0.029$	$\bm{0.460\pm 0.034}$
SinerGym (Test)	$0.400\pm 0.087$	$\bm{0.900\pm 0.025}$	$0.790\pm 0.046$	$\bm{0.970\pm 0.010}$
HumEnv (Train)	$0.440\pm 0.025$	$0.440\pm 0.044$	$0.440\pm 0.025$	$0.440\pm 0.024$
Industrial Benchmark (Train)	$\bm{1.020\pm 0.001}$	$0.940\pm 0.008$	$\bm{1.020\pm 0.001}$	$1.010\pm 0.001$

Appendix

Table 34: Gaussian and Flow Matching Heads single-domain models comparison

Domain	Vintix Online	Vintix II Online	Vintix Offline	Vintix II Offline
Bi-DexHands	$-0.45\pm 0.08$	$\bm{-0.43\pm 0.08}$	$-0.42\pm 0.08$	$\bm{0.07\pm 0.05}$
CityLearn	$0.01\pm 0.02$	$\bm{0.78\pm 0.00}$	$0.77\pm 0.00$	$\bm{0.78\pm 0.00}$
ControlGym	$0.97\pm 0.03$	$\bm{0.99\pm 0.31}$	$1.00\pm 0.00$	$1.00\pm 0.00$
HumEnv	$0.04\pm 0.06$	$\bm{0.09\pm 0.02}$	$0.06\pm 0.02$	$\bm{0.07\pm 0.01}$
Industrial Benchmark	$0.42\pm 0.05$	$\bm{0.98\pm 0.00}$	$0.86\pm 0.05$	$\bm{0.96\pm 0.01}$
Kinetix	$0.20\pm 0.05$	$\bm{0.23\pm 0.11}$	$0.17\pm 0.05$	$\bm{0.23\pm 0.05}$

Appendix

Table 35: Comparison between Vintix and Vintix II in online and offline settings across domains on test set

In-Context RL

In-context reinforcement learning studies how agents adapt their policies from interaction histories without updating model parameters.

See all 45 papers

Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation
Sep 28, 2026Yuanqing Ma, Zhenrui Zheng, Chenjun XiaoTransformer In-Context RL
The Chinese University of Hong Kong, Shenzhen
Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models
Jun 23, 2026Thibaut KulakReinforcement Learning Transformer
NeoInstinct SA
Agentic In-Context Learning: What Algorithms Can Transformers Learn in Closed-Loop Interaction?
Oct 16, 2026Shiguang Wu, Yaqing Wang, Quanming YaoIn-Context Learning In-Context RL
Tsinghua University · Beijing Institute of Mathematical Sciences and Applications

Sep 28, 2026J/K move · Enter open · S save

Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation

Yuanqing Ma, Zhenrui Zheng, Chenjun Xiao

The Chinese University of Hong Kong, Shenzhen

Transformer In-Context RL Context Compression

Open paper AskPDF arXiv

Algorithm Distillation (AD) has demonstrated the remarkable ability of Transformers to perform in-context reinforcement learning without explicit weight updates. However, capturing long-term learning progress necessitates expansive context windows, which incur prohibitive memory costs and limit scalability in complex, long-horizon tasks. To address this bottleneck, we propose Recurrent Algorithm Distillation (RAD). RAD employs a dual-component architecture: a Compression Transformer that distills extended interaction histories into compact latent tokens, and an AD Transformer that auto-regressively generates actions using a hybrid context of these compressed memories and recent transitions. By maintaining a fixed-size latent buffer, RAD decouples the effective history length from computational complexity, functionally providing the model with a long-horizon memory. Empirical evaluations across diverse environments demonstrate that RAD matches the asymptotic performance of standard AD with significantly reduced context window sizes, offering a scalable solution for efficient in-context decision-making.

Abstract

In-Context RL

Explore similar work

Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation

Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models

Agentic In-Context Learning: What Algorithms Can Transformers Learn in Closed-Loop Interaction?