FaST: Factored Synthesis of Transitions for Offline Reinforcement Learning
Authors: Daniel Campa, Spandan Das, Srinjoy Das
Organizations: School of Mathematical and Data Sciences, West Virginia University · Department of Computer Science and Engineering, Jadavpur University
Synthetic data augmentation promises to scale offline reinforcement learning beyond fixed logged datasets, but leading diffusion-based generators model the transition tuple (state, action, next state, reward, terminal) as a monolith: no single factor can be natively redrawn in isolation. We present FaST (Factored Synthesis of Transitions), which factors transitions via the chain rule into state, action, dynamics, reward, and terminal stages, fitting each with an estimator matched to its statistical properties. Across twelve D4RL locomotion and maze benchmarks, TD3+BC and IQL agents trained on FaST transitions match the SynthER diffusion baseline while generating transitions over 90x faster. The explicit action-given-state conditional can also be re-queried for a fixed state, enabling targeted action synthesis: drawing N candidate actions per generated state and keeping the highest-advantage one under a value function fit to the same data, thereby synthesizing improved behavior during generation rather than filtering for it post-hoc. Best-of-10 synthesis beats training on real data on 11 of 12 medium and medium-replay settings (+8.9 D4RL points on average, up to +28.4), and is competitive with advantage-filtered SynthER at a fraction of its generation cost. On expert mixtures both mechanisms degrade identically, tracing the failure to the advantage estimate itself rather than to either targeting mechanism.
Figures & tables
Figure 1: Each chain-rule factor of Eq. 1 is fit by a stage-specialized estimator (§ 3.2 –§ 3.5 ). Dashed: training, independent per stage on real D . Solid: ancestral sampling, stage k conditions on the synthetic outputs of stages <k . Orange: the optional best-of- N intervention re-samples Pr(A∣S) alone, keeping the candidate of highest IQL advantage while every other stage is reused verbatim (§ 3.6 ).
TD3+BC
IQL
Environment
Dataset
Real
SynthER
FaST
Real
SynthER
FaST
halfcheetah-v2
medium
48.2 ± 0.1
48.2 ± 0.1
48.0 ± 0.2
48.1 ± 0.1
48.8 ± 0.1
48.1 ± 0.2
medium-replay
44.5 ± 0.2
45.2 ± 0.2
44.5 ± 0.3
44.0 ± 0.1
46.2 ± 0.1
45.3 ± 0.5
medium-expert
88.6 ± 1.6
77.0 ± 2.1
82.0 ± 4.3
93.1 ± 1.1
91.3 ± 0.7
92.9 ± 0.6
hopper-v2
medium
58.5 ± 0.7
61.3 ± 0.3
60.7 ± 0.5
63.0 ± 1.6
68.3 ± 1.0
67.4 ± 4.0
medium-replay
62.4 ± 6.1
62.1 ± 3.2
52.9 ± 11.5
86.9 ± 4.0
97.4 ± 3.1
96.5 ± 3.0
Table 1: D4RL normalized scores for offline RL agents trained on purely synthetic data from each generator. Evaluation with uniform protocol and mean ± std as described in § 4.1 .
TD3+BC
IQL
Environment
Dataset
Real
FaST
+ filter
+ BoN-10
Real
FaST
+ filter
+ BoN-10
halfcheetah-v2
medium
48.2 ± 0.1
48.0 ± 0.2
50.4 ± 0.3
53.0 ± 0.3
48.1 ± 0.1
48.1 ± 0.2
50.4 ± 0.0
52.5 ± 0.1
medium-replay
44.5 ± 0.2
44.5 ± 0.3
47.1 ± 0.1
49.5 ± 0.2
44.0 ± 0.1
45.3 ± 0.5
47.4 ± 0.2
49.9 ± 0.2
medium-expert
88.6 ± 1.6
82.0 ± 4.3
67.0 ± 1.8
65.8 ± 6.4
93.1 ± 1.1
92.9 ± 0.6
70.2 ± 3.2
71.2 ± 2.3
hopper-v2
medium
58.5 ± 0.7
60.7 ± 0.5
70.2 ± 2.1
77.0 ± 1.7
63.0 ± 1.6
67.4 ± 4.0
68.3 ± 2.4
71.6 ± 3.7
medium-replay
62.4 ± 6.1
52.9 ± 11.5
87.2 ± 2.9
90.8 ± 8.8
86.9 ± 4.0
96.5 ± 3.0
77.0 ± 6.4
93.8 ± 8.0
Table 2: Advantage-targeted generation across the locomotion suite (uniform protocol of § 4.1 ). Real : the D4RL dataset D . FaST : the untargeted pipeline ( N=1 ). + filter : top- 20% of a 5× FaST pool (size-matched). + BoN-10 : targeted action synthesis with N=10 . Both targeted arms build on FaST and use the same advantage estimate.
TD3+BC
IQL
Dataset
Data source
Untargeted
+ filter
+ BoN-10
Untargeted
+ filter
+ BoN-10
halfcheetah-mr
FaST
44.5 ± 0.3
47.1 ± 0.1
49.5 ± 0.2
45.3 ± 0.5
47.4 ± 0.2
49.9 ± 0.2
SynthER
45.2 ± 0.2
47.3 ± 0.1
—
46.2 ± 0.1
47.7 ± 0.2
—
hopper-mr
FaST
52.9 ± 11.5
87.2 ± 2.9
90.8 ± 8.8
96.5 ± 3.0
77.0 ± 6.4
93.8 ± 8.0
SynthER
62.1 ± 3.2
99.7 ± 0.9
—
97.4 ± 3.1
101.8 ± 2.5
—
walker2d-mr
FaST
79.1 ± 0.9
83.3 ± 1.3
90.7 ± 1.0
77.0 ± 1.4
73.6 ± 3.3
85.3 ± 1.8
Table 3: Each intervention of § 3.6 applied to each generator, on the medium-replay datasets (real-data scores for these three: Table 2 ). Untargeted : the generator’s output with no targeting applied. Every arm is size-matched at 106 transitions. Filter columns are the top- 20% of a 5×106 pool from their own generator. Best-of- N has no SynthER counterpart (—): a joint generator cannot natively redraw Pr(A∣S) for a fixed state (§ 2.2 ).
SynthER
FaST
Ratio
Training (min)
42.5
128.7
0.33 ×
Inference / 1M (s)
1736.0
18.6
93.3 ×
Table 4: Wall-clock cost on a single RTX 2070 SUPER, averaged over the twelve datasets. Inference is the time to draw 1M transitions; ratio is SynthER ÷ FaST. Per-environment numbers and per-stage breakdown in Appendix D .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Architecture
Epochs
Batch
LR
Weight decay
Dropout
Patience
ARMDN Pr(S)
K=15 , h=256
300
8192
2.8e−3
1e−4
0.10
30
Action MDN Pr(A∣S)
K=15 , (512,512,256)
500
1024
1e−3
1e−4
0.05
30
Delta ensemble Pr(S′∣S,A)
M=7 , (512,512,512,256)
500
4096
2e−3
5e−4
0.04
25
Reward MLP Pr(R∣⋅)
(128,128)
100
256
3e−4
1e−4
0.05
15
Terminal Pr(d∣⋅)
ExtraTrees ×50 , balanced
—
—
—
—
—
—
Appendix
Table 5: Per-stage architecture and training hyperparameters. This configuration is used for every dataset in this paper. The ARMDN models each conditional with K mixture components over a two-hidden-layer MLP of width h (SiLU activations, LayerNorm). The action MDN places K components over a residual MLP of the listed hidden widths. The delta ensemble averages M independently-trained residual MLPs. The reward stage is a single residual MLP. The terminal stage is a scikit-learn ExtraTreesClassifier with 50 trees.
Stage
Architecture
Params
Train (min)
Sample ( μ s/trans.)
Pr(S)
quantile–logit ARMDN
0.80M
7.25
2.71
Pr(A∣S)
logit MDN
1.35M
3.15
0.79
Pr(S′∣S,A)
delta ensemble ×7
13.0M
21.78
6.93
Pr(R∣⋅)
residual MLP
70.3K
6.79
0.23
Pr(d∣⋅)
ExtraTrees ×50
12.2K nodes
0.04
0.41
Appendix
Table 6: Per-stage cost on hopper-medium-replay-v2 . Params counts the trained parameters of the stage. Sampling cost is microseconds per transition. No stage is refit for advantage targeting (Algorithm 1 ). Best-of- N over-generates only from the fitted action stage, whose sampling cost is a small fraction of the pipeline’s.
Training (min)
Inference / 1M (s)
Environment
SynthER
FaST
SynthER
FaST
Speedup
halfcheetah-v2
40.2
111.4
1735.08 ± 1.25
18.98 ± 7.27
91.4 ×
hopper-v2
40.0
110.1
1732.73 ± 2.17
16.06 ± 5.61
107.9 ×
walker2d-v2
40.3
181.3
1740.54 ± 2.06
18.74 ± 7.39
92.9 ×
maze2d-v1
49.4
111.9
1735.82 ± 0.18
20.66 ± 9.97
84.0 ×
Mean
42.5
128.7
1736.04
18.61
93.3 ×
Appendix
Table 7: Training and inference wall-clock cost on a single RTX 2070 SUPER. Each row is averaged over the three D4RL variants of that environment: medium, medium-replay, medium-expert for locomotion and umaze, medium, large for maze2d. Inference is the time to draw 1M transitions (mean ± std over 3 repeats). Speedup is the SynthER -to-FaST inference ratio.
Figure 2: Quantile-logit transform applied per-dimension to state dimensions 0,1,4,7,10 of hopper-medium-replay-v2 . Top: raw marginals, multimodal and heavy-tailed with spikes at bounded extremes. Middle: after the empirical CDF F^j , yielding Uniform(0,1) by construction. Bottom: after the logit, yielding a standard logistic distribution and the dashed overlay shows the close Gaussian approximation that the MDN’s components exploit.
Figure 3: Action marginals on dimensions 0,1,2 of hopper-medium-replay-v2 , before (top) and after (bottom) the logit transform. Raw actions pile up at the actuator limits ±1 . Under T , the boundary saturation is remapped to finite locations on R and the resulting distribution is well-approximated by a single Gaussian (red dashed overlay), giving the MDN a target that Gaussian components can fit.
K
NLL
KS
1
4.636 ± 0.014
0.0151 ± 0.0041
3
4.589 ± 0.013
0.0201 ± 0.0022
5
4.586 ± 0.013
0.0123 ± 0.0005
8
4.577 ± 0.001
0.0153 ± 0.0022
15
4.583 ± 0.003
0.0148 ± 0.0012
Appendix
Table 8: Action-model component sweep on hopper-medium-replay-v2 , mean ± std over 3 seeds. NLL is the negative log-likelihood of the logit-transformed action on a held-out split (lower is better). KS is the Kolmogorov–Smirnov statistic between synthetic and real action marginals.
Figure 4: Next-state marginals (top) against delta marginals δj=sj′−sj (bottom) for dimensions 0,1,4,7,10 of hopper-medium-replay-v2 , on a set of non-terminal transitions. Raw next-state marginals are broad and multimodal with mass piled at joint and velocity limits (dim 4 ), while the corresponding deltas are unimodal and concentrated at zero. The red dashed curve is a normal matched to each panel’s mean and variance, drawn as a fixed reference for width and shape only. The model does not assume normality.
M
R next-obs2
R δ2
RMSE
1
0.9931 ± 0.0004
0.8299 ± 0.0144
0.1184 ± 0.0046
3
0.9945 ± 0.0001
0.8615 ± 0.0039
0.1027 ± 0.0026
5
0.9950 ± 0.0002
0.8724 ± 0.0055
0.0985 ± 0.0032
7
0.9951 ± 0.0004
0.8769 ± 0.0083
0.0968 ± 0.0029
Appendix
Table 9: Ensemble size sweep on hopper-medium-replay-v2 . R δ2 measures prediction accuracy on the delta target. RMSE in next-obs space. Mean over 3 seeds.
Figure 5: Reward residuals r−fθR(s,a,s′) on non-terminal transitions of each locomotion environment. All three distributions are sharply peaked and unimodal, which is all the MSE choice of § 3.5 requires. The matched Gaussian overlay (red) badly under-covers the central spike, confirming the conditional is unimodal but non-Gaussian. The excess kurtosis κ^ above each panel measures the other half of that misfit, the heavy tails, which lie outside the plotted window. The maze2d datasets are omitted because their reward is binary; they are treated separately below.
Figure 6: Advantage of the transitions from multiple sources on hopper-medium-replay-v2 . The untargeted FaST pool tracks the real data, so fidelity alone cannot raise action quality. Best-of-10 shifts the bulk right (mean +1.15 ) while keeping the low-advantage tail. The top- 20% filter reaches +2.53 instead by truncating it (10th percentile +1.23 , against −0.77 ).
N
1 (pool)
5
10
20
Score
79.1±0.9
90.0±0.6
90.7±1.0
91.2±1.1
Appendix
Table 10: Best-of- N dose-response on walker2d-medium-replay-v2 (TD3+BC). N=1 is the untargeted pool. Scores are means over the final 500 k steps, 3 generation ×3 training seeds, ± std across the generation-seed means, as in the uniform protocol.