Many multi-step off-policy temporal-difference methods handle policy mismatch through products of per-step trace coefficients, leading to a trade-off between variance control and long-horizon signal. The maximum trace length consequently becomes a sensitive hyperparameter. We propose MINT (Meeting-time INdicators for Truncation), which replaces these products with a binary stopping indicator: one until coupled behavior and target trajectories first meet, zero thereafter. At the true value function, the indicator's idempotence yields a variance bound governed by meeting-time survival. For practical implementation, a single sampled action provides a model-free surrogate for the stopping rule, requiring no transition model or mixing-time estimate. We derive separate sufficient conditions for contraction of the state-coupled operator and its practical surrogate under fixed-policy evaluation. On MuJoCo and DeepMind Control tasks, MINT outperforms the reported multi-step baselines; the main MuJoCo comparison uses a common cap without per-task cap-length tuning. We also demonstrate the effectiveness of MINT on Atari 100k, RWRL, and OGBench experiments; MINT adapts the backup horizon to reduce target variability relative to long fixed-horizon backups across tasks and dynamics.
Figures & tables
Figure 1: Cap-length sensitivity on MuJoCo Hopper (SAC). (a) Bias and variance at 100 K and 1 M steps for caps {3,5,10,20} (darker shades indicate shorter caps). MINT exhibits a favorable empirical bias–variance balance across the tested caps; several baseline configurations have higher variance at longer caps. (b) Learning curves for the same caps. MINT attains high returns across the tested values, whereas baseline performance varies more with the cap.
Operator
Trace / return rule
E[Wt2]
Conservative?
Fixed point
Retrace [ 31 ]
ci=min(1,π/β)
E[∏i=1tci2]
Yes
Qπ
Peng’s Q(λ) [ 24 ]
ci=1†
1
No
Qπλ
Uncorrected n -step
Nn‡
1{t<n}
No
β -dependent
MINT
ci=Ai∈{0,1}
Pr(T>t)
No
Qπ
Table 1: Multi-step off-policy operators for behavior and target policies. Wt excludes (γλ)t ; E[Wt2] describes squared weights, not the full estimator variance.
Figure 2: Single-Trap chain: empirical second moment. (a) Chain MDP. (b) Second moment of the multi-step Bellman residual Δ^ over trace lengths at γ=0.9 , λ=1 (log–log axes). Error bars ( ±1 standard deviation, 30 replicates) are smaller than the markers. At t=50 , Retrace and Peng’s Q(λ) are 1.43× and 2.28× MINT , whose curve levels off over the tested lengths. Setup: Section D.2 .
Figure 3: State-coupled horizon budget. At γ=0.9 , green satisfies the sufficient condition in Corollary C.1 ; pink is inconclusive. Guides show discount-only bounds for λ∈{0.7,0.5} .
Figure 4: Adaptive trace lengths and mismatch visualization on 11-task DMC. Points show mean trace lengths vs the action-based mismatch proxy χλproxy . Hollow/filled markers denote measurements at ≤140 K/ >140 K steps. Green contours show the fraction of discount-grid values satisfying an approximate contraction inequality; Blue lines mark calibrated transition-based horizon references (group-mean Kref ); Gray curves show empirical 90th-quantile references. Trace lengths vary across tasks and steps. More details are in Section F.2.3 .
Figure 5: Performance and adaptive truncation behavior on MuJoCo. (a–b) Aggregate normalized scores (IQM) of SAC (fixed- α ) and TD3 variants over training steps; shaded regions show 95% stratified bootstrap confidence intervals across 5 seeds per task. (c–d) Mean truncation lengths by MINT across five MuJoCo environments, with ±1 standard deviation across seeds. SAC horizons shorten over training, while TD3 horizons remain closer to the cap.
Figure 6: 11-task DMC aggregate performance at 500K steps (5 seeds). Scores are normalized by 1000 . Rectangles show 95% bootstrap intervals; black ticks mark estimates.
Kmax
Keff
Stops before cap (%)
5
3.9±0.1
36.7±3.3
50
4.8±0.8
99.5±0.7
Table 2: RWRL cap ablation. Mean ± std. across three seeds; actuator-noise levels pooled.
Figure 7: DMC target diagnostics at 200K (RQ3). Green: MINT ; red: fixed λ -TD traces (square: K=10 ); blue: n -step. Dark-to-light: length 1–10. Five training seeds per task. Axes: absolute aggregated signed bias and target std., relative to 100 ; scales differ. Full results: Section F.2.4 .
Table 3: OGBench frozen-critic evaluation ( h=0.5 ). Approximate Monte-Carlo critic; 1 paired run per cell over 100 stratified evaluation states. Bold marks the lower value within each metric pair.
Appendix figures & tables61 assets
Supplementary material from the paper’s appendix.
Appendix
Result
Required assumptions
Additional conditions
State-coupled results and chain bounds
Lemma 3.3 Variance
Adapted joint law
Bounded TD error
Theorem 4.1 Contraction
State coupling
κ<1 in Equation 9
Corollary B.1 Closed form
Cross-Doeblin; kernel TV d
—
Lemma B.2 Disagreement
Cross-Doeblin; kernel TV d
—
Lemma B.5 Perturbation
Kernel TV d ; Assumption B.3
Stationary laws; explicit bπ(t)
Appendix
Table 4: Assumption usage by result. No listed result requires deterministic dynamics. State-coupled and action-level surrogate results have separate sufficient conditions.
Figure 9: Geometric-overlap specialization at γ=0.99 . Under the cross-Doeblin and uniform initial-step overlap hypotheses of Corollary B.1 , γ[1+2Dˉwgtλ(1−ρ)]<1 ensures contraction of the fixed-policy state-coupled operator. Dark-to-light green bands identify the satisfying subsets of λ∈{1,0.7,0.5,0.3} specified in the legend. Pink is inconclusive for all four displayed values; it does not indicate divergence. For each λ , its curve marks equality and the condition holds strictly above it. The inset enlarges the high-overlap region; open markers are hypothetical boundary points at Dˉwgt=0.2 . The comparison holds the summary coordinate Dˉwgt fixed. These state-coupled conditions do not certify action-level surrogate contraction or deep-RL convergence.
Dˉwgt
Regime
λ∗
Kλmax ( λ=1 )
Kλmax ( λ=0.5 )
0.001
(A)
any
∞
∞
0.05
(A)
any
∞
∞
0.10
(B)
0.556
2.00
∞
0.30
(B)
0.185
1.20
1.20
0.50
(B)
0.111
1.11
1.11
1.00
(B)
0.0556
1.05
1.05
Appendix
Table 5: State-coupled contraction budget at γ=0.9 ( Corollary C.1 ). λ∗=(1−γ)/(2γDˉwgt) is the critical λ below which the sufficient condition holds for any meeting-time distribution. Finite entries require Kλ<Kλmax ; ∞ denotes no additional horizon restriction beyond the discount-only bound, not an infinite realized horizon.
Figure 10: Diagnostic study on tabular CliffWalking. (a-top) Environment visualization for CliffWalking . (a-bottom) State-wise total variation distance d(s) between an optimal policy and a uniformly random policy, illustrating pronounced off-policy discrepancies near the cliff region. (b) Pairwise overlap matrix ρ(s,s′) between transition kernels induced by the two policies; near-zero overlap across most state pairs highlights a strong structural mismatch. (c) Time-averaged expected-disagreement bounds from Lemma B.2 for illustrative (d,ρ) pairs; these curves do not certify a meeting-time bound for the displayed environment.
Figure 11: Data-processing gap on the slippery Single-Trap chain ( N=50 , γ=0.9 , λ=1 , slippery ξ varied). (a) Action-level Ds (red) is unchanged by slippery dynamics, while kernel-level ps (green) shrinks at the trap state. (b) The plotted policy-TV and kernel-TV weighted diagnostics separate with ξ ; the smaller kernel diagnostic is not substituted into the contraction condition. (c) Cross-Doeblin ρ=0 for the chain MDP without restart. (d) The left-hand side of the sufficient condition in Corollary B.1 exceeds 1 throughout; the restart modification introduces ρ>0 .
qrst
measured ρ
MINT LHS
Reference LHS
1/(1−ρ)
0.00
0.000
1.101
1.101
1.000
0.05
0.050
1.064
1.072
1.053
0.10
0.100
1.023
1.036
1.111
0.20
0.200
0.985✓
1.006
1.250
0.30
0.300
0.957✓
0.982✓
1.429
0.50
0.500
0.930✓
0.960✓
2.000
Appendix
Table 6: Uniform-restart diagnostic: MINT and the no-survival reference bound on the restart chain ( N=30 , γ=0.9 , λ=1 , ξ=0.1 ). The contraction condition is γ(1+2Dˉwgtλ(1−ρ))<1 for MINT and the ρ=0 specialization for the reference. At qrst=0.20 : MINT contracts ( ✓ ) while the reference bound is inconclusive.
Figure 12: Uniform-restart diagnostic: cross-Doeblin overlap and the region satisfying MINT ’s sufficient contraction condition on the restart chain ( N=30 , γ=0.9 , ξ=0.1 ). (a) Measured ρ matches restart probability qrst exactly. (b) The plotted policy-TV and kernel-TV diagnostics decrease with ρ . Only the policy-TV quantity enters the stated contraction bound. (c) MINT ’s contraction LHS (green) drops below 1 at qrst=0.20 , while the no-survival reference LHS (red) does not until qrst=0.30 . (d) Empirical region ratio matches the theoretical 1/(1−ρ) to three decimal places.
Ds
Variant
Keff
Variance
Backup MSE
0.90
Surrogate, no additional cap
10.0
0.129
13.8±12.1
Surrogate, cap 10
6.52
0.128
9.5±6.5
0.99
Surrogate, no additional cap
39.5
0.143
40.8±46.5
Surrogate, cap 10
9.57
0.141
16.1±12.6
Appendix
Table 7: High-mismatch surrogate and cap comparison at a 50 -step evaluation window. Backup MSE is mean ± SD over 20 perturbation seeds; variance is measured at Qπ . No additional cap means that the trace is still limited by the 50 -step window.
Figure 13: Representative high-mismatch ablation: Ds=0.90 . Policies are β(+1∣s)=0.05 and π(+1∣s)=0.95 at every state. (a) Variance versus mean retained horizon, including random controls matched separately to the surrogate and state-coupled reference. (b) The symmetric log-ratio heuristic saturates and retains all 50 steps; the surrogate has lower variance here. (c) Cap 10 reduces the measured horizon from 10.0 to 6.52 with little change in variance. The inset displays the right-censored stopping time min(T^,50) ; the final bin includes T^≥50 , and about 39% of traces reach cap 10 . (d) Sampled backup MSE at Q0=Qπ+ξ , ξ∼N(0,I) ; the cap lowers the mean from 13.8 to 9.5 . Error bars: (a)–(c), ±1 SD across 20 trajectory replicates; (d), bootstrap percentile 95% CI for the mean across 20 perturbation seeds. These MSEs are not expected-operator contraction measurements.
Figure 14: Additional high-mismatch comparison: Ds=0.99 . Policies are β(+1∣s)=0.005 and π(+1∣s)=0.995 at every state; panel definitions and error-bar conventions match Figure 13 . Cap 10 reduces the measured surrogate horizon from 39.5 to 9.57 and mean backup MSE from 40.8 to 16.1 , while variance changes only from 0.143 to 0.141 . About 91% of traces reach cap 10 . The inset is right-censored at 50 , so the final bin includes T^≥50 . The state-coupled reference has horizon 44.5 and is not a minimum-MSE benchmark. The matched-random controls and variance comparisons do not establish a general surrogate advantage across policy pairs.
Algorithm 2 MINT multi-step target construction — ordinary-residual form matching Algorithm 1 ; SAC/TD3 substitutions are specified below.
Method
Target or continuation rule
One-step
r0+γv1 .
Uncorrected L -step
∑i=0L−1γiri+γLvL .
Retrace
YL(c) with ci=min{1,π(ai∣si)/β(ai∣si)} ; use Equation 62 .
Peng’s Q(λ)
PL−1=rL−1+γvL ; Pi=ri+γ[(1−λ)vi+1+λPi+1] for i=L−2,…,0 ; return P0 .
MINT
YL(c) with ci=A^i , A^i∼Bernoulli(h(si,ai)) ; equivalently q0+∑i=0K^−1(γλ)iδi .
Appendix
Table 8: Consistent finite-cap targets for the ordinary Bellman residual used in the main text. Here L is deterministic; MINT ’s realized prefix length is K^≤L . The maximum-entropy substitutions are specified below.
Hyperparameter
Value
Actor learning rate
1×10−3
Critic learning rate
1×10−3
Batch size
100
Replay buffer size
1×106
Discount factor γ
0.99
Polyak averaging coefficient ( τ )
0.995
Appendix
Table 9: TD3 Hyperparameters
Hyperparameter
Value
Actor learning rate
1×10−3
Critic learning rate
1×10−3
Entropy coefficient α
0.2 (fixed; no automatic tuning)
Batch size
100
Replay buffer size
1×106
Discount factor γ
0.99
Appendix
Table 10: SAC Hyperparameters (MuJoCo main experiments). The entropy coefficient is fixed at α=0.2 (the SpinningUp default) for MINT and all multi-step baselines; no automatic entropy tuning is used in these runs. The DMC ablation and RWRL experiments use their separately stated entropy settings; the fixed- α statement here is restricted to the main MuJoCo SAC comparison.
Figure 15: Performance with stochastic truncation. Across five MuJoCo benchmarks, MINT attains higher returns than the multi-step baselines on most tasks and at earlier training checkpoints. For baseline comparisons, we follow the convention of [ 24 ] and fix the cap length to n=5 for all multi-step methods.
Figure 16: Evaluation of TD3 variants over OpenAI MuJoCo environments.
Figure 17: Comparison of SAC, SAC- MINT , and MBPO across four MuJoCo benchmarks. We use different multi-step configurations per environment: halfcheetah and walker use a standard multi-step setting (cap length =5 , λ=0.7 ), whereas ant and hopper follow the longer-horizon model-based configuration inspired by MBPO (cap length =25 , λ=0.1 ). The curves compare the sample efficiency of SAC, MINT , and MBPO under these task-specific settings; MINT often attains returns comparable to MBPO while using a model-free algorithm.
Environment
Random Score
Expert Score
Hopper
−20.27
3234.3
HalfCheetah
−280.18
12135.0
Walker 2d
1.63
4592.3
Ant
−325.6
3879.7
Humanoid
78.85
9024.95
Appendix
Table 11: Reference scores for min–max normalization. Random and expert performance values are taken from D4RL [ 13 ] and Minari [ 12 ] .
task
λ=0.1
λ=0.3
λ=0.5
λ=0.7
λ=0.9
Ant-v2
5812±81
5991±92
5466±98
3922±553
3417±1594
Hopper-v2
3453±8
3291±60
3280±3
3151±161
3262±137
Appendix
Table 12: λ sweep for SAC- MINT (fixed α=0.2 , Kmax=5 , 1M steps; mean ± s.d. across 2–3 seeds per cell ). The main-table value is λ=0.7 .
Evaluation
Steps and seeds
Target settings
Score window
Primary eleven-task
500K; five seeds per method/task
Cap 10 , γ=0.99 , automatic α ; MINT λ=0.9
Last five evaluations; aggregate return/ 1000
Supplementary five-task
1M; three-seed table, separate five-seed curves and CartPole re-run
Cap 5 on CartPole , 10 elsewhere; γ=0.999 on sparse tasks, 0.99 on Hopper-Stand ; automatic α , MINT λ=0.9
Last 100K steps (mean 10 ); raw returns
Appendix
Table 13: DMC evaluation protocols. Supplementary evaluation scores are not pooled into the primary eleven-task aggregates. The supplementary evaluation’s five-seed curves and CartPole re-run retain their separately reported configurations.
Task
SAC 1-step
SAC 10-step
SAC- MINT
Rare ( <5% ; 2 tasks)
Cartpole swingup sparse
431±353
291±229
493±196
Pendulum swingup
249±176
560±264
284±253
Occasional ( 10 – 50% ; 4 tasks)
Finger turn easy
492±54
498±43
663±62
Finger turn hard
323±59
288±74
541±105
Appendix
Table 14: 11-task DMC final returns by reward accessibility. Last-five-evaluation average at 500K steps; mean ± standard deviation across five seeds. Groups use the fraction of uniform-random episodes containing positive reward. Gray literature stars in the task-curve figure use a different protocol and are not matched baselines.
Figure 18: 11-task DMC evaluation-time IQM. All eleven tasks and the original six-/five-task A/B split. Bands are pointwise 95% stratified-bootstrap intervals; final-window aggregate scores average five evaluations per run.
Figure 19: 11-task DMC final-score profiles. Fraction of pooled task/seed runs above return/ 1000 threshold τ , with pointwise 95% bootstrap bands. Each task contributes five seeds.
Figure 20: 11-task DMC task learning curves. Mean over five seeds; bands are one standard error, not 95% intervals. Each evaluation averages ten episodes. Gray stars are published SAC reference values with a different action-repeat protocol.
Figure 21: DMC reward-accessibility groups. Endpoint IQM with 95% confidence-interval rectangles and black estimate ticks. Group sizes are 2/4/5; grouping is descriptive rather than a universal exploration taxonomy.
Figure 22: DMC action-dimensionality groups. Endpoint IQM, bootstrap intervals and estimates. The two-dimensional group contains only Finger tasks, confounding action dimension with task identity.
Figure 23: DMC learned-horizon groups. Endpoint IQM grouped by MINT mean length after training. Grouping is post hoc and does not certify an optimal horizon.
Task
100K
200K
300K
500K
Cartpole sparse
5
5
5
5
Pendulum
5
4 (2)
3 (2,3)
1 (1,2,3,4)
Finger easy
4 (1)
5
5
4 (2)
Finger hard
3 (0,1)
3 (0,4)
4 (3)
2 (0,1,2)
Hopper stand
5
5
4 (1)
4 (1)
Hopper hop
5
4 (1)
4 (1)
4 (1)
Appendix
Table 15: Seeds retained in target diagnostics. Each cell gives the retained count, followed by excluded zero-based seed IDs in parentheses when nonempty. Training and control-score aggregates use all five seeds.
Figure 24: DMC target-spread diagnostics with the weighted TD-error reference. 33 task–stage cases at 200K/300K/500K, conditional on the seed filter in Table 15 . (a) Fixed- Kλ -weighted TD trace standard deviation divided by its length-one value; gray lines are cases and the violet line is their pointwise median. (b) MINT /reference standard-deviation ratios; black diamonds are medians. The cap-ten reference is the fixed TD trace, not the Peng mixture. The mean- K comparison interpolates the fixed-length standard-deviation curve; it is not a randomized matched-length estimator.
Figure 25: DMC temperature associations. Each of 55 runs uses 50 training records. Gray points are seeds; pink diamonds are task medians and segments are seed min–max ranges, not confidence intervals. Weak overall median correlations do not imply independence.
Figure 26: All-task DMC bias–variance plane at 100K. Eleven task panels; 51/55 task–seed cells retained ( Table 15 ). Red: fixed- Kλ -weighted TD trace, square: K=10 ; blue: pure n -step; green diamond: MINT . Dark-to-light points mark length 1–10. Horizontal: absolute aggregated signed bias; vertical: target standard deviation. Both are percentages relative to the normalization value 100 , not a soft-value upper bound. Panel scales differ and no confidence intervals are shown. The available behavior and target checkpoints coincide. MINT is one-step by construction; overlapping marks can hide its diamond. The other fixed-length targets remain useful reference curves with learned critics.
Figure 27: All-task DMC bias–variance plane at 200K. Eleven task panels; 50/55 task–seed cells retained ( Table 15 ). Red: fixed- Kλ -weighted TD trace, square: K=10 ; blue: pure n -step; green diamond: MINT . Dark-to-light points mark length 1–10. Horizontal: absolute aggregated signed bias; vertical: target standard deviation. Both are percentages relative to the normalization value 100 , not a soft-value upper bound. Panel scales differ and no confidence intervals are shown. The first off-policy stage includes current and older behavior checkpoints. Finger / Hopper and Walker panels show different trade-offs; compare both coordinates within a panel rather than reading a single universal frontier.
Figure 28: All-task DMC bias–variance plane at 300K. Eleven task panels; 50/55 task–seed cells retained ( Table 15 ). Red: fixed- Kλ -weighted TD trace, square: K=10 ; blue: pure n -step; green diamond: MINT . Dark-to-light points mark length 1–10. Horizontal: absolute aggregated signed bias; vertical: target standard deviation. Both are percentages relative to the normalization value 100 , not a soft-value upper bound. Panel scales differ and no confidence intervals are shown. This figure includes all eleven tasks at 300K, including the Acrobot counterexample. Finger-Easy , Walker-Walk , Fish , and Acrobot show spread savings, modest changes, and a negative case under the same stage-specific aggregation.
Figure 29: All-task DMC bias–variance plane at 500K. Eleven task panels; 45/55 task–seed cells retained ( Table 15 ). Red: fixed- Kλ -weighted TD trace, square: K=10 ; blue: pure n -step; green diamond: MINT . Dark-to-light points mark length 1–10. Horizontal: absolute aggregated signed bias; vertical: target standard deviation. Both are percentages relative to the normalization value 100 , not a soft-value upper bound. Panel scales differ and no confidence intervals are shown. These panels use the 500K checkpoints. Pendulum retains one seed and Finger-Hard two; these panels do not characterize all five trained runs. Negative cases remain visible, so lower spread or bias is not asserted uniformly.
Figure 30: (a) CartPole-Balance-Sparse : the original λ∈{0.3,0.7,0.9,1} comparison. MINT succeeds at λ=0.3 and 0.7 ; the poor-performing 0.9 and 1 settings are also retained. (b) Normalized next-state distances on Hopper (median ≈0.36 , 90th percentile ≈0.64 ), used as descriptive dynamics diagnostics; they do not certify the surrogate contraction condition of Theorem B.9 .
Task
SAC- MINT
SAC-multistep
Vanilla SAC
Discriminative
CartPole-Swingup-Sparse
719±46
24
9±16
Hopper-Stand
891±24
57±90
—
Finger-Turn-Easy
686±26
767±62
—
Scope (non-discriminative)
Ball-in-Cup-Catch
976±2
—
976±1
Appendix
Table 24: DMControl ablation. Mean evaluation return at 1M environment steps (mean 10 ± std, 3 seeds). Bold marks the best per row. SAC- MINT improves over uncorrected multi-step on CartPole and Hopper; both learn on Finger, saturate on Ball-in-Cup, and fail on the tested Acrobot setting.
Figure 31: Supplementary DMC learning curves. Five seeds, 95% bootstrap intervals, 1M steps. The Hopper-Stand one-step reference is its final return. These curves retain their original settings, separately from the three-seed table and the CartPole re-run ( Tables 13 and 24 ).
Figure 32: Fixed/auto-temperature comparison on CartPole-Swingup-Sparse . Cap L=10 ; entropy coefficient (left) and realized length (right). The original plotted ranges are preserved. This protocol comparison is distinct from the within-run correlations in Section F.2.6 .
Method
Runs/game
Median
IQM
Mean
Gap
MINT ( λ=0.9 )
5
0.152
0.251
0.512
0.627
DER
100
0.189
0.183
0.350
0.698
DrQ( ε )
100
0.313
0.280
0.465
0.631
SPR
100
0.396
0.337
0.616
0.577
EfficientZero
3
1.116
1.020
1.945
0.371
BBF (author scores)
50
0.883
1.034
2.232
0.339
Appendix
Table 25: Atari 26-game aggregate point estimates. MINT : primary, five seeds/game. Literature rows use published point estimates [ 3 , 37 ] ; the BBF row aggregates 50 runs/game from the authors’ released scores. Bottom rows are approximate estimates recovered from the rliable paper’s 10-run figure. The 10-run and 100-run DrQ references are reported separately. Higher is better except optimality gap.
Metric
MINT (5 runs/game)
BBF (50 runs/game)
Median
0.152[0.120,0.180]
0.883[0.825,0.942]
IQM
0.251[0.225,0.279]
1.034[0.999,1.070]
Mean
0.512[0.470,0.554]
2.232[2.183,2.281]
Optimality Gap
0.627[0.609,0.646]
0.339[0.331,0.347]
Appendix
Table 26: Full-suite estimates and uncertainty. Point estimate [95% stratified-bootstrap interval] from the archived aggregate results. Entries use numerical interval endpoints, which may differ slightly from those in the figures. BBF is published context, not a matched comparison.
Figure 33: Pooled-run Atari performance profile. Fraction of game/run scores above HNS threshold, with pointwise 95% intervals. MINT: primary, five runs/game; BBF: 50 author runs/game. DrQ is the approximate 10-run profile recovered from the rliable paper, not the 100-run game-mean reference used in the group analysis. Lines connect the 201 evaluated thresholds without smoothing or additional interpolation; bands report the archived pointwise intervals. This differs from the game-mean profile in Figure 36 .
Nominal steps
Median
IQM
Mean
Gap
10K
0.008[0.001,0.011]
0.002[−0.001,0.006]
0.020[−0.020,0.061]
0.989[0.954,1.025]
50K
0.068[0.063,0.087]
0.108[0.096,0.123]
0.271[0.246,0.296]
0.770[0.753,0.788]
100K
0.152[0.120,0.179]
0.251[0.224,0.279]
0.512[0.469,0.554]
0.627[0.608,0.645]
Appendix
Table 27: Primary MINT evaluation checkpoints. Point estimate [95% interval]; 26 games and five training seeds. The larger final evaluation budget differs from the intermediate budget.
Figure 34: Atari task groups under Bellemare et al. [ 6 ] . E/H: easy/human-optimal; E/X: easy/score-exploit; H/D: hard/dense; H/S: hard/sparse (11/6/7/2 games). (a) Equal-game mean HNS. (b) Game-mean differences and group means against published DrQ( ε ). No baseline-run confidence intervals or subgroup pooled-run IQMs are inferred.
Wins
Mean
Median
IQM
Group
/games
MINT
DrQ
MINT
DrQ
MINT
E/H
7/11
0.689
0.463
0.669
0.429
0.521
E/X
2/6
0.741
0.875
0.398
0.423
0.400
H/D
1/7
0.077
0.151
0.075
0.124
0.069
H/S
2/2
0.372
0.343
0.372
0.343
0.364
Appendix
Table 28: Atari group performance. Mean HNS and median across game means for both methods; IQM pools MINT ’s game/seed scores only. DrQ group IQM is unavailable. Wins count MINT game means above the published DrQ reference, not a probability of improvement. DrQ values use rounded published returns.
Figure 35: Group mean, median, and IQM with seed uncertainty. E/H: easy/human-optimal; E/X: easy/score-exploit; H/D: hard/dense; H/S: hard/sparse. Pink rectangles are MINT ’s 95% percentile seed-bootstrap intervals (50,000 resamples, seeds independently resampled within each fixed game; no task resampling). Black vertical ticks are estimates; open diamonds are DrQ point references from published game means. Median is over game means, while IQM pools game/run scores. DrQ group IQM, baseline intervals, and difference intervals are unavailable. The sparse group’s tight seed interval does not establish robustness over task selection.
Group
Mean difference
Contribution to suite mean
Easy/human
+0.225587
+0.095440
Easy/exploit
−0.133670
−0.030847
Hard/dense
−0.073158
−0.019696
Hard/sparse
+0.028792
+0.002215
Total
+0.047112
Appendix
Table 29: Where the Atari mean difference comes from. Equal-game arithmetic decomposition against rounded published game means, without a confidence interval for the difference.
Figure 36: Atari game-mean profile and IQM progression. Left: fraction of 26 game means above HNS threshold τ ; MINT averages five seeds/game and baselines use published game means. This is not a pooled-run profile. Right: MINT IQM by nominal training step (five evaluation episodes at 10K–90K, 100 at the endpoint); DrQ/DER horizontal lines are final-IQM references, not baseline learning curves. Bands show pointwise bootstrap intervals for MINT ; final-reference comparisons do not establish a learning-speed advantage.
Figure 37: Primary Atari agreement and horizon diagnostics. Group curves are equal-game averages of five-seed iteration means. Final scatter points are five-seed game means; correlations are descriptive across 26 games. Steps are nominal iteration counts times 10K; no smoothing is applied. Published DrQ game means are point references.
Figure 38: Additional Atari λ ablation. λ=0.9 (seeds 0–4) versus 0.7 (seeds 5–9). IQM uses filled 95% bootstrap confidence-interval bars and black estimate ticks. Other panels retain horizon and game-group HNS differences. This is supplementary to the primary 0.9 result.
Game
MINT return
DrQ return
MINT HNS
DrQ HNS
Δ HNS
E/H: easy/human-optimal
Assault
844.1±255.1
579.6
1.1965
0.6875
+0.5090
Asterix
859.3±119.1
763.6
0.0783
0.0668
+0.0115
BattleZone
4466.0±1460.7
10165.3
0.0605
0.2241
−0.1636
Boxing
19.4±2.8
9.0
1.6102
0.7417
+0.8685
Breakout
21.0±9.1
19.8
0.6691
0.6285
+0.0406
Appendix
Table 30: Primary Atari performance by game and task group. MINT raw return: mean ± standard deviation over five seeds; DrQ return: published point reference. The final columns are human-normalized game means and their difference. Standard deviations are not confidence intervals; no uncertainty for the difference is inferred. Baseline protocols are not matched ablations. Values are rounded, including PrivateEye ’s near-zero HNS.
Kmax
Keff/Kmax
Eβ[p^]
5
0.79
0.85
50
0.10
0.70
Appendix
Table 31: Additional diagnostics for the fixed- α RWRL comparison in Table 2 : α=0.05 , 200k steps, three seeds, noise levels pooled.
Kmax
Keff
Pr[K^<Kmax]
learned α
5
4.34→4.86→4.85
23.9%→5.5%→5.7%
0.0023
10
8.45→9.49→9.69
27.2%→9.3%→5.9%
0.0082
50
23.98→29.25→21.86
76.7%→61.8%→75.9%
0.0058 – 0.0138
Appendix
Table 32: Auto- α arm: Keff and Pr[K^<Kmax] at 20k → 100k → 780k steps (15 trials per cap, σ pooled), and the learned α at the end of training (5-seed mean).
σ
Kmax=10
Kmax=50
0.0
938.3±39.7
963.2±6.0
0.1
878.1±104.7
967.6±6.7
0.3
865.7±79.3
963.7±3.5
pooled ( n=15 )
894.0±80.4
964.8±5.5
Appendix
Table 33: Eval return on RWRL walker_walk by transition-noise level (auto- α arm, 780k steps, mean ± s.d. over 5 seeds; max return 1000).
Kmax
Keff
ΔKeff
return
Δ return
50
21.86
—
965
—
10
9.69
−56%
894
−7%
5
4.85
−50%
358
−60%
Appendix
Table 34: Return as a function of the cap (auto- α arm, 780k steps, 15 trials per cell; the only experiment with a Kmax=10 condition).
Figure 39: Recent-behavior stopping distribution. Auto- α , λ=0.9 ; π at 1M, β at 900K, training cap 50 . Panels pool three seeds, 40 states/seed, and 256 draws/state. Dashed caps 5/10/50 are applied to the same stopping draws. Means are 7.06/10.56 , medians 5.5/6 , and maxima 56/75 at σ=0.1/0.3 ; neither condition has censored draws. Horizontal “meeting time” T denotes the surrogate stopping time T^ .
Figure 40: Surrogate stopping survival across behavior ages. Fixed π at 1M; solid/dashed/dotted curves use β at 900K/500K/100K (ages 100K/500K/900K). Each noise–age condition pools three seeds and 30,720 indicator draws. Dashed vertical lines mark caps 5/10/50 ; the horizontal axis is logarithmic. Recent and intermediate conditions have no censoring. At age 900K, 33.5%/3.0% of draws remain censored at the observation horizon 1000 for σ=0.1/0.3 ; their unobserved tails are not extrapolated.
σ
β / age (K)
Mean T^
Median
E[K50]
Cut at 50 (%)
Censored (%)
0.1
900/100
7.06
5.5
7.06
0.007
0
0.1
500/500
14.95
9
14.21
4.47
0
0.1
100/900
≥447.2
242
45.96
79.73
33.5
0.3
900/100
10.56
6
10.52
0.583
0
0.3
500/500
22.38
13
18.10
10.38
0
0.3
100/900
≥243.9
159
44.61
75.70
3.0
Appendix
Table 35: Behavior age and cap- 50 intervention. Pooled draw summaries, fixed π at 1M. Values marked ≥ are lower bounds obtained by recording censored draws at 1000 . For L≤100 , E[KL] and Pr(T^>L) are fully observed empirical summaries despite tail censoring. Percentages are descriptive, without an independent-draw confidence claim.
checkpoint
Eβ[p^]
matching diagnostic mϵ(s)
29%
0.981
0.755
57%
0.874
0.473
71%
0.909
0.655
100%
0.735
0.365
Appendix
Table 36: Action-level TV versus the ϵ -tolerant matching diagnostic on RWRL walker_walk ( Kmax=50 arm, seed 0) at four checkpoints. Eβ[p^]≥mϵ(s) at all 250 retained states; reported Spearman correlations are 0.95 across cells and 0.86 across states. The matching statistic is not an estimate of exact kernel TV.
coefficients (λ,γ)
mean κ^
median
s.d.
κ^<1
(0.7,0.99) (deployed, Section 5 )
0.90
0.88
0.13
78%
(0.9,0.999) (training coefficients)
1.23
1.11
0.44
27%
Kmax at (0.7,0.99)
mean κ^
κ^<1
K^ˉprobe
Pr[K^probe=min(Kmax,L)]
5
0.84
90%
4.68
88%
50 (effective 10 )
0.96
67%
7.75
61%
λ=0 gate: γ
κ^
κ^/γ
Appendix
Table 37: Empirical backup gain κ^ on RWRL walker_walk with transition noise. Top: operating point (pooled over cap, noise, seed). Middle: deployed coefficients by nominal cap. Bottom: comparison at λ=0 with the one-step Bellman reference γ . Probes use windows of length L=10 , so the Kmax=50 row has an effective cap of 10 . These are fitted probe measurements, not certificates of the global surrogate sufficient condition.
Dataset
Bias: MINT
Bias: Retrace
t
large-navigate (det.)
0.35
2.77
2.5
large-stitch (det.)
0.38
3.81
4.0
teleport-navigate (stoch.)
0.16
2.14
2.3
teleport-stitch (stoch.)
0.22
2.60
3.2
Appendix
Table 38: Additional frozen-critic statistics at h=0.5 for the comparison in Table 3 . t : paired t -statistic of the per-state absolute-error difference, MINT vs. Retrace.
dataset
h
operator
median
mean ± s.d.
unstable seeds
large-navigate (det.)
1.0
MINT
17.15
17.15±0.55
0/5
Retrace
19.92
19.92±0.53
0/4
shuffled- K^
16.62
16.62±0.56
0/5
teleport-navigate (stoch.)
1.0
MINT
17.55
17.20±1.63
0/5
Retrace
12.04
12.04±0.43
0/5
shuffled- K^
16.98
16.17±1.98
0/5
Appendix
Table 40: Learned-critic (fitted-Q evaluation) RMSE against the Monte-Carlo Qπ at 300k gradient steps, 5 seeds (one Retrace cell has 4). Bold marks the best median in each cell.
Temporal-difference learning estimates values by bootstrapping from later predictions, enabling online learning and control from sequential experience.
Nanjing University of Posts and Telecommunications, Nanjing, China · Department of Computer Science and Technology, Nanjing University, Nanjing, China · College of Electronic Countermeasure, National University of Defense Technology, Hefei, China