Off-policy evaluation (OPE) estimates the expected return of a target policy from previously collected data without additional environment interaction. While OPE methods for flat Markovian policies are well studied, little work has addressed evaluating hierarchical policies in which a high-level policy selects temporally extended options that generate primitive actions until a termination condition is met. The primary existing approach applies per-decision importance sampling at the option level, but requires option-annotated trajectories and exhibits variance that still grows with the horizon. For non-hierarchical policies, fitted Q-evaluation (FQE) typically achieves lower error than importance sampling by leveraging the classic Bellman equation for policy evaluation; however, a naive extension of FQE to hierarchical policies results in a biased policy value estimate. In this paper, we introduce Intra-Option Fitted Q-Evaluation (IO-FQE), which estimates a value function in the augmented state-option space; this approach enables OPE of hierarchical policies even when we lack annotations of what option was run in the data. We develop two continuous-action instantiations of IO-FQE and show on long-horizon hierarchical continuous control tasks (AntMaze navigation and OGBench Puzzle manipulation) that IO-FQE eliminates the bias incurred by a naive application of FQE to hierarchical policies and substantially lowers mean squared-error compared to option-level importance sampling.
Figures & tables
Figure 1: The committed hierarchical policy Π (left) executes an option until termination is sampled from βo(⋅) before drawing a new one from μ(⋅∣s) , whereas πM (right), the policy flat FQE actually evaluates, resamples an option from μ(⋅∣s) at every primitive step. The two induce different actions along the same state trajectory; the resulting return gap Δ(Π)=∣J(Π)−J(πM)∣ is formalized in Proposition 2.3 .
Flat FQE
Regime
Environment
Option-PDIS
IO-FQE-BE
IO-FQE-AC
vs J(Π)
vs J(πM)
Hierarchical
AntMaze-U
6.35±1.76
0.14±0.10
0.39±0.31
1.07±0.34
0.23±0.04
AntMaze-M
55.5±3.9
0.74±0.14
0.66±0.35
7.52±1.63
1.09±0.23
AntMaze-L
79.2±2.5
1.51±0.91
0.84±0.49
14.85±1.29
1.96±0.25
Puzzle- 3×3 ( ×103 )
7.2±3.8
10.6±4.4
0.6±0.3
5.3±1.2
0.7±0.1
Puzzle- 4×6 ( ×103 )
N/A
436.1±54.1
46.4±21.4
301.7±56.6
67.4±19.8
Table 1: MSE of policy value estimates ( ↓ ). Ground-truth values are obtained by Monte Carlo rollout and reported in Appendix D.3 . Under the hierarchical regime, data are collected by a hierarchical behavior policy with option commitment; under the flat regime, by a flat primitive policy. Option-PDIS requires option-annotated trajectories and is inapplicable in the flat regime (marked —). For Puzzle environments, MSE values are scaled by 10−3 . Values report mean with 95% bootstrapped confidence intervals over 10 seeds. Paired-bootstrap comparisons between methods are reported in Appendix D.5 .
Figure 2: MSE vs. dataset size on AntMaze-L and Puzzle- 3×3 under flat and hierarchical data regimes. Dataset size varies from 10% to 200% of the full dataset. Shaded regions indicate 95% bootstrapped confidence intervals over 10 seeds.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: AntMaze layouts used in our experiments: UMaze from D4RL Fu et al. [2020] , Medium and Large from OGBench Park et al. [2024] . The yellow figure is the Ant agent at its start position; the pink marker indicates the goal. Layouts are shown at their relative scales.
Figure 4: Puzzle (Lights Out) environments from OGBench Park et al. [2024] : 3×3 (9 buttons) and 4×6 (24 buttons). Pressing a button toggles its own state and that of its 4-connected neighbors. The button colors shown are the initial states under the rendering seed and do not carry semantic meaning.
Flat
Hierarchical
Environment
Episodes
Mean Length
Episodes
Mean Length
AntMaze-U
200
412.6
200
404.3
AntMaze-M
200
504.7
200
477.5
AntMaze-L
200
764.7
200
733.2
Puzzle 3×3
200
4,579.1
200
4,764.3
Puzzle 4×6
200
6,753.8
200
6,488.6
Appendix
Table 2: Dataset sizes by environment and regime. Each dataset consists of 200 episodes.
Environment
J(Π)
J(πM)
AntMaze-U
−94.30±0.32
−94.87±0.28
AntMaze-M
−95.24±0.28
−98.35±0.57
AntMaze-L
−96.42±0.41
−98.78±0.62
Puzzle 3×3
−226.52±3.90
−285.35±4.53
Puzzle 4×6
−895.33±32.14
−801.64±29.33
Appendix
Table 3: Ground-truth policy values estimated by Monte Carlo rollout. Values report mean ± one standard deviation.
Hyperparameter
Value
Discount factor γ
0.99
Network
MLP with hidden layers [256,256]
Optimizer
AdamW (learning rate 3×10−4 , weight decay 10−5 )
Target-network update rate τ
0.005
Gradient steps
105
Batch size
256
Appendix
Table 4: Hyperparameter configuration shared by all FQE-based estimators on AntMaze and Puzzle, fixed from standard defaults before any evaluation.
Figure 5: Tabular validation on a 5×5 deterministic gridworld with two strict-commitment options. Left: estimated J(Π) as p=μ(o0∣s) varies; flat FQE tracks J(πM) while both IO-FQE variants recover J(Π) . Right: absolute bias vs. dataset size at p=0.5 ; flat FQE’s bias plateaus at ∣Δ(Π)∣ regardless of N . Mean over 5 seeds; standard deviation is below 10−11 .
MSE ( ↓ )
Density Estimator
Hierarchical
Flat
Oracle (true pD )
0.35±0.11
—
MDN ( K=3 )
1.51±0.91
0.32±0.11
Gaussian
22.7±4.0
0.57±0.12
Appendix
Table 5: Effect of behavior density estimation on IO-FQE-BE (AntMaze Large). Oracle uses the true behavior policy network to compute pD(a∣s) ; MDN and Gaussian are trained on the offline dataset. The oracle is omitted under flat data because the IQL behavior policy is unimodal and both trained density models already approximate it well. MSE is computed against J(Π) ; mean with 95% CI over 10 seeds.
MSE ( ↓ )
Method
H=4
H=8
Flat FQE
2.40[2.32,2.49]×10−3
2.37[2.34,2.39]×10−2
Option-PDIS
6.14[4.05,8.56]×10−3
1.06[0.71,1.47]×10−3
IO-FQE-BE
1.5[1.0,2.1]×10−5
1.2[0.8,1.7]×10−4
IO-FQE-AC
2.5[1.7,3.5]×10−5
1.5[1.0,2.1]×10−4
Appendix
Table 6: MSE against J(Π) under time-dependent termination ( 15×15 four-rooms gridworld, options of duration H ). Each entry reports the mean with a 95% bootstrap confidence interval over 50 independently generated datasets.
Flat FQE
Scale
Option-PDIS
IO-FQE-BE
IO-FQE-AC
vs J(Π)
vs J(πM)
15×15
33[22,47]
11[6.8,17]
7.0[4.4,11]
1840[1820,1860]
1.1[0.74,1.6]
9×9
6.7[3.6,11]
5.1[3.0,7.5]
3.0[1.8,4.5]
1850[1830,1870]
0.56[0.36,0.79]
Appendix
Table 7: MSE ( ×10−5 ) in the short-horizon four-rooms gridworld with discrete actions. Each entry reports the mean with a 95% bootstrap confidence interval over 10 seeds.
Method
MSE ( ↓ )
IO-FQE-AC ( M=1 )
1.63[0.61,2.87]
IO-FQE-AC ( M=10 )
0.89[0.39,1.39]
IO-FQE-AC ( M=50 )
0.82[0.36,1.28]
IO-FQE-BE
0.32[0.21,0.43]
Appendix
Table 8: MSE of IO-FQE-AC on AntMaze-L under the flat regime as the number of Monte Carlo samples M varies, with IO-FQE-BE as a reference. Each entry reports the mean with a 95% bootstrapped confidence interval.