Simultaneous Forward and Inverse Human-in-the-Loop Optimization

Kyeongwon Park; Steven H. Collins

cs.ROSep 18, 2026

Simultaneous Forward and Inverse Human-in-the-Loop Optimization

Authors: Kyeongwon Park, Steven H. Collins

Organizations: Department of Mechanical Engineering, Stanford University, United States

Abstract

Subjective user experience is important to human-robot interaction, but the outcomes users value, and how those preferences vary across individuals and contexts, are often unknown. While inverse learning approaches using human data can help identify user rewards, in many assistive settings the experimental costs of executing a control policy, measuring biomechanical or physiological outcomes, and collecting user feedback often limit the number of queries and optimization iterations. Here, we present Simultaneous Forward and Inverse Human-In-the-Loop Optimization (SFIHILO), which efficiently infers individual-specific reward functions from preferences over human outcomes and identifies a final control policy that maximizes the learned reward. SFIHILO bootstraps a forward model to predict user outcomes from control policies, uses this model for active querying to accelerate inverse reward learning, and then optimizes the final control policy without additional user trials. We score candidate policies by their expected reduction in uncertainty across both forward and inverse beliefs, targeting a regional preference boundary to robustly inform this simultaneous learning process. In simulation, we show that SFIHILO was effective across user heterogeneity, outcome dimensionalities, precision requirements, noise levels, and nonstationarity; compared with mutual information approaches, the proposed active querying strategy significantly improved sample efficiency in inverse learning while preserving forward model accuracy. This approach demonstrates the potential to infer latent human goals, enabling more transferable and effective human-robot interaction.

Figures & tables

Figure 1: SFIHILO framework overview. (a) A real or simulated human compares a newly executed robot policy with the previous one; in simulation, this process is represented by a ground-truth outcome model and a true preference model. (b) The forward model learns a mapping from control policies to human outcomes. (c) The inverse model infers an outcome-space reward posterior from preference choices. (d) Active querying leverages both models to select informative policy comparisons. (e–f) After querying, the learned reward is used for final policy optimization.

Figure 2: Simulation evaluation of SFIHILO across precision requirements, outcome dimensionalities, noise levels, nonstationarity, and feature misspecification. Unless varied along an axis, panels use the default settings: outcome dimensionality

d=8

, practical noise levels, and reward-gap-balanced test sets; the feature-misspecification panels (H and I) use

d=10

. Boxplots summarize independently sampled users. Annotations report the relative change from Random querying within each condition; asterisks denote Holm-adjusted paired t-tests [ 18 ] (

\ast

p<0.05

\ast\ast

p<0.01

\ast\ast\ast

p<0.001

Simulated-user family	Forward RMSE $\downarrow$	Preference error (%) $\downarrow$	Final reward gap to optimum $\downarrow$
RBF (matched)	$0.0635\pm 0.0024$	$6.88\pm 0.72$	$0.0630\pm 0.0224$
Matérn- $5/2$	$0.0473\pm 0.0011$	$6.83\pm 1.16$	$0.0496\pm 0.0189$
Additive polynomial	$0.0411\pm 0.0004$	$6.39\pm 0.64$	$0.0864\pm 0.0321$
Random MLP	$0.0535\pm 0.0010$	$7.67\pm 0.68$	$0.0808\pm 0.0227$

Table 1: Robustness across simulated-user families after 1,000 queries (mean

\pm

95% CI;

n=30

Figure 3: Final reward gap to the optimum after 1,000 preference queries under (A) optimization over the fixed global policy space and (B) reoptimization across 210 balanced feasible sets (

\binom{8}{4}

constrained-parameter subsets

\times

three complementary low/mid/high assignments).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Model-behavior diagnostics for simulated users. (A) Policy-to-outcome sweeps for four independently sampled simulated users, showing smooth and diverse outcome responses to control variation. (B) Outcome-to-reward sweeps for the same users, showing smooth nonlinear preference structure over human outcomes. (C) Nonstationary outcome drift for motor adaptation, where the current policy-to-outcome map exponentially transitions from an initial response map to a steady-state response map with time constant

\approx 67

queries, reaching 95% transition by query 200.

Quantity	Policy-to-outcome map $f_{\star}$	Outcome-to-reward map $r_{\star}$
Input	Control policy $\tau\in[0,1]^{d_{\tau}}$	Outcome $\phi\in[0,1]^{d_{\phi}}$
Output	One scalar head per outcome dimension	One scalar reward head
RFF features	$M_{f}=1024$	$M_{r}=512$
Length scale	$\ell_{f}\sim\mathrm{LogUniform}(0.35,0.65)$	$\ell_{r}\sim\mathrm{LogUniform}(0.50,0.80)$
Frequencies	$\omega_{m}\sim\mathcal{N}(0,I)$	$\omega_{m}\sim\mathcal{N}(0,I)$
Phases	$b_{m}\sim\mathrm{Unif}(0,2\pi)$	$b_{m}\sim\mathrm{Unif}(0,2\pi)$

Appendix

Table 2: RFF hyperparameters used to sample simulated users. These settings define the simulated-user response functions, not the learned inverse reward model.

Truth family	Median chord range	Fraction of chords with an interior peak	Affine-residual nonlinearity
RBF–RFF	0.203	0.486	0.826
Matérn- $5/2$	0.192	0.407	0.582
Additive polynomial	0.179	0.424	0.902
Random MLP	0.210	0.403	0.408

Appendix

Table 3: Ground-truth complexity diagnostics across simulated-user families. Values are averaged over 30 users evaluated with identical random-policy probes.

Noise source	Std.	Scale and role
Execution	$\sigma_{\tau}=0.02$	2% FS implementation jitter in the commanded policy
Outcome measurement	$\sigma_{\phi}=0.05$	5% FS sensing noise in measured outcomes
Reward evaluation	$\sigma_{\tilde{r}}=0.02$	ambiguity for small reward-gap comparisons, with reversal rates determined by $\Delta r$

Appendix

Table 4: Default stochastic-noise settings. Standard deviations are in normalized units; FS denotes the full scale of a normalized coordinate.

Figure 5: Estimation of the 10% preference-error query threshold. Each column shows a representative run for one querying rule. (A) When the observed held-out error crosses 10% within the finite query budget, the threshold is estimated by linear interpolation between adjacent checkpoints; no curve fit is used. (B) Only for runs that do not cross within

q_{\max}=1000

queries, we fit a logarithmic learning curve to the observed checkpoints and solve for the crossing. Points denote observed errors, dark curves denote the fitted log trend, and light curve segments denote the extrapolated portion. The dashed horizontal line marks 10% error, and square markers indicate the estimated threshold

\hat{q}_{10\%}

. The fitted examples show that the logarithmic trend captures the observed decrease before extrapolation.

Quantity	Setting
Forward model	Exact/full RBF kernel ridge regression
Kernel	$k_{\ell_{f}}(\tau,\tau^{\prime})=\exp(-\\|\tau-\tau^{\prime}\\|_{2}^{2}/(2\ell_{f}^{2}))$
Refitting schedule	Exact KRR refit after each new policy–outcome observation
Hyperparameter reselection	Performed at evaluation checkpoints using a validation split
Validation fraction	$20\%$ of available forward observations
RBF length-scale candidates	$\ell_{f}\in\{0.3,0.4,0.5,0.6,0.7\}$

Appendix

Table 5: Learner-side forward KRR settings for the stationary forward model. Recency weighting for nonstationary motor adaptation is described separately.

Quantity	Setting
Observed temporal variable	Query index $c_{i}$ for each forward observation
Estimated progress coordinate	$\hat{\alpha}_{i}(T_{d})=1-\exp(-c_{i}/T_{d})$
Recency schedule parameters	$T_{d}$ and $h_{d}$
Weight floor	$\omega_{\min}=10^{-4}$
Learned-decay selection	Joint validation-based selection of $T_{d},h_{d},\ell_{f},\lambda_{f}$
Search method	TPE hyperparameter search [ 4 ]

Appendix

Table 6: Recency-weighted forward-learning settings for nonstationary experiments.

Quantity	Setting
Reward representation	RBF-GP approximated by RFFs
RFF features	$m=256$
Reward-kernel length scale	$\ell_{r}=0.65$
RFF frequencies	$\Omega_{kj}\sim\mathcal{N}(0,1)$
RFF phases	$b_{k}\sim\operatorname{Unif}(0,2\pi)$
Reward-weight prior	$w\sim\mathcal{N}(0,I_{m})$

Appendix

Table 7: Learner-side inverse GP hyperparameters. These settings define the inverse reward model used by SFIHILO, not the simulated-user reward function used to generate preference labels.

Stage	Computation
Input	Anchor outcome $\bar{\phi}_{t}$ , forward belief $p_{f}^{(t)}(\phi\mid\tau)$ , reward posterior samples $\{\Theta_{s}\}_{s=1}^{S}$ , and histories $\mathcal{D}_{t}=(\mathcal{D}_{f}^{(t)},\mathcal{D}_{r}^{(t)})$ .
Pool	Sample a Monte Carlo policy pool $\mathcal{P}_{t}=\{\tau^{(m)}\}_{m=1}^{M_{\mathrm{pool}}}\subset\mathcal{T}$ .
Cheap scoring	For each $\tau\in\mathcal{P}_{t}$ , compute the forward mean $\mu_{f}^{(t)}(\tau)$ , the uncertainty decomposition $v_{f}^{(t)}=v_{\mathrm{epi}}^{(t)}+v_{\mathrm{obs}}^{(t)}$ , the reducible-uncertainty fraction $\rho_{f}^{(t)}(\tau)$ , and the mean-outcome reward-side mutual information $a_{\mathrm{MI,mean}}^{(t)}(\tau)$ . Define $s_{\mathrm{rank}}^{(t)}(\tau)=a_{\mathrm{MI,mean}}^{(t)}(\tau)\rho_{f}^{(t)}(\tau).$
Target set	Select the target boundary set $\mathcal{Z}_{t}=\operatorname{TopK}_{K_{\mathrm{tar}}}\{s_{\mathrm{rank}}^{(t)}(\tau):\tau\in\mathcal{P}_{t}\}.$ This finite set approximates the current anchor-relative preference-boundary region.
Shortlist	Construct a deduplicated candidate shortlist $\mathcal{S}_{t}=\mathcal{S}_{\mathrm{main}}\cup\mathcal{S}_{\mathrm{var}}\cup\mathcal{S}_{\mathrm{rand}},$ where $\mathcal{S}_{\mathrm{main}}$ contains top-ranked policies under $s_{\mathrm{rank}}^{(t)}$ , $\mathcal{S}_{\mathrm{var}}$ contains policies with high reducible forward-variance score $s_{\mathrm{var}}^{(t)}$ , and $\mathcal{S}_{\mathrm{rand}}$ is a random guard set.
Current utility	Estimate $\mathcal{U}_{t}(\mathcal{Z}_{t})$ using reward posterior samples and $M_{\mathrm{tar}}$ forward samples $\Phi_{\zeta}^{(m)}\sim p_{f}^{(t)}(\cdot\mid\zeta)$ for each $\zeta\in\mathcal{Z}_{t}$ .

Appendix

Table 8: Implemented Boundary Lookahead querying at round

t

Quantity	Setting
Candidate pool size	$M_{\mathrm{pool}}=\max(4000,400d_{\tau})$
Target set size	$K_{\mathrm{tar}}=32$
Target forward samples	$M_{\mathrm{tar}}=3$
Candidate outcome approximation	$\tilde{\phi}_{\tau}=\mu_{f}^{(t)}(\tau)$
Candidate-side forward integral	Collapsed to one mean observation
Target-set forward uncertainty	Retained through forward sampling and one-step variance collapse

Appendix

Table 9: Boundary Lookahead implementation settings.

Human-in-the-Loop AI

Human-in-the-loop AI combines human feedback, oversight, and decision-making with machine learning and autonomous systems.

See all 288 papers

PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning
Sep 25, 2026Hang Yu, James Staley, Cheng Xi Tsou +8Reinforcement Learning Robotic RL
Multi-Objective Human-in-the-Loop Bayesian Optimization of a Lower-Limb Exoskeleton
Sep 25, 2026Neil Janwani, Matthew T. Lerner, Aaron J. Young +1Human-in-the-Loop AI Bayesian Optimization
Georgia Institute of Technology
Towards Interaction Regulation from Human Feedback via Free Energy Minimization
Sep 16, 2026Maria Paula Diaz Monfort, Cinzia Tomaselli, Michael Richardson +1Human-Robot Interaction
Scuola Superiore Meridionale, Napoli, Italy · Macquarie University, Sydney, Australia · University of Salerno, Salerno, Italy

Sep 25, 2026J/K move · Enter open · S save

PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning

Hang Yu, James Staley, Cheng Xi Tsou, Xiujin Liu+7

Reinforcement Learning Robotic RL Data-Efficient Robot Learning Reward Design for RL

Open paper AskPDF arXiv

Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.

Abstract

Human-in-the-Loop AI

Explore similar work

PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning

Multi-Objective Human-in-the-Loop Bayesian Optimization of a Lower-Limb Exoskeleton

Towards Interaction Regulation from Human Feedback via Free Energy Minimization