Subjective user experience is important to human-robot interaction, but the outcomes users value, and how those preferences vary across individuals and contexts, are often unknown. While inverse learning approaches using human data can help identify user rewards, in many assistive settings the experimental costs of executing a control policy, measuring biomechanical or physiological outcomes, and collecting user feedback often limit the number of queries and optimization iterations. Here, we present Simultaneous Forward and Inverse Human-In-the-Loop Optimization (SFIHILO), which efficiently infers individual-specific reward functions from preferences over human outcomes and identifies a final control policy that maximizes the learned reward. SFIHILO bootstraps a forward model to predict user outcomes from control policies, uses this model for active querying to accelerate inverse reward learning, and then optimizes the final control policy without additional user trials. We score candidate policies by their expected reduction in uncertainty across both forward and inverse beliefs, targeting a regional preference boundary to robustly inform this simultaneous learning process. In simulation, we show that SFIHILO was effective across user heterogeneity, outcome dimensionalities, precision requirements, noise levels, and nonstationarity; compared with mutual information approaches, the proposed active querying strategy significantly improved sample efficiency in inverse learning while preserving forward model accuracy. This approach demonstrates the potential to infer latent human goals, enabling more transferable and effective human-robot interaction.
Figures & tables
Figure 1: SFIHILO framework overview. (a) A real or simulated human compares a newly executed robot policy with the previous one; in simulation, this process is represented by a ground-truth outcome model and a true preference model. (b) The forward model learns a mapping from control policies to human outcomes. (c) The inverse model infers an outcome-space reward posterior from preference choices. (d) Active querying leverages both models to select informative policy comparisons. (e–f) After querying, the learned reward is used for final policy optimization.
Figure 2: Simulation evaluation of SFIHILO across precision requirements, outcome dimensionalities, noise levels, nonstationarity, and feature misspecification. Unless varied along an axis, panels use the default settings: outcome dimensionality d=8 , practical noise levels, and reward-gap-balanced test sets; the feature-misspecification panels (H and I) use d=10 . Boxplots summarize independently sampled users. Annotations report the relative change from Random querying within each condition; asterisks denote Holm-adjusted paired t-tests [ 18 ] ( ∗ : p<0.05 , ∗∗ : p<0.01 , ∗∗∗ : p<0.001 ).
Simulated-user family
Forward RMSE ↓
Preference error (%) ↓
Final reward gap to optimum ↓
RBF (matched)
0.0635±0.0024
6.88±0.72
0.0630±0.0224
Matérn- 5/2
0.0473±0.0011
6.83±1.16
0.0496±0.0189
Additive polynomial
0.0411±0.0004
6.39±0.64
0.0864±0.0321
Random MLP
0.0535±0.0010
7.67±0.68
0.0808±0.0227
Table 1: Robustness across simulated-user families after 1,000 queries (mean ± 95% CI; n=30 ).
Figure 3: Final reward gap to the optimum after 1,000 preference queries under (A) optimization over the fixed global policy space and (B) reoptimization across 210 balanced feasible sets ( (48) constrained-parameter subsets × three complementary low/mid/high assignments).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Model-behavior diagnostics for simulated users. (A) Policy-to-outcome sweeps for four independently sampled simulated users, showing smooth and diverse outcome responses to control variation. (B) Outcome-to-reward sweeps for the same users, showing smooth nonlinear preference structure over human outcomes. (C) Nonstationary outcome drift for motor adaptation, where the current policy-to-outcome map exponentially transitions from an initial response map to a steady-state response map with time constant ≈67 queries, reaching 95% transition by query 200.
Quantity
Policy-to-outcome map f⋆
Outcome-to-reward map r⋆
Input
Control policy τ∈[0,1]dτ
Outcome ϕ∈[0,1]dϕ
Output
One scalar head per outcome dimension
One scalar reward head
RFF features
Mf=1024
Mr=512
Length scale
ℓf∼LogUniform(0.35,0.65)
ℓr∼LogUniform(0.50,0.80)
Frequencies
ωm∼N(0,I)
ωm∼N(0,I)
Phases
bm∼Unif(0,2π)
bm∼Unif(0,2π)
Appendix
Table 2: RFF hyperparameters used to sample simulated users. These settings define the simulated-user response functions, not the learned inverse reward model.
Truth family
Median chord range
Fraction of chords with an interior peak
Affine-residual nonlinearity
RBF–RFF
0.203
0.486
0.826
Matérn- 5/2
0.192
0.407
0.582
Additive polynomial
0.179
0.424
0.902
Random MLP
0.210
0.403
0.408
Appendix
Table 3: Ground-truth complexity diagnostics across simulated-user families. Values are averaged over 30 users evaluated with identical random-policy probes.
Noise source
Std.
Scale and role
Execution
στ=0.02
2% FS implementation jitter in the commanded policy
Outcome measurement
σϕ=0.05
5% FS sensing noise in measured outcomes
Reward evaluation
σr~=0.02
ambiguity for small reward-gap comparisons, with reversal rates determined by Δr
Appendix
Table 4: Default stochastic-noise settings. Standard deviations are in normalized units; FS denotes the full scale of a normalized coordinate.
Figure 5: Estimation of the 10% preference-error query threshold. Each column shows a representative run for one querying rule. (A) When the observed held-out error crosses 10% within the finite query budget, the threshold is estimated by linear interpolation between adjacent checkpoints; no curve fit is used. (B) Only for runs that do not cross within qmax=1000 queries, we fit a logarithmic learning curve to the observed checkpoints and solve for the crossing. Points denote observed errors, dark curves denote the fitted log trend, and light curve segments denote the extrapolated portion. The dashed horizontal line marks 10% error, and square markers indicate the estimated threshold q^10% . The fitted examples show that the logarithmic trend captures the observed decrease before extrapolation.
Quantity
Setting
Forward model
Exact/full RBF kernel ridge regression
Kernel
kℓf(τ,τ′)=exp(−∥τ−τ′∥22/(2ℓf2))
Refitting schedule
Exact KRR refit after each new policy–outcome observation
Hyperparameter reselection
Performed at evaluation checkpoints using a validation split
Validation fraction
20% of available forward observations
RBF length-scale candidates
ℓf∈{0.3,0.4,0.5,0.6,0.7}
Appendix
Table 5: Learner-side forward KRR settings for the stationary forward model. Recency weighting for nonstationary motor adaptation is described separately.
Quantity
Setting
Observed temporal variable
Query index ci for each forward observation
Estimated progress coordinate
α^i(Td)=1−exp(−ci/Td)
Recency schedule parameters
Td and hd
Weight floor
ωmin=10−4
Learned-decay selection
Joint validation-based selection of Td,hd,ℓf,λf
Search method
TPE hyperparameter search [ 4 ]
Appendix
Table 6: Recency-weighted forward-learning settings for nonstationary experiments.
Quantity
Setting
Reward representation
RBF-GP approximated by RFFs
RFF features
m=256
Reward-kernel length scale
ℓr=0.65
RFF frequencies
Ωkj∼N(0,1)
RFF phases
bk∼Unif(0,2π)
Reward-weight prior
w∼N(0,Im)
Appendix
Table 7: Learner-side inverse GP hyperparameters. These settings define the inverse reward model used by SFIHILO, not the simulated-user reward function used to generate preference labels.
Sample a Monte Carlo policy pool Pt={τ(m)}m=1Mpool⊂T .
Cheap scoring
For each τ∈Pt , compute the forward mean μf(t)(τ) , the uncertainty decomposition vf(t)=vepi(t)+vobs(t) , the reducible-uncertainty fraction ρf(t)(τ) , and the mean-outcome reward-side mutual information aMI,mean(t)(τ) . Define srank(t)(τ)=aMI,mean(t)(τ)ρf(t)(τ).
Target set
Select the target boundary set Zt=TopKKtar{srank(t)(τ):τ∈Pt}. This finite set approximates the current anchor-relative preference-boundary region.
Shortlist
Construct a deduplicated candidate shortlist St=Smain∪Svar∪Srand, where Smain contains top-ranked policies under srank(t) , Svar contains policies with high reducible forward-variance score svar(t) , and Srand is a random guard set.
Current utility
Estimate Ut(Zt) using reward posterior samples and Mtar forward samples Φζ(m)∼pf(t)(⋅∣ζ) for each ζ∈Zt .
Appendix
Table 8: Implemented Boundary Lookahead querying at round t .
Quantity
Setting
Candidate pool size
Mpool=max(4000,400dτ)
Target set size
Ktar=32
Target forward samples
Mtar=3
Candidate outcome approximation
ϕ~τ=μf(t)(τ)
Candidate-side forward integral
Collapsed to one mean observation
Target-set forward uncertainty
Retained through forward sampling and one-step variance collapse
Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.
Human-in-the-loop optimization (HILO) is a common approach for optimizing the control of assistive devices to account for the wearer's unique biomechanics and subjective preferences. However, despite research suggesting that a person may have a different prioritization of objectives depending on time-varying factors such as the environment, their mood, or energy levels, existing HILO approaches only consider a single objective or enforce a fixed weighting on a set of objectives. Neither approach is capable of representing an individual's preferences over objectives. In this work, we propose Multi-Objective Human-in-the-loop Bayesian Optimization (MO-HILBO), which builds on explicit multi-objective Bayesian optimization to efficiently infer a personalized set of Pareto-optimal controllers. We compare our approach with an existing multi-objective HILO method and experimentally demonstrate MO-HILBO on a lower-limb exoskeleton across two objectives: metabolic cost (efficiency) and ordinal human feedback (comfort). We find that MO-HILBO (1) discovers Pareto-optimal controllers, and (2) that the pairwise ordering of points on the Pareto front itself is consistent with validation trials. Lastly, we open-source mohilo, a Python package for running both HILO and MO-HILBO on wearable devices: https://dynamicmobility.github.io/mohilo/.
Neil Janwani, Matthew T. Lerner, Aaron J. Young +1
A central challenge across control and learning is the design of mechanisms regulating the interactions between humans and autonomous agents. Inspired by the free energy principle from computational neuroscience, we introduce a control-theoretical framework to integrate human preferences online into an agent policy. We turn the framework into an open control architecture and validate our approach using a human-in-the-loop experimental testbed involving a rover navigating via onboard sensing. The human, remotely located and equipped with virtual reality headsets, shares the same sensory information as the rover. Human preferences are provided to the rover via gestures which introduce both cooperative and competitive interactions between the agent goal and the preferences. The experiments show that interactions are regulated, validating the proposed approach.
Maria Paula Diaz Monfort, Cinzia Tomaselli, Michael Richardson +1
Scuola Superiore Meridionale, Napoli, Italy · Macquarie University, Sydney, Australia · University of Salerno, Salerno, Italy