Fully online embodied learning requires synaptic adaptation to acquire new behaviors while preserving previously learned dynamics during ongoing interaction. We extend the Predictive-Coding-inspired Variational Recurrent Neural Network (PV-RNN) to continuously adapt its synaptic weights and propose Free-Energy-Gated Plasticity (FEGP), which regulates the effective learning rate according to variational free energy. In real-time physical human-robot interaction, a randomly initialized network acquired three cyclic motor patterns without offline pretraining, replay, or task-boundary signals, with all three patterns emerging in autonomous rollouts. Controlled experiments over ten randomized teaching streams and five network initializations per stream showed that FEGP substantially improved repertoire coverage and retention of previously acquired patterns after they left the recent observation window. Neither a constant learning rate matched to the gate's time-averaged effective rate nor replay of the same gain values with disrupted temporal organization reproduced these improvements. These results indicate that the temporal allocation of plasticity relative to model-environment mismatch, rather than simply its average magnitude or distribution, is critical for maintaining previously acquired behaviors during continued online learning.
Figures & tables
Figure 1: Overview of the fully online learning loop and the offline evaluation pipeline. Left: each model update incorporates the newest sensory observation into the sliding past window, performs posterior-state inference through error regression, adapts the synaptic weights through gated BPTT, and generates a future trajectory for motor control and subsequent analysis. Right: saved trajectories are segmented and classified post hoc using the evaluation pipeline described in Sec. 3.3 . The evaluation pipeline does not feed information back into the online learning loop. The numerical settings used in the experiments are listed in Table 1 .
Figure 2: Schematic illustration of Free-Energy–Gated Plasticity. The plasticity gain g(st) is a sigmoid function of the gate signal st , defined as the logarithm of the window-averaged variational free energy (Eq. 6 ). In the single-threshold formulation, one threshold λ determines the transition between lower and higher plasticity. In the hysteretic formulation, the active threshold is selected from λlow and λhigh according to the current plasticity regime. The single-threshold gate is recovered when λlow=λhigh=λ .
Component
Parameter
Value
Network
Sensory layer
softmax units, σ
10, 0.05
PV-RNN L1
d , z , τ
40, 4, 3
PV-RNN L2
d , z , τ
20, 2, 9
Both PV-RNN layers
meta-prior w
0.01
Online inference and learning
Table 1: Shared model, inference, and optimization settings.
Condition
Setting
FEGP
(λlow,λhigh)=(−9,−7) , T=0.1
Constant-plasticity baseline
g(st)=1
Single-threshold sweep
λ∈{−10.0,−9.5,…,−6.0} , T=0.1
Hysteretic-gate sweep
λlow∈{−10,−9,−8} ,
λhigh∈{−8,−7,−6}
Constant-rate control
αconst=αg(st) under FEGP
Table 2: Conditions in Experiment 1.
Figure 3: Effect of hysteresis on the temporal dynamics of the plasticity gate. (a) Representative gain trajectories under the single-threshold setting λ=−8 and the hysteretic setting (λlow,λhigh)=(−9,−7) . (b) Number of threshold crossings or regime transitions during the interaction. (c) Fraction of model updates with intermediate gain, defined by 0.1<g(st)<0.9 . The hysteretic formulation strongly reduces rapid switching and yields more clearly separated low- and high-plasticity regimes.
Condition
Cov. (%)
Ret. (%)
Δ vs. FEGP (pp)
FEGP
45.9
23.1
—
Constant
18.9
0.0
−27.0/−23.1
Matched rate
12.5
0.2
−33.4/−22.9
Permuted gain
21.3
0.5
−24.6/−22.6
Reversed gain
11.4
0.8
−34.5/−22.3
Shifted gain
17.1
2.2
−28.8/−20.9
Table 3: Repertoire coverage and retention under FEGP and its control conditions. Differences are relative to FEGP and are reported as coverage / retention in percentage points.
Figure 4: Retention of movement patterns as a function of the time elapsed since they left the 500-update observation window. The value at 0 corresponds to patterns still present within the observation window. Under FEGP, availability drops after a pattern leaves the window but remains substantially above the two constant-rate controls over longer absence intervals. In contrast, retention under constant plasticity and the matched-rate control falls to near zero shortly after the pattern leaves the window.
Figure 5: Development of the generated movement repertoire during real-time physical interaction in Experiment 2. Each row shows one independent session. Left: temporal evolution of the number of identified movement types and the unidentified-segment ratio in the open-loop rollouts generated during learning. Right: autonomous trajectories generated using parameters saved at model updates t=1,200 , 2,000 , and 6,000 . The trajectories were segmented and classified using the evaluation pipeline; segments assigned to patterns A, B, and C are color-coded, while unclassifiable segments are labeled “?”.
Setting
Precision
Recall
F1
Matching tolerance, at threshold 0.03
±3 model updates
1.000
1.000
1.000
±5 / ±10 / ±15
1.000
1.000
1.000
Detection threshold, at tolerance ±5
0.03 (operating point)
1.000
1.000
1.000
0.10
1.000
1.000
1.000
Table S1: Boundary detection against ground truth (four streams, 459 boundaries).
Stream
Boundary-step / within-cycle median
F1 ( ±5 )
Class accuracy
Unmodified
1.94
1.000
1.000
Crossfade (width 9)
1.21
1.000
1.000
Smoothed (width 5)
0.44
1.000
1.000
Time-warped ±10%
1.96
1.000
1.000
Warped + crossfade
1.25
1.000
1.000
Table S2: Boundary detection after removing the concatenation artifact.
Trajectory set
Summed score, thr. 35
Rule used
Shape distance (median)
Recorded cycles (30)
0.00
0.00
0.36
Time-reversed
0.00
0.02
0.64
Block-permuted
0.00
0.98
2.35
Two patterns spliced
0.00
1.00
3.43
Half cycle
0.80
1.00
2.62
Two cycles concatenated
0.35
1.00
2.59
Table S3: Rejection performance on recorded and control trajectories.
Boundary detection
Detector
Tol.
Prec.
Rec.
F1
All 30 cycles
±3
1.000
1.000
1.000
All 30 cycles
±5
1.000
1.000
1.000
Demos 1–5
±3
0.967
0.967
0.967
Demos 1–5
±5
1.000
1.000
1.000
Cycle classification
Table S4: Evaluation on a stream assembled only from held-out demonstrations.
Figure S1: Late-stage summary of the single-threshold sweep in Experiment 1. Left: weight-update magnitude averaged over the final 2,000 model updates as a function of λ for four representative teaching streams (Seq1–4). Right: corresponding late-stage log-scaled free energy for the same streams. Increasing λ progressively suppresses weight adaptation while residual free energy increases.
Figure S2: Composition of the open-loop rollout over online learning under the single-threshold plasticity gate in Experiment 1. Rows correspond to threshold values λ∈{−10.0,…,−6.0} , and columns show the same four representative teaching streams (Seq1–4) used throughout this section. At each online-learning update, the model’s 3,000-step rollout was segmented and classified as movement pattern A, B, C, or Unknown using the evaluation pipeline in Sec. III-C of the main text. The displayed values use a 10-update moving average. For λ≤−8 , the rollout develops and maintains all three instructed patterns, whereas plasticity is strongly suppressed for λ≥−6.5 .
Figure S3: Composition of the open-loop rollout under Synaptic Intelligence with constant plasticity in Experiment 1. Rows correspond to Synaptic Intelligence coefficients c∈{0,10−6,10−5,10−4,10−3,10−2,10−1,1.0} , and columns show the same four representative teaching streams (Seq1–4) used in Figs. S1 and S2 . Colors are as in Fig. S2 . Weak regularization produces outcomes similar to the unregularized baseline, whereas strong regularization prevents stable acquisition.
Predictive coding characterizes adaptive behavior as a dynamic balance between internally generated predictions and external sensory evidence, yet how an embodied cognitive system can regulate this balance online during ongoing interaction remains poorly understood. This study proposes a predictive-coding-based mechanism for regulating internally generated and externally coupled processing during physical human--robot interaction. The framework employs a predictive-coding-inspired variational recurrent neural network (PV-RNN), in which a meta-prior controls the degree to which posterior inference is constrained by learned prior dynamics. We extend this architecture with an online mechanism that uses reconstruction error accumulated over recent interaction history to select between predefined meta-prior regimes. The mechanism was evaluated across three physical human--robot interaction tasks involving fixed structured, changing structured, and less-constrained interaction. Across all tasks, lower meta-prior values produced the expected increase in posterior--prior divergence and reduction in reconstruction error. More importantly, reconstruction-history-driven regime selection was also associated with reduced prospective prediction error and robot-side physical interaction conflict, demonstrating consequences beyond the retrospective reconstruction objective itself. Task~3 further showed that recent sensory observations can be successfully accommodated while subsequent human motion still departs from the model's prior-generated future trajectory. Overall, these findings show that accumulated reconstruction mismatch can provide an endogenous signal for regulating how strongly subsequent inference relies on learned internal dynamics relative to ongoing sensory input during embodied interaction.
Henrique Oyama, Hiroki Sawada, Jun Tani
Cognitive Neurorobotics Research Unit Okinawa Institute of Science and Technology Graduate University Okinawa
Biological neural networks continuously adapt and modify themselves in response to experiences throughout their lifetime - a capability largely absent in artificial neural networks. Hebbian plasticity offers a promising path toward rapid adaptation in changing environments. Here, we introduce Hebbian Attractor Networks (HAN), a class of plastic neural networks in which local weight update normalization induces emergent attractor dynamics. Unlike prior approaches, HANs employ dual-timescale plasticity and temporal averaging of pre- and postsynaptic activations to induce either co-dynamic limit cycles or fixed-point weight attractors. Using simulated locomotion benchmarks, we gain insight into how Hebbian update frequency and activation averaging influence weight dynamics and control performance. Our results show that slower updates, combined with averaged pre- and postsynaptic activations, promote convergence to stable weight configurations, while faster updates yield oscillatory co-dynamic systems. We further demonstrate that these findings generalize to high-dimensional quadrupedal locomotion with a simulated Unitree Go1 robot. These results highlight how the timing of plasticity shapes neural dynamics in embodied systems, providing a principled characterization of the attractor regimes that emerge in self-modifying networks.
Alexander Dittrich, Fuda van Diggelen, Dario Floreano
Laboratory of Intelligent Systems, EPFL, Switzerland
Human-in-the-loop Reinforcement Learning has become a popular approach to training, finetuning, and aligning robot behavior with user preferences. Our paper explores the feasibility of using brain signals via functional near-infrared spectroscopy (fNIRS) to modulate robot learning in simulation. We compare agents trained on passive (observational) versus active (demonstrative) interaction tasks, and test multiple methods for enhancing the RL algorithm with the neural signal, focusing on parameter augmentation rather than replacement. We further examine how model granularity and noise affect agent learning. Our results show that this framework is effective: the neural signal improves learning when augmenting trajectory priorities and state-action q-values. Additionally, the framework learns successfully from offline data, offering a practical alternative for settings where real-time BCI setups are impractical or only limited data is available.
Julia Santaniello, Madelaine Brower, Benson Jiang +3