Learning When to Act: Interval-Aware Reinforcement Learning with Predictive Temporal Structure

Davide Di Gioia

cs.LGMar 23, 2026

Learning When to Act: Interval-Aware Reinforcement Learning with Predictive Temporal Structure

Authors: Davide Di Gioia

Organizations: University College London

Abstract

Autonomous agents operating in continuous environments must decide not only what to do, but when to act. We introduce a lightweight adaptive temporal control system that learns the optimal interval between cognitive ticks from experience, replacing ad hoc biologically inspired timers with a principled learned policy. The policy state is augmented with a predictive hyperbolic spread signal (a "curvature signal" shorthand) derived from hyperbolic geometry: the mean pairwise Poincare distance among n sampled futures embedded in the Poincare ball. High spread indicates a branching, uncertain future and drives the agent to act sooner; low spread signals predictability and permits longer rest intervals. We further propose an interval-aware reward that explicitly penalises inefficiency relative to the chosen wait time, correcting a systematic credit-assignment failure of naive outcome-based rewards in timing problems. We additionally introduce a joint spatio-temporal embedding (ATCPG-ST) that concatenates independently normalised state and position projections in the Poincare ball; spatial trajectory divergence provides an independent timing signal unavailable to the state-only variant (ATCPG-SO). This extension raises mean hyperbolic spread (kappa) from 1.88 to 3.37 and yields a further 5.8 percent efficiency gain over the state-only baseline. Ablation experiments across five random seeds demonstrate that (i) learning is the dominant efficiency factor (54.8 percent over no-learning), (ii) hyperbolic spread provides significant complementary gain (26.2 percent over geometry-free control), (iii) the combined system achieves 22.8 percent efficiency over the fixed-interval baseline, and (iv) adding spatial position information to the spread embedding yields an additional 5.8 percent.

Figures & tables

Panel of Figure 1 — Figure 1: A general adaptive cognitive pacing loop. While instantiated here in reinforcement learning, the same structure applies to agents that dynamically allocate computation, planning, or tool use.

Regime	Construction	$r$	$\kappa$	$\kappa/\kappa_{\text{confident}}$
Conflicted uncertainty	Opposing dominant features at boundary	$\approx 0.90$	$6.22$	$10.4\times$
Confident prediction	Stable signal, ball interior	$\approx 0.37$	$0.60$	$1.0\times$
Uninformative noise	Near-zero state, Euclidean limit	$\approx 0.12$	$0.21$	$0.35\times$

Table 1: MC-dropout predictive curvature

\kappa

across three geometric regimes (

N=200

samples,

p=0.20

d=6

). The Poincaré boundary amplification selectively elevates conflicted states.

Variant	Efficiency $\eta$	$\Delta$ vs Full	Info access

Full model (ATCPG, blind)	$0.0290$	—	none

$-$ Learning (fixed interval)	$0.0131$	$-54.8\%$	none

$-$ Spread	$0.0214$	$-26.2\%$	none

Table 2: Efficiency scores

\eta

( 18 ) for the full ATCPG model, ablated variants (averaged over 3 seeds), the fixed-interval baseline (averaged over 5 seeds), and TemporalController as a privileged upper bound.

\Delta

is the relative change versus the full model.

Metric	TC (privileged)	ATCPG (blind)
Efficiency (primary)	$0.0275$	$0.0474$ ( +72.5% )
Avg interval (s)	$28.8$	$16.7$
Performance score	$0.792$	$0.792$
$\kappa$ overload	$2.03$	$4.06$
$\kappa$ normal	$0.10$	$1.20$
$\kappa$ discriminability	$20\times$	$3.4\times$

Table 3: Head-to-head comparison on a shared 500-tick environment. TC directly observes the overload flag

o_{t}

(privileged, same run as the TC † row in Table 2 ); ATCPG infers load geometrically from future embeddings (blind). The

+72.5\%

efficiency advantage for ATCPG is a lower bound: TC’s information advantage is eliminated in partial-observability settings.

† Lower ratio reflects positions raising baseline $\kappa$ in normal ticks;
Variant	Efficiency $\eta$	Mean $\kappa$	$\kappa$ disc. (ol/nol)
ATCPG-SO (state only)	$0.0336$	$1.88$	$4.65\times$
ATCPG-ST (+ positions)	$0.0355$	$3.37$	$1.98\times$
Gain (ST vs. SO)	$+5.8\%$	$+1.79\times$	—
the absolute overload $\kappa$ is strictly higher for ATCPG-ST.

Table 4: Spatial ablation: ATCPG state-only vs. joint spatio-temporal embedding on a shared 500-tick trajectory. Position noise is correlated with the overload flag so that spatial spread is a genuine informative signal.

Strategy	Success	Total tokens	$\eta$	vs. Fixed
Fixed	$1.00$	$6{,}924$	$0.00217$	—
Reactive	$0.27$	$2{,}564$	$0.00156$	$-28.1\%$
ATCPG	$1.00$	$6{,}553$	$0.00229$	$+5.7\%$
ATCPG-ST	$1.00$	$6{,}172$	$0.00243$	$+12.2\%$

Table 5: Real-LLM pacing benchmark. GPT-4.1 (OpenAI, gpt-4.1 ). 15 episodes (

5\times 3

seeds), 5 LLM calls/episode, overload

=30\%

Paper Concept (RL)	Agent / LLM Analog
Action interval	Reasoning or planning frequency
Acting sooner	Replanning / tool invocation
Acting later	Waiting / batching computation
Hyperbolic spread	Branching of imagined futures
Interval-aware reward	Compute–quality trade-off

Table 6: Cross-domain conceptual mapping for Adaptive Cognitive Pacing.

Continuous-Time RL

Continuous-Time Reinforcement Learning studies how agents learn policies for systems that evolve continuously or through asynchronous jumps.

See all 33 papers

Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance
May 11, 2026Adam Haroon, Erick J. Rodríguez-Seda, Cody Fleming +1Policy Learning Lyapunov-Based Control
Department of Mechanical Engineering, Iowa State University, Ames, IA, USA · Navy Center for Applied Research in Artificial Intelligence (NCARAI), U.S. Naval Research Laboratory, Washington, D.C., USA · Department of Weapons, Robotics, and Control Engineering, United States Naval Academy, Annapolis, MD, USA
Chronocooked: A Benchmark for Interval Timing in Reinforcement Learning Agents
Aug 17, 2026Amrapali Pednekar, Alvaro Garrido-Perez, Yara Khaluf +1RL Benchmarks
Department of Information Technology IDLab, Ghent University - imec · Computer Science Department Vrije Universiteit Amsterdam
When to Plan: Learning to Select Between Reactive Control and Deliberative Planning
Jul 17, 2026Adam Labiosa, Josiah P. HannaRobot Policy Learning Robotic RL
Department of Computer Sciences, University of Wisconsin–Madison, USA

May 11, 2026J/K move · Enter open · S save

Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance

Adam Haroon, Erick J. Rodríguez-Seda, Cody Fleming, Tristan Schuler

Department of Mechanical Engineering, Iowa State University, Ames, IA, USA · Navy Center for Applied Research in Artificial Intelligence (NCARAI), U.S. Naval Research Laboratory, Washington, D.C., USA · Department of Weapons, Robotics, and Control Engineering, United States Naval Academy, Annapolis, MD, USA

Policy Learning Lyapunov-Based Control RL Control Robot Safety Feedback Control Runtime Assurance+2

Open paper AskPDF arXiv

Safe reinforcement learning (RL) typically asks

\textit{what}

an agent should do. We ask

\textit{when}

it needs to act, and show that a single policy can jointly learn control inputs and communication-efficient timing decisions under a pointwise Lyapunov safety shield. We focus on stabilization around a known equilibrium, where CARE-based LQR backups, Lyapunov certificates, and classical Lyapunov-STC are well defined, enabling clean comparison against analytical baselines. A run-time assurance (RTA) layer overrides the policy via a one-step-ahead Lyapunov prediction and a precomputed LQR backup, providing a strictly stronger guarantee than constrained MDP methods that enforce safety only in expectation. On an inverted pendulum, cart--pole, and planar quadrotor, the learned policy achieves

1.91\times

1.45\times

, and

3.51\times

higher mean inter-sample interval (MSI) than a Lyapunov-triggered baseline; a fixed LQR controller at the same average rate is unstable on all three plants, showing that adaptive timing, not a lower average rate, makes sparsity safe. A CARE-derived Lyapunov reward transfers across environments without redesign, with a single weight

w_c

controlling the stability--communication tradeoff; ablations confirm the RTA shield is essential, with its removal reducing MSI by

1.27

1.84\times

and degrading state norms. A preference-conditioned extension recovers the full tradeoff frontier from one model at

\tfrac{2}{11}

of training compute, and SAC experiments show the results are algorithm-agnostic across discrete and continuous domains. A 12-state 3D quadrotor case study extends the framework to higher-dimensional systems where classical STC is intractable, and robustness to

\pm30\%

mass variation and disturbances shows graceful degradation, with the RTA absorbing what the learned policy cannot.

Abstract

Continuous-Time RL

Explore similar work

Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance

Chronocooked: A Benchmark for Interval Timing in Reinforcement Learning Agents

When to Plan: Learning to Select Between Reactive Control and Deliberative Planning