Learning When to Act: Interval-Aware Reinforcement Learning with Predictive Temporal Structure

Davide Di Gioia

cs.LGMar 23, 2026

Learning When to Act: Interval-Aware Reinforcement Learning with Predictive Temporal Structure

Authors: Davide Di Gioia

Organizations: University College London

Abstract

Autonomous agents operating in continuous environments must decide not only what to do, but when to act. We introduce a lightweight adaptive temporal control system that learns the optimal interval between cognitive ticks from experience, replacing ad hoc biologically inspired timers with a principled learned policy. The policy state is augmented with a predictive hyperbolic spread signal (a "curvature signal" shorthand) derived from hyperbolic geometry: the mean pairwise Poincare distance among n sampled futures embedded in the Poincare ball. High spread indicates a branching, uncertain future and drives the agent to act sooner; low spread signals predictability and permits longer rest intervals. We further propose an interval-aware reward that explicitly penalises inefficiency relative to the chosen wait time, correcting a systematic credit-assignment failure of naive outcome-based rewards in timing problems. We additionally introduce a joint spatio-temporal embedding (ATCPG-ST) that concatenates independently normalised state and position projections in the Poincare ball; spatial trajectory divergence provides an independent timing signal unavailable to the state-only variant (ATCPG-SO). This extension raises mean hyperbolic spread (kappa) from 1.88 to 3.37 and yields a further 5.8 percent efficiency gain over the state-only baseline. Ablation experiments across five random seeds demonstrate that (i) learning is the dominant efficiency factor (54.8 percent over no-learning), (ii) hyperbolic spread provides significant complementary gain (26.2 percent over geometry-free control), (iii) the combined system achieves 22.8 percent efficiency over the fixed-interval baseline, and (iv) adding spatial position information to the spread embedding yields an additional 5.8 percent.

Figures & tables

Panel of Figure 1 — Figure 1: A general adaptive cognitive pacing loop. While instantiated here in reinforcement learning, the same structure applies to agents that dynamically allocate computation, planning, or tool use.

Regime	Construction	$r$	$\kappa$	$\kappa/\kappa_{\text{confident}}$
Conflicted uncertainty	Opposing dominant features at boundary	$\approx 0.90$	$6.22$	$10.4\times$
Confident prediction	Stable signal, ball interior	$\approx 0.37$	$0.60$	$1.0\times$
Uninformative noise	Near-zero state, Euclidean limit	$\approx 0.12$	$0.21$	$0.35\times$

Table 1: MC-dropout predictive curvature

\kappa

across three geometric regimes (

N=200

samples,

p=0.20

d=6

). The Poincaré boundary amplification selectively elevates conflicted states.

Variant	Efficiency $\eta$	$\Delta$ vs Full	Info access

Full model (ATCPG, blind)	$0.0290$	—	none

$-$ Learning (fixed interval)	$0.0131$	$-54.8\%$	none

$-$ Spread	$0.0214$	$-26.2\%$	none

Table 2: Efficiency scores

\eta

( 18 ) for the full ATCPG model, ablated variants (averaged over 3 seeds), the fixed-interval baseline (averaged over 5 seeds), and TemporalController as a privileged upper bound.

\Delta

is the relative change versus the full model.

Metric	TC (privileged)	ATCPG (blind)
Efficiency (primary)	$0.0275$	$0.0474$ ( +72.5% )
Avg interval (s)	$28.8$	$16.7$
Performance score	$0.792$	$0.792$
$\kappa$ overload	$2.03$	$4.06$
$\kappa$ normal	$0.10$	$1.20$
$\kappa$ discriminability	$20\times$	$3.4\times$

Table 3: Head-to-head comparison on a shared 500-tick environment. TC directly observes the overload flag

o_{t}

(privileged, same run as the TC † row in Table 2 ); ATCPG infers load geometrically from future embeddings (blind). The

+72.5\%

efficiency advantage for ATCPG is a lower bound: TC’s information advantage is eliminated in partial-observability settings.

† Lower ratio reflects positions raising baseline $\kappa$ in normal ticks;
Variant	Efficiency $\eta$	Mean $\kappa$	$\kappa$ disc. (ol/nol)
ATCPG-SO (state only)	$0.0336$	$1.88$	$4.65\times$
ATCPG-ST (+ positions)	$0.0355$	$3.37$	$1.98\times$
Gain (ST vs. SO)	$+5.8\%$	$+1.79\times$	—
the absolute overload $\kappa$ is strictly higher for ATCPG-ST.

Table 4: Spatial ablation: ATCPG state-only vs. joint spatio-temporal embedding on a shared 500-tick trajectory. Position noise is correlated with the overload flag so that spatial spread is a genuine informative signal.

Strategy	Success	Total tokens	$\eta$	vs. Fixed
Fixed	$1.00$	$6{,}924$	$0.00217$	—
Reactive	$0.27$	$2{,}564$	$0.00156$	$-28.1\%$
ATCPG	$1.00$	$6{,}553$	$0.00229$	$+5.7\%$
ATCPG-ST	$1.00$	$6{,}172$	$0.00243$	$+12.2\%$

Table 5: Real-LLM pacing benchmark. GPT-4.1 (OpenAI, gpt-4.1 ). 15 episodes (

5\times 3

seeds), 5 LLM calls/episode, overload

=30\%

Paper Concept (RL)	Agent / LLM Analog
Action interval	Reasoning or planning frequency
Acting sooner	Replanning / tool invocation
Acting later	Waiting / batching computation
Hyperbolic spread	Branching of imagined futures
Interval-aware reward	Compute–quality trade-off

Table 6: Cross-domain conceptual mapping for Adaptive Cognitive Pacing.

Continuous-Time RL

Continuous-Time Reinforcement Learning studies how agents learn policies for systems that evolve continuously or through asynchronous jumps.

See all 33 papers

May 11, 2026

Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance

Safe reinforcement learning (RL) typically asks

\textit{what}

an agent should do. We ask

\textit{when}

it needs to act, and show that a single policy can jointly learn control inputs and communication-efficient timing decisions under a pointwise Lyapunov safety shield. We focus on stabilization around a known equilibrium, where CARE-based LQR backups, Lyapunov certificates, and classical Lyapunov-STC are well defined, enabling clean comparison against analytical baselines. A run-time assurance (RTA) layer overrides the policy via a one-step-ahead Lyapunov prediction and a precomputed LQR backup, providing a strictly stronger guarantee than constrained MDP methods that enforce safety only in expectation. On an inverted pendulum, cart--pole, and planar quadrotor, the learned policy achieves

1.91\times

1.45\times

, and

3.51\times

higher mean inter-sample interval (MSI) than a Lyapunov-triggered baseline; a fixed LQR controller at the same average rate is unstable on all three plants, showing that adaptive timing, not a lower average rate, makes sparsity safe. A CARE-derived Lyapunov reward transfers across environments without redesign, with a single weight

w_c

controlling the stability--communication tradeoff; ablations confirm the RTA shield is essential, with its removal reducing MSI by

1.27

1.84\times

and degrading state norms. A preference-conditioned extension recovers the full tradeoff frontier from one model at

\tfrac{2}{11}

of training compute, and SAC experiments show the results are algorithm-agnostic across discrete and continuous domains. A 12-state 3D quadrotor case study extends the framework to higher-dimensional systems where classical STC is intractable, and robustness to

\pm30\%

mass variation and disturbances shows graceful degradation, with the RTA absorbing what the learned policy cannot.

Adam Haroon, Erick J. Rodríguez-Seda, Cody Fleming +1

Department of Mechanical Engineering, Iowa State University, Ames, IA, USA · Navy Center for Applied Research in Artificial Intelligence (NCARAI), U.S. Naval Research Laboratory, Washington, D.C., USA · Department of Weapons, Robotics, and Control Engineering, United States Naval Academy, Annapolis, MD, USA

Policy Learning Lyapunov-Based Control RL Control Robot Safety Feedback Control Runtime Assurance+2

Ask PDF

Aug 17, 2026

Chronocooked: A Benchmark for Interval Timing in Reinforcement Learning Agents

Interval timing is extensively studied as an important aspect of human behaviour. As artificial agents are increasingly designed to function alongside humans, their interval timing abilities also needs to be studied. However, research in this area remains limited and scattered. This paper presents Chronocooked, a reinforcement learning (RL) benchmark environment that enables a systematic study of interval timing abilities in RL agents. Inspired by Overcooked, the suite comprises cooking scenarios involving interval timing tasks drawn from the psychology literature. The tasks and reward functions are designed such that temporal information is unobserved but critical for optimal performance. The environment is intentionally kept simple to enable controlled experiments and support biologically plausible models. Each task is accompanied by evaluation metrics to study different aspects of interval timing in RL agents, namely, task performance, human-like timing and scalability. We report baselines using a non-recurrent architecture (CNN), a recurrent architecture (LSTM), and a biologically inspired recurrent architecture (CTRNN). The baseline model analysis shows that, although RL agents can successfully perform time-dependent tasks, they do not necessarily process and perceive time in the same way as humans. Understanding these differences is important for anticipating their impact on human-robot interactions (HRI).

Amrapali Pednekar, Alvaro Garrido-Perez, Yara Khaluf +1

Department of Information Technology IDLab, Ghent University - imec · Computer Science Department Vrije Universiteit Amsterdam

RL Benchmarks

Ask PDF

Jul 17, 2026

When to Plan: Learning to Select Between Reactive Control and Deliberative Planning

It has long been recognized that humans have the ability to switch between fast, reactive decision-making and slower, deliberative planning. In this paper, we study the question of how to learn this ability, known as meta-reasoning, in artificial agents. We model reactive decision-making as a policy that directly maps state observations to actions. Such policies can be trained with reinforcement learning (RL) or imitation learning, but may generalize poorly outside of their training distribution. Alternatively, model-based decision-time planning is more likely to produce good actions across a broader set of states but requires additional computation time, which delays acting. In this work, we introduce an RL method for training a meta-reasoning policy that allocates computation by conditioning on a reactive-policy uncertainty score. This score enables it to predict when the reactive policy is likely to perform poorly and when planning is needed. We conduct an empirical study on motion planning and navigation environments, showing that this design enables the meta-reasoning policy to learn when the reactive policy provides a good-enough action versus when decision-time planning is needed. Additionally, we show that our design enables the meta-agent to shift toward fully reactive control as the reactive policy improves.

Adam Labiosa, Josiah P. Hanna

Department of Computer Sciences, University of Wisconsin–Madison, USA

Robot Policy Learning Robotic RL Robotics Policy Learning

Ask PDF

Abstract

Continuous-Time RL

Explore similar work

Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance

Chronocooked: A Benchmark for Interval Timing in Reinforcement Learning Agents

When to Plan: Learning to Select Between Reactive Control and Deliberative Planning