When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Making
Authors: Jun Liu, Pu Zhao, Zhenglun Kong, Xuan Shen, Peiyan Dong, Fan Yang, Lin Cui, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, Gaowen Liu, Yanzhi Wang, Dong Huang
Organizations: Robotics Institute, Carnegie Mellon University · Northeastern University · Harvard University · Cornell University · MIT · Fujitsu Research of America · Tsinghua University · Peking University · University of Georgia · Florida International University · Cisco Systems · EmbodyX Inc
Embodied robotic systems increasingly rely on large language model (LLM)-based agents to support high-level reasoning, planning, and decision-making during interactions with the environment. However, invoking LLM reasoning introduces substantial computational latency and resource overhead, which can interrupt action execution and reduce system reliability. Excessive reasoning may delay actions, while insufficient reasoning often leads to incorrect decisions and task failures. This raises a fundamental question for embodied agents: when should the agent reason, and when should it act? In this work, we propose RARRL (Resource-Aware Reasoning via Reinforcement Learning), a hierarchical framework for resource-aware orchestration of embodied agents. Rather than learning low-level control policies, RARRL learns a high-level orchestration policy that operates at the agent's decision-making layer. This policy enables the agent to adaptively determine whether to invoke reasoning, which reasoning role to employ, and how much computational budget to allocate based on current observations, execution history, and remaining resources. Extensive experiments, including evaluations with empirical latency profiles derived from the ALFRED benchmark, show that RARRL consistently improves task success rates while reducing execution latency and enhancing robustness compared with fixed or heuristic reasoning strategies. These results demonstrate that adaptive reasoning control is essential for building reliable and efficient embodied robotic agents.
Figures & tables
Fig. 1: Overview of the proposed single-agent embodied architecture. A reinforcement learning (RL) policy operates at the decision-making layer to regulate when the agent should act directly and when to invoke expensive LLM-based reasoning modules under resource constraints. The reward signal is used during training to update the orchestration policy based on task outcome and execution latency.
Fig. 2: Decision process and training pipeline of the proposed orchestration policy. The embodied agent observes the current task state, execution history, and remaining computational budget, which are encoded and provided to a reinforcement learning–based policy. At each decision step, the policy determines whether to execute a high-level action directly or to invoke an LLM-based reasoning module. Execution feedback, including task outcome and execution latency, is used to compute rewards and update the orchestration policy during training.
Fig. 3: Illustration of a representative multi-step embodied robotic task. The agent performs navigation, inspection, and delivery stages, during which reasoning demands vary. Complex stages benefit from high-level reasoning, while routine stages favor direct execution, motivating adaptive reasoning orchestration.
Navigation
Inspection
Delivery (Pick–Place)
Method
TSR ↑
Time ↓
LLM-Time ↓
Tok ↓
TSR ↑
Time ↓
LLM-Time ↓
Tok ↓
TSR ↑
Time ↓
LLM-Time ↓
Tok ↓
Full Reasoning [ 27 ]
84.0 ± 1.8
42.3 ± 2.1
31.5 ± 1.9
4100 ± 180
78.5 ± 2.0
48.7 ± 2.4
36.8 ± 2.1
4300 ± 200
71.2 ± 2.3
55.1 ± 2.7
40.4 ± 2.4
4500 ± 220
Heuristic [ 21 ]
72.4 ± 2.1
27.6 ± 1.6
15.8 ± 1.2
1700 ± 140
66.3 ± 2.4
31.4 ± 1.8
18.1 ± 1.3
1850 ± 150
60.5 ± 2.6
36.9 ± 2.0
21.7 ± 1.5
2100 ± 170
Constrained PPO
79.3 ± 1.9
30.8 ± 1.7
18.4 ± 1.3
1950 ± 150
73.1 ± 2.2
34.6 ± 1.9
20.7 ± 1.4
2150 ± 160
65.8 ± 2.5
39.2 ± 2.1
23.9 ± 1.6
2400 ± 180
Ours (RARRL)
82.7 ± 1.7 ∗
25.1 ± 1.4 ∗∗
12.3 ± 1.0 ∗∗
980 ± 90 ∗∗
76.4 ± 1.9 ∗
29.8 ± 1.6 ∗∗
14.1 ± 1.1 ∗∗
1100 ± 100 ∗∗
69.5 ± 2.1 ∗
33.7 ± 1.8 ∗∗
16.8 ± 1.3 ∗∗
1350 ± 120 ∗∗
TABLE I: ALFRED runtime evaluation with real LLM inference. Results are reported as mean ± standard error over 50 episodes per task category. Time is measured in wall-clock seconds. TSR: Task Success Rate (%), Time: Average wall-clock time per episode (s), LLM-Time: Average LLM inference time per episode (s), Tok: Average token consumption per episode. ∗p<0.05 , ∗∗p<0.01 compared to Constrained PPO (paired t-test).
Task Decomposition
Structured Navigation
Method
TSR (%) ↑
EL ↓
RE ↑
RF ↓
Tok ↓
TSR (%) ↑
EL ↓
RE ↑
RF ↓
Tok ↓
No Reasoning
45.2 ± 2.1
12.3
0.92
0.0
0
52.1 ± 1.8
15.4
1.05
0.0
0
Full Reasoning [ 27 ]
85.4 ± 1.5
45.6
0.75
50.0
4200
88.3 ± 1.3
48.2
0.78
50.0
4100
Fixed ( k=3 ) [ 28 ]
68.7 ± 1.9
28.4
0.85
16.7
1450
72.5 ± 2.0
30.1
0.88
16.7
1380
Heuristic [ 21 ]
72.1 ± 1.6
25.7
0.89
13.2
1120
75.4 ± 1.4
27.3
0.91
13.5
1090
Ours(RARRL)
82.3 ± 1.2
18.5
0.95
7.4
620
84.6 ± 1.1
20.2
0.97
6.9
580
TABLE II: Main results across abstract embodied tasks (mean ± std over 5 seeds). EL execution latency, RE resource efficiency, RF reasoning frequency, and Tok average LLM token usage per episode. The proposed orchestration policy consistently approaches the success rate of full reasoning while significantly reducing execution latency, reasoning frequency, and token consumption, demonstrating an improved success–efficiency trade-off under limited computational budgets.
Fig. 4: Performance ceiling analysis. Each point shows mean task success over 5 random seeds with standard deviation. Execution and reasoning strength jointly determine the attainable performance ceiling, while adaptive orchestration enables closer approach to this ceiling.
Fig. 5: Robustness to latency uncertainty. Task success rate under increasing execution and reasoning latency variability. Points show mean performance over 5 random seeds, with error bars indicating standard deviation. The proposed orchestration policy degrades more gracefully than heuristic strategies as latency uncertainty increases.
Before Shock
After Shock
Method
TSR (%) ↑
RI ↓
TSR (%) ↑
RI ↓
Heuristic [ 21 ]
78.5
14.2
61.8
15.1
Ours(RARRL)
87.6
11.3
74.9
7.6
TABLE III: Robustness to budget shock. Task success rate (TSR, %) and average number of reasoning invocations per episode (RI) before and after an abrupt reduction of remaining computational budget. Results are averaged over 5 random seeds.
Variant
TSR (%) ↑
RI ↓
Cost ↓
Full (Ours)
87.6
11.3
1.00
w/o Budget State ( −bt )
80.9
15.8
1.18
w/o History ( −ht )
78.7
13.9
1.12
Planner-only (no verifier)
83.1
12.5
1.06
Verifier-only (no planner)
79.4
12.1
1.05
Fixed Budget (no c )
82.6
13.7
1.10
TABLE IV: Ablation study on key components of the orchestration policy. We report task success rate (TSR, %), average reasoning invocations per episode (RI), and normalized execution cost (Cost; lower is better). Results are averaged over 5 random seeds.
Reasoning-based robotic policies using large language and vision-language models achieve strong semantic planning capabilities but mostly suffer from a high inference latency that limits practical real-time deployment. In this work, we observe that robotic reasoning workloads contain substantial temporal redundancy, where consecutive observations frequently produce identical actions and subgoals. Based on this insight, we present REIS, a human cognition inspired robotic decision-making framework that minimizes unnecessary reasoning while preserving semantic adaptability. REIS combines lightweight scene gating, KV-steered affordance routing, and deliberative reasoning to accelerate robotic control under embodied constraints. Experiments on ALFRED, and real-world robotic tasks demonstrate that REIS significantly suppresses reasoning overhead while maintaining competitive task performance.
Joonhee Lee, Hyunseung Shin, Hyunmi Kim +2
School of Integrated Technology, College of Computing, Yonsei University · Department of Hyperscale AI SoC Research Section, ETRI · 3EECS - Electrical and Computer Engineering, University of Michigan
Reasoning-enhanced large language models have achieved remarkable improvements in planning tasks, yet their deployment in embodied systems remains impractical due to prohibitive inference delays-often exceeding minutes per planning instance. The fundamental bottleneck stems from the serial nature of existing paradigms: models must complete all reasoning before any action execution, leaving execution time windows entirely unexploited. We introduce PACE (Planning with Adaptive Cognitive Effort), a framework that enables interleaved reasoning and execution through two key innovations: an Interleaved Think-Act architecture that pipelines cognitive processing with action execution, and a Dynamic Budget Allocator that adapts reasoning token budgets to available execution time windows. On the Robotouille benchmark using Qwen3-8B-AWQ, PACE achieves a 10% success rate-representing a 67% improvement over the ReAct+Think baseline-while delivering 6.9 times acceleration in thinking time compared to unconstrained reasoning. The framework hides 66.8% of thinking time within execution windows, demonstrating that strategic cognitive effort allocation can simultaneously improve both planning quality and time efficiency. These results provide evidence that time-aware architectural innovations enable reasoning models to operate in latency-sensitive embodied domains where they were previously impractical.
In order to flexibly act in an everyday environment, a robotic agent needs a variety of cognitive capabilities that enable it to reason about plans and perform execution recovery. Large language models (LLMs) have been shown to demonstrate emergent cognitive aspects, such as reasoning and language understanding; however, the ability to control embodied robotic agents requires reliably bridging high-level language to low-level functionalities for perception and control. In this paper, we investigate the extent to which an LLM can serve as a core component for planning and execution reasoning in a cognitive robot architecture. For this purpose, we propose a cognitive architecture in which an agentic LLM serves as the core component for planning and reasoning, while components for working and episodic memories support learning from experience and adaptation. An instance of the architecture is then used to control a mobile manipulator in a simulated household environment, where environment interaction is done through a set of high-level tools for perception, reasoning, navigation, grasping, and placement, all of which are made available to the LLM-based agent. We evaluate our proposed system on two household tasks (object placement and object swapping), which evaluate the agent's reasoning, planning, and memory utilisation. The results demonstrate that the LLM-driven agent can complete structured tasks and exhibits emergent adaptation and memory-guided planning, but also reveal significant limitations, such as hallucinations about the task success and poor instruction following by refusing to acknowledge and complete sequential tasks. These findings highlight both the potential and challenges of employing LLMs as embodied cognitive controllers for autonomous robots.
Shinas Shaji, Fabian Huppertz, Alex Mitrevski +1
Institute of AI and Autonomous Systems (A2S), Hochschule Bonn-Rhein-Sieg, Sankt Augustin, Germany · Fraunhofer Institute for Intelligent Analysis and Information Systems, Sankt Augustin, Germany