Swarms of small robots are an attractive way to survey lunar caves and lava tubes. Missions such as NASA's CADRE will fly cooperating rovers for distributed mapping, but only in daylight. The 354 h lunar night drops the surface towards 95 K, and the radioisotope heaters that let single rovers survive it cannot be supplied at swarm scale. In this study we investigate whether a swarm can instead use its own members as thermal shielding. We train one policy, shared by all twenty robots, by reinforcement learning with an evolution-strategies estimator. The policy receives a single reward, once, at the end of the night: the mean temperature of the five coldest robots. Nothing in that reward refers to position, spacing, exposure or motion. Two behaviors nonetheless emerge. The swarm aggregates, and robots circulate between the sheltered core and the exposed rim, the same pair of behaviors with which emperor penguins survive the Antarctic winter. We measure the survival threshold, the smallest stored energy per robot at which every robot is still alive at sunrise. Across ten runs it is 155.6 +/- 1.3 pcm-h, where one pcm-h is 7.2 kJ of stored heat. That is 37% below a motionless huddle and within 13% of a physics-based lower bound, and at a budget where the motionless huddle loses six robots, every robot survives. The behavior needs no shared spatial reference and no robot-robot contact, and transfers from twenty robots to forty without retraining.
Figures & tables
Fig. 1: Twenty simulated robots during the 354h lunar night, from one rollout of the learned policy. Ring color is each robot’s sky exposure χ on the scale of Fig. 3 . Top: the scattered start at hour 0. Every robot is red: with no neighbor close enough to block it, each one radiates to the full sky. Bottom: hour 73, after the swarm has gathered. The interior is blue, shielded by the bodies around it, and only the rim stays warm-colored; all twenty are alive.
Parameter
Value
Parameter
Value
N
20
Tset
273.15K
R
0.085m
Tenv
95K
M
2.0kg
Tdeath
233.15K
tnight
354h
Cth
2000JK−1
Δt
54.1s
Gsky
0.0108WK−1
vcmd
0.015ms−1
Qmax
2.0W
TABLE I: Model parameters.
Fig. 2: Overview of the learning system. Left: fourteen local observation channels enter a 64-unit LSTM whose state persists across thermal steps; a skip connection passes the input directly to the output layer, which produces a velocity command. ES perturbs the shared parameters θ and updates them from a single scalar returned after the full night. Middle: the simulated swarm, colored by sky exposure χ . Right: the control loop deployed on every robot, using the trained θ unchanged.
Training budget
Training budget
Resulting threshold
Static
( × bound)
(pcm-h)
(pcm-h)
survivors
1.10
151.9
155.8
14/20
1.18
163.0
155.7
14/20
1.40
193.3
176.4
19/20
TABLE II: Sensitivity of the survival threshold to the training budget. The budget is an input, the threshold an outcome; both are in pcm-h. One policy per budget, same seed.
Fig. 3: Emergence from a scattered start; all rows share one initial configuration. Color is each robot’s time-averaged exposure ⟨χ⟩ up to that frame; crosses are dead robots. Under each frame: robots alive and the swarm’s mean ⟨χ⟩ . The thresholds of Table III are a separate quantity, bisected on the budget rather than read off these runs. Top: the learned policy gathers within five hours and circulates, every average converges, 20/20 survive. Middle: the gather-only rule, which aggregates but never circulates, forms the same huddle, but a core–rim gradient persists all night and nine robots are lost. Bottom: under zero command the swarm stays scattered and all are lost by 225 h.
Condition
n
Threshold
× bound
Dispersed, no command
—
328.5
2.38
Motionless huddle
—
246.0
1.78
Gather-only rule
—
250.3
1.81
Learned, packed, shared ref.
3
155.7±1.3
1.13
Learned, packed, no shared ref.
2
154.3±1.6
1.12
Learned, scattered, shared ref.
2
155.4±0.6
1.13
TABLE III: Survival thresholds in pcm-h. Learned entries pool converged runs.
Fig. 4: Transfer to larger swarms. Change in survival requirement relative to N=20 , the size at which all five policies were trained; they were evaluated at N=30 and N=40 without retraining. Squares are the mean across policies with bars spanning best and worst; faint points are the individual policies. The grey region is the change available from the equalization bound.
Channel zeroed
s0
s1
s2
None
157.1
155.8
155.4
Neighbor stored energy
161.4
157.5
155.8
Neighbor temperature
179.8
154.9
159.6
Own sky exposure
157.1
159.2
242.6
Both neighbor channels
175.5
154.9
268.4
TABLE IV: Observation ablation results, reported per policy.
Electrically-actuated quadrupedal robots possess high mobility on complex terrains, but their motors tend to accumulate heat under high-torque cyclic loads, potentially triggering overheat protection and limiting long-duration tasks. This work proposes a thermal-aware control method that incorporates motor temperatures into reinforcement learning locomotion policies and introduces thermal-constraint rewards to prevent temperature exceedance. Real-world experiments on the Unitree A1 demonstrate that, under a fixed 3 kg payload, the baseline policy triggers overheat protection and stops within approximately 7 minutes, whereas the proposed method can operate continuously for over 27 minutes without thermal interruptions while maintaining comparable command-tracking performance, thereby enhancing sustainable operational capability.
Letian Qian, Yuhang Wan, Shuhan Wang +1
State Key Laboratory of Intelligent Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, China
Motor thermal management is often overlooked in the context of electrically-actuated robots, particularly legged robots, but motor overheating is a key factor that limits long-duration locomotion especially under payload conditions. This paper integrates a whole-body thermal model of a quadruped robot into the reinforcement learning pipeline to update motor temperatures, and proposes a two-stage training framework for motor thermal management. In this framework, a nominal policy is first pre-trained as a locomotion baseline capable of traversing diverse terrains. A residual policy is then trained on top of the nominal policy to provide corrective actions based on the robot's thermal state, ensuring high performance under low-temperature conditions and preventing motor overheating under high-temperature conditions. Simulation results demonstrate that the proposed policy achieves an effective balance between motor thermal safety and locomotion performance. Real-world experiments on a Unitree A1 quadruped robot further validate the approach: under a 3 kg payload, the robot achieves stable locomotion across multiple terrains for over 13 minutes, while the nominal policy alone leads to motor overheating in about 5 minutes.
Yuhang Wan, Weixian Lin, Letian Qian +5
School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, 430074, China
The explore/exploit trade-off is the core problem of reinforcement learning (RL). We show that a minimal foraging swarm approaches its optimum - the ideal-free / Nash allocation of a multi-agent resource-sharing game (an equilibrium, not a return maximum) - with no learner and no reward signal. Agents explore by heavy-tailed (Levy) flights, recruit neighbours to a discovery through a threshold cascade on their proximity network, and disperse as a patch crowds; this cascade cycle balances exploration and exploitation and drives foragers toward the ideal free distribution. The swarm attains the shape of this optimum (committed foragers track richness with slope 0.94) but not its exact equilibrium - it stays measurably exploitable - reaching it dynamically (~150 steps) through the coupling of recruitment to movement: remove the coupling and matching collapses (0.94 -> 0.08). The RL sits at the level of the problem and the allocation reached, not the micro-rule: no step evaluates payoff (reward-free by construction, though the crowd signal is informationally tied to payoff), and the swarm reaches the allocation without learning it (quality plateaus and re-settles after resources move, no cumulative gain). Against a trained baseline the comparison is two-sided: in an easier, disembodied game a learner is more precise, while in the same embodied environment the reward-free swarm reaches ~65% of a validated oracle ceiling at zero training and leads until trained PPO and centralised-critic (CTDE) learners overtake it (~74-75%, 16 seeds, all above the swarm mean) after ~4,000 of 15,000 training steps - its advantage is in budget, not asymptote. In this one setting we read it not as beating RL but as a strong zero-shot, reward-free operating point: a demonstration that the self-organizing physics of a minimal swarm can carry a collective close to the optimum RL seeks, with none of RL's machinery.