Reality and Perceptual Gaps in Expressive Swarm Robotics
Authors: Ali Imran, Abdalwhab Bakheet Mohammed Abdalwhab, Rafael Gomes Braga, Hélène Duval, Audrey Rochette, Vivek Shankar Varadhajan, Yann Bouteiller, Jean Mazerolle, Giovanni Beltrame, David St-Onge
Organizations: INIT Robots Laboratory, Department of Mechanical Engineering, École de technologie supérieure, Montréal, Québec, Canada . · Department of Dance, Université du Québec à Montréal, Montréal, Québec, Canada . · MIST Lab, Department of Computer and Software Engineering, Polytechnique Montréal, Montréal, Québec, Canada .
Moving robot swarms from simulation into the physical world can be difficult because small discrepancies in hardware, timing and communication can cascade across interacting robots and alter group behaviour. This challenge is known as the reality gap. When swarms operate in human environments, a second challenge emerges: a perceptual gap between what the swarm physically does and what human onlookers actually perceive. This study examines the same decentralised coordination code deployed across three robotic swarms-tabletop, aerial and human-scale mobile manipulators-during live artistic performances. We find that while the underlying algorithms executed reliably on all three platforms, physical embodiment drastically altered visible motion-distorting execution timescales by more than a factor of three and reshaping trajectory geometry. More importantly, audience surveys revealed that human interpretation does not mirror technical success: how observers attribute meaning (perceived purpose) and how they react emotionally (comfort versus threat) depend heavily on physical form. Meaning attribution and emotional responses did not vary together. Our results demonstrate that successfully transferring swarm software to physical hardware is insufficient for real-world deployment. Addressing the reality gap does not guarantee that the resulting collective motion will be readily interpreted by human observers. Swarms destined for human environments must be engineered not only for decentralised stability but also for the expressive physical dynamics through which humans interpret collective motion.
Figures & tables
Figure 1: From specified collective organisation to human interpretation: the sequential reality and perceptual gaps examined in this work. The physically realised collective forms the stimulus available to human perception.
Figure 2: The three DESSAIM robotic swarms seen in their performing settings.
Figure 3: Example trajectories of the three behaviours on each platform: (a) Fireworks, (b) cyclic Pursuit, and (c) Lennard–Jones. Each behaviour runs from the same script on the Doodies, CrazyCognies and Sushis. Lines show individual robots, coloured by time.
Figure 4: Distribution of each robot’s direction of motion relative to the outward radial direction from the swarm centroid, α (0° outward, 180° inward, ± 90° tangential), for Fireworks ( a ), cyclic Pursuit ( b ), and Lennard–Jones ( c ) on the Doodies (motion capture), CrazyCognies (motion capture), and Sushis (overhead video). Radial motion produces a vertical bow-tie distribution, sustained circulation produces a lateral lobe, and motion without a preferred radial or tangential orientation produces an approximately circular distribution. Each outline represents one take; the dotted circle denotes a uniform angular distribution, and the polar-density scale is set independently for each panel. Column headers report the ψ span between the most radial Fireworks take and the most tangential Pursuit take.
Figure 5: Median robot speed against the ratio of median tangential to median radial speed relative to the swarm centroid, on logarithmic axes. Above the dashed line, motion is predominantly tangential; below it, predominantly radial. Outlines group takes from the same platform–behaviour combination. Left of the axis break are the three Sushi Lennard–Jones takes; their very low speeds mean that the ratio is dominated by near-stationary periods rather than the brief corrective motions that maintained spacing.
Doody
CrazyCogni
Sushi
Fireworks
Median Rg (m)
1.45±0.24
1.53±0.05
0.166±0.004
Median speed (mm s -1 )
130±40
166±25
34.8±1.0
ψ
+0.57±0.07
+0.16±0.07
+0.653±0.015
P
0.33±0.08
0.29±0.07
0.157±0.015
M
0.175±0.029
0.17±0.04
0.055±0.008
Table 1: Rg , radius of gyration; ψ , radial index; P , polarisation; M , milling order; modulation depth, peak-to-trough swing of the phase-folded Rg over its median. Measurements were obtained from motion capture for the Doodies and CrazyCognies and overhead video for the Sushis.
Figure 6: polarisation P (shared heading; median over time of ∥⟨v^i⟩∥ ) against milling order M (rotation about the centroid; median over time of ∣⟨(r^i×v^i)z⟩∣ ), one point per take. Dashed boxes indicate the fish-school states of Tunstrøm et al. [ 33 ] : disordered ( P,M<0.35 ), milling ( P<0.35 , M>0.65 ), and polarised ( P>0.65 , M<0.35 ). These thresholds were defined for groups of 30–300 fish; with four to six robots, chance alone gives P≈0.4 . Grey regions show the union of the 95% null intervals across takes, each estimated from 200 randomised copies in which every robot’s velocity series was independently time-shifted and rotated while positions were preserved.
Figure 7: Local median position error relative to each behaviour’s own median, pooled across behaviours, for the aerial (left) and ground (right) platforms. Stippling marks locations where the local value cannot be distinguished from the platform’s typical error, covering 68% of the aerial field and 83% of the ground field. Circles indicate the smoothing bandwidth, which differs between platforms because of the number of samples (72,184 ground rows versus 26,120 aerial rows).
Platform
Behaviour
n
Horizontal error (m)
Samples >0.5 m
median [Q1, Q3]
(%)
CrazyCognies
Fireworks
13
0.077 [0.049, 0.115]
1.1
Cyclic pursuit
15
0.077 [0.050, 0.111]
0.2
Lennard–Jones
13
0.078 [0.048, 0.111]
3.0
Doodies
Fireworks
14
0.128 [0.081, 0.224]
6.0
Cyclic pursuit
14
0.107 [0.065, 0.155]
2.1
Table 2: Localisation uncertainty during expressive behaviours. Horizontal position error of the onboard estimate relative to motion capture, by platform and behaviour. Nine takes were recorded per platform, three per behaviour. n gives the number of robot-takes passing the correspondence criterion. Position error is reported as median [Q1, Q3]. The final column gives the fraction of synchronised samples with error exceeding 0.5 m.
Figure 8: The three DESSAIM robotic platforms.
Figure 9: PyBuzz architecture used to preserve a common Buzz behavioural representation across simulation and heterogeneous physical platforms. Each logical robot executes an independent Buzz Virtual Machine (BVM), while platform adapters translate localisation, sensing, inter-robot communication, and motion commands between the behavioural layer and platform-specific interfaces.
Behaviour
Platforms
Designed collective organisation
Compositional function
Aggregation
S
Convergence toward the local centroid
Contraction; increasing density
Attraction point
S
Convergence toward an external reference
Directed collective displacement
Diffusion
S
Motion away from the local centroid
Expansion; decreasing density
Repulsion point
S
Motion away from an external reference
Directed collective avoidance
Fireworks
S, C, D
Cyclic aggregation–diffusion
Rhythmic contraction–expansion
Lennard–Jones
S, C, D
Regulation of inter-agent spacing
Cohesive, deformable spatial structure
Table 3: Expressive behaviour vocabulary used in DESSAIM. S: Sushis; C: CrazyCognies; D: Doodies.
Figure 10: Iterative research-creation process used in DESSAIM. Robotic experimentation, choreographic and movement exploration, performance, observation, public feedback, and interdisciplinary debriefing were linked through repeated cycles of reconfiguration. Technical, artistic, and perceptual observations could be reinvested in subsequent iterations, allowing changes to robot behaviour, localisation, dramaturgy, staging, embodiment, and mediation.
As groups of robots increasingly collaborate with humans, understanding how humans perceive them is critical for designing effective human-robot teams. While prior research examined how humans interpret and evaluate the abilities and intentions of individual agents, social perception of robot teams remains relatively underexplored. Drawing on the competence-warmth framework, we conducted two studies manipulating swarm behaviors in completing a collective search task and measured the social perception of swarm behaviors when human participants are either observers (Study 1) and operators (Study 2). Across both studies, our results show that variations in swarm behaviors consistently influenced participants' perceptions of warmth and competence. Notably, longer broadcast durations increased perceived warmth; larger separation distances increased perceived competence. Interestingly, individual robot speed had no effect on either of the perceptions. Furthermore, our results show that these social perceptions predicted participants' team preferences more strongly than task performance. Participants preferred robot teams that were both warm and competent, not those that completed tasks most quickly. These findings demonstrate that human-robot interaction dynamically shapes social perception, underscoring the importance of integrating both technical and social considerations when designing robot swarms for effective human-robot collaboration.
Genki Miyauchi, Roderich Groß, Chaona Chen
School of Electrical and Electronic Engineering, University of Sheffield, Sheffield, UK · Department of Computer Science, Technical University of Darmstadt, Darmstadt, Germany · School of Computer Science, University of Sheffield, Sheffield, UK
This work investigates human-swarm interaction (HSI) for steering large collectives through an intermediary robot helper. While prior robot-swarm control and HSI techniques commonly assume swarm members are addressable, these assumptions often fail for ``ungovernable'' swarms such as wildlife flocks and schools of fish. Motivated by pastoral shepherding, we compare low-level teleoperation to a higher-autonomy motion-primitives interface in a virtual reality (VR) shepherding task under varied swarm cohesion dynamics. Results indicate that motion primitives that operate on collective shape (e.g., boundary-based control) improve task performance and reduce perceived workload.
Arihant Yadav, Sophie Jörg, Eakta Jain
Arihant Yadav & Eakta Jain are with Department of Computer & Information Science & Engineering , University of Florida, Gainesville, USA · Sophie Jörg is a Professor in the Information Systems and Applied Computer Sciences Department at University of Bamberg, Germany
Robots in shared spaces often move in ways that are difficult for people to interpret, placing the burden on humans to adapt. High-DoF robots exhibit motion that people read as expressive, intentionally or not, making it important to understand how such cues are perceived. We present an online video study evaluating how different signaling modalities, expressive motion, lights, text, and audio, shape people's ability to understand a quadruped robot's upcoming navigation actions (Boston Dynamics Spot). Across four common scenarios, we measure how each modality influences humans' (1) accuracy in predicting the robot's next navigation action, (2) confidence in that prediction, and (3) trust in the robot to act safely. The study tests how expressive motions compare to explicit channels, whether aligned multimodal cues enhance interpretability, and how conflicting cues affect user confidence and trust. We contribute initial evidence on the relative effectiveness of implicit versus explicit signaling strategies.
Jonathan Albert Cohen, Kye Shimizu, Allen Song +3
MIT Media Lab Cambridge USA · New England Innovation Academy Cambridge USA