Organizations: Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology, Moscow, Russian Federation. · NLP Research Center, Moscow, Russian Federation
A photorealistic 3D view tells a teleoperator where a robot is, but not what the scene contains, how well each object has been observed, or how to turn pointing and speech into robot action. CognitiveReality turns a robot's RGB-D stream into a live, semantically indexed Gaussian-TSDF map shared by an operator in virtual reality and a tool-using language agent. One mapper binary serves any platform through configuration alone: it ingests poses from robot SLAM, joint kinematics, motion capture or an inline visual tracker, bridges localization outages with a shadow tracker and keyframe-anchored PnP, and maintains open-vocabulary instance identities with per-object quality at 2 Hz. Speech and controller rays are grounded against persistent scene objects through validated typed tools and operator-confirmed robot actions. In the controlled agent evaluation, the deployed local Qwen3-VL-8B router reaches 81.24% tool exact match, while merge-aware replay correctly redirects 101 absorbed object identifiers. On robot data CognitiveReality exceeds a Gaussian-plus-SDF baseline by 2-8 dB; pose error through 5-40 s SLAM outages stays within 1-8 cm. Deployed live on two quadrupeds, the agent executed 26 of 30 navigation requests and 20 of 20 re-observation requests, raising object quality by 2-5 dB.
Figures & tables
Fig. 1: Shared 3D scene memory for robot-agent-VR interaction. RGB-D observations from a robot build an online TSDF-Gaussian map enriched with semantic labels, object identities, and observation quality. The agent grounds the operator’s language and pointing in this shared memory, enabling scene queries, object selection, highlighting, and viewpoint control. A common object ID connects robot perception, scene reconstruction, and VR interaction.
Fig. 2: Architecture of CognitiveReality, showing the data flow between the robot backend, online geometric and semantic reconstruction, the shared scene-aware agent, and the two VR delivery modes. A common 3D scene representation connects robot sensing, semantic object state, agent reasoning, operator interaction, and robot control.
Fig. 3: Example of the semantic coverage without (left) and with (right) Label Splats on a chair in the Replica office0 scene.
Fig. 4: Example of operator interaction in the VR client. The controller ray provides spatial grounding for a robot navigation request, which is presented to the operator for confirmation before execution.
Fig. 5: Quadruped robotic platforms used in this work: ZSL-1 (left) and ZSM-1 (right), with mast-mounted RGB-D cameras.
LAB1
Quadruped A a
UR10 b
Drone bag3
Drone new1
Tien Kung
handheld, 720p
ORB-SLAM3, 720p
robot poses, 720p
Vicon, VGA
Vicon, VGA
head cam, 720p
GPS
CR
GPS
CR
GPS
CR
GPS
CR
GPS
CR
GPS
CR
PSNR ↑
21.50
22.17
22.86
26.54
22.83
24.23
16.11
21.91
11.44
19.54
15.32
23.20
SSIM ↑
0.689
0.738
0.744
0.839
0.839
0.843
0.557
0.639
0.428
0.591
0.439
0.738
LPIPS ↓
0.358
0.319
0.313
0.212
0.145
0.143
0.417
0.362
0.496
0.407
0.456
0.253
FPS
64.8
65.4
97.1
51.4
10.9
29.4
178
158
175
132
98.9
66.1
TABLE I: Online mapping on the same sequences with matched settings.
Method
mIoU
mAcc
f-mIoU
f-Acc
HoloAgent-Memory
29.93
43.60
57.00
65.39
CR, closed vocab., 10 fps / 1 Hz
29.53
46.63
65.73
78.33
CR, closed vocab., 20 fps / 2 Hz
29.22
45.54
65.71
78.25
CR, closed vocab., 30 fps / 3 Hz
27.56
43.35
62.99
76.04
CR, open vocab., default prompt
17.23
26.80
57.66
73.06
CR, open vocab., tuned prompt
22.45
33.12
62.73
77.47
TABLE II: Zero-shot 3D semantic segmentation on Replica. IoU/Acc are averaged over classes (m) or weighted by class frequency (f). CR - CognitiveReality, HoloAgent-Memory - baseline. Best in bold, second best underlined.
Replay condition
Ep. / events
Ep. success
Redirects
Silent wrong
Natural / frozen
167 / 87
0 / 167
0
142
Natural / latest
167 / 87
16 / 167
0
40
Natural / causal
167 / 87
117 / 167
101
40
TABLE III: Natural temporal replay over 167 episodes from 87 recorded source events. A redirect is a correct successor identifier, not navigation.
Condition
Named target
Injected class
Conflict ack.
Image only
0.247
–
0.048
Metadata only
0.993
–
0.027
Image + metadata
0.959
–
0.007
Wrong metadata
0.021
0.918
0.185
Screenshot + metadata
0.897
–
0.144
TABLE IV: VLM metadata controls. Named target measures agreement with the map label; injected class is reported under conflicting metadata.
Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.
Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology, Moscow, Moscow, Russian Federation · NLP Research Center, Moscow, Russian Federation