Reliable monitoring of astronauts is essential for safety, situational awareness, and autonomous assistance in space missions. While modern computer vision enables accurate human detection and pose estimation on Earth, its robustness in space environments remains largely unquantified due to limited data availability. We present AstroPIE-53, a pilot dataset of 53 annotated astronaut videos collected from publicly available mission footage. To respect copyright restrictions, we release video identifiers, reconstruction scripts, and dense annotations rather than raw videos. The dataset includes bounding boxes, challenge classes, and human-annotated pose failure tags. We benchmark state-of-the-art person detection and pose estimation models under realistic zero-shot deployment conditions and perform the first systematic failure analysis for astronaut monitoring. Our results reveal substantial performance degradation caused by occlusions, unusual body orientations, low contrast, and microgravity motion patterns.
Figures & tables
Figure 1: Example images from AstroPIE-53. We propose six different challenge classes common in astronaut detection.
Figure 2: Three-stage data pipeline for constructing AstroPIE-53: (1) metadata-driven video search, (2) filtering and synchronization of relevant recordings, and (3) frame extraction and annotation of astronaut activities.
Figure 3
Analyzed Failure Cases
Dataset / Work
Real
#Frames
2D GT
3D GT
Env.
Public
Occl.
Dist.
Suit
Contr.
Microg.
Quant.
SpacesuitPose [ 16 ]
✓
400
19 joints
–
Lab
Yes
✓
–
✓
✓
–
No
NASA Challenge [ 15 ]
✓
∼ 200 + video
✓
✓
Mixed
No
–
–
✓
–
–
No
AstroPose [ 14 ]
Astron. Figure
–
–
–
Lab
No
–
–
✓
–
–
No
μ Pose [ 3 ]
Synthetic
312 (aug.)
✓
–
Inside SC
Yes
–
–
–
–
✓
No
ROpenPose [ 22 ]
✓
∼ 9k
Unclear
–
Inside SC
No
✓
–
–
–
✓
No
Table 1: Overview of monocular methods and datasets related to astronaut monitoring. Our AstroPIE-53 benchmark includes a detailed failure case analysis regarding occlusion, person-camera distance, spacesuit existence, etc., whilst existing publications do not provide systematic quantification across detection and pose estimation. Some datasets are recorded in labs, others provide real data (inside/outside spacecraft (SC))
Spacesuit
Orientation
Env.
Contrast
Distance
Occlusion
Suit
No-Suit
Front
Back
Side
Multi
Ind.
Out.
High
Low
Close
Far
Occ.
Non-Occ.
Videos
43
10
7
3
25
18
15
38
20
18
46
7
9
44
Frames [%]
86.0
14.0
8.4
7.0
34.6
50.0
21.0
79.0
39.3
60.7
68.8
31.2
17.1
82.9
Table 2: Statistics of our AstroPIE-53 dataset. It is divided into 14 different challenge label subsets across six challenge classes.
DINO [ 13 ]
DINO [ 13 ]
RF-DETR [ det:rf-detr:robinson ]
YOLO26 [ 8 ]
Categories [%]
(Astronaut)
(Person)
Suit
46.03
46.88
16.78
14.85
No-Suit
86.90
86.87
85.76
84.40
Front
94.91
94.57
86.15
86.22
Back
13.62
24.64
6.52
2.18
Side
76.24
73.93
30.17
23.53
Table 3: AP@50 (%) of tested person detectors ( τconf=0.3 , τIoU=0.5 ) on AstroPIE-53 evaluated per challenge category. Detection performance drops sharply for suit presence/outdoors, back views, low contrast and far distances. Overlapping, imbalanced categories yield associations, not isolated effects.
Failure Type [%]
MeTRAbs
NLF
ViTPose
Misaligned head
3.9
—
—
Misaligned torso
0.9
0.1
—
Misaligned arms
1.3
4.6
0.1
Misaligned legs
8.5
7.9
1.5
Scale
—
3.4
—
Global orientation
1.8
3.8
0.3
Table 4: Human-annotated failure analysis of three pose estimators on AstroPIE-53. Each cell reports the percentage of 19912 annotated bounding-box keyframes exhibiting the respective failure type, with respect to 20489 total video frames. As multiple misalignments can occur on the same bounding box frame the total sum of percentages will not necessary add up to 100%. Correct Estimations denotes keyframes with no failure tag.
Undergraduate Student, Wellesley College, Wellesley, MA 02481, USA · Undergraduate Student, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Research Scientist, Massachusetts Institute of Technology, Cambridge, MA 02139, USA +5
University of Chinese Academy of Sciences, Beijing, China · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China · Nanjing University of Aeronautics and Astronautics, Nanjing, China