Reading the Table: Sim-to-Real Game State Detection for the Chef's Hat Card Game
Organizations: Knowledge Technology, Dept. of Informatics, University of Hamburg, Germany
Abstract
Tabletop games offer a compelling scenario to study the interaction between artificial and human agents, combining social interaction with game state tracking and physical manipulation. To assess the game state, a robot needs to precisely detect closely related game objects, such as different card designs, requiring dedicated training data that is costly to annotate in the real world. Instead, we develop a fully automated synthetic data collection pipeline, using re-projection of known 3D coordinates to generate labels automatically and domain randomization to bridge the reality gap. This approach is used to collect a dataset of 25413 samples to finetune an object detection and a segmentation model, which are combined to assign detected cards to game-relevant zones and infer the game state. When evaluated on a real NICO humanoid robot, our pipeline achieves an F1-score of 0.906 for card-to-zone assignment, thus providing the perceptual foundation for a robot that can autonomously play Chef's Hat against human players.
Figures & tables
| Model | Infer. (ms) | P | R | F1 | mAP 50 | mAP 50:95 |
|---|---|---|---|---|---|---|
| YOLO26n | 3.5 | 0.878 | 0.815 | 0.845 | 0.850 | 0.657 |
| YOLO26s | 7.8 | 0.862 | 0.822 | 0.842 | 0.845 | 0.652 |
| YOLO26s † | 8.7 | 0.606 | 0.513 | 0.556 | 0.514 | 0.382 |
| YOLO11n | 3.5 | 0.845 | 0.781 | 0.812 | 0.811 | 0.619 |
| YOLO11s | 4.7 | 0.879 | 0.802 | 0.839 | 0.838 | 0.647 |
| † Finetuned with canonical data only. | ||||||
| Models | P | R | F1 | missing avg | |
|---|---|---|---|---|---|
| YOLO26n | & YOLO26n-seg | 0.902 | 0.911 | 0.906 | 1.52 |
| YOLO26s | & YOLO26s-seg | 0.899 | 0.906 | 0.902 | 1.61 |
| YOLO26s † | & YOLO26s-seg † | 0.727 | 0.551 | 0.627 | 7.02 |
| YOLO11n | & YOLO11n-seg | 0.835 | 0.908 | 0.870 | 1.38 |
| YOLO11s | & YOLO11s-seg | 0.873 | 0.918 | 0.895 | 1.34 |
| † Both models finetuned with canonical data only. | |||||
Robotic Perception
Robotic perception enables robots to interpret scenes, objects, motion, and contact from cameras and other sensors to guide action.
See all 337 papers