cs.RONov 11, 2026

Reading the Table: Sim-to-Real Game State Detection for the Chef's Hat Card Game

Authors: Connor Gäde, Matthias Kerzel, Hauke Bünning, Jan-Gerrit Habekost, Mariana Barros, Stefan Wermter

Organizations: Knowledge Technology, Dept. of Informatics, University of Hamburg, Germany

Abstract

Tabletop games offer a compelling scenario to study the interaction between artificial and human agents, combining social interaction with game state tracking and physical manipulation. To assess the game state, a robot needs to precisely detect closely related game objects, such as different card designs, requiring dedicated training data that is costly to annotate in the real world. Instead, we develop a fully automated synthetic data collection pipeline, using re-projection of known 3D coordinates to generate labels automatically and domain randomization to bridge the reality gap. This approach is used to collect a dataset of 25413 samples to finetune an object detection and a segmentation model, which are combined to assign detected cards to game-relevant zones and infer the game state. When evaluated on a real NICO humanoid robot, our pipeline achieves an F1-score of 0.906 for card-to-zone assignment, thus providing the perceptual foundation for a robot that can autonomously play Chef's Hat against human players.

Figures & tables

Explore similar work

CardsList
  1. TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation

    Dec 1, 2025Ziqian Wang, Yonghao He, Licheng Yang +6Robotic Data CollectionRobotics Simulation

  2. Twin: Playing an Unknown Game with a Test-Time Digital Twin

    Aug 14, 2026Alexy Skoutnev, Kirill Acharya, Gaston Longhitano +3Game-Playing AgentsCode World Models