Long-horizon tabletop games pose a distinct systems challenge for robotics: small perceptual or execution errors can invalidate accumulated task state, propagate across decision-making modules, and ultimately derail interaction. This paper studies how to maintain internal state consistency in turn-based, multi-human robotic tabletop games through deliberate system design rather than isolated component improvement. Using Mahjong as a representative long-horizon setting, we present an integrated architecture that explicitly maintains perceptual, execution, and interaction state, partitions high-level semantic reasoning from time-critical perception and control, and incorporates verified action primitives with tactile-triggered recovery to prevent premature state corruption. We further introduce interaction-level monitoring mechanisms to detect turn violations and hidden-information breaches that threaten execution assumptions. Beyond demonstrating complete-game operation, we provide an empirical characterization of failure modes, recovery effectiveness, cross-module error propagation, and hardware-algorithm trade-offs observed during deployment. Our results show that explicit partitioning, monitored state transitions, and recovery mechanisms are critical for sustaining executable consistency over extended play, whereas monolithic or unverified pipelines lead to measurable degradation in end-to-end reliability. The proposed system serves as an empirical platform for studying system-level design principles in long-horizon, turn-based interaction.
Figures & tables
图 2: Cross-module error propagation in long-horizon tabletop gameplay. Perceptual, execution, or interaction errors can propagate through the maintained game history and internal state, corrupting downstream reasoning and actions if not detected and recovered. The red dashed lines represent propagation paths that could immediately create visible insecurity.
图 3: Representative deployment environments and gameplay settings. Including 1) Top-down views of the Mahjong table layout and per-player tile walls. 2) Close-up of manipulation during discarding and claiming. 3) Multi-human turn-based gameplay in laboratory conditions. 4) Public exhibition deployment with multiple participants.
Decision Context
Model
n
Evaluation Metrics (mean ± std)
Fluency
Independence
Rigor
Interpretability
Correctness
Overall
Discard (Missing Suit)
GPT-5.2
74
4.230.91
4.051.16
4.051.08
3.821.28
4.720.67
3.911.09
Ours
85
4.381.05
4.271.13
4.271.15
4.01.23
4.850.45
4.311.03
Discard (no Missing Suit)
GPT-5.2
55
3.761.13
3.61.43
3.531.44
3.41.43
3.981.39
3.491.26
Ours
44
4.231.13
4.271.12
4.111.21
4.091.28
4.570.86
4.20.94
Claiming Decision
GPT-5.2
15
3.471.02
3.01.41
2.81.56
2.41.58
3.131.5
2.531.36
表 I: Human evaluation of decision traces on a Likert scale. Fluency measures grammatical correctness and naturalness; Independence assesses whether the reasoning appears like human thought rather than tool-assisted; Rigor evaluates logical soundness and evidential support (including appropriate use of quantification and certainty); Interpretability measures whether the final decision is stepwise explainable and rule-consistent; Correctness assesses the strategic quality of the chosen action; Overall reflects the perceived human-likeness of the reasoning style. We randomly sampled 300 game states from 40 matches and distributed all model outputs evenly to six blinded annotators; five samples were incomplete. Values are reported as mean ± standard deviation, and n denotes the number of evaluated states per model and decision context.
Mode
Games
Robot
Right
Opp.
Left
Draw
Characters
46
1
19
14
8
4
Normal
122
34
29
27
22
13
表 II: Distribution of winning outcomes under two gameplay configurations. In the Characters mode, used during debugging, the robot records the declared missing suit of all players as Characters. The Normal mode corresponds to the full system described earlier, where each player’s declared missing suit is determined through speech interaction and perception. Columns Right , Opp. , and Left denote the three human players relative to the robot \CJK@punctchar \CJK@uniPunct 0”80”99s seating position. Multiple players may win from the same discard, so the sum of wins across players can exceed the number of games.
Task
VLM
YOLO / DINO
Tile recognition accuracy
<94.3%
98.9% (YOLO)
Latency (ms)
≥ 380
15 ± 2 (YOLO)
Inspection P/R
0.967 / 0.989
0.956 / 0.960 (DINO)
Turn attribution P/R
0.937 / 0.976
0.949 / 0.931 (DINO)
Cross-identity error
0%
3.0% (DINO)
表 III: Comparison of architectural alternatives for tile perception and interaction-level violation detection. VLM achieves 94.3% per-tile recognition accuracy rather than per-image. Latency is measured per image. Precision/Recall are reported for positive-class detection.
图 4: Smoothed hardware error probability as a function of continuous operation time. Error frequency increases noticeably after approximately 20,000 seconds of sustained execution.
图 5: Training pipeline for strategic reasoning. Stage 1 distills a conventional RL-trained policy into the VLM via supervised fine-tuning with LLM-synthesized reasoning traces. Stage 2 applies single-step RL (GRPO) to optimize decision quality beyond imitation. Stage 3 uses self-play with DPO to discover strategies that surpass the original teacher policy.
Long-horizon manipulation requires actions whose validity depends on earlier interactions, yet visual progress can conceal missing task prerequisites. We propose RoboLongR, an offline-to-online framework that learns history-dependent rewards to guide policy adaptation to new tasks. Its central objective is causal consistency: aligning rewarded progress with prerequisites supported by execution history. Stage-shuffle supervision encourages execution-order sensitivity, recurrent memory preserves evidence of earlier interactions, and two-round training refines trajectory segments using predicted progress. The resulting rewards guide stage-conditioned policy refinement from behavior cloning to online reinforcement learning. Sparse terminal feedback connects policy experience to reward learning by triggering correction for repeated high-scoring failures. Pretrained on demonstrations from 88 RLBench tasks and Open-X data, RoboLongR achieves 31.8% average success on 18 long-horizon tasks excluded from reward pretraining, exceeding the strongest evaluated baseline by 9.2 percentage points. Across four real-robot tasks, it achieves 57.5% success, a 32.5-percentage-point improvement over the comparison method. Reward-level controls probe sensitivity to synthetic order perturbations, while ablations examine the contributions of individual components to policy improvement.
Zixuan Chen, Lin Cao, Fenggang Zhang +5
Nanjing University · Southeast University · Fudan University
Robotic systems operating over extended tasks must maintain a world state assembled from observations arriving at different times, with varying confidence and potential revisions. Conventional representations emphasize latest estimates, hindering fact provenance, decision reproduction, or execution auditing. We present Traceable World State (TWS), a middleware-neutral semantic representation and reference runtime for provenance-aware robot world state. A TWS snapshot captures entities, relations, observations, confidence, and revision metadata. Validated update operations transform snapshots immutably, ordered updates support deterministic replay, and a canonical SHA-256 hash chain ensures tamper-evident logs. We evaluate TWS through schema conformance, complete state lifecycles, deterministic replay, and fault injection. Passing 38 tests across Python 3.10-3.14, the framework detects record corruptions, broken hash links, sequence discontinuities, and world mismatches. Across ten public BEHAVIOR-1K task definitions, TWS imported 153 entities and 146 relations with successful validation. On 103 NVIDIA Unitree G1 simulated trajectories containing 78,369 frames, TWS achieved exact terminal-state replay in all episodes and detected 412/412 injected corruptions with a 1.72% storage overhead over Plain JSONL.
Hierarchical robotic systems executing long-horizon manipulation tasks must make high-level semantic decisions that orchestrate stochastic low-level skills. In this setting, failed rollouts are ambiguous: a poor downstream state may reflect an invalid high-level decision, partial observation, or a valid decision whose physical execution failed. Traditional supervised learning lacks data for such recovery states, while reinforcement learning struggles with sparse rewards and non-local credit assignment. We propose MAGMA-GEN, an on-policy data-generation pipeline that converts ambiguous failed rollouts into validated recovery supervision. MAGMA-GEN first uses a privileged coach to hypothesize an early decision-level error and propose localized correction or recovery actions. Because this diagnosis is fallible, candidates are retained only if re-execution from the same state under matched conditions improves downstream progress. This produces supervised examples from the agent's own failure distribution without per-step human demonstrations. Evaluated on interactive long-horizon manipulation tasks, MAGMA-GEN improves task success and recovery capabilities, against distillation and trajectory-repair baselines under evolving task constraints in both simulation and real-robot execution.