A team of Gemini 4 Argon agents beats Minecraft

We present TickingCollabBench, a Minecraft-based multi-agent benchmark for a novel class of time-sensitive complementary collaboration tasks. Our benchmark reflects four core characteristics of real-world collaboration: agent heterogeneity, mandatory collaboration, dynamic environments, and strict real-time constraints with failure risks. To enable this, we develop the TickingCollab framework, which supports the generation of diverse dynamic environments and abstracts Minecraft's primitive APIs to enable declarative YAML task specifications for composing these events. Building on this, we design a feasibility-aware automated benchmark generation pipeline, where an LLM drafts structurally diverse task configurations and feasibility verifier filters out invalid ones using approximate constraints. Evaluations demonstrate that lang latency and inherent difficulty of coordinating under partial observability and agent heterogeneity cause LLMs to frequently fail under dynamic environments and fall significantly short of a global-knowledge oracle.

With the rapid progress of large language models (LLMs), an interesting question has emerged: how do LLM-based agents perform in Minecraft? To date, most Minecraft benchmarks evaluate them under fixed game mechanics, which are extensively documented online. High performance in these settings does not show whether an agent can continue making progress when familiar recipes, drops, and other rules change. In this paper, we introduce MirrorCraft, a paired benchmark for evaluating agents under hidden rule changes in Minecraft. Each Mirror world is a copy of its paired Vanilla world, with selected server-side rules silently modified by the datapack of one rule suite. Terrain, spawn, resource placement, task, interface, and action budget remain matched within every Vanilla-Mirror pair. MirrorCraft includes 10 Vanilla worlds in five controlled biomes, six rule suites, and three progression tasks, and we evaluate two LLMs and six agent configurations under a shared Mineflayer interface. We measure progress with server-verified advancement milestones and completion with success rate, and define the Rule Intervention Effect (RIE) as the performance change between paired Vanilla and Mirror worlds. Experiments with 8,640 episodes show that hidden rule changes have sharply different effects across suites. Among the configurations evaluated without rule descriptions, ReAct, Reflexion, and Voyager achieve the highest pooled Mirror Scores, but ReAct loses the most under compressed resource drops. Providing ReAct with the exact rules yields modest gains in average progress and completion across all three tasks, yet most of the difference between suites remains once the rules are known. MirrorCraft extends Minecraft evaluation beyond fixed mechanics and provides a controlled setting for studying how agents use gameplay outcomes when the rules of the current world differ from familiar ones.

Long-horizon embodied intelligence requires agents to improve through interaction, not merely to execute plans generated from static goals. A central challenge is therefore to transform past executions into knowledge that can shape future decisions. Minecraft provides a representative testbed for this problem, where tasks such as crafting tools, building redstone components, and obtaining diamond equipment involve long prerequisite chains and are frequently disrupted by missing tools, blocked paths, GUI failures, or stagnant execution. To this end, we propose MineEvolve, a knowledge-driven self-evolution framework that converts execution feedback into actionable behavioral knowledge. MineEvolve first uses \underline{\emph{\ding{182}Monitor}} to convert each subgoal execution into typed feedback, including state changes, inventory changes, failure types, progress signals, and stagnation indicators. \underline{\emph{\ding{183}Inducer}} then derives reusable skills from successful executions and remedies from failed or stagnant executions. \underline{\emph{\ding{184}Curator}} validates, merges, filters, and retrieves these knowledge entries, while \underline{\emph{\ding{185}Adaptor}} uses them to repair the unfinished part of the plan under repeated failures or stagnation. Experiments on the Minecraft MCU long-horizon task suite show that MineEvolve consistently improves performance across multiple language-model planners, with larger gains on high-dependency task groups. Ablation and knowledge-accumulation studies further demonstrate that converting execution signals into structured behavioral knowledge is an effective path toward self-evolving embodied agents in long-horizon environments. Our code is available at https://github.com/xzw-ustc/MC-MineEvolve.