cs.AIOct 16, 2026
SaveSelf-Improving Video-to-Action Models
Organizations: Computer Science & Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA 02139, USA
Abstract
Video generative models are an emerging tool for embodied agents: they can serve as scalable visual planners, generating videos that depict how to complete previously unseen tasks. However, executing these plans typically requires a video-to-action or "inverse dynamics" model trained on large, human-collected video-action datasets, severely limiting scalability. We introduce Mirror, a self-supervised approach that learns video-to-action mappings through environment interactions, without prior action data or hand-crafted rewards. Mirror learns by attempting to follow a visual plan by executing its best guess of the underlying actions in the environment. Mirror then trains on the resulting video-action pairs, incrementally self-improving and following the visual planner more closely. Across Atari, OGBench, and DeepMind Control, Mirror outperforms leading learning-from-observation methods with learned video plans, and achieves near-expert performance with oracle plans. On Minecraft BASALT, it outperforms methods with access to expert actions or hand-engineered task knowledge. In Minecraft survival, it is the first method to acquire an iron pickaxe through self-improvement alone, without rewards or expert action labels. In the offline setting, with expert action labels but no environment interaction, it sets a new state of the art in obtaining diamonds. Our results suggest that advances in video models can translate into increasingly capable agents without a corresponding need for hand-crafted rewards or large collections of expert action data.
Figures & tables
Figure 1: Enabling long-horizon exploration with a video planner. Left: Random exploration struggles to reach later stages of long-horizon tasks that require specific sequences of actions. Right: The video planner guides exploration. As the V2A model learns from achieved transitions, the agent follows the plan more accurately and reaches progressively later stages of the task. These new transitions further improve the model, eventually enabling the agent to execute the full plan.
Figure 2: Re-planning holds the agent at the milestone until noise supplies the critical action.
Algorithm 1 Learning without action labels: VP-guided self-training of the V2A model
Figure 3: The causal video planner. (a) Tokenized frames and text are fed into the 14B-parameter Cosmos3-Nano ( NVIDIA, 2026 ) model, forming a causal stream of contexts whose keys and values are cached and reused. The denoiser generates future frames while cross-attending to the context KV cache, so each observation and instruction is encoded only once. (b) Attention mask. Video history is encoded block-causally. The future plan attends to this history and bidirectionally to itself. Sliding-window attention lets the planner generalize beyond the window while keeping the KV cache within a constant budget.
Figure 4: Learning from human demonstration. Mirror can complete a variety of open-ended BASALT tasks from self-play without any expert action labels or reward supervision. (a) The VP inherits long-horizon planning capabilities learned from video pre-training, while the V2A model learns to ground the video plans to action via online interaction. (b) The constructed waterfalls are diverse and similar to human creations. (c) Mirror achieves near-human performance on FindCave.
| No action labels | With action labels | Vision-language-action (VLA) models | Heuristic | Human | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Task | Mirror (Ours) | VIPER | Mirror (Ours) | KAIROS | BC (VPT-SFT) | UniTeam | STEVE-1 | GROOT | JARVIS- VLA | OpenHA | GoUp | |
| FindCave | 0% | 27% | 27% | 64% | 45% | 45% | 36% | 45% | 36% | 95% | ||
| MakeWaterfall | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 36% † | 100% | ||
| CreateAnimalPen | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 18% † | 95% |
| BuildVillageHouse | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 91% † | 100% |
Table 1: Solving fuzzy tasks from BASALT. Success rates for methods without action labels (our self-improving method and VIPER ( Escontrela et al., 2023 ) ), action-supervised and heuristic BASALT baselines ( Milani et al., 2022 ) , vision-language-action (VLA) models, and humans.
Figure 5: Minecraft Diamond Challenge. Success rates across twelve acquisition milestones. Offline methods learn from VPT human data without environment interaction. Online methods learn by interacting with the environment for about 10M steps ( 140 in-game hours) without expert action labels. In the offline setting, Mirror (supervised) sets a new state of the art. In the online setting, Mirror (self-improving) is the only method to reach the iron pickaxe without action labels or rewards. Mirror is evaluated on 128 episodes of 60 minutes with respawning, following VPT and Dreamer V4. Offline baselines are from Dreamer V4 ( Hafner et al., 2025b ) . Online baselines at 10M environment steps are from Dreamer V3 ( Hafner et al., 2025a ) .
Figure 6: Emergent in-context learning from failure. (a) We challenge our agent to escape a lava lake without fine-tuning. (b) Retaining context between deaths causes a directional bias in the agent’s trajectories, indicating in-context adaptation. (c) Shared history allows the agent to learn from its past failures in-context, leading to higher escape rates. Retaining recurrent state does not provide a comparable benefit for VPT BC.
Figure 7: Text-conditioned video planning. We evaluate text-conditioned control on 10 tasks across two unseen environments (see Appendix C.1.4 ). Each task is specified only through a text prompt, with no task-specific fine-tuning. VIPER ( Escontrela et al., 2023 ) is included as a reward-free video-based baseline. Combined with our self-improving V2A approach, the agent achieves high success rates although it has never seen expert demonstrations on these tasks. Left: Success rates of each method. Right: Text guidance leads the agent to complete different tasks.
Figure 8: Imitation learning performance across three benchmark suites. Left: learning curves on 13 DeepMind Control (DMC) tasks, four OGBench tasks, and seven Atari games. Colors identify methods; dashed method curves use an oracle VP and solid method curves use a generative VP. The horizontal dashed line marks expert performance. Curves average tasks equally, with shaded regions showing standard error across three seeds per task. The Atari VIPER curve is digitized from Escontrela et al. (2023) over its first 1M decisions and rescaled to % of expert. Right: final performance at 1M environment transitions (mean SE, equal task weights), separated by planner type.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Variable prediction length training. Prediction windows of variable length over the same 32 s clip. The clean history is encoded once by the context encoder and shared by all prediction windows. With sliding-window attention, the clean history tokens and the noisy plan windows attend to the preceding 12 s and 8 s clean frames, respectively.
| Steps | GPUs | Global batch | Learning rate | Wall-clock |
|---|---|---|---|---|
| 0–30k | 8 H200 | 8 | 25 h | |
| 30k–80k | 16 H200 (2 nodes) | 16 | 42 h | |
| 80k–215k | 32 H200 (4 nodes) | 32 | 115 h | |
| 215k–220k | 8 H200 | 32 | (cooldown) | 10 h |
Table 2: Pretraining schedule of the video planner on VPT dataset.
Figure 10: Rolling cache at inference. (1) Each new observation is encoded once by the context encoder, which attends to the 12 s cache. (2) The resulting keys and values are appended and the oldest context is evicted, keeping the cache size fixed. (3) The denoiser generates the next 1.6 s of video while attending to the last 8 s of cached context and, optionally, to a control-level caption. Cache cells schematically group frames.
| Count | Per video (median) | Span (s) | Tokens / caption | |
|---|---|---|---|---|
| Videos | 58,121 | – | 241.0 (duration) | – |
| Descriptions | 13,352,359 | 244 | 3.5 [1.0, 8.0] | 19.8 |
| Instructions | 13,155,310 | 237 | 3.5 [1.0, 8.0] | 15.2 |
| Task segments | 1,506,066 | 26 (5 partitions) | 31.5 [12.0, 62.6] | 16.1 |
| Total tokens | 489.1M | |||
Table 3: Caption corpus statistics for the VPT dataset. Spans are median [10th, 90th percentile]. Tokens are counted with the Cosmos3-Nano tokenizer.
Figure 11: Subgoal task planner. (1) Once per plan, the planner reads the agent’s inventory, health, and whether a GUI is open, and sends a subgoal caption and a guidance scale to the video planner; the V2A model never sees them. (2) Guidance-scale rules; the dashed one is used only during training. (3) Subgoals in the order the planner picks them from an empty inventory. Each 3 3 craft happens at a crafting table placed for it and picked up afterwards.
Figure 12: Video post-processing for the diamond task. (a) The original VPT video recordings do not include the cursor inside GUI menus. (b) As in VPT’s data loader ( Baker et al., 2022 ) , we draw the cursor from the mouse position log. However, VPT’s original 16 16 cursor is too small for the Wan2.2 VAE, which blurs or drops it (Figure 13 ). Thus, for the diamond task we draw a high-visibility cursor: VPT’s cursor enlarged 2 , tinted coral, and outlined in black and white. (c) The VAE also blurs the small item counts or turns them into other digits, so we crop the 184 24-pixel hotbar from the video, enlarge it 3 with nearest-neighbor upsampling, and paste it back at the bottom center, moving the health and food bars up. (d) Finally, we resize the frames to with bilinear interpolation and antialiasing. Of the filters we tried, it is the blurriest but keeps all task-relevant detail and causes the fewest VAE artifacts (reconstruction PSNR 32.7 dB, versus 24.5–30.4 dB for bilinear without antialiasing, bicubic, area, and Lanczos).
Figure 13: The Wan2.2 VAE loses the VPT cursor. The same frame with (a) VPT’s cursor and (b) the high-visibility cursor, before and after a round trip through the Wan2.2 VAE. VPT’s small cursor nearly vanishes in the reconstruction, which is why we draw a larger, outlined cursor for GUI-heavy tasks.
| Method | Inputs | Outputs | Minecraft Data (hours) | ||
|---|---|---|---|---|---|
| Offline | Web | Online | |||
| Dreamer V3 | Video, inventory state | Keyboard, camera, abstract crafting | – | – | 1.4K |
| Dreamer V4 | Video | Keyboard, mouse | 2.5K | – | – |
| VPT (BC) | Video | Keyboard, mouse | 2K | 70K | – |
| VPT (RL) | Video | Keyboard, mouse | 2K | 70K | 230K |
| Video Planner (Ours) | Video, text prompt (optional) | Video | 3K | – | – |
Table 4: Data scale of different Minecraft agents
| Milestone | PPO | Rainbow | IMPALA | Dreamer V3 | Mirror (self-improving) |
|---|---|---|---|---|---|
| Log | 75.2 | 74.4 | 71.0 | 95.0 | 99.2 |
| Planks | 74.6 | 73.3 | 69.7 | 95.0 | 99.2 |
| Crafting table | 69.8 | 65.4 | 61.8 | 91.3 | 99.2 |
| Stick | 74.2 | 72.3 | 69.3 | 94.6 | 99.2 |
| Wooden pickaxe | 51.2 | 48.4 | 45.5 | 76.9 | 94.5 |
| Cobblestone | 26.2 | 23.9 | 31.0 | 67.4 | 57.0 |
Table 5: Diamond Challenge under the Dreamer V3 protocol. Success rate (%) per milestone after about 10M environment steps, with 30-minute episodes that end at the first death. Baseline values are read from Figure 9 of Hafner et al. (2025a) ; Mirror is evaluated on 128 episodes.
Figure 14: Text-conditioned planning on Camp. We demonstrate text-conditioned planning on the camp environment. While our self-improving approach is able to solve many tasks, it struggles on tasks with fine-grained environment interactions such as placing torches in the cave. Meanwhile, VIPER, our action-free baseline, struggles to learn the action space of Minecraft. BC solves some tasks, but remains limited in its generalization ability.
| Environment | Ours | Random-V2A | BCO-V2A | Grounding |
|---|---|---|---|---|
| Cube Single | 78.3 2.8 | 25.3 4.2 | 8.5 2.5 | 5.0 0.8 |
| Cube Double | 27.0 4.6 | 20.7 2.8 | 2.0 1.6 | 1.0 0.7 |
| Scene | 32.5 3.5 | 50.0 5.4 | 1.5 1.5 | 3.0 0.5 |
| AntMaze Large | 25.8 1.5 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
Table 6: Transfer to unseen OGBench goals. Success rate (%, mean SE across five seeds), averaged over target goals. All methods use target-task oracle planners; BCO-V2A uses BCO’s saved V2A model.
Figure 15: Per-task performance on DeepMind Control. Episode return on 13 tasks. Curves show the mean one standard error over three seeds.
Figure 16: Per-task performance on OGBench. Success rate on four single-task environments. Curves show the mean one standard error over three seeds.
Figure 17: Per-game performance on Atari. Scores on seven games, expressed as a percentage of each game’s expert score. Curves show the mean one standard error over three seeds.
Figure 18: MakeWaterfall comparisons. Mirror is capable of the long horizon planning necessary for MakeWaterfall. The agent must make a waterfall, remember its location, and move to another location to take a picture of it. Even though BC, UniTeam and KAIROS are trained with action labels, they achieve 0% success on this long-horizon task. They often go out of distribution. GoUp, which relies on scripted actions, cannot adapt to unseen scenarios such as when underwater. While it is able to place the water bucket with a pre-programmed instruction, it does not know what to do when underwater. The VLA approaches, STEVE-1, GROOT, JARVIS-VLA, and OpenHA, struggle at following the task instructions. VIPER, our action-free baseline, struggles to learn the Minecraft action space.
More in this topic
Self-Improving Agents
Self-improving agents use experience and feedback to update their models, memories, skills, or workflows for better future task performance.
See all 160 papers