Learning to Live: A Step Toward Lifelong Embodied Intelligence
Organizations: Agibot Human Intelligence Project · AgibotTech/LIFE
Abstract
Embodied agents are usually organized around individual tasks, yet the physical world keeps changing between tasks and during execution. We introduce LIFE (Lifelong Intelligence from Experience), a framework in which a robot observes continuously, remembers selectively, and turns experience into new capabilities. RoboBeats, a family of lightweight streaming gates, tracks conditions set by the agent and discovers new events, deciding when an observation warrants a response or a memory write. Selected observations enter Plastic Memory, a flat store of images and structured records that keeps the source evidence, so new fields and indexes can be added when later questions need them. When existing tools fall short, Explore-Distill-Mount (EDM) turns solutions found during problem solving into validated, reusable tools and skills. Under matched protocols, RoboBeats substantially improves condition tracking over its base models and, at the same writing budget, retains more useful evidence for memory question answering than uniform sampling. A case study of limited scale, replaying 24 hours of continuous video recorded by a real robot, illustrates how the integrated system builds memory from a day of experience and uses it to answer everyday questions. LIFE takes a step toward robots that become more familiar with their surroundings and more capable through experience: learning not only to complete tasks, but to live in the world.
Figures & tables
| Method | Size | EPM | ASI | HLD | BT Avg. | EPM+ASI Avg. |
|---|---|---|---|---|---|---|
| Proprietary MLLMs | ||||||
| Gemini 1.5 Pro ( Gemini Team Google, 2024 ) | — | 58.6 | 76.4 | 52.6 | 62.5 | 67.5 |
| GPT-4o ( OpenAI, 2024 ) | — | 57.9 | 75.7 | 48.7 | 60.8 | 66.8 |
| Open-Source Offline MLLMs | ||||||
| Qwen2-VL-72B ( Wang et al., 2024a ) | 72B | 52.5 | 60.8 | 57.5 | 57.0 | 56.7 |
| LLaVA-Video-7B ( Zhang et al., 2024 ) | 7B | 56.2 | 57.4 | 7.5 | 40.4 | 56.8 |
| Method | Size | Charades-STA [email protected] | StreamingBench PO s | SPOT-Bench T-F1@5 |
|---|---|---|---|---|
| Streaming methods, published results, original protocols | ||||
| StreamingVLM ( Chatterjee et al., 2026 ) | 7B | – | – | 3.9 |
| LiveStar ( Yang et al., 2026b ) | 8B | – | 13.6 | – |
| Dispider ( Qian et al., 2025 ) | 7B+1.5B | – | 25.3 | – |
| MMDuet2 ( Wang et al., 2025d ) | 3B | – | – | 14.2 |
| JoyAI-VL-Interaction | 8B+4B | – | – | 17.0 |
| Method | Explicit proactive tasks | Implicit proactive tasks | Context QA | Overall | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OR | AP | TRU | OL | OSC | EOL | EOSC | AR | Avg. | OFR | IFR | NAR | TU | Avg. | ORC | TRC | Avg. | ||
| Existing Methods | ||||||||||||||||||
| MMDuet | 6.7 | 9.6 | 17.1 | 9.8 | 3.8 | 5.6 | 8.6 | 7.5 | 8.6 | 9.5 | 7.3 | 48.0 | 68.5 | 33.3 | 17.4 † | 23.1 † | 20.3 † | 17.8 † |
| Qwen2-VL | 12.6 | 12.0 | 14.8 | 28.3 | 7.1 | 14.2 | 15.8 | 10.1 | 14.4 | 16.5 | 18.7 | 54.5 | 61.0 | 37.7 | 13.0 † | 7.7 † | 10.4 † | 21.3 † |
| MiniCPM-V | 13.2 | 15.5 | 16.2 | 24.9 | 6.5 | 11.7 | 11.6 | 12.3 | 14.0 | 14.3 | 20.0 | 44.2 | 60.9 | 34.8 | 24.6 | 27.8 | 26.2 | 22.9 † |
| VideoLLM-EyeWO | 19.4 | 24.4 | 18.7 | 23.1 | 16.1 | 20.4 | 20.9 | 19.9 | 20.4 | 23.6 | 26.3 | 52.4 | 34.8 | 34.3 | 39.4 | 54.9 | 47.1 | 28.2 |
| Reader | Memory source | EPM | ASI | EPM+ASI Avg. | Gain (pp) |
|---|---|---|---|---|---|
| Qwen3.8-27B | OneStreamer PHCM | 41.1 | 59.5 | 50.3 | — |
| RoboBeats-gated | 46.8 | 66.2 | 56.5 | +6.2 | |
| OneStreamer-4B | OneStreamer PHCM | 44.8 | 59.5 | 52.1 | — |
| RoboBeats-gated | 48.1 | 68.2 | 58.2 | +6.1 |
| Reader | Memory source | Writer size | Writer input [-1pt](M tokens/h) | EPM | ASI | EPM+ASI Avg. |
|---|---|---|---|---|---|---|
| Qwen3.8-27B | All frames | 27B | 0.338 | 67.7 | 72.3 | 70.0 |
| Most recent 16 frames | — | 0 | 65.0 | 63.5 | 64.2 | |
| OneStreamer PHCM | 4B | 23.8 | 65.3 | 68.2 | 66.8 | |
| RoboBeats-gated | 27B | 0.186 | 62.3 | 70.9 | 66.6 | |
| OneStreamer-4B | All frames | 27B | 0.338 | 65.3 | 66.9 | 66.1 |
| Most recent 16 frames | — | 0 | 62.3 | 64.2 | 63.2 |
| Method | Image | Object | Language | Overall |
|---|---|---|---|---|
| Literature results reported in ReSight | ||||
| DynaMem ( Liu et al., 2025 ) | 16.1 | 7.5 | 5.1 | 9.6 |
| VLMaps ( Huang et al., 2023 ) | 43.3 | 69.7 | 61.0 | 58.8 |
| LagMemo ( Zhou et al., 2025 ) | 56.4 | 88.4 | 66.8 | 70.8 |
| OpenFunGraph ( Zhang et al., 2025a ) | 53.4 | 67.1 | 69.0 | 63.1 |
| HOV-SG ( Werby et al., 2024 ) | – | 70.8 | 82.3 | 76.5 |
| Method | Binary (%) | Text (%) | Position error (m) | 10 m (%) | Time error (min) |
|---|---|---|---|---|---|
| Literature results reported in ReSight | |||||
| ReMEmbR-2B ( Anwar et al., 2024 ) | 43.2 | – | 53.47 | – | 2.287 |
| ReMEmbR-8B | 46.3 | – | 55.89 | – | 4.106 |
| ConceptGraphs ( Gu et al., 2023 ) | 29.9 | – | 111.29 | – | – |
| DAAAM-Mini ( Gorlo et al., 2026a ) | 71.1 | – | 41.75 | – | 1.792 |
| Sol execution configurations | |||||
| Method | Spatial | Temporal | Multi-goal | Overall | Tokens/q (k) |
| Capability-construction episodes: 91–100, 596 queries | |||||
| Literature results from RAVEN, as reported in ReSight | |||||
| Random | 13.4 | 6.9 | 0.5 | 9.2 | – |
| Embedder Only | 31.0 | 8.1 | 3.4 | 18.8 | – |
| VLM Only-Gemini | 36.0 | 28.1 | 2.2 | 28.2 | – |
| ReMEmbR-Gemini ( Anwar et al., 2024 ) | 36.7 | 17.1 | 4.5 | 25.0 | – |
| Success | Tool calls | Tokens/run (k) | ||||
| Task | w/o EDM | w/ EDM | w/o EDM | w/ EDM | w/o EDM | w/ EDM |
| A. Capability-construction tasks | ||||||
| Warmth Corner (8F) | 2/5 | 5/5 | 8.4 | 1.0 | 60.0 | 10.4 |
| Farthest point (8F) | 1/5 | 5/5 | 7.2 | 1.0 | 38.8 | 10.0 |
| Booth count (8F) | 0/5 | 5/5 | 7.6 | 1.0 | 50.2 | 12.9 |
| B. Subsequent tasks: a new query (805) and new memory (1F) | ||||||
| Question / metric | Codex A | Codex B | LIFE |
|---|---|---|---|
| Question-level scores | |||
| Two cups | |||
| Cushion handover and entry | |||
| Jacket location (asked at 12:30) | |||
| Early-morning visitor | |||
| First identified arrival | |||
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Size | Per-source AUROC | Overall | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Charades | TSU | Edinburgh | IKEA | Perception Test | Ego4D | EGTEA | EPIC | AUROC | AP | Recall | ||
| General VLMs | ||||||||||||
| Qwen3-VL-4B | 4B | 0.726 | 0.659 | 0.847 | 0.447 | 0.706 | 0.720 | 0.784 | 0.705 | 0.683 | 0.738 | 37.1% |
| Qwen3-VL-8B | 8B | 0.727 | 0.686 | 0.849 | 0.404 | 0.709 | 0.711 | 0.811 | 0.702 | 0.680 | 0.739 | 39.3% |
| InternVL3.5-8B | 8B | 0.728 | 0.668 | 0.799 | 0.443 | 0.724 | 0.731 | 0.812 | 0.728 | 0.670 | 0.730 | 36.2% |
| MiniCPM-V-4.5 | 8B | 0.727 | 0.686 | 0.876 | 0.407 | 0.697 | 0.712 | 0.780 | 0.690 | 0.683 | 0.732 | 34.0% |
| Method | Readout | [email protected] | [email protected] | [email protected] | mIoU |
|---|---|---|---|---|---|
| Offline methods, published results | |||||
| 2D-TAN ( Zhang et al., 2019 ) | Offline | 57.3 | 45.8 | 27.9 | 41.0 |
| MMN ( Wang et al., 2021 ) | Offline | 65.4 | 53.3 | 31.5 | 46.5 |
| TimeSuite ( Zeng et al., 2024 ) | Offline | 69.9 | 48.7 | 24.0 | – |
| FV-Action (Qwen2.5-VL-7B) ( Huang et al., 2026 ) | Offline | 70.1 | 56.8 | 29.7 | – |
| UniVTG ( Lin et al., 2023 ) | Offline | 72.6 | 60.2 | 38.6 | 52.2 |
| Method | s | s | s |
|---|---|---|---|
| Published, original protocols: closed-source | |||
| Claude Fable 5 | 71.6 | – | – |
| GPT-6 | 73.2 | – | – |
| Gemini 3.7 Flash | 80.0 | – | – |
| Published, original protocols: open-source | |||
| Qwen3.5-4B (ProAgent base) | 11.6 | – | – |
| Method | Size | T-score@5 | T-F1@5 | Ego-Exo4D T-F1@5 | HTStep T-F1@5 |
|---|---|---|---|---|---|
| Published Results | |||||
| StreamingVLM | 7B | 4.8 | 3.9 | — | — |
| Qwen2.5-VL | 7B | 41.8 | 10.0 | — | — |
| Qwen3-VL | 8B | 40.5 | 10.7 | — | — |
| MMDuet2 | 3B | 13.9 | 14.2 | — | — |
| MiniCPM-o 4.5 | 9B | — | 15.4 | — | — |
| Method | Reader | Overall (%) | Question type | Observation duration (min) | |||||
|---|---|---|---|---|---|---|---|---|---|
| Temporal localization (247) | Event order & repetition (202) | Object/person changes (169) | (33) | 10–15 (154) | 15–20 (240) | (191) | |||
| Native end-to-end answering | |||||||||
| JoyAI-VL-Interaction | Native | 24.1 | 7.3 | 27.7 | 44.4 | 42.4 | 24.0 | 22.5 | 23.0 |
| OneStreamer | Native | 21.8 | 9.3 | 19.3 | 43.2 | 36.4 | 19.5 | 21.7 | 21.5 |
| Direct frames, no memory | |||||||||
| Recent context (32 s) | Qwen3.8-27B | 13.6 | 1.2 | 16.3 | 28.4 | 39.4 | 9.1 | 12.9 | 13.6 |
| Reader | Memory source | EPM | ASI | HLD | BT Avg. |
|---|---|---|---|---|---|
| Text memory only | |||||
| All frames | 51.18 | 64.19 | 98.39 | 71.25 | |
| OneStreamer PHCM | 44.78 | 59.46 | 99.46 | 67.90 | |
| OneStreamer-4B | RoboBeats-gated | 48.15 | 68.24 | 98.39 | 71.59 |
| All frames | 53.20 | 66.89 | 88.71 | 69.60 | |
| OneStreamer PHCM | 41.08 | 59.46 | 85.48 | 62.01 | |
| Method | BT Avg. | EPM | ASI | HLD | Input tokens (M) | Output tokens (M) |
|---|---|---|---|---|---|---|
| DeltaStreamer | 59.19 | 52.86 | 68.24 | 56.45 | 2,925.2 | 63.2 |
| DeltaStreamer + RoboBeats | 56.00 | 54.88 | 64.19 | 48.92 | 1,278.0 | 32.1 |
| Field | Content |
|---|---|
| ref | Frame reference identifying the stored image. |
| t | Capture time. |
| pose | Capture position, body heading, head rotation, and map. When the writer does not supply a pose, the pose timeline is queried at the capture time. |
| source , trigger | How the frame was written (route sampling, on-site user confirmation, or proactive perception) and the reason, such as a state change, an event, state maintenance, a fired condition, or a person’s arrival. |
| event | Activity associated with the write: type, identifier, and text, such as a state or event description, the original condition, or a navigation task. |
| segment , tracks | Perceptual state segment covering the capture time and the tracks of people present. |
Continual Robot Learning
Continual robot learning develops robots that acquire new skills and adapt to changing conditions while retaining previously learned behavior.
See all 96 papers