cs.CVSep 6, 2026

Generalist Open-World Temporal Perception

Authors: Cristian Sminchisescu

Organizations: Google DeepMind · Lund University · IMAR

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The next generation of artificial intelligence systems will likely be natively temporal and multimodal in both inputs and outputs: able to converse, perceive, predict, reason, and synthesize through a shared world representation. Realizing this requires a temporal perceptual substrate integrating sensory streams, language, and structured outputs within a multimodal world model. We seek a generalist open-world perceptual system that represents biological forms, natural physical structures, and artifacts, and their interactions, as a coherent, temporally persistent process. The model should infer geometry, articulation, semantics, interaction structure, and uncertainty from raw multimodal streams; maintain identity through occlusion and viewpoint change; generalize across species, forms, mechanisms, and materials; and abstain or expand its ontology when encountering the unknown. The objective is a structured world state supporting understanding, prediction, counterfactual reasoning, and controllable synthesis. Recent work suggests that some cross-modal and reasoning-like capabilities can emerge from large-scale generative video pretraining, reminiscent of language-model scaling. Yet these capabilities are often accessed through language probes or expressed through photorealistic video, leaving explicit semantic, geometric, or temporal structure largely unexposed. This paper articulates an alternative and complementary paradigm: perception and synthesis as distinct conditionings within a shared Generalist Open-World Temporal Perception Architecture (GOWTPA). Recognition, structured prediction, and simulation arise as different conditionings of the same generative substrate, while reasoning and embodiment-specific policies build upon the resulting world state. This positions generalist temporal perception as a potential foundation layer for broader multimodal intelligence and physical AI.

Figures & tables

Figure 1: Generalist Open-World Temporal Perception Architecture. GOWTPA is a conceptual multimodal architecture for investigating perception and generation as complementary conditionings of shared temporal perceptual state. Different instantiations may condition on, and predict, subsets of text, audio, haptic or proprioceptive signals, images, video, and structured semantic or geometric variables. Controls select the conditioning sources, requested outputs, and operating mode. The resulting perceptual state and readouts can support downstream reasoning, simulation, planning, and policy or control.
Figure 2: The open-world category and statistics gap. Biological diversity is highly uneven across taxonomic groups and remains only partially characterized at the level needed for perceptual modeling. The left panel shows a source-specific snapshot of reported vertebrate species across major vertebrate groups; the middle panel shows mammal species by major order using the Mammal Diversity Database MDD2 taxonomy; the right panel shows approximate described insect species by major order, not estimates of total extant insect diversity. Even within comparatively well-studied groups, taxonomic richness is highly uneven, and source checklists remain incomplete, aggregated at different levels, and subject to revision. More importantly, species names alone do not provide the quantitative structure required for perception: morphology, motion, behavior, interaction, and multimodal sensory statistics are often missing. The figure illustrates why open-world multimodal perception cannot be reduced to fixed supervised label spaces. Counts are source- and version-dependent, based on major taxonomic checklists and compilations for fishes, birds, amphibians, reptiles, mammals, and insects ( Fricke et al., 2026 ; AviList Core Team, 2025 ; AmphibiaWeb, 2026 ; Uetz et al., 2026 ; Burgin et al., 2025 ; Zhang, 2013 ) .
Figure 3: Conceptual illustration of correspondence and articulation as partially transferable perceptual structure. Left: dense correspondence can provide a reusable coordinate structure across time, pose, viewpoint, instance variation, and sometimes related categories, while higher-genus objects may require cuts, patches, or local charts. Right: articulation transfer should be understood as the reuse of motion constraints and structural motifs—such as local rigidity, joints, axial deformation, and limited degrees of freedom—rather than as the instantiation of one universal skeleton across all categories.
Figure 4: Perception and synthesis as conditional generation. A shared Temporal Foundation Model backbone can support both synthesis and perception as conditional generation over latent token fields. In synthesis mode, conditioning signals such as text, control inputs, or other modalities are combined with an initial noisy token field, and the model produces sensory outputs through iterative denoising and generative decoding. In perception mode, observed video, audio, and optional task prompts are encoded into an observed token field, and the same backbone architecture and synthesis-pretrained substrate produce structured world-state projections through efficient one-step conditional inference or, when needed, multi-step conditional sampling. Dense structured readouts include segmentation, depth, normals, and dense correspondence; sparse or entity-level readouts can include pose, tracks, entities, parts, events, or relations. The latent field, together with any persistent entity or query tokens, acts as an internal carrier of perceptual state; the decoded outputs are explicit task-conditioned projections. Neither a single latent output field nor a single decoded readout is equated with the complete world state. The common conditional form is pθ(F∣I,c)p_{\bm{\theta}}(\mathbf{F}\mid\mathbf{I},\mathbf{c}) , where I\mathbf{I} denotes the initial token field: noise in synthesis and observed sensory tokens in perception. Synthesis and perception differ mainly in initialization, sampling regime, and decoder/readout, while sharing the same temporally conditioned latent substrate. Structured perceptual layers may therefore operate in either direction: as outputs inferred from sensory evidence or as controls supplied to synthesis.
Gap addressedCapabilityStatusIntegration pathway within conditional generation
Fragmented sensory streams and output spacesMultimodal integration★Joint audio–video tokenization, shared encoders, bidirectional prediction, and reconstruction of missing modalities.
Lack of persistent explicit state within an observation episodeStructured working state★Recurrent state or query tokens updated across processing windows, with identity and evidence links maintained as context evolves.
No durable memory across separated encountersEpisodic and semantic memory❍Selective storage and retrieval, provenance-aware episodic memory, and gradual consolidation with mechanisms for controlling interference and forgetting.
Weak long-horizon temporal persistenceLong-horizon temporal prediction★Long-horizon prediction, counterfactual rollouts, temporal contrast, and intervention-style conditioning.
Insufficient metric and physical groundingGeometric and physical grounding★Depth, normals, correspondence, pose, contact, deformation, and physically informed consistency losses.
Closed-world categories and brittle transferGeneralization and adaptivity★Parameter-efficient adaptation, uncertainty-driven novelty clustering, and open-world ontology growth.
Table 1: Capabilities, unresolved gaps, current support, and integration pathways for open-world temporal perception. The table connects the earlier diagnosis of why perception remains unsolved to concrete mechanisms within a conditional–generative framework. Legend: ✓ demonstrated in the motivating systems or closely related work; ★ partially demonstrated or plausible but incomplete; ❍ future extension.
More in this topic

World Model Learning

World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.

See all 414 papers

Explore similar work

CardsList
Feb 26, 2026

The Trinity of Consistency as a Defining Principle for General World Models

The construction of World Models capable of learning, simulating, and reasoning about objective physical laws constitutes a foundational challenge in the pursuit of Artificial General Intelligence. Recent advancements represented by video generation models like Sora have demonstrated the potential of data-driven scaling laws to approximate physical dynamics, while the emerging Unified Multimodal Model (UMM) offers a promising architectural paradigm for integrating perception, language, and reasoning. Despite these advances, the field still lacks a principled theoretical framework that defines the essential properties requisite for a General World Model. In this paper, we propose that a World Model must be grounded in the Trinity of Consistency: Modal Consistency as the semantic interface, Spatial Consistency as the geometric basis, and Temporal Consistency as the causal engine. Through this tripartite lens, we systematically review the evolution of multimodal learning, revealing a trajectory from loosely coupled specialized modules toward unified architectures that enable the synergistic emergence of internal world simulators. To complement this conceptual framework, we introduce CoW-Bench, a benchmark centered on multi-frame reasoning and generation scenarios. CoW-Bench evaluates both video generation models and UMMs under a unified evaluation protocol. Our work establishes a principled pathway toward general world models, clarifying both the limitations of current systems and the architectural requirements for future progress.

Jingxuan Wei, Siyuan Li, Yuhang Xu +21

World ModelsWorld Model LearningWorld Model EvaluationUnified Multimodal Models
AskPDF
Oct 13, 2026

Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence

World models have emerged as a promising paradigm for embodied AI, yet existing systems often remain fragmented in the modalities they model and the functions they support. In particular, visual dynamics, geometric structure, and executable action are commonly coupled through separate interfaces or downstream modules rather than being jointly represented and evolved within a shared dynamics model. In this work, we present Thinker-WM, a unified multimodal world model that jointly models and generates RGB, metric depth, and robot action through a shared intermediate Unified Dynamics Transformer with modality-specific encoders and decoders. The model performs multimodal unification at the representation level: appearance, geometry, and action are transformed into a common dynamic representation and jointly evolved by shared parameters, while their distinct output structures are preserved. The same parameterization supports multiple functions, including joint video-action generation, world prediction, policy inference, action-conditioned generation, video-to-action generation, and inverse dynamics. To examine whether the shared representation captures action-related information, we introduce a lightweight motion probe that shows the frozen visual representation contains recoverable motion information. Thinker-WM is trained on a heterogeneous corpus comprising approximately 20K hours of egocentric video, 10K hours of real-robot data, and 1K hours of simulation data, including approximately 4K hours of real-robot data with aligned depth supervision. Extensive experiments demonstrate strong performance across embodied world-model evaluations. Thinker-WM ranks first on EWMBench, WorldOlympiad and WorldArena, while controlled studies further demonstrate the benefits of action conditioning, geometric interaction, and multi-function training within the shared representation.

Thinker Team, Peng Gao, Jichao Jiao +8

Ubtech

World Models for RoboticsEmbodied World ModelsUnified Multimodal ModelsAction-Conditioned World ModelsCross-Modal Representation Learning
AskPDF
May 28, 2026

World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications

World models, internal simulators that learn the structure and dynamics of an environment, have emerged as a central paradigm in the pursuit of artificial general intelligence, enabling agents to predict, plan, and reason within learned representations. Despite rapid progress across reinforcement learning, robotics, autonomous driving, and video generation, the field lacks a unified framework integrating its diverse architectural choices, training methods, reasoning mechanisms, and application settings. This survey addresses that gap with a multi-axis taxonomy organized along four dimensions: (i) architecture, encompassing representation format, dynamics formulation, input modality, learning paradigm, and downstream application; (ii) methodological family, including state-space and recurrent approaches, transformer-based models, diffusion-based generators, physics-informed networks, and language-augmented multimodal systems; (iii) reasoning strategy, covering imagination-based planning, latent policy learning, counterfactual reasoning, and planning under uncertainty; and (iv) application domain, spanning robotics, autonomous driving, video prediction, multimodal agents, reinforcement learning, scientific modeling, medical imaging, educational measurement, and business and finance. Tracing the field from early cognitive-science foundations to milestone systems such as PlaNet, the Dreamer family, MuZero, Sora, Cosmos, and Genie, we examine how these dimensions interact and highlight the recent convergence of chain-of-thought reasoning with world-model imagination. We review evaluation protocols and benchmarks, identify persistent challenges such as compounding prediction errors, sim-to-real transfer, and fragmented evaluation, and outline future directions toward unified multimodal world models, foundation-scale interactive simulators, and safe deployment in safety-critical domains.

Arif Hassan Zidan, Yi Pan, Hanqi Jiang +23

School of Computer and Cyber Sciences, Augusta University, Augusta, GA, USA · School of Computing, University of Georgia, Athens, GA, USA · Tandon School of Engineering, New York University, Brooklyn, NY, USA +9

World Model LearningWorld Model-Based PlanningLatent World ModelsWorld Model EvaluationInteractive World ModelsEmbodied World Models
AskPDF