cs.CVSep 16, 2026

Can MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model

Authors: Haoyu Zhao, Zihao Zhao, Tianyu Deng, Ziqin Xu, Zihao Zhang, Xudong Wang, Jinxiang Guo, Chen Gao, 

Organizations: National University of Singapore · Fudan University · Tencent

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent Omni-Modal Generative Models (Omni-Models) have advanced content generation toward unified modeling of text, images, video, and audio. MiniMax-H3 exemplifies this transition by combining multimodal context understanding with joint audio-visual generation in a shared latent framework. Its unified architecture raises a fundamental question: Can multimodal alignment improve the model's world reasoning, and what new evaluation paradigms do omni-modal inputs enable? To investigate this question, this work introduces a comprehensive evaluation framework organized around four complementary dimensions of physical world reasoning. Unlike existing evaluation frameworks for video generation and world models, which are often constrained by limited input modalities and evaluation settings where prompts closely match the target video content, our evaluation is specifically designed to exploit the multimodal inputs of Omni-Model. We construct a diverse set of novel tasks that require models to integrate complementary information across modalities. Specifically, we consider four scenarios, including implicit prompts paired with multiple frames, audio-image, prefix-videos, and audio-video inputs. Every single modality provides only partial evidence about the underlying event, requiring the model to jointly reason over the complementary semantic cues to infer latent event states and future dynamics. Across 517 evaluation instances, MiniMax-H3 achieves an overall success rate of 41.97%. Video-based Decision Reasoning yields the highest success rate at 56.00%, while Audio-based Disambiguation Reasoning is the weakest, reaching only 27.40%. These results indicate that effective multimodal integration remains key to fully exploiting the benefits of diverse input modalities. The project is available at https://github.com/gulucaptain/MiniMax-H3-Reason.

Figures & tables

Figure 1: Overview of our evaluation for the reasoning capability of MiniMax-H3 . First, we illustrate how MiniMax-H3 progresses from world observation and modality perception to omni-modal model training, where the acquired multimodal knowledge is transformed into task-oriented decision-making through omni-modal reasoning. Second, we construct implicitly paired multimodal data to systematically evaluate it across four complementary dimensions of physical-world reasoning.
Evaluation Sets# ExamplesInput ModalitiesPrompt TypeMulti ScenePhys. AawareReasoning
VBench ( Huang et al., 2024 )800T + IExplicit✗✗✗
TC-Bench ( Feng et al., 2024 )150T + IExplicit✗✗✗
VideoPhy ( Bansal et al., 2025 )688T + IExplicit✗✓✗
WorldModelBench ( Li et al., 2026b )350T + IExplicit✗✓✗
Ours517T + I + A + VImplicit✓✓✓
Table 1: Comparison with existing evaluation benchmarks. Our evaluation extends beyond explicit text-image conditioning by introducing omni-modal inputs and implicit prompts, enabling the evaluation of multi-scene understanding, physical awareness, and multimodal reasoning.
Figure 2: Video-audio generation pipeline of MiniMax-H3 , which is an open-weight, general-purpose, omni-modal generation model.
Figure 3: Pipeline of evaluation data construction. We build implicit multimodal condition–prompt pairs in which the information required for correct generation is not explicitly stated in the prompt, but must be inferred from complementary visual and acoustic evidence.
Figure 4: Our evaluation set consists of 4 domains and 29 subdomains, totaling 517 paired conditions.
Figure 5: MSR evaluation inputs. Household and tabletop manipulation scenes include substantial viewpoint variation, occlusion, and diverse object configurations.
Figure 6: ADR evaluation inputs. Scenes contain alternative candidate sound sources, including animals, instruments, tools, and appliances.
Figure 7: VDR evaluation inputs. Selected video frames illustrate human and animal behavior, physical interactions, puzzles, and scene-memory settings.
Figure 8: AVIR evaluation inputs. Selected frames span activities, animation, animals, and cues. Video is paired with environmental sounds or spoken constraints for continuation or localization.
MSRVDRADRAVIR
SubcategoryNNSR (%)SubcategoryNNSR (%)SubcategoryNNSR (%)SubcategoryNNSR (%)
Deformation3441.20Humans3455.88Vocalizations1631.30Activities1457.14
Cleaning2733.30Cartoons520.00Alerts2623.10Animation633.33
Articulation2634.60Animals1163.64Music1625.00Making2152.38
Arrangement2347.80Traffic9100.00Machinery3417.60Animals1030.00
Transport2334.80Physics1050.00Contact3221.90Cues2050.00
Table 2: Human-evaluated performance of MiniMax-H3 across four reasoning scenarios. Scenario-level and overall success rates are weighted by sample count. NN : number of samples. SR: success rate ( ↑\uparrow ). Blank cells indicate no additional subcategories.
Figure 9: MSR results under paired-view conditioning. Each example shows the paired views together with the corresponding prompt.
Figure 10: VDR results conditioned on prefix videos. Each row shows observed frames on the left and generated continuation frames on the right, together with the corresponding instruction.
Figure 11: ADR results conditioned on visual and acoustic inputs. Each row shows the input scene, acoustic cue, and generated frames. The cabinet-opening and typewriter examples produce visible events consistent with the supplied sounds.
Figure 12: AVIR results for discrepancy localization and audiovisual continuation. The upper examples localize visual content that conflicts with spoken constraints, while the lower examples continue the video from audiovisual inputs.
Figure 13: Representative ADR failure cases. Each row shows uniformly sampled video frames and identifies the input sound and failure type: (a) audio-visual semantic mismatch, (b) implausible visual content, (c) confusion between similar sound sources, and (d) failure to ground a small-object interaction. Red boxes highlight the regions relevant to each failure.
Figure 14: Representative VDR failure cases. Green-bordered frames indicate the observed prefix, followed by sampled continuation frames. Panels (a), (b), and (f) illustrate violations of physical constraints; (c) and (d) illustrate fine-grained manipulation errors; and (e) and (g) illustrate temporal inconsistencies in object states and scene content.
More in this topic

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 512 papers

Explore similar work

CardsList
May 21, 2026

LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose LatentOmni, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct LatentOmni-Instruct-35K, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.

Yifan Dai, Zhenhua Wu, Bohan Zeng +18

School of AI, Shanghai Jiao Tong University · 2Kling Team, Kuaishou Technology · 3Peking University +5

Cross-Modal LearningAudio-Visual UnderstandingMultimodal Large Language ModelsMultimodal CoT ReasoningMultimodal Foundation ModelsMultimodal Understanding
AskPDF
Sep 22, 2026

OmniFysics-Nano-V2 Technical Report: Understanding the Physical World Across Modalities

Omni-modal models have expanded multimodal interaction across vision, audio, speech, and language. However, their training is predominantly organized around semantic descriptions and general-purpose objectives, leaving physical attributes, interaction states, and causal mechanisms only partially specified. This gap is not simply a matter of modality coverage: adding more modalities does not by itself provide the supervision needed to connect observations with the physical structure of the world. We present OmniFysics-Nano-V2, a compact omni-modal model for physical-world perception and understanding. The model supports image, video, audio, speech, and text inputs within a shared reasoning framework, together with text and speech generation. To address the lack of explicit physical supervision, we construct a dual-branch physics-aware data pipeline that grounds salient objects in structured physical attributes and aligns visual changes with acoustic events, intermediate responses, and interaction outcomes. To address homogeneous training objectives, we curate reinforcement-learning prompts by reward diversity and adopt a two-stage Group Relative Policy Optimization curriculum that progresses from general task correctness to fine-grained physical perceptual reasoning. Experiments across multimodal, audio-visual, and physical reasoning benchmarks show that the proposed data and training strategy improves physical-world understanding while preserving broad omni-modal competence. The proposed model achieves leading result on 17 of 21 benchmarks against SOTA omni-modal models. By equipping AI systems with both omni-modal and physical-world perception capabilities, OmniFysics-Nano-V2 is poised to become a cornerstone of next-generation Physical AI.

Yizhou Liu, Jinghang Han, Kaixiang Qiu +8

Physical Superintelligence Lab, Fysics AI · College of Intelligent Robotics and Advanced Manufacturing, Fudan University

Multimodal GroundingPhysical ReasoningMultimodal Foundation Models
AskPDF
Sep 30, 2026

OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning

Recent advances have enabled unified omni-modal models in understanding audio, vision, and language. However, existing benchmarks, training data, and learning methods largely treat the modalities independently, leaving the capability of audio-visual joint reasoning poorly evaluated and insufficiently elicited. We address this gap with a benchmark, data engine, and learning method. First, we introduce OmniReasoningBench, a benchmark where both audio and visual evidence are indispensable. It comprises 1,150 multiple-choice and open-ended questions across two tasks, reasoning over video and reasoning beyond video. Second, we develop a data engine OmniQA. It automatically constructs evidence-grounded QA pairs that explicitly necessitate audio-visual joint reasoning, together with time-stamped clue chains that guide the annotation of thinking process. Besides our benchmark, this engine produces training data OmniReasoning-SFT-112K and OmniReasoning-RL-19K. Finally, we propose an on-policy self-distillation method Modality-Factored Self-Distillation (MFSD). It evaluates each sampled response under modality-specific clue contexts, disentangling the contributions of individual clues and their cross-modal interactions for token-level credit assignment. With our training data and learning method, our model OmniReasoning-30B-A3B achieves 50.0% on OmniVideoBench and 42.5% on OmniReasoningBench, improving the base model Qwen3-Omni-30B-A3B-Thinking by 12.8 and 9.3 percentage points, respectively. Moreover, it delivers substantial gains on general and long-video benchmarks, including Video-MME-v2. We hope our work offers a solid step for facilitating future research in omni-modal joint reasoning.

Junming Lin, Yuxuan Wang, Zhenxin Lei +11

School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group

On-Policy Self-DistillationMultimodal ReasoningVideo ReasoningAudio-Visual QA
AskPDF