Multimodal Understanding

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, up 233% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 413

All topics

Orca: The World is in Your Mind
Jun 29, 2026Yihao Wang, Yuheng Ji, Mingyu Cao +54World Model Learning Multimodal Pretraining
Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models
Jun 29, 2026Kai Jiang, Ruishu Zhu, Siqi Huang +2Visual Question Answering Multimodal Large Language Models
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University · Institute of Artificial Intelligence, China Telecom (TeleAI) · 3Fudan University +1
Consistency as Inductive Bias: Learning Cross-View Invariance for Robust Multimodal Reasoning
Jun 29, 2026Xin Zou, Haolin Deng, Yibo Yan +6Multi-View Consistency Multimodal Robustness
HKUST (GZ) · HKUST · UIC +1
Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation
Jun 29, 2026Xin Zou, Haolin Deng, Yibo Yan +5Multimodal Large Language Models Multimodal Hallucination
HKUST (GZ) · HKUST · OPPO AI Center
Reliability-Prioritized Fine-Grained Generation in Multimodal Large
Jun 28, 2026Xiaomeng Fan, Wei Wu, Yuwei Wu +9Direct Preference Optimization Multimodal Large Language Models
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University · Department of Electrical and Computer System Engineering, Monash University
MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein
Jun 28, 2026Hong-Han Wang, Yuntao Wang, Hu DingLarge Vision-Language Models VLM Reasoning
University of Science and Technology of China, Hefei, China
Personalizing MLLMs via Reinforced Multimodal Reference Game
Jun 27, 2026Deepayan Das, Davide Talon, Yiming Wang +2Multimodal Large Language Models Personalized Language Models
University of Trento, Trento, Italy · Fondazione Bruno Kessler, Trento, Italy
COMPASS: Grounding Composition-Intent Guidance in Unified Multimodal Models
Jun 27, 2026Ziqi Zhou, Weize Quan, Mining Tan +6Multimodal Grounding Unified Multimodal Models
University of Edinburgh, United Kingdom · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China +1
Understanding How MLLMs Describe Artworks Using Token Activation Maps
Jun 26, 2026Nicola Fanelli, Pasquale De Marinis, Raffaele Scaringi +3VLM Interpretability Vision-Language Grounding
Department of Computer Science University of Bari Aldo Moro Bari, Italy
MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy
Jun 26, 2026Zhiyuan Han, Beier Zhu, Wenwen Tong +8RL for Language Model Reasoning Multimodal Large Language Models
University of Science and Technology of China · 2SenseTime Research · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +1
Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards
Jun 25, 2026Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar +5Unified Multimodal Models Multimodal Generation
1Mohamed bin Zayed University of Artificial Intelligence · 4Linköping University · 3Australian National University +1
ViQ: Text-Aligned Visual Quantized Representations at Any Resolution
Jun 25, 2026Xumin Yu, Zuyan Liu, Zhenyu Yang +5Visual Representation Learning Multimodal Pretraining
Tencent HY Vision Team · Tsinghua University · Institute of Automation, CAS +1
See & Sniff: Learning Visuo-Olfactory Representations
Jun 25, 2026Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu +2Representation Learning Cross-Modal Representation Learning
Korea Advanced Institute of Science and Technology, Korea · Hankuk University of Foreign Studies, Korea · Ulsan National Institute of Science and Technology, Korea
Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
Jun 25, 2026Jinyu Liu, Xincheng Shuai, Henghui Ding +1Unified Multimodal Models Multimodal Foundation Models
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
Bridging Vision and Language Concepts through Optimal Transport Semantic Flow
Jun 25, 2026Chenyang Zhang, Anqi Dong, Guangming Zhu +4Concept Bottleneck Models VLM Interpretability
School of Computer Science and Technology, Xidian University, Xi’an, China · KTH Royal Institute of Technology, Stockholm, Sweden
SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context
Jun 25, 2026Qinkai Zhang, Yanyan Zhao, Xin Lu +3LLM Personalization Multimodal Large Language Models
Harbin Institute of Technology
Position Rebinding Cache Reuse: Replay-Free Visual Revisiting for Interleaved Multimodal Reasoning
Jun 25, 2026Mengzhao Wang, Yanli Ji, Wangmeng Zuo +2Efficient Multimodal Inference KV Caching
Sun Yat-sen University · Shenzhen Loop Area Institute · Harbin Institute of Technology (HIT) +3
What We are Missing in Multimodal LLM Evaluation?
Jun 24, 2026Po-han Li, Shenghui Chen, Sandeep Chinchali +1Spatiotemporal Reasoning Multimodal Large Language Models
The University of Texas at Austin, USA
Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models
Jun 24, 2026Akshay Paruchuri, Sanmi Koyejo, Ehsan AdeliMultimodal Robustness LLM Auditing
Stanford University
TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs
Jun 24, 2026Yu-Yang Chen, Lan-Zhe GuoSpatial Reasoning Benchmarks Visual Reasoning
School of Intelligence Science and Technology, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
Jun 24, 2026Haoxiang Sun, Tao Wang, Li Yuan +2Vision-Language Models Multimodal Large Language Models
School of Computer Science, Sichuan University, No. 24 South Section 1, Yihuan Road, Chengdu, 610065, Sichuan, China · School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School, 2199 Lishui Road, Shenzhen, 518055, Guangdong, China · Institute of Artificial Intelligence (TeleAI), China Telecom and Northwestern Polytechnical University, 127 Youyi West Road, Xi’an, 710072, Shaanxi, China
SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity
Jun 24, 2026Tianchen Guo, Chen Liu, Ling Chen +1Visual Question Answering Multi-View Consistency
The University of Queensland · Follow Me AI Pty LTD · University of Technology Sydney +1
Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
Jun 24, 2026Zhiyuan Han, Beier Zhu, Wenwen Tong +6Multimodal Reasoning Multimodal Hallucination
University of Science and Technology of China · SenseTime · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +1
iFLYTEK-Embodied-Omni Technical Report
Jun 24, 2026Yuan Zhang, Jingfei Ni, Guanchen Lu +12Unified Multimodal Models Robot Motion Generation
1iFLYTEK · University of Science and Technology of China · 2LindenBot
ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering
Jun 23, 2026Zhentao Guo, Chen Duan, Tongkun Guan +3Multimodal Large Language Models Video QA
Meituan · Shanghai Jiao Tong University
S1-Omni-Image: A Unified Model for Scientific Image Understanding, Generation, and Editing
Jun 23, 2026Qingxiao Li, Zikai Wang, Qingli Wang +1AI for Science Unified Multimodal Models
XScience Lab, Wenge AI
AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression
Jun 23, 2026Yijing Chen, Wenhui Tan, Xiaoyi Yu +7Audio-Visual Understanding Multimodal Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China · Huawei Inc.
Latent Visual States for Efficient Multimodal Reasoning
Jun 23, 2026Xiuwei Chen, Wentao Hu, Yongxin Wang +8Efficient Multimodal Inference Large Vision-Language Models
Sun Yat-sen University · The Hong Kong Polytechnic University · MBZUAI +2
4DVLT: Dynamic Scene Understanding with Worldline-Centered Vision-Language Tracking
Jun 21, 2026Chaoyue Li, Boxue Yang, Shengyao Zhou +3Visual Object Tracking Vision-Language Grounding
Huazhong University of Science and Technology · Shanghai Jiao Tong University · Zhejiang University
Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
Jun 21, 2026Zhuoran Jin, Kejian Zhu, Hongbang Yuan +5Multimodal Reasoning VLM Reasoning
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Jun 29, 2026, cs.CVJ/K move · Enter open · S save

Orca: The World is in Your Mind

Yihao Wang, Yuheng Ji, Mingyu Cao, Yanqing Shen+53

World Model Learning Multimodal Pretraining Latent World Models Multimodal Foundation Models Embodied World Models Multimodal Understanding

Open paper AskPDF arXiv

We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a general world foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.

Momentum

Related topics

Latest papers 413