Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Task

Yurui Dong; Ziyue Wang; Shuyun Lu; Dairu Liu; Xuechen Liu; Fuwen Luo; Peng Li; Yang Liu

cs.CVMar 16, 2026

Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Task

Authors: Yurui Dong, Ziyue Wang, Shuyun Lu, Dairu Liu, Xuechen Liu, Fuwen Luo, Peng Li, Yang Liu

Organizations: Fudan University, Shanghai, China · Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · School of Computer & Communication Engineering, University of Science and Technology Beijing, Beijing, China · College of Software, Nankai University, Tianjin, China ·

Abstract

Multimodal Large Language Models (MLLMs) have recently made rapid progress toward unified Omni models that integrate vision, language, and audio. However, existing environments largely focus on 2D or 3D visual context and vision-language tasks, offering limited support for temporally dependent auditory signals and selective cross-modal integration, where different modalities may provide complementary or interfering information, which are essential capabilities for realistic multimodal reasoning. As a result, whether models can actively coordinate modalities and reason under time-varying, irreversible conditions remains underexplored. To this end, we introduce EscapeCraft-4D, a customizable 4D environment for assessing selective cross-modal perception and time awareness in Omni models. It incorporates trigger-based auditory sources, temporally transient evidence, and location-dependent cues, requiring agents to perform spatio-temporal reasoning and proactive multimodal integration under time constraints. Building on this environment, we curate a benchmark to evaluate corresponding abilities across powerful models. Evaluation results suggest that models struggle with modality bias, and reveal significant gaps in current model's ability to integrate multiple modalities under time constraints. Further in-depth analysis uncovers how multiple modalities interact and jointly influence model decisions in complex multimodal reasoning environments.

Figures & tables

Figure 1: Our EscapeCraft-4D environment setup. It incorporates auditory, visual, and time-aware clues to evaluate multimodal reasoning. The system is designed to test the abilities of agents to coordinate cross-modal perception, such as leveraging spatially grounded auditory cues and visually presented clues or making time-aware decisions under strict temporal constraints. This figure illustrates the sequencially arranged tasks according to three stages: exploration, time-aware search, and finally escape, with dynamic auditory cues playing central roles in guiding the decision-making process. The trigger system dynamically introduces time-aware clues that challenge the agent to act within limited timeframes, showcasing the integration of auditory and visual modalities.

Figure 2: Examples of our designed levels with successful runs from GPT-4o. (a) Basic game design included both auditory and visual clues. (b) Misleading levels included distractors which would play misleading auditory clues. Models should distinguish which auditory clue was genuinely useful and which was the distractor. (C) Time-aware level design included clues only visible in limited time. Models should find these clues in the time limit or the clue would disappear.

Category	Specification / Value
Forward Speed	2.0 m/s
Rotation Speed	60 ∘ /s
Grab/Trigger	0.5 s (fixed cost)
Audio Types	Examples
Ambient Sound	Wind sound from exit
Auditory Clues	Passwords or prompts

Table 1: Time-cost Configuration and Audio Categories

Level Type	Scenes	Total Objs	Avg/Scene
Difficulty-1	11	152	13.82
Difficulty-2	11	151	13.73
Difficulty-3	11	186	16.91
Difficulty-2-M	11	188	17.09
Difficulty-3-M	11	192	17.45
Difficulty-2-T	11	163	14.82

Table 2: Statistical Summary of MM-Escape4D

Models	Difficulty-1				Difficulty-2						Difficulty-3							AVG ER (%) $\uparrow$
Models	ER (%) $\uparrow$	Steps $\downarrow$	GSR (%) $\uparrow$	GR (%)	ER (%) $\uparrow$	Steps $\downarrow$	GSR (%) $\uparrow$	GR (%)	TSR (%) $\uparrow$	TR (%)	ER (%) $\uparrow$	Prop (%) $\uparrow$	Steps $\downarrow$	GSR (%) $\uparrow$	GR (%)	TSR (%) $\uparrow$	TR (%)	AVG ER (%) $\uparrow$
Agentic Approach
GPT-4o Omni-Agent	90.91	16.82	48.52	24.58	81.82	38.82	12.75	33.66	55.30	9.15	45.45	45.45	59.27	8.39	34.73	34.85	24.09	72.73
Omni Models
Gemini3-Pro-Preview	100.00	16.82	59.25	21.95	72.73	27.73	37.12	22.67	81.82	6.42	27.27	27.27	50.36	7.82	29.69	55.11	4.52	66.67
Qwen3-Omni-Thinking	63.64	24.00	24.80	54.97	27.27	56.73	5.50	29.97	39.86	22.24	0.00	0.00	81.00	0.00	27.27	42.29	19.53	30.30

Table 3: Results of basic game setting. Prop: Prop Gain; Steps: average steps used to complete the game; GSR: Grab Success Rate, the precision of grabbing; GR: Grab Ratio, the portion of grabbing actions regarding the total consumed steps. TSR: Trigger Success Rate. TR: Trigger Ratio. Note that Difficulty-1 and Difficulty requires no prop, and the prop gain is therefore omitted, and so as TSR and TR The max allowed steps are 50, 65, 80 for Difficulty-1, -2, -3 respectively. The best score of each metrics is bolded and the second is underlined . The GPT-4o Omni Agent refers to our agentic approach built on GPT-4o models, gpt-4o for vision and language, together with gpt4o-audio-preview for audio. We use Qwen3-Omni-30B-A3B version for Qwen3-Omni models in this table.

Models	Time-Aware	Misleading Levels
Models	TCSS $\uparrow$	MAT $\downarrow$	AMR $\downarrow$	D2-M-ER $\uparrow$	D3-M-ER $\uparrow$	D2-M-Steps $\downarrow$	D3-M-Steps $\downarrow$
Agentic Approach
GPT-4o Omni-Agent	67.95%	63.64%	100.00%	72.73%	63.64%	40.55	60.09
Omni Models
Qwen3-Omni-Thinking	55.87%	59.09%	23.08%	0 0.00%	0 0.00%	66.00	81.00
Gemini-3-Pro-Preview	47.04%	40.91%	22.22%	63.64%	63.64%	28.82	53.82

Table 4: Comprehensive Experimental Results on MM-Escape4D. (I) Performance on Time-aware Levels measured by TCSS. (II) Robustness against Misleading Auditory cues: MAT refers to ratio of scenes where m isleading a udio was t riggered; AMR represents the ratio of scenes where the agent was deceived given that the misleading cue was triggered, within 3 following steps. Dn-M-ER and Dn-M-Step denote the Escape Rate and average steps for Difficulty-n Misleading levels. The best and second-best results are highlighted in bold and underline , respectively.

Figure 3: Path density analysis for different levels in our MM-Escape4D benchmark. Exits of different scenes are aligned as mentioned in Section 5.1 The heatmaps represent the log-normalized density ( $\log(1+\text{density})$ ) of agent positions, with blue gradients for successful runs and red for failed ones. The dashed contours represent the spatial distribution of key items (e.g., recorders for auditory clues and boxes).

Figure 4: Analysis of Ambient Sound via Audio and No-Audio settings. (A-B) Path density heatmaps showing trajectory concentrations, using the same spatial aligning method as in Figure 3 . (C) Delta distributions for sign-aligned metrics ( Steps, Timer, Frechet, PathLen ); the red median lines below zero indicate the superiority of the audio condition. (D) The winning rate of audio guidance across nine navigation metrics. Metrics where lower values are better: Steps , Time , PathLen , Turn , and Frechet . Efficiency metrics where higher values are preferred: ProgEff , PathEff , and Mono . (E) Mean distance-to-exit over normalized episode progress; the shaded area represents

\pm 1

standard deviation. (F) Kaplan-like survival plot showing the percentage of models that have not yet reached the 1.5m exit threshold. Statistical significance for AUC and endpoint success is determined via paired Wilcoxon tests (

*p<0.05

indicates a statistically significant difference).

Figure 5: Paired HII comparison under auditory conditioning for Difficulty-2 and Difficulty-3. Each row corresponds to one difficulty level. Left column: unpaired success vs fail comparison in audio-active segments. Middle and right columns: paired audio vs non-audio comparison within success and fail runs. Each dot is one run-level

\mathrm{HII}_{\mathrm{norm}}

; lines connect paired points from the same run; error bars show mean

\pm

standard deviation. The figure shows consistent concentration increase during audio-active periods, while success vs fail separation in audio periods is significant in Difficulty-2 and not significant in Difficulty-3.

* Note: $P<0.05$ indicates a statistically significant difference.
Test	HII				JS Distance		Visual Effects
Difficulty	$P_{A\text{ vs }NA}^{s}$	$P_{A\text{ vs }NA}^{f}$	$P_{aud}^{s/f}$	$P_{non-aud}^{s/f}$	$P^{\mathrm{sf}}_{\mathrm{L_{1}},\,\mathrm{A\ vs\ NA},\,S}$	$P^{\mathrm{sf}}_{\mathrm{JSD},\,\mathrm{A\ vs\ NA},\,F}$	$P^{\mathrm{Mantel}}_{\mathrm{box\leftrightarrow path}}$
Difficulty-1	–	–	–	0.3612	–	–	–
Difficulty-2	0.0004*	0.0215*	0.0088*	0.2388	0.00033*	0.00033*	–
Difficulty-3	0.0039*	0.0028*	0.4001	0.3568	0.00033*	0.00033*	0.0047*

Table 5: P-values for the hypothesis tests used .

Test Symbol	Test Type	What Is Being Tested
$P^{\mathrm{AUC}}_{\mathrm{A\ vs\ NA}}$	Paired Wilcoxon signed-rank Appendix C.2	Whether audio changes the mid-trajectory distance-to-exit behavior between Difficulty-1 audio and non-audio settings (AUC on normalized progress window).
$P^{\mathrm{end}}_{\mathrm{A\ vs\ NA}}$	Paired Wilcoxon signed-rank Appendix C.2	Whether audio changes endpoint not-reached status between Difficulty-1 audio and non-audio settings (Kaplan-like endpoint comparison).
$P^{\mathrm{perm}}_{\mathrm{L1}}$ , $P^{\mathrm{perm}}_{\mathrm{JSD}}$	Point-label permutation test Appendix C.3	Whether the density between difficulty-1 audio/non-audio settings changes significantly using $L_{1}$ and $JS$ distance.
$P^{\mathrm{A\ vs\ NA}}_{\mathrm{S/F}}$	Paired Wilcoxon signed-rank Appendix C.2	Within successful/failed runs, whether HII behaviors change among audio/non-audio segments.
$P^{s/f}_{aud}$ / $P^{s/f}_{non\text{-}aud}$	Mann-Whitney U test Appendix C.1	Whether audio-active-segment/non-audio-segment HII differs between success and fail runs.
$P^{\mathrm{sf}}_{\mathrm{JSD},\,\mathrm{A\ vs\ NA},\,S/F}$	Paired sign-flip permutation Appendix C.5	Whether density between audio and non-audio segments changes significantly within successful/failed runs using $JS$ distance.

Table 6: Tests used for analysis.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Models	GPT-4o Omni Agent	Gemini3-Pro-Preview	Qwen3-Omni-Thinking
Scores: $C_{IO}(GSR)$	37.08 (8.39)	53.75 (7.82)	27.55 (0.00)

Appendix

Table 7: Consistency compared with GSR on Difficulty-3.

Instruction Prompt

You find yourself locked inside a room, and your ultimate goal is to escape the room. i.e. the room escape game.

You can explore the room, interact with objects, inspect items, and resolve puzzles. You can adopt the following actions to explore the room and interact with objects:

Operation Prompt

- move_forward: float, ranged between [-10, 10]. This is the number of meters you want to move forward (negative value means moving backward).

- rotate_right: float, ranged between [-180, 180]. This is the number of degrees you want to turn right (negative value means turn left).

Appendix

Table 8: The System Prompt

Interaction Result

{interaction_result}

===

Inventory

The items in your bag usable include:

{bag_desc}

===

Appendix

Table 9: The Step Prompt

Consistency Evaluation Prompt
You are a reasoning consistency evaluator for a multimodal agent benchmark. Your task is to determine whether the agent’s intention (described in the “rationale”) matches the actual interaction result (described in the “response”).

Specifically, you are given:
- The agent’s rationale for why it attempted an interaction, describing its goal or belief about the current environment.
- The response from the environment after the interaction, which includes the actual result (e.g., what item was interacted with and what was obtained).

Appendix

Table 10: The Consistency Evaluation Prompt

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
Jan 29, 2026Sangyun Chung, Se Yeon Kim, Youngchae Chee +1Multimodal Robustness Hallucination in Language Models
Integrated Vision Language Lab, KAIST, South Korea
C $^3$ PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
Aug 5, 2026Swapnanil Mukherjee, Agyeya Negi, Tanuja Ganu +1Modality Imbalance Multimodal Reasoning
Microsoft Research India · IIIT Hyderabad, India
Unlocking Cognitive Capabilities and Analyzing the Perception-Logic Trade-off
Feb 27, 2026Longyin Zhang, Shuo Sun, Yingxu He +15VLM Reasoning Multimodal CoT Reasoning
Institute for Infocomm Research (I2R), A*STAR, Singapore

Jan 29, 2026J/K move · Enter open · S save

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Sangyun Chung, Se Yeon Kim, Youngchae Chee, Yong Man Ro

Integrated Vision Language Lab, KAIST, South Korea

Multimodal Robustness Hallucination in Language Models Multimodal Large Language Models Multimodal Hallucination Contrastive Decoding Multimodal Understanding

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8% and 2.0% improvements for VideoLLaMA2-AV, 8.7% and 4.7% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at https://github.com/top-yun/MAD

Abstract

Multimodal Reasoning

Explore similar work

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

C3^33PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models

Unlocking Cognitive Capabilities and Analyzing the Perception-Logic Trade-off

C $^3$ PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models