OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering

Yiduo Jia; Muzhi Zhu; Hao Zhong; Mingyu Liu; Yuling Xi; Hao Chen; Bin Qin; Yongjie Yang

cs.CVApr 9, 2026

OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering

Authors: Yiduo Jia, Muzhi Zhu, Hao Zhong, Mingyu Liu, Yuling Xi, Hao Chen, Bin Qin, Yongjie Yang,

Organizations: Zhejiang University · Xiaomi Inc.

Abstract

To extend the reinforcement learning post-training paradigm to omni-modal models for concurrently bolstering video-audio understanding and collaborative reasoning, we propose OmniJigsaw, a generic self-supervised framework built upon a temporal reordering proxy task. Centered on the chronological reconstruction of shuffled audio-visual clips, this paradigm strategically orchestrates visual and auditory signals to compel cross-modal integration through three distinct strategies: Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking. Recognizing that the efficacy of such proxy tasks is fundamentally tied to puzzle quality, we design a two-stage coarse-to-fine data filtering pipeline, which facilitates the efficient adaptation of OmniJigsaw to massive unannotated omni-modal data. Our analysis reveals a ``bi-modal shortcut phenomenon'' in joint modality integration and demonstrates that fine-grained clip-level modality masking mitigates this issue while outperforming sample-level modality selection. Extensive evaluations on 15 benchmarks show substantial gains in video, audio, and collaborative reasoning, validating OmniJigsaw as a scalable paradigm for self-supervised omni-modal learning.

Figures & tables

Figure 2: Data filtering pipeline for efficient adaptation of OmniJigsaw. Raw videos are subjected to signal-based filtering to ensure omni-modal integrity and dynamism, followed by semantic-based screening incorporating CoT reasoning for the assessment of narrative logic and state transitions.

Method	AoT Bench	TUNA -Bench	Temp Compass	Video -TT	Video -Holmes	MLVU -Test	Video -MME	MLVU
Infer w/ Audio
Omni-R1	52.09	53.84	63.10	36.80	40.72	52.59	63.20	65.69
HumanOmniV2	48.58	49.16	63.86	40.30	42.90	49.00	65.60	66.70
Qwen3-Omni-30B	64.88	62.57	70.63	44.30	50.84	57.97	72.10	70.01
VideoJigsaw	67.45 $\uparrow$ 2.57	63.41 $\uparrow$ 0.84	72.28 $\uparrow$ 1.65	44.90 $\uparrow$ 0.60	51.99 $\uparrow$ 1.15	60.16 $\uparrow$ 2.19	72.90 $\uparrow$ 0.80	71.90 $\uparrow$ 1.89
OmniJigsaw (JMI)	66.83 $\uparrow$ 1.95	62.78 $\uparrow$ 0.21	71.08 $\uparrow$ 0.45	44.70 $\uparrow$ 0.40	50.24 $\downarrow$ 0.60	58.76 $\uparrow$ 0.79	72.90 $\uparrow$ 0.80	71.39 $\uparrow$ 1.38

Table 1: Performance comparison on video reasoning. Results are reported across two inference modes (

w/

and

w/o

audio). Bold : best; green : gains over baseline.

Method	MMAU -Pro	MMAU -test-mini	MMSU	MMAR
Omni-R1	52.36	77.70	61.87	59.70
HumanOmniV2	53.49	75.90	60.83	61.80
Qwen3-Omni-30B	56.61	74.40	70.16	68.50
AudioJigsaw	57.67 $\uparrow$ 1.06	75.40 $\uparrow$ 1.00	70.30 $\uparrow$ 0.14	70.70 $\uparrow$ 2.20
OmniJigsaw (JMI)	58.33 $\uparrow$ 1.72	74.50 $\uparrow$ 0.10	69.80 $\downarrow$ 0.36	69.10 $\uparrow$ 0.60
OmniJigsaw (SMS)	58.46 $\uparrow$ 1.85	75.80 $\uparrow$ 1.40	70.48 $\uparrow$ 0.32	69.50 $\uparrow$ 1.00

Table 2: Performance comparison on audio reasoning. Bold : best; green : gains over baseline.

Method	Daily -Omni	Intent Bench	Omni Video Bench
Omni-R1	54.14	64.18	32.40
HumanOmniV2	58.48	68.21	33.50
Qwen3-Omni-30B	69.92	67.40	38.80
OmniJigsaw (JMI)	70.26 $\uparrow$ 0.34	67.95 $\uparrow$ 0.55	40.10 $\uparrow$ 1.30
OmniJigsaw (SMS)	70.26 $\uparrow$ 0.34	68.21 $\uparrow$ 0.81	40.20 $\uparrow$ 1.40
OmniJigsaw (CMM)	71.09 $\uparrow$ 1.17	68.89 $\uparrow$ 1.49	40.50 $\uparrow$ 1.70

Table 3: Performance comparison on omni-modal collaborative reasoning. Bold : best; green : gains over baseline.

Video

Audio

Omni-Modal

Method

AoT

TUNA

TempC

V-TT

V-Holmes

MLVU-T

V-MME

MLVU

MMAU-P

MMAU-TM

MMSU

MMAR

D-Omni

Intent

OVB

OmniJigsaw

66.83

66.20

72.34

46.50

48.29

62.75

69.30

73.46

58.59

76.30

70.70

71.00

71.09

68.89

40.50

w/o Filtering

64.94

\downarrow

1.89

65.29

\downarrow

0.91

71.14

\downarrow

1.20

45.40

\downarrow

1.10

47.41

\downarrow

0.88

58.76

\downarrow

3.99

68.60

\downarrow

0.70

72.59

\downarrow

0.87

57.67

\downarrow

0.92

76.00

\downarrow

0.30

70.40

\downarrow

0.30

68.90

\downarrow

2.10

70.01

\downarrow

1.08

66.77

\downarrow

2.12

40.10

\downarrow

0.40

w/o DF

66.00

\downarrow

0.83

64.11

\downarrow

2.09

71.27

\downarrow

1.07

44.70

\downarrow

1.80

47.96

\downarrow

0.33

61.95

\downarrow

0.80

68.60

\downarrow

0.70

72.36

\downarrow

1.10

57.81

\downarrow

0.78

75.90

\downarrow

0.40

70.48

\downarrow

0.22

69.30

\downarrow

1.70

69.76

\downarrow

1.33

67.66

\downarrow

1.23

39.50

\downarrow

1.00

Table 4: Ablation study on data quality and reward discount factor across video, audio, and omni-modal benchmarks under CMM.

Figure 3: Performance comparison of JMI, CMM, and uni-modal Jigsaw across video, audio, and omni-modal benchmarks. CMM’s consistent superiority and JMI’s performance degradation relative to uni-modal Jigsaw baselines compellingly support the “bi-modal shortcut phenomenon”.

Figure 4: Comparison of CoT reasoning between CMM and JMI at training step 800. CMM (left) compels the model to jointly analyze visual and auditory cues by masking less salient modalities (dashed boxes) to create an information bottleneck, while JMI (right) exhibits a bi-modal shortcut by “solely relying [on] linguistic cues” (underlined text) and bypassing the necessary visual analysis.

Figure 5: Optimization dynamics w/ and w/o Discount Factor.

Figure 6: Task reward (w/o

R_{rep},R_{fmt}

) dynamics of JMI, CMM, VideoJigsaw and AudioJigsaw.

Figure 7: Sub-capability performance comparison between CMM and SMS across fine-grained dimensions. CMM’s predominant superiority over SMS highlights the efficacy of clip-level orchestration in capturing temporally non-uniform audio-visual cues, whereas sample-level arbitration often misses local high-value modal information.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Parameter	Value
Visual Input	Maximum Sampled Frames	200
Visual Input	Maximum Resolution (pixels)	100,352
Sampling Params	Temperature ( $\tau$ )	0 (Greedy Decoding)
	Top- $p$	1.0
	Top- $k$	-1 (Disabled)
	Repetition Penalty	1.05

Appendix

Table 5: Inference configurations for semantic screening via Qwen2.5-VL-7B-Instruct [ 1 ] .

Source Dataset	Raw Samples	After Stage 1	After Stage 2
YouCook2 [ 59 ]	2,000	327	327
FineVideo [ 5 ]	43,751	7,737	6,986
LLaVA-Video-178K [ 54 ] (NextQA)	3,868	982	907
Total	49,619	9,046	8,220

Appendix

Table 6: Data filtering statistics for OmniJigsaw-8K curation. The pipeline progressively distills massive unannotated omni-modal data into high-fidelity samples suitable for the jigsaw proxy task.

Hyperparameter	Value
Model & Infrastructure
Base Policy Model	Qwen3-Omni-30B-A3B-Instruct [ 42 ]
Hardware Infrastructure	$8\times$ NVIDIA H200 GPUs
Frozen Components	Vision Tower, Audio Tower, Router
Training Dynamics (VeRL)
Total Training Steps	1,000

Appendix

Table 7: Hyperparameter settings for OmniJigsaw GRPO post-training implemented via VeRL [ 31 ] .

Modality	Benchmark	Evaluation File
Video	AoTBench [ 43 ]	data_files/AoTBench_QA.json
	TUNA-Bench [ 12 ]	TUNA-MCQ.json
	TempCompass [ 19 ]	multi-choice/test-00000-of-00001.parquet
	Video-TT [ 53 ]	data/test-00000-of-00001.parquet
	Video-Holmes [ 3 ]	test_Video-Holmes.json
	MLVU-Test [ 58 ]	test_multi_choice_tasks.json

Appendix

Table 8: Detailed list of evaluation benchmarks and files.

Parameter	Value
Temperature ( $\tau$ )	0 (Greedy Decoding)
Top- $p$	1.0
Top- $k$	-1 (Disabled)
Max New Tokens	2,048
Repetition Penalty	1.05

Appendix

Table 9: Evaluation sampling parameter configurations. These settings are uniformly applied across all benchmark evaluations to ensure deterministic and reproducible outputs.

Panel of Figure 8 — Figure 8: Qualitative example of Sub-Scene Captioning. Comparison between the Qwen3-Omni-30B-A3B-Instruct [ 42 ] baseline and its OmniJigsaw (CMM)-post-trained variant.

Method	Temporal Perception	Spatial Perception	Attribute Perception	Action Recognition	Object Recognition	OCR Problems	Counting Problem	Temporal Reasoning	Spatial Reasoning	Action Reasoning	Object Reasoning	Information Synopsis
Qwen3-Omni-30B	80.0	70.4	77.0	69.0	76.6	74.1	46.3	57.1	78.6	57.2	65.9	80.5
OmniJigsaw (JMI)	78.2	74.1	78.8	71.2	74.6	75.5	47.4	62.7	76.8	58.6	65.9	80.8
OmniJigsaw (SMS)	80.0	72.2	80.2	71.2	74.6	74.1	48.1	61.6	73.2	59.3	67.0	80.5
OmniJigsaw (CMM)	76.4	75.9	78.8	73.8	76.3	77.0	44.4	62.1	73.2	61.8	65.9	80.8

Appendix

Table 10: Sub-capability evaluation results on Video-MME [ 7 ] . Green Bold indicates performance surpassing the baseline.

	Task-wise			Difficulty-wise
Method	Sound	Music	Speech	Easy	Hard	Medium
Qwen3-Omni-30B	76.88	72.75	73.57	70.09	70.76	77.78
OmniJigsaw (JMI)	77.18	73.65	72.67	70.54	68.22	78.89
OmniJigsaw (SMS)	78.98	73.65	74.77	70.98	68.64	80.93
OmniJigsaw (CMM)	79.28	75.15	74.47	73.66	72.03	79.26

Appendix

Table 11: Sub-capability evaluation results on MMAU-test-mini [ 28 ] . Green Bold indicates performance surpassing the baseline.

Method	Attribute Comparison	Bkg. & Music Understanding	Causal Reasoning	Counting	Ego Reasoning	Fine-grained Perception	Hypothetical Reasoning	Reference Reasoning	Relationship Reasoning	Sentiment Analysis	Spatial Understanding	Summarization	Temporal Understanding
Qwen3-Omni-30B	28.57	34.04	47.14	25.16	45.07	38.46	58.33	41.84	70.83	38.27	22.58	63.27	33.58
OmniJigsaw (JMI)	61.90	31.91	49.29	21.29	54.93	47.25	45.83	45.92	66.67	38.27	22.58	55.10	32.85
OmniJigsaw (SMS)	38.10	27.66	50.00	26.45	47.89	41.76	41.67	37.76	79.17	44.44	30.65	61.22	34.31
OmniJigsaw (CMM)	38.10	29.79	46.43	24.52	43.66	41.76	58.33	44.90	70.83	41.98	32.26	55.10	40.15

Appendix

Table 12: Sub-capability evaluation results on OmniVideoBench [ 15 ] . Green Bold indicates performance surpassing the baseline.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning
Sep 30, 2026Junming Lin, Yuxuan Wang, Zhenxin Lei +11On-Policy Self-Distillation Multimodal Reasoning
School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group
LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
May 21, 2026Yifan Dai, Zhenhua Wu, Bohan Zeng +18Cross-Modal Learning Audio-Visual Understanding
School of AI, Shanghai Jiao Tong University · 2Kling Team, Kuaishou Technology · 3Peking University +5
Omni-o3: Deep Nested Omnimodal Deduction for Deliberative Audio-Visual Reasoning
Apr 27, 2026Zhicheng Zhang, Wentao Gu, Weicheng Wang +5Audio-Visual Understanding Tree Search
Nankai University · Kuaishou Technology · Pengcheng Laboratory +1

Sep 30, 2026J/K move · Enter open · S save

OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning

Junming Lin, Yuxuan Wang, Zhenxin Lei, Yuxin Liu+10

School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group

On-Policy Self-Distillation Multimodal Reasoning Video Reasoning Audio-Visual QA

Open paper AskPDF arXiv

Recent advances have enabled unified omni-modal models in understanding audio, vision, and language. However, existing benchmarks, training data, and learning methods largely treat the modalities independently, leaving the capability of audio-visual joint reasoning poorly evaluated and insufficiently elicited. We address this gap with a benchmark, data engine, and learning method. First, we introduce OmniReasoningBench, a benchmark where both audio and visual evidence are indispensable. It comprises 1,150 multiple-choice and open-ended questions across two tasks, reasoning over video and reasoning beyond video. Second, we develop a data engine OmniQA. It automatically constructs evidence-grounded QA pairs that explicitly necessitate audio-visual joint reasoning, together with time-stamped clue chains that guide the annotation of thinking process. Besides our benchmark, this engine produces training data OmniReasoning-SFT-112K and OmniReasoning-RL-19K. Finally, we propose an on-policy self-distillation method Modality-Factored Self-Distillation (MFSD). It evaluates each sampled response under modality-specific clue contexts, disentangling the contributions of individual clues and their cross-modal interactions for token-level credit assignment. With our training data and learning method, our model OmniReasoning-30B-A3B achieves 50.0% on OmniVideoBench and 42.5% on OmniReasoningBench, improving the base model Qwen3-Omni-30B-A3B-Thinking by 12.8 and 9.3 percentage points, respectively. Moreover, it delivers substantial gains on general and long-video benchmarks, including Video-MME-v2. We hope our work offers a solid step for facilitating future research in omni-modal joint reasoning.

Abstract

Audio-Visual Understanding

Explore similar work

OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning

LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning

Omni-o3: Deep Nested Omnimodal Deduction for Deliberative Audio-Visual Reasoning