The Topology of Multimodal Fusion: Why Current Architectures Fail at Creative Cognition

Xiujiang Tan

cs.AIApr 6, 2026

The Topology of Multimodal Fusion: Why Current Architectures Fail at Creative Cognition

Authors: Xiujiang Tan

Organizations: Guangzhou Academy of Fine Arts, Guangzhou, China · Guangzhou Academy of Fine Arts

Abstract

This paper identifies a structural limitation in current multimodal AI architectures that is topological rather than parametric. Contrastive alignment (CLIP), cross-attention fusion (GPT-4V/Gemini), and diffusion-based generation share a common geometric prior -- modal separability -- which we term contact topology. The argument rests on three pillars with philosophy as the generative center. The philosophical pillar reinterprets Wittgenstein's saying/showing distinction as a problem rather than a conclusion: where Wittgenstein chose silence, the Chinese craft epistemology tradition responded with xiang (operative schema) -- the third state emerging when saying and showing interpenetrate. A cruciform framework (dao/qi x saying/showing) positions xiang at the intersection, executing dual huacai (transformation-and-cutting) along both axes. This generates a dual-layer dynamics: chuanghua (creative transformation as spontaneous event) and huacai (its institutionalization into repeatable form). The cognitive science pillar reinterprets DMN/ECN/SN tripartite co-activation through the pathological mirror: overlap isomorphism vs. superimposition collapse in a 2D parameter space (coupling intensity x regulatory capacity). The mathematical pillar formalizes these via fiber bundles and Yang-Mills curvature, with the cruciform structure mapped to fiber bundle language. We propose UOO implementation via Neural ODEs with topological regularization, the ANALOGY-MM benchmark with error-type-ratio metric, and the META-TOP three-tier benchmark testing cross-civilizational topological isomorphism across seven archetypes. A phased experimental roadmap with explicit termination criteria ensures clean exit if falsified.

Figures & tables

Figure 1: The Three Topological Regimes of Multimodal Integration. Panel A: Contact topology (current AI)—separated manifolds with interface-only alignment. Panel B: Overlap topology (creative emergence / UOO target)—non-separable interior with persistent $\beta_{1}$ loops maintaining structural tension. Panel C: Superimposition collapse (psychosis / computational failure)—loss of transversality and topological singularity. Panel D: Corresponding persistence diagrams as quantitative diagnostic: $\tau(X)=\beta_{1}/\beta_{0}$ total persistence.

Hypothesis	$\beta_{0}$ prediction	$\beta_{1},\beta_{2}$ prediction
Rapid sequential processing	Two persistent independent components	Trivial
Overlap Zone (inseparable)	Single connected component	Persistent non-trivial loops and voids

Table 2

Figure 2: The Cruciform Structure. Xiang (tu-xiang, operative schema) occupies the intersection of two philosophical axes: the vertical dao / qi axis (metaphysical/physical) and the horizontal saying/showing axis (propositional/presentational). Xiang simultaneously executes dual huacai (transformation-and-cutting) along both axes. Four materials from the Chinese craft tradition each occupy a precise coordinate within this structure.

Figure 3: The UOO Computation Graph. Left: mathematical framework (base space $B$ , fibers $E_{b}$ , connection $\nabla$ , curvature $F_{\nabla}$ , harmonic maps). Right: computational implementation (bilinear entanglement $\to$ Neural ODE $\to$ topological regularization). Correspondence arrows map each mathematical construct to its computational realization.

Outcome pattern	Implication for theory
Cross-modal ETR significantly > single-modal ETR	Supports contact topology diagnosis
No ETR difference across conditions	Opposes reality of topological type distinction
Some architecture shows no ETR shift on cross-modal	That architecture may have partially achieved overlap topology

Table 5

Figure 4: Cognitive-Pathological Parameter Space and ANALOGY-MM Benchmark. Left: two-dimensional parameter space (coupling intensity $\times$ regulatory capacity) with four quadrants—Overlap Zone (high coupling, high regulation), Superimposition Collapse (high coupling, low regulation), Conventional Cognition (low coupling, high regulation), and Cognitive Slowing (low coupling, low regulation). $\tau(X)$ contour lines indicate predicted structural tension. Right: ANALOGY-MM cross-modal analogy workflow with 2 $\times$ 2 forced-choice design and ETR computation.

	Structure correct	Structure wrong
Content domain correct	A (correct answer)	B (ETR target error)
Content domain wrong	C (structure transfer, wrong domain)	D (random / double error)

Table 7

Figure 5: META-TOP — Three-Tier Benchmark System. Left: progressive three-tier structure testing increasingly deep cross-modal topological understanding. Tier L1 (ANALOGY-MM) tests structural mapping with ETR. Tier L2 (META-TOP Simple) tests dynamical pattern recognition with TSAS. Tier L3 (META-TOP Full) tests cross-civilizational topological isomorphism across seven archetypes. Right: the seven topological archetypes, with Archetype 7 ( chuanghua , creative transformation) as the core test—detecting the dynamic emergence of topology itself.

Phase	Scope	Timeline	Termination criterion
Phase 0	White paper revision; candidate definitions for $B$ , $G$ , $E_{b}$ ; pathological mirror integration	1–2 months	—
Phase 1	Synthetic data PoC + NS validation + synthetic TDA pipeline	1–3 months	Separable bias $\geq$ non-separable on zero-shot transfer → terminate
Phase 1.5	TDA pipeline pre-validation on public fMRI creativity datasets	1–2 months	TDA metrics fail to distinguish high/low creativity → redesign
Phase 2	Behavioral experiment 1 + ANALOGY-MM first round	3–6 months	No cross-dimension semantic propagation → reassess framework
Phase 3	Neuroimaging experiment 2 + dimensional pathological mirror test	6–12 months	No non-decomposable emergent component → fiber bundle needs alternative
Phase 4	Cross-domain isomorphism (Expts 3–4) + causal TMS	12–24 months	Topological signatures differ across modality pairs → isomorphism falsified

Table 9

Design dimension	Enhancement
Participant design	Dimensional individual-differences design : recruit 150–200 healthy participants, continuously measuring creativity (AUT originality + DAT + CAQ) and schizotypy (O-LIFE scale), using continuous regression instead of between-group ANOVA
Primary metric	Automated semantic distance (SemDis) as primary quantification; human ratings as validation only
Demand characteristic control	Low-level perturbation control condition (non-semantic perturbation) to exclude confounds
Temporal resolution	Continuous trajectory tracking (mouse trajectory/force-sensitive input) throughout 5-second response window
Sequential processing control	10Hz forced alternation control group: if inseparability signature collapses under high-frequency alternation, this evidences the Overlap Zone’s dependence on temporal co-presence
Network framework	DMN/ECN/SN tripartite analysis; SN interpreted as coupling regulator (§4.2)

Table 10

Layer	Method	Decision criterion
Behavioral	Continuous trajectory tracking	Rapid switching → discrete stepwise polyline; Overlap Zone → smooth global topological deformation
Electrophysiological	Zero-lag phase locking (MEG)	DMN and ECN must simultaneously fire Gamma bursts within the same low-frequency Theta phase slot—sharing a metronome, not alternating
State space	HMM microstate analysis	Must reveal a third dynamical attractor state whose Betti number characteristics cannot be reconstructed as linear superposition of “pure propositional” and “pure imagistic” states
Causal (information-theoretic)	Bidirectional transfer entropy	Perturbation must trigger instantaneous nonlinear global network reorganization (not sequential A→B causal chain)
Causal (model-based)	DCM (fMRI) + Granger Causality (MEG)	SN→DMN-ECN gating model must win Bayesian Model Selection; forward Granger causality must exceed reverse

Table 11

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 772 papers

Gestalt: Large Multimodal Interplay Model
Sep 30, 2026Zequn Yang, Yu Miao, Haotian Ni +8Unified Multimodal Models Cross-Modal Alignment
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beihang University +3
Discovering Functionally Selective Brain Regions with a Deep Topographic Multimodal Model
Jun 8, 2026Badr AlKhamissi, Johannes Mehrer, Lara Marinov +3Computational Cognitive Modeling Unified Multimodal Models
NeuroAI Lab, EPFL
The Giant Hippocampus: From Structural Monoculture to a System of Systems
Jul 22, 2026Jaeho SeolTransformer Inductive Bias
Independent Researcher

Sep 30, 2026J/K move · Enter open · S save

Gestalt: Large Multimodal Interplay Model

Zequn Yang, Yu Miao, Haotian Ni, Ziheng Chen+7

Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beihang University+3

Unified Multimodal Models Cross-Modal Alignment Multimodal Generation Multimodal Diffusion Models Multimodal Understanding

Open paper AskPDF arXiv

In this paper, we propose Gestalt, a new paradigm of large multimodal model built around multimodal interplay. Despite rapid advances, large multimodal models are reaching a bottleneck: existing approaches focus primarily on accommodating additional modalities while overlooking the distinct characteristics of each modality and the relations among them. Motivated by the multistage property of human multisensory perception, we propose a multimodal interplay pyramid that organizes multimodal modeling as a progression from modality-specific processing, through cross-modal alignment, to deeper multimodal integration. Guided by this pyramid, Gestalt adopts a unified discrete diffusion framework and an interplay-partitioned architecture, with learnable interplay tokens mediating cross-modal exchange and integration. The pyramid also structures its data organization and training strategy. Strong performance across image generation, multimodal understanding, and text-only evaluation shows that Gestalt significantly improves cross-modal integration while preserving modality-specific information, effectively harnessing the strengths of diffusion-based multimodal models and offering a promising path toward unified multimodal intelligence. Project page: https://GeWu-Lab.github.io/Gestalt.

Abstract

Multimodal Fusion

Explore similar work

Gestalt: Large Multimodal Interplay Model

Discovering Functionally Selective Brain Regions with a Deep Topographic Multimodal Model

The Giant Hippocampus: From Structural Monoculture to a System of Systems