Beyond Perturbation Magnitude: Direction-Dependent Responses in Multimodal Geometric Representations

Yongsheng Luo; Wengan He; Yu Li; Rouying Wu; Wei Lv

cs.CVOct 6, 2026

Beyond Perturbation Magnitude: Direction-Dependent Responses in Multimodal Geometric Representations

Authors: Yongsheng Luo, Wengan He, Yu Li, Rouying Wu, Wei Lv

Organizations: Zhuhai College of Science and Technology, Zhuhai, China · Zhuhai UNO Technology Co., Ltd., Zhuhai, China

Abstract

Geometric alignment scores based on Gram determinants provide a compact way to model higher-order consistency among modalities, yet how such scores respond to modality degradation is poorly understood. This paper asks whether the response of a multimodal geometric score is determined primarily by the magnitude of the perturbation-induced displacement. Using frozen cohorts from MSR-VTT (N=878) and DiDeMo (N=980), we apply controlled video blur and audio noise and analyze the response in the relational geometry on which the score is defined. Displacement magnitude explains at most 15% of the out-of-sample variance in the absolute response, and magnitude-matched pairs respond systematically differently, so scalar magnitude does not organize the response. The closed-form first-order expansion of the Gramian volume yields the Directional Geometric Response (DGR): the projection of the displacement onto the local volume gradient, which jointly captures the clean operating point, displacement magnitude, and displacement direction. The absolute first-order DGR term explains the observed response with out-of-sample R^2 of 0.838-0.969, matched-magnitude ranking accuracies of 0.864-0.963, and response-sign accuracies of 0.909-0.989, whereas the tested direction-free alternatives remain weak or unstable under the corresponding evaluation protocols. A pre-specified gain-normalization candidate, V/(g_V+eps), fails its predictability and clean-order gates. DGR uses the observed degraded-state displacement and is therefore an explanatory quantity, not a deployment-time predictor: geometric response depends on where the representation operates, how far degradation moves the relational geometry, and in which direction it moves.

Figures & tables

Fig. 1: Mechanism overview of direction-dependent responses in multimodal geometric representations. A clean multimodal representation defines an operating point $\mathbf{o}_{c}$ in relational geometry; a controlled degradation produces a displacement $\Delta\mathbf{o}$ , and the first-order volume response is governed jointly by the local operating-point gain $\|\nabla V(\mathbf{o}_{c})\|$ , the displacement magnitude $D=\|\Delta\mathbf{o}\|$ , and the direction factor $\cos\theta$ , whose product defines the Directional Geometric Response (DGR). Empirically, magnitude alone provides weak response predictability, the direction-free gain–magnitude product remains insufficient, and the direction-aware first-order quantity explains substantially more of the response heterogeneity. DGR is an explanatory quantity because it uses the observed degraded-state displacement.

Fig. 2: Local geometric decomposition of the first-order response. A degradation moves the relational geometry by

\Delta\mathbf{o}

from the clean operating point

\mathbf{o}_{c}

; the induced volume change is governed by the projection of

\Delta\mathbf{o}

onto the local gradient

\nabla V(\mathbf{o}_{c})

, i.e., local gain

g_{V}\!\times

magnitude

D\!\times

\cos\theta

Fig. 3: Absolute volume response

|\Delta V|

versus relational displacement magnitude

D

under single-axis degradation (top: MSR-VTT; bottom: DiDeMo; left: video blur; right: audio noise). Magnitude leaves most of the response variance unexplained, and magnitude-matched samples respond systematically differently.

Candidate	Information retained	OOS $R^{2}$	Matched- $D$ ranking	Decision
$D$	displacement magnitude	$-0.004$ – $0.146$	$0.509$ – $0.523$	Reject
S1 context	how compromised the sample state is	— †	0.447 / 0.420 †	Reject
S2 self-sensitivity	probe-induced displacement	— †	0.503 / 0.488 †	Reject
S3 self-consistency	two-view agreement	— †	0.505 / 0.460 †	Reject
$\\|\nabla V\\|D$	operating-point gain $\times$ magnitude (no direction)	$0.002$ – $0.224$	$0.517$ – $0.574$	Unstable
$\|\mathrm{DGR}\|$	gain $\times$ magnitude $\times$ direction (projection onto $\nabla V(\mathbf{o}_{c})$ )	$\mathbf{0.838}$ – $\mathbf{0.969}$	$\mathbf{0.864}$ – $\mathbf{0.963}$	Supported

TABLE I: Falsification ladder for explanations of geometric response. The table summarizes the falsification sequence, from scalar perturbation summaries to increasingly explicit geometric descriptions.

R^{2}

ranges span the four dataset–degradation cells; ranking accuracies are matched-

D

pairwise accuracies on

|\Delta V|

at the primary bin width (

\varepsilon=0.01

; chance

0.5

). † Proxy statistics were computed only for the MSR-VTT cohort under the Phase-3A protocol (video/audio axes); “—” indicates that the metric was not evaluated under the four-cell

|\Delta V|

protocol for that candidate. The supported quantity is an explanatory first-order term that uses the observed degraded-state displacement, not a deployment-time predictor.

	OOS $R^{2}$ for $\|\Delta V\|$			Matched- $D$ ranking acc.		Sign accuracy
Cell	$D$	$\\|\nabla V\\|D$	$\|\mathrm{DGR}\|$	$D$	$\|\mathrm{DGR}\|$	$\mathrm{DGR}$
MSR-VTT video ( $N=878$ )	0.146	0.224	0.969	0.523	0.963	0.989
MSR-VTT audio ( $N=878$ )	$-0.004$	0.002	0.925	0.515	0.903	0.950
DiDeMo video ( $N=980$ )	0.136	0.152	0.925	0.509	0.961	0.983
DiDeMo audio ( $N=980$ )	0.021	0.059	0.838	0.517	0.864	0.909

TABLE II: Main four-cell results.

D

is the relational displacement magnitude;

\|\nabla V\|D

is the direction-free gain–magnitude product;

|\mathrm{DGR}|

is the absolute first-order directional response, which uses the degraded-state displacement and is therefore explanatory only. Matched-

D

ranking is pairwise accuracy within narrow magnitude bins, with chance at

0.5

Fig. 4: Geometric explanation ladder: from magnitude to directional response. (a) Out-of-sample

R^{2}

for

|\Delta V|

under three nested descriptions of the same perturbation: magnitude

D

; the direction-free gain–magnitude product

\|\nabla V\|D

; and the directional first-order projection

|\mathrm{DGR}|

. (b) The same comparison under matched-

D

pairwise ranking (chance

0.5

). Direction, rather than the local gain alone, carries the missing structure.

Fig. 5: Pointwise validation of the Directional Geometric Response. Each panel compares the observed volume response $\Delta V$ with the first-order directional quantity $\mathrm{DGR}=\nabla V(\mathbf{o}_{c})^{\top}\Delta\mathbf{o}$ for one dataset–degradation cell. The dashed diagonal denotes the ideal first-order correspondence $\Delta V=\mathrm{DGR}$ . The strong alignment across samples confirms that the directional first-order term captures the dominant response structure, while the visible deviations from the diagonal, particularly for the audio axes, quantify non-negligible higher-order effects.

Fig. 6: The gain-normalization intervention fails both pre-specified gates. (a) Predictability gate: the bootstrap CI of

\Delta R^{2}

excludes zero in only one of four cells. (b) Clean-order gate: despite high median Spearman correlation (

\rho_{\rm med}

), the clean top-1 order keep rate falls below the

0.98

threshold on both datasets under the isotropic candidate, and collapses under the anisotropic variant.

Cell	$\Delta R^{2}$ CI	Clean $\rho_{\rm med}$	Top-1 keep
MSR-VTT video	$[0.0014,\,0.0290]$	0.9918 †	0.9715 †
MSR-VTT audio	CI includes 0	0.9918 †	0.9715 †
DiDeMo video	CI includes 0	0.9880 †	0.9296 †
DiDeMo audio	CI includes 0	0.9880 †	0.9296 †
Anisotropic variant	not rescued	—	0.485 / 0.386

TABLE III: Gain-normalization audit.

\Delta R^{2}

is the bootstrap

95\%

CI of the out-of-sample

R^{2}

change under

\widetilde{V}=V/(g_{V}+\varepsilon)

; clean-order statistics are computed at

\varepsilon=0.1\times

median(

g_{V}

). The candidate passes neither gate.

Explore similar work

Learning Where Outcomes Change:Credit-Addressable Reasoning for Multimodal Geometry
Aug 31, 2026Jiani Guo, Junjie Wang, Jie Wu +5Multimodal Reasoning
1Tsinghua University · 3Zhejiang University · 2Microsoft Research
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Jul 20, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive Learning Multimodal Alignment
Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University
Anisotropic Modality Align
May 8, 2026Xiaomin Yu, Yijiang Li, Yuhui Zhang +8Modality Alignment
HKUST(GZ) · NUS · Stanford +2

Aug 31, 2026, cs.LGJ/K move · Enter open · S save

Learning Where Outcomes Change:Credit-Addressable Reasoning for Multimodal Geometry

Jiani Guo, Junjie Wang, Jie Wu, Pengxiang Zhao+4

1Tsinghua University · 3Zhejiang University · 2Microsoft Research

Multimodal Reasoning

Open paper AskPDF arXiv

Multimodal geometry reasoning requires VLMs to extract precise visual relations and preserve them through multi-step deduction. Existing free-form traces obscure the decisions that determine the answer, and trajectory-level reinforcement learning distributes a single terminal signal across the entire response. We introduce credit-addressable reasoning, in which the semantic units exposed during inference also define where learning compares alternatives and assigns credit. We instantiate this principle with Code-CoT, which retains the diagram, represents visual relations as line-addressable executable code, and organizes reasoning into typed events, and CE-GRPO, which selects event boundaries using structural priors and type-normalized entropy, samples complete continuations from shared prefixes, and converts outcome differences into localized advantages. Across nine geometry benchmarks, CE-GRPO achieves an average accuracy of 76.04, outperforming Qwen3-VL-8B and trajectory-level GRPO by

8.09

and 3.43 points, respectively. Its relative advantage increases with the number of intermediate events, demonstrating the value of representation--optimization co-design for long, dependency-heavy multimodal reasoning.

Abstract

Explore similar work

Learning Where Outcomes Change:Credit-Addressable Reasoning for Multimodal Geometry

Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

Anisotropic Modality Align