cs.CVApr 13, 2026

Lost in Adaptation: Layer-Selective Recovery of Temporal Reasoning in Video-Language Models

Authors: Zihang Fu, Haonan Wang, Jian Kang, Kenji Kawaguchi, Jiaying Wu

Organizations: National University of Singapore · MBZUAI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal adaptation can weaken temporal reasoning (TR) in video-language models (VLMs) even when the visual evidence needed for an answer is correctly perceived. This weakened competence, however, remains partly recoverable from the model's paired pre-adaptation language checkpoint. We introduce MERIT, a gradient-free framework that selectively restores this competence across the language backbone while protecting temporal perception (TP). It assigns each self-attention layer a VLM-dominant or LLM-dominant interpolation and uses the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to search the recipe space under a perception-constrained objective. Across three VLM families and five video benchmarks, MERIT improves reasoning-aligned performance in all 15 model--benchmark evaluations while preserving TP across all three Video-MME settings. Recipes selected using 232 diagnostic examples transfer to four unseen benchmarks, with relative gains up to 27.8%, and outperform full-backbone, uniform-attention, and alpha-tuned random-layer merging. Interventional masking shows that MERIT-selected layers are disproportionately important for TR, while frame-level attribution reveals increased reliance on temporally distributed, causally relevant evidence. These results show that adaptation-induced reasoning degradation can be repaired post hoc through selective reuse of pre-adaptation competence, without learning new parameters or adding inference overhead.

Figures & tables

Figure 1: Multimodal adaptation can preserve what a VLM sees while weakening how its language backbone reasons over time. (a) The adapted language backbone underperforms its pre-adaptation LLM on text-only temporal reasoning (TR), directly isolating adaptation-induced degradation within the language component (§ B ). (b) In video TR, the VLM correctly perceives the relevant visual entities but fails to infer the temporal-causal structure required for the answer.
Figure 2: Overview of MERIT . Given an adapted VLM and its paired pre-adaptation LLM, MERIT searches for a layer-wise self-attention merging recipe in which each layer is VLM-dominant or LLM-dominant. CMA-ES optimizes this discrete structure under a perception-constrained objective that rewards TR recovery while penalizing TP degradation. The selected recipe is compiled into a single merged VLM with no additional inference components.
Figure 3: MERIT selects reasoning-sensitive intervention points. Suppressing the self-attention outputs of the selected layers in the unmodified VLM causes Reasoning to degrade markedly faster than Overall and Others .
Figure 4: MERIT shifts temporal grounding from a salient local cue to distributed causal evidence. In this Video-Holmes example (Question ID: 463), Base anchors on the phone-call scene and incorrectly attributes the victim’s death to delayed help. MERIT instead assigns greater support to two temporally separated events, the victim being restrained and later found motionless, and recovers the correct causal explanation.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: MERIT recovers temporal reasoning weakened within the adapted language backbone. Across all three model families, multimodal adaptation reduces text-only TempReason performance, while MERIT restores part or all of the lost capability. Percentages beside the brackets show relative gains over the adapted backbone.
VLMα\boldsymbol{\alpha}#Layersu^ℓ∗=1\boldsymbol{\hat{u}_{\ell}^{\ast}=1} self-attention layer indices
LongVA-7B1.0111, 5, 7–8, 11, 13–14, 16, 19–20, 23
InternVL3-8B0.994, 7, 12, 14, 16–18, 24, 26
Qwen3-VL-4B1.0160, 4, 8, 10, 13, 17–18, 21, 23, 26, 28–32, 35
Appendix
Table 3: MERIT discovers distinct recovery recipes across model families. We report the selected interpolation weight α\alpha , number of LLM-dominant self-attention layers ( u^ℓ∗=1\hat{u}_{\ell}^{\ast}=1 ), and their 0-based indices.
Figure 6: CMA-ES outperforms matched-budget uniform random search. Best-so-far perception-constrained objective F(g)\mathcal{F}(g) over 1,600 candidate evaluations under the same α\alpha , recipe space, objective, and budget. Curves show the mean over three runs; shaded regions indicate ±1\pm 1 standard deviation.
Figure 7: Case study from Video-Holmes (Question ID: 1040). Both models answer correctly, but MERIT grounds its prediction in a longer event chain linking the apparent theft, the pursuit, and the later awakening. This temporal structure supports the inference that the preceding events were a dream, while the base model remains more locally grounded.
Figure 8: Case study from Video-Holmes (Question ID: 565). The base model anchors on the murderer’s entry through the unlocked door and incorrectly treats it as the direct cause of death. MERIT instead links the staged scare, the first non-lethal encounter, and the ignored second scream, correctly identifying the missed warning as the decisive causal factor.
Figure 9: Case study from Video-Holmes (Question ID: 527). Both models select the correct answer, but MERIT links repeated wake-up and gun-firing sequences to infer a recurring pattern in which each gunshot triggers a shift in reality. The base model relies primarily on a single observed transition.
VLMα\boldsymbol{\alpha}#LayersRunu^ℓ=1\boldsymbol{\hat{u}_{\ell}=1} self-attention layer indices
LongVA-7B1.01115, 7–8, 10–11, 17, 20–22, 26–27
0.920, 3–6, 8–9, 14–15, 19, 27
0.531, 4, 7, 11–12, 15, 20–21, 23, 26–27
0.241, 6–7, 15, 17–18, 20–22, 25, 27
0.955, 8–9, 12–15, 21, 24–25, 27
InternVL3-8B0.6915, 8, 10, 12–13, 16, 23–24, 27
Appendix
Table 10: Alpha-tuned random- kk baseline recipes. For each model, we report five independent random binary layer patterns, each assigning exactly kk self-attention layers to the u^ℓ=1\hat{u}_{\ell}=1 branch of Eq. 4 , with kk matching the number of layers assigned u^ℓ∗=1\hat{u}_{\ell}^{\ast}=1 in the corresponding MERIT recipe. The interpolation coefficient α\alpha is tuned independently for every run on the TP/TR diagnostic set. Because α\alpha may be below 0.50.5 , the u^ℓ=1\hat{u}_{\ell}=1 branch is not necessarily LLM-dominant. Layer indices are 0-based.
More in this topic

Video-Language Models

Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.

See all 167 papers

Explore similar work

CardsList

  1. Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs

    Oct 1, 2026Youngwoo Shin, Yusung Ro, Minseo Kim +1Video-Language ModelsActivation Steering

    Korea Advanced Institute of Science and Technology (KAIST)

  2. Temporal-Aware Reasoning Optimization for Video Temporal Grounding

    Jun 8, 2026Minghang Zheng, Zihao Yin, Yi Yang +2Temporal Video GroundingTemporal Reasoning

    Wangxuan Institute of Computer Technology, Peking University · Central Media Technology Institute, Huawei Technologies Ltd. · State Key Laboratory of General Artificial Intelligence, Peking University.

  3. TTA-Vid: Generalized Test-Time Adaptation for Video Reasoning

    Apr 1, 2026Soumya Shamarao Jahagirdar, Edson Araujo, Anna Kukleva +7Test-Time AdaptationVideo Reasoning

    University of Tübingen · Max Planck Institute for Informatics, SIC · MIT +3