cs.LGAug 12, 2026

Reading the Gate, Not the Interference: Output-Side Interference Measurement Does Not Track Merge Collapse

Authors: Chencheng Zhu

Organizations: UNSW Sydney

Abstract

Task-arithmetic merging works until it doesn't, and the field diagnoses why by measuring interference inside the merged model. We take the most direct such measure, the exact layerwise activation cross-term of a factorial ledger, establish its causal anatomy, and then ask what it tracks. The anatomy is clean: each block mostly transports and amplifies the cross-term rather than generating it; erased, it is regenerated by the untouched marginal paths to 99% of its norm unless removed late; its output effect varies monotonically with the displacement's angle (orthogonal displacements make interference worse), and a two-assumption model derives the angle law and retro-dicts the dose curve (R^2 >= 0.99). What the measure tracks is not what the field assumes. Behavioural expert-likeness is decoupled from it across four instruments. Its cross-condition behaviour is denominator-dominated: an instruction template pins the main effect to within 1% while the absolute interaction grows 111x from two to six merged tasks, suppressing expressed interference at k=2 and amplifying it at k=6. And where merging actually collapses, the cross-term is a bystander, not the carrier: across two collapse parameterizations at two scales, even erased persistently at every position, removing it entirely repairs none of the collapse. There the output-side ratio carries no method information under a common counterfactual, while two state-space measures the field already uses rank methods correctly at both scales. All 81 predictions were frozen before their data; falsifications are reported as such. Output-side interference measurement reads the gate, the denominator, and the displacement budget, not the interference. What fails a merge is the carrier-bystander split: collapse rides in the marginal displacements while the cross-term merely accompanies it, and only state space sees the carrier.

Explore similar work

Jun 15, 2026cs.AI

Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing

Model merging has become a practical post-training strategy for building a single multi-task large language model (LLM) by combining multiple task-specialized models. However, most existing approaches rely on post-hoc merging, in which task-specific models are merged only once after training. This one-shot aggregation often suffers from task interference, leading to information erasure across individual tasks. In this work, we show that replacing post-hoc merging with an iterative many-shot merging protocol is effective in improving multi-task performance. Building on this insight, we propose METIS, Mitigating Erasure from Task Interference for Stable many-shot merging. METIS is a loss-aware many-shot merging method that addresses information erasure in post-hoc merging through task-wise loss-gap weighting and consensus-based masking. Notably, METIS exhibits significant performance improvement on the worst-performing task, effectively mitigating information erasure. (Project page: https://imkyungjin.github.io/METIS/)
Kyungjin Im, Miru Kim, Chanin Eom +1
Jul 20, 2026cs.AI

Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment? Existing heterogeneous fusion methods typically introduce distillation, adapters, learned latent spaces, routing, or feature alignment, leaving open whether a simpler recipe can work for genuinely different billion-parameter checkpoints. We revisit this counterintuitive question through training-free dimensional adaptation followed by ratio-controlled interpolation. In union-style merging, we expand the smaller model into the larger parameter space; in intersection-style merging, we truncate the larger model into the smaller parameter space. Across Qwen-family model pairs and benchmarks covering mathematical reasoning, code generation, language understanding, commonsense reasoning, knowledge, and instruction following, deterministic expansion largely preserves the source model function, and small-ratio interpolation can improve over strong source checkpoints by transferring complementary capabilities. However, near-balanced interpolation often collapses, and task-level results reveal a seesaw effect in which gains on some capabilities coexist with regressions on others. These results show that simple parameter averaging, when paired with lightweight dimensional adaptation and carefully controlled ratios, is a surprisingly strong baseline for heterogeneous LLM merging, suggesting that the limits of direct weighted fusion may also bound what more complex heterogeneous merging methods can achieve at scale.
Jiahe Fan, Yinghao Hou, Si Chen +3
Jun 12, 2026stat.ME

A Two-Stage Statistical Framework for Evaluating Associative Interference in Large Language Models

Large language models (LLMs) are increasingly evaluated for bias using adaptations of human psychological paradigms, yet methodological limitations-particularly the conflation of refusal behavior with task performance-have hindered clear interpretation. Here, we adapt the Implicit Association Test (IAT) to a controlled, forced-choice framework and introduce a two-stage modeling approach that separates response compliance from task-consistent classification. Across three contemporary LLMs (Claude Sonnet-4, Gemini 2.5 Pro, and GPT-5), we evaluate associative interference, defined as reduced task-consistency in incongruent relative to congruent conditions. While compliance with the structured response format was uniformly high, interference effects varied substantially across models and domains. Claude Sonnet-4 exhibited strong interference in the Gender--Career domain (DeltaP = 0.086, 95% CrI [0.026, 0.173]) and smaller but credible effects in Gender--Science. Gemini 2.5 Pro showed attenuated interference, and GPT-5 exhibited minimal or no detectable interference across domains. These findings demonstrate that IAT-style associative asymmetries are not a universal property of LLMs, but instead depend on model-specific characteristics. By isolating interference from compliance and modeling item-level variability, this study provides a principled framework for evaluating structured response patterns in LLMs. The results highlight the importance of model-specific assessment and suggest that associative interference can be substantially mitigated in modern systems.
Achraf Cohen, Andrew Kincaid