cs.LGOct 29, 2026

Characterizing Recoverable Expert-Utilization Signatures in Fine-Tuned Mixture-of-Experts LLMs

Authors: Fei Wang, Baochun Li, Bo Li

Organizations: University of Toronto, Department of Electrical and Computer Engineering, Toronto, Ontario, Canada · Hong Kong University of Science and Technology, Department of Computer Science and Engineering, Hong Kong, China

Abstract

As Mixture-of-Experts (MoE) large language models are increasingly deployed in distributed environments, from federated learning to edge-cloud systems, their architecture harbors an underexplored landscape of security vulnerabilities. Moving beyond textual analysis, we uncover structural memorization: a novel vulnerability where a fine-tuned model memorizes not only its training data but also the internal pathway used to process it, creating a characteristic routing fingerprint unique to the training process. We provide comprehensive empirical evidence that structural memorization emerges consistently even from a few samples, and that its intensity is directly influenced by the choice of trainable modules. We show that these memorized expert pathways can be stealthily recovered through the model's unconditional generation, using only routing and output logits, with no knowledge of the fine-tuning data or procedure. In distributed deployments, this vulnerability is amplified as routing information is exposed at multiple trust boundaries, including federated aggregation points, edge-cloud interfaces, and multi-tenant monitoring systems. Recovery exposes a critical blind spot: traditional membership inference metrics, viewed through the lens of expert routing, fail to identify memorized data corresponding to these structural pathways. To close this gap, we devise composite metrics combining reference model tail likelihood with complexity-aware normalization, significantly improving detection of memorized routing structures. This enables adversaries to pinpoint critical experts shaped by fine-tuning, whether dominant or dormant, and execute targeted attacks such as router manipulation or backdoor injection in realistic distributed settings where attackers lack training data but can observe routing patterns across system components.

Figures & tables

Explore similar work

CardsList
  1. When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry

    Oct 7, 2026Yixin Tan, Jiayang Liu, Lu Sun +3Privacy Leakage in Language ModelsMembership Inference Attacks

  2. BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts

    Apr 24, 2025Qingyue Wang, Qi Pang, Xixun Lin +2LLM Backdoor AttacksBackdoor Attacks

  3. RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs

    May 1, 2026Zhiyuan Xu, Joseph Gardiner, Sana Belguith +1Mixture-of-Experts Language ModelsAdversarial Attacks on LLMs