Trust-Aware Routing for Distributed Generative AI Inference at the Edge

Chanh Nguyen; Erik Elmroth

cs.DCMar 30, 2026

Trust-Aware Routing for Distributed Generative AI Inference at the Edge

Authors: Chanh Nguyen, Erik Elmroth

Organizations: Department of Computing Science, Ume˚a University, SE-90187, Sweden

Abstract

Emerging deployments of Generative AI increasingly execute inference across decentralized and heterogeneous edge devices rather than on a single trusted server. In such environments, a single device failure or misbehavior can disrupt the entire inference process, making traditional best-effort peer-to-peer routing insufficient. Coordinating distributed generative inference therefore requires mechanisms that explicitly account for reliability, performance variability, and trust among participating peers. In this paper, we present G-TRAC, a trust-aware coordination framework that integrates algorithmic path selection with system-level protocol design to ensure robust distributed inference. First, we formulate the routing problem as a Risk-Bounded Shortest Path computation and introduce a polynomial-time solution that combines trust-floor pruning with Dijkstra's search, achieving sub-millisecond median routing latency at practical edge scales, and remaining below 10 ms at larger scales. Second, to operationally support the routing logic in dynamic environments, the framework employs a Hybrid Trust Architecture that maintains global reputation state at stable anchors while disseminating lightweight updates to edge peers via background synchronization. Experimental evaluation on a heterogeneous testbed of commodity devices demonstrates that G-TRAC significantly improves inference completion rates, effectively isolates unreliable peers, and sustains robust execution even under node failures and network partitions.

Figures & tables

Panel of Figure 1 — Fig. 1 : Overview of a smart-city edge scenario and the role of G-TRAC in coordinating trust-aware, distributed GenAI execution.

Work	Primary Goal	Routing / Coordination Logic	Trust / Reliability Model	Target Environment
Petals (ACL’23) [ 22 ]	Democratize access to large models	DHT-based discovery with latency-prioritized chain selection	Best-effort (cooperative assumption)	Consumer GPUs
Parallax [ 26 ]	Maximize throughput under dynamic availability	DAG-based scheduling with pipeline parallelism	Best-effort (performance variability only)	Decentralized GPU pools
DeServe [ 27 ]	Affordable offline inference (batching)	Cost-aware workload distribution	Best-effort (cost-driven cooperation)	Underutilized, preemptible GPUs in shared data centers
Helix (ASPLOS’25) [ 28 ]	Maximize throughput in heterogeneous clusters	Max-flow / min-cost optimization	Managed infrastructure	Heterogeneous GPU Clusters
HexGen (ICML’24) [ 29 ] and HexGen-2 (ICLR’25) [ 30 ]	Optimize disaggregated prefill/ decoding phases	Graph partitioning & Max-Flow scheduling	Managed infrastructure	Heterogeneous GPU Clusters
$\text{MoE}^{2}$ (MobiCom’25) [ 31 ]	Energy-accuracy expert optimization	Expert ensemble selection	Trust-agnostic optimization	Edge Servers

TABLE I : Comparison of decentralized and cloud–edge GenAI inference systems

Fig. 2 : G-TRAC architecture and execution flow.

Type	Processor	Mem	#
$P$ (server)	AMD Opteron 6272	47 GB	4
$R$ (legacy server)	Intel Xeon E5430	15 GB	4
$D$ (desktop)	Intel Core i5-13500	31 GB	1
$L$ (laptop)	Intel Core i7-1260P	15 GB	1
$S$ (phone)	ARMv8 SoC (2×A76 + 6×A55)	3.6 GB	2

TABLE II : Experimental testbed with heterogeneous nodes.

Param.	Value	Description
Model	GPT2-L	LLM 36 layers
$\tau$	$0.96$	Trust floor
$\beta$	$0.30$	Latency EWMA factor
$\ell_{init}$	$250$ ms	Initial latency estimate
$\Delta r^{+}$	$0.03$	Trust reward
$\Delta r^{-}$	$0.2$	Trust penalty

TABLE III: System parameters and configuration.

Fig. 3 : Service success rate under different generation lengths.

Fig. 4 : Per-token end-to-end latency distribution under different generation lengths. Black diamonds denote the P99 latency.

Fig. 5 : Distribution of inference chain length (hop count).

Fig. 6 : Distribution of selected peers by Trust and Latency (

L_{tok}=50

Fig. 7 : Algorithm scalability: Decision time vs Network size.

Panel of Figure 11 — Fig. 8 : Per-token time analysis: (a) end-to-end inference latency per token and (b) CPU execution time per token.

Panel of Figure 12 — Fig. 9 : Resource consumption for monolithic vs. distributed configurations.

Distributed Inference

Distributed inference runs machine-learning models across multiple devices or sites to meet compute, latency, memory, privacy, and reliability needs.

See all 75 papers

HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference
Aug 1, 2026Xin Yuan, Ning Li, Wenchao Xu +2Edge Inference Cost-Aware Inference
School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China · School of Artificial Intelligence, University of Science and Technology Beijing, China · Division of Integrative Systems and Design, Hong Kong University of Science and Technology, Hong Kong +1
$E^3$ -Agent: An Executable and Evolving Agent for Resource Management of Edge Generative Inference
May 21, 2026Rui Bao, Yaping Sun, Zhiyong Chen +4LLM Inference Efficiency Edge Inference
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai 200240, China · Department of Broadband Communication, Pengcheng Laboratory, Shenzhen 518000, China · China Mobile Research Institute, Beijing 100053, China
TrustedARI: Towards Trust-Native Agentic Routing Infrastructure for Agentic AI
Jun 14, 2026Qi Li, Zhenhua Zou, Shuo Li +2Verifiable Outsourced Computation LLM Agent Security
Tsinghua University

Aug 1, 2026J/K move · Enter open · S save

HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference

Xin Yuan, Ning Li, Wenchao Xu, Song Guo+1

School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China · School of Artificial Intelligence, University of Science and Technology Beijing, China · Division of Integrative Systems and Design, Hong Kong University of Science and Technology, Hong Kong+1

Edge Inference Cost-Aware Inference Mixture-of-Experts Inference Expert Routing Distributed Inference

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models have become a dominant architecture for large-scale AI services, yet deploying them over geo-distributed heterogeneous edge servers remains challenging. When the Top-k activated experts of a token are spread across multiple servers, the optimal routing depends jointly on cross-server link bandwidth, heterogeneous GPU computing capability, GPU-CPU expert loading delay, instantaneous queueing backlog, and replica-level quantization quality loss. Existing distributed inference and MoE serving methods address these factors separately and do not provide a unified framework for online multi-server collaborative routing. In this paper, we propose HetRoute, a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference. HetRoute introduces a unified per-assignment cost model that explicitly captures four cost components: cross-server transmission, GPU-CPU offloading, GPU computation with queueing, and quantization-induced quality penalty. Guided by this model, the offline stage determines expert server placement, GPU-CPU residency, and replica precision through a routing-cost-coupled deployment algorithm, while the online stage routes the Top-k activated expert set as a whole by minimizing the bottleneck layer cost via exact enumeration or beam search. Theoretical analysis establishes fallback feasibility, a bound on the number of participating servers, per-layer optimality for small candidate domains, and online computational complexity. Trace-driven evaluation on three MoE models over a heterogeneous 10-server edge testbed shows that HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%, cuts cross-server traffic by up to 72.1%, and achieves 2.13x throughput improvement compared with representative baselines, while keeping quality degradation within the configured budget.

Abstract

Distributed Inference

Explore similar work

HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference

E3E^3E3-Agent: An Executable and Evolving Agent for Resource Management of Edge Generative Inference

TrustedARI: Towards Trust-Native Agentic Routing Infrastructure for Agentic AI

$E^3$ -Agent: An Executable and Evolving Agent for Resource Management of Edge Generative Inference