Cornserve: A Distributed Serving System for Any-to-Any Multimodal Models

Jae-Won Chung; Jeff J. Ma; Jisang Ahn; Yizhuo Liang; Akshay Jajoo; Myungjin Lee; Mosharaf Chowdhury

cs.LGMar 12, 2026

Cornserve: A Distributed Serving System for Any-to-Any Multimodal Models

Authors: Jae-Won Chung, Jeff J. Ma, Jisang Ahn, Yizhuo Liang, Akshay Jajoo, Myungjin Lee, Mosharaf Chowdhury

Organizations: University of Michigan · University of Southern California · Cisco Research

Abstract

Any-to-Any models are an emerging class of multimodal models that accept combinations of multimodal data (e.g., text, image, video, audio) as input and generate them as output. Serving these models are challenging; different requests with different input and output modalities traverse different paths through the model computation graph, and each component of the model have different scaling characteristics. We present Cornserve, a distributed serving system for generic Any-to-Any models. Cornserve provides a flexible task abstraction for expressing Any-to-Any model computation graphs, enabling component disaggregation and independent scaling. The distributed runtime dispatches compute to the data plane via an efficient record-and-replay execution model that keeps track of data dependencies, and forwards tensor data between components directly from the producer to the consumer. Built on Kubernetes with approximately 23K new lines of Python, Cornserve supports diverse Any-to-Any models and delivers up to 3.81 $\times$ higher throughput and 5.79 $\times$ lower tail latency. Cornserve is open-source, and the demo video is available on YouTube.

Figures & tables

Model	Input	Output
Qwen 2.5 Omni ( 18 ) , Qwen 3 Omni ( 19 )	T, I, V, A	T, A
Qwen 2.5 VL ( 2 ) , InternVL 3 ( 23 )	T, I, V	T
DeepSeek Janus ( 16 , 3 )	T, I	T, I
LTX-2 ( 5 )	T, I	V, A
Qwen Image ( 15 ) , GLM Image ( 21 )	T	I

Table 1. Modality breakdown of recent Any-to-Any multimodal models. The input and output modalities ( T ext, I mage, V ideo, A udio) supported by a model can vary significantly.

Panel of Figure 1 — Figure 1. Computation graphs of (a) InternVL 3 ( 23 ) , a multimodal input model, and (b) Qwen Omni ( 18 , 19 ) , a multimodal input and output model. Different requests invoke different components and take different paths on the graph. $E$ stands for Encoder, $L$ for LLM, and $G$ for Generator. $L_{\text{th}}$ and $L_{\text{ta}}$ stand for thinker and talker LLMs, respectively.

Figure 2. Cornserve architecture and flows for deployment (black) and inference (red).

Panel of Figure 3 — Figure 3. Simplified MLLMTask composite task. When encoder_fission is true, LLMTask is instantiated with recv_embeds=True , which is recognized by its task descriptor to disable the encoder and receive embeddings from the sidecar instead. Cornserve’s record-and-replay captures unit task invocations and data dependencies automatically.

Panel of Figure 4 — Figure 4. Cornserve app serving multiple Gemma 3 models ( 13 ) . The serve function defines the app’s computation: it invokes multiple MLLM tasks concurrently, merges their streaming responses, and yields structured results, demonstrating how apps can express arbitrary logic in plain Python.

Panel of Figure 5 — Figure 5. Monolith vs. Cornserve comparisons for Qwen 2.5 Omni 7B ( 18 ) throughput and latency CDF, and Qwen 3 Omni 30B ( 19 ) throughput. $\times$ indicate GPU out-of-memory.

Panel of Figure 6 — Figure 6. Cornserve deployment configurations for Qwen 3 Omni on 8 and 16 GPUs. Each box represents a GPU. Model fission allows each component to scale independently: the thinker (LLM) uses tensor parallelism while talkers and generators are replicated to balance throughput.

LLM Inference Serving

Large language model inference serving studies how to run model requests efficiently while meeting latency, throughput, cost, and energy goals.

See all 72 papers

M*: A Modular, Extensible, Serving System for Multimodal Models
Jun 10, 2026Atindra Jha, Naomi Sagan, Keisuke Kamahori +9Efficient Multimodal Inference Distributed Inference
Stanford University · University of Washington · Carnegie Mellon University
TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inference
Mar 27, 2026Konstantinos Papaioannou, Thaleia Dimitra DoudaliLLM Serving LLM Inference Scheduling
IMDEA Software Institute, Universidad Politécnica de Madrid
Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
May 5, 2026Yixuan Mei, Zikun Li, Zixuan Chen +5LLM Serving Cost-Aware Inference
Carnegie Mellon University · Peking University

Jun 10, 2026J/K move · Enter open · S save

M*: A Modular, Extensible, Serving System for Multimodal Models

Atindra Jha, Naomi Sagan, Keisuke Kamahori, Irmak Sivgin+8

Stanford University · University of Washington · Carnegie Mellon University

Efficient Multimodal Inference Distributed Inference

Open paper AskPDF arXiv

We are entering a new era of composite model architectures that integrate diverse components such as vision encoders, language backbones, diffusion and flow heads, audio codecs, action generators, and world-model predictors. Such architectures underpin a broad class of multimodal models, including unified multimodal models, omni models, speech-language models, vision-language-action policies, and world models. However, existing model serving frameworks were built on narrow assumptions about model structure, making them ill-suited to accommodate this new architectural diversity. Here we present M*, a universal serving system for efficient serving of composite AI models. M* represents models as dataflow graphs, processing requests spanning diverse modalities and tasks as traversals over these graphs. The core insight is a modular abstraction that supports arbitrary composition of model components, flexible placement onto a physical cluster, and model-agnostic optimizations within a distributed runtime. We call this abstraction the Walk Graph and show how it can concisely capture composite models from a broad range of families. We instantiate M* on representative models and find that it achieves, on average, 20% lower end-to-end latency than vLLM-Omni for text-to-image workloads on BAGEL, while delivering up to 2.9x lower real-time factor and 2.7x higher throughput for text-to-speech workloads on Qwen3-Omni. M* also outperforms the V-JEPA 2-AC rollout baseline for robotic planning by up to 12.5x. Thus, our work paves the road towards more efficient serving of complex models with minimal developer effort.

Abstract

LLM Inference Serving

Explore similar work

M*: A Modular, Extensible, Serving System for Multimodal Models

TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inference

Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs