Graph-Based Orchestration of Heterogeneous AI Models: A Control Node Approach to Composite Intelligence

G. Michael Youngblood; Filip Dvořák; Slavo Švancár; Tomáš Balyo; Michal Ficek; Martin Dousek

cs.AIOct 13, 2026

Graph-Based Orchestration of Heterogeneous AI Models: A Control Node Approach to Composite Intelligence

Authors: G. Michael Youngblood, Filip Dvořák, Slavo Švancár, Tomáš Balyo, Michal Ficek, Martin Dousek

Organizations: Filuta AI, Inc., 1606 Headway Cir STE 9145, Austin, TX 78754, USA

Abstract

We present a graph-based framework for Composite AI systems that integrate multiple AI techniques to address complex problems. We describe an architecture using specialized control nodes to orchestrate diverse models including planning, machine learning, constraint programming, and large language models. The system employs directed acyclic graphs and behavior trees to manage model relationships and execution sequencing. Our methodology encompasses data connectors, computational services, and a containerized deployment framework supporting hierarchical, sequential, and concurrent model composition patterns. Through case studies, we demonstrate how this architecture enables the creation of composite systems that leverage the strengths of multiple AI techniques within a unified operational schema, overcoming the limitations of single-model approaches.

Figures & tables

Figure 2: Composition framework. Each control node is unique being configured for their specific downstream components through introspection or manually, but use a common code base.

Figure 4: An end-to-end system for Specification, Codification, and Deployment of Composite AI Systems

M*: A Modular, Extensible, Serving System for Multimodal Models
Jun 10, 2026Atindra Jha, Naomi Sagan, Keisuke Kamahori +9Efficient Multimodal Inference Distributed Inference
Stanford University · University of Washington · Carnegie Mellon University
Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systems
Jun 12, 2026Milos Gravara, Andrija Stanisic, Stefan NasticCost-Aware Inference
Distributed Systems Group TU Wien
Gypscie: A Cross-Platform AI Artifact Management System
Apr 11, 2026Fabio Porto, Eduardo Ogasawara, Gabriela Moraes Botaro +4
LNCC, Petrópolis, Brazil · CEFET/RJ, Rio de Janeiro, Brazil · Inria, Univ Montpellier, CNRS, LIRMM, France

Jun 10, 2026J/K move · Enter open · S save

M*: A Modular, Extensible, Serving System for Multimodal Models

Atindra Jha, Naomi Sagan, Keisuke Kamahori, Irmak Sivgin+8

Stanford University · University of Washington · Carnegie Mellon University

Efficient Multimodal Inference Distributed Inference

Open paper AskPDF arXiv

We are entering a new era of composite model architectures that integrate diverse components such as vision encoders, language backbones, diffusion and flow heads, audio codecs, action generators, and world-model predictors. Such architectures underpin a broad class of multimodal models, including unified multimodal models, omni models, speech-language models, vision-language-action policies, and world models. However, existing model serving frameworks were built on narrow assumptions about model structure, making them ill-suited to accommodate this new architectural diversity. Here we present M*, a universal serving system for efficient serving of composite AI models. M* represents models as dataflow graphs, processing requests spanning diverse modalities and tasks as traversals over these graphs. The core insight is a modular abstraction that supports arbitrary composition of model components, flexible placement onto a physical cluster, and model-agnostic optimizations within a distributed runtime. We call this abstraction the Walk Graph and show how it can concisely capture composite models from a broad range of families. We instantiate M* on representative models and find that it achieves, on average, 20% lower end-to-end latency than vLLM-Omni for text-to-image workloads on BAGEL, while delivering up to 2.9x lower real-time factor and 2.7x higher throughput for text-to-speech workloads on Qwen3-Omni. M* also outperforms the V-JEPA 2-AC rollout baseline for robotic planning by up to 12.5x. Thus, our work paves the road towards more efficient serving of complex models with minimal developer effort.

Abstract

Explore similar work

M*: A Modular, Extensible, Serving System for Multimodal Models

Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systems

Gypscie: A Cross-Platform AI Artifact Management System