Compound AI System
Compound AI systems integrate multiple large language models (LLMs) and other components to perform complex tasks exceeding the capabilities of individual LLMs. Current research focuses on optimizing these systems through architectures like "Networks of Networks," which leverage verification processes to improve accuracy, and developing frameworks like TextGrad for automated optimization via textual feedback. This approach is significant because it addresses limitations of single LLMs, improving performance on diverse tasks ranging from question answering to drug molecule design, and impacting fields like data management and AI safety.
Papers
SpecTool: A Benchmark for Characterizing Errors in Tool-Use LLMs
Shirley Kokane, Ming Zhu, Tulika Awalgaonkar, Jianguo Zhang, Thai Hoang, Akshara Prabhakar, Zuxin Liu, Tian Lan, Liangwei Yang, Juntao Tan, Rithesh Murthy, Weiran Yao, Zhiwei Liu, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, Caiming Xiong, Silivo Savarese
SoK: A Systems Perspective on Compound AI Threats and Countermeasures
Sarbartha Banerjee, Prateek Sahu, Mulong Luo, Anjo Vahldiek-Oberwagner, Neeraja J. Yadwadkar, Mohit Tiwari
A Blueprint Architecture of Compound AI Systems for Enterprise
Eser Kandogan, Sajjadur Rahman, Nikita Bhutani, Dan Zhang, Rafael Li Chen, Kushan Mitra, Sairam Gurajada, Pouya Pezeshkpour, Hayate Iso, Yanlin Feng, Hannah Kim, Chen Shen, Jin Wang, Estevam Hruschka
CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systems
Yanlin Feng, Sajjadur Rahman, Aaron Feng, Vincent Chen, Eser Kandogan