cs.SEAug 19, 2026

Flama: a Python framework for development and deployment of production-ready APIs, machine learning, and LLM services

Authors: José A. Perdiguero López, Miguel A. Durán-Olivencia

Organizations: Vortico Tech, Málaga, Spain

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present Flama, an open-source Python framework for developing and deploying production-ready web APIs, machine learning services, and large-language-model (LLM) applications. Built on the Asynchronous Server Gateway Interface (ASGI), Flama offers a type-driven, async-first programming model that unifies REST API development, predictive model serving, and generative AI inference in one architecture. It is organised around seven subsystems: a component-based dependency injection system resolving handler parameters from type annotations at startup; a pluggable schema layer supporting Pydantic, Marshmallow and Typesystem behind a single adapter; an automatic CRUD generator turning a SQLAlchemy table and a schema class into REST endpoints backed by the Repository and Unit of Work patterns; a portable binary format (.flm) packaging models from scikit-learn, TensorFlow, PyTorch and Hugging Face Transformers with their metadata for zero-code deployment; a multi-backend LLM server running vLLM (Linux/CUDA) or MLX (Apple Silicon) and exposing four wire protocols (OpenAI, Anthropic, Ollama, and a native streaming dialect) through a shared codec; a Rust-accelerated core compiled via Maturin for routing, JSON encoding, compression and parsing; and a Model Context Protocol module turning any application into an MCP server over JSON-RPC 2.0. Built-in capabilities include JWT authentication, two pagination strategies, background tasks in threads or processes, WebSocket endpoints, Server-Sent Event and NDJSON streaming, OpenAPI 3.2.0 generation from handler signatures, and a command-line interface for running applications and for serving, packaging and inspecting models. We describe the architecture, present the programming model through worked examples, and compare Flama with existing frameworks, model serving platforms and LLM inference engines.

Figures & tables

Figure 1: Layered architecture of Flama . The ASGI protocol provides the server interface. The Flama Core layer, with Rust-compiled performance-critical paths, supplies HTTP abstractions, route resolution, path and host matching, JSON encoding, compression, and multipart and cookie parsing. The Application layer adds dependency injection, component management, and lifecycle hooks. Modules extend the application with schema generation, CRUD resources, ML and LLM model serving, and the Model Context Protocol.
Figure 2: Request processing pipeline. The request flows left to right through the middleware stack, route resolution, dependency injection, and handler execution. The response flows back through the middleware stack before being sent to the client.
Figure 3: Request validation pipeline. Each component resolves one aspect of the incoming data and passes validated, typed values to downstream components or the handler.
Figure 4: Resolution tree for a handler with four parameters. The injector traverses the tree bottom-up: context values are read from the ASGI scope, component values are produced by calling resolve () , and primitive parameters are extracted from the URL path. The tree is compiled at startup and flattened into an ordered sequence of resolution steps for efficient request-time execution. Node fill indicates where a value comes from, as given in the key.
Figure 5: Domain-driven design integration. The handler receives a Worker via dependency injection. The Worker manages repositories within a transactional boundary. All operations within the async with block either commit together or roll back together.
Figure 6: Structure of the FLM binary format (protocol version 1). The outer header identifies the format version and compression algorithm. The body contains three sections: compressed JSON metadata, compressed model weights, and a sequence of compressed named artifact entries. Each section can be decompressed independently.
FeatureFlamaFlaskTF ServingTorchServeBentoMLvLLMOllama
ASGI / async-native✓––––––
WebSocket & streaming✓––––––
SSE / NDJSON responses✓––––✓✓
Type-driven validation✓––––––
Pluggable schema libs✓––––––
Component-based DI✓––––––
Table 1: Feature comparison across frameworks, serving platforms, and LLM inference engines. A checkmark indicates built-in support without third-party extensions. Flama is the only system that provides full web API capabilities, native multi-framework ML model serving, multi-backend LLM inference with simultaneous multi-dialect exposure, and MCP support within a single unified architecture.
More in this topic

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 207 papers

Explore similar work

CardsList
May 21, 2026

HarnessAPI: A Skill-First Framework for Unified Streaming APIs and MCP Tools

Every Python function deployed as an LLM tool must today exist in two forms: an HTTP endpoint for human-facing clients and CI pipelines, and an MCP tool registration for agent runtimes such as Claude and Cursor. These representations share business logic yet diverge in all the surrounding machinery (routing, validation, serialisation, streaming, and schema maintenance), and they drift apart as the underlying code evolves. We present HarnessAPI, a Python framework that eliminates this duplication by treating a typed skill folder as the single source of truth. From one handler.py plus Pydantic schemas, the framework automatically derives a streaming HTTP endpoint with Server-Sent Events, an interactive OpenAPI/Swagger UI, and a zero-configuration MCP tool, all served from a single process. Dual-mode content negotiation lets the same handler serve SSE-streaming and JSON-returning clients with no handler changes. A dynamic code-generation mechanism ensures Pydantic type annotations propagate correctly to FastMCP's inspection layer, resolving a technical limitation that prevents naive closure-based registration. Measured across six representative skills using cloc, HarnessAPI reduces framework-facing boilerplate by 74% compared with a manually maintained dual-stack implementation (FastAPI server + FastMCP server). HarnessAPI subclasses FastAPI, inheriting its full middleware, dependency-injection, and deployment ecosystem. It is available at https://github.com/edwinjosechittilappilly/harnessapi and on PyPI (pip install harnessapi)

Edwin Jose

Department of Computer Science Western Michigan University Kalamazoo, MI 49008, USA

LLM Tool Use
AskPDF
Mar 13, 2026

Orla: A Library for Serving LLM-Based Multi-Agent Systems

We introduce Orla, a library for constructing and running LLM-based agentic systems. Modern agentic applications consist of workflows that combine multiple LLM inference steps, tool calls, and heterogeneous infrastructure. Today, developers typically build these systems by manually composing orchestration code with LLM serving engines and tool execution logic. Orla provides a general abstraction that separates request execution from workflow-level policy. It acts as a serving layer above existing LLM inference engines: developers define workflows composed of stages, while Orla manages how those stages are mapped, executed, and coordinated across models and backends. It provides agent-level control through three mechanisms: a stage mapper, which assigns each stage to an appropriate model and backend; a workflow orchestrator, which schedules stages and manages their resources and context; and a memory manager, which manages inference state such as the KV cache across workflow boundaries. We demonstrate Orla with a customer support workflow that exercises many of its capabilities. We evaluate Orla on two datasets, showing that stage mapping improves latency and cost compared to a single-model vLLM baseline, while workflow-level cache management reduces time-to-first-token.

Rana Shahout, Hayder Tirmazi, Minlan Yu +1

Harvard University MA, USA · Boston University MA, USA

LLM Agent OrchestrationLLM Agent ServingLLM Inference ServingAI Agent ArchitectureMulti-Agent LLM SystemsLLM Agent Workflow Optimization
AskPDF
Apr 10, 2026

LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation

The rapid proliferation of Large Language Model (LLM) providers--each exposing proprietary API formats--has created a fragmented ecosystem where applications become tightly coupled to individual vendors. Switching or bridging providers requires O(N2)O(N^2) bilateral adapters, impeding portability and multi-provider architectures. We observe that despite substantial syntactic divergence, the major LLM APIs share a common semantic core: the practical challenge is the combinatorial surface of syntactic variations, not deep semantic incompatibility. Based on this finding, we present LLM-Rosetta, an open-source translation framework built on a hub-and-spoke Intermediate Representation (IR) that captures the shared semantic core--messages, content parts, tool calls, reasoning traces, and generation controls--in a 9-type content model and 10-type stream event schema. A modular Ops-composition converter architecture enables each API standard to be added independently. LLM-Rosetta supports bidirectional conversion (provider-to-IR-to-provider) for both request and response payloads, including chunk-level streaming with stateful context management. We implement converters for four API standards (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Google GenAI), covering the vast majority of commercial providers. Empirical evaluation demonstrates lossless round-trip fidelity, correct streaming behavior, and sub-100 microsecond conversion overhead--competitive with LiteLLM's single-pass approach while providing bidirectionality and provider neutrality. LLM-Rosetta passes the Open Responses compliance suite and is deployed in production at Argonne National Laboratory. Code is available at https://github.com/Oaklight/llm-rosetta.

Peng Ding

University of Chicago

LLM Serving
AskPDF