LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation

Peng Ding

cs.SEApr 10, 2026

LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation

Authors: Peng Ding

Organizations: University of Chicago

Abstract

The rapid proliferation of Large Language Model (LLM) providers--each exposing proprietary API formats--has created a fragmented ecosystem where applications become tightly coupled to individual vendors. Switching or bridging providers requires $O(N^2)$ bilateral adapters, impeding portability and multi-provider architectures. We observe that despite substantial syntactic divergence, the major LLM APIs share a common semantic core: the practical challenge is the combinatorial surface of syntactic variations, not deep semantic incompatibility. Based on this finding, we present LLM-Rosetta, an open-source translation framework built on a hub-and-spoke Intermediate Representation (IR) that captures the shared semantic core--messages, content parts, tool calls, reasoning traces, and generation controls--in a 9-type content model and 10-type stream event schema. A modular Ops-composition converter architecture enables each API standard to be added independently. LLM-Rosetta supports bidirectional conversion (provider-to-IR-to-provider) for both request and response payloads, including chunk-level streaming with stateful context management. We implement converters for four API standards (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Google GenAI), covering the vast majority of commercial providers. Empirical evaluation demonstrates lossless round-trip fidelity, correct streaming behavior, and sub-100 microsecond conversion overhead--competitive with LiteLLM's single-pass approach while providing bidirectionality and provider neutrality. LLM-Rosetta passes the Open Responses compliance suite and is deployed in production at Argonne National Laboratory. Code is available at https://github.com/Oaklight/llm-rosetta.

Figures & tables

Aspect	OpenAI Chat	Anthropic	Google GenAI	OpenAI Responses
Message unit	messages[]	messages[]	contents[]	input[] items
System prompt	role in array	top-level param	role in array	instructions
Content model	string or parts	block array	parts[]	items with type
Tool calls	tool_calls[]	content block	functionCall	item with type
Streaming	delta chunks	event types	candidate deltas	response events
Reasoning	—	thinking	thought	reasoning item

Table 1: API format divergence across major LLM providers.

Feature	LangChain	LiteLLM	AI Gw.	Open R.	LLM-Rosetta
Provider-neutral IR	×	×	×	n/a	✓
Bidir. format conv.	×	×	×	n/a	✓
Lossless round-trip	×	×	×	n/a	✓
Streaming support	✓	✓	✓	✓	✓
App framework	✓	×	×	×	×
Providers ( $\geq$ 50)	✓	✓	✓	×	×

Table 2: Comparison of LLM-Rosetta with related approaches. × = not supported; n/a = not applicable (specification, not implementation).

Figure 1: Overview of the IR schema. Arrows indicate containment relationships.

Figure 2: Hub-and-spoke architecture of LLM-Rosetta . Each provider converter translates bidirectionally between its native format and the IR . Cross-provider translation composes two converters through the IR hub.

Category	OpenAI Chat	Anthropic	Google	Responses
Content parts	22	31	33	43
Messages	38	23	28	27
Tool defs/calls	25	28	31	46
Config/params	23	26	47	44
Full round-trip	31	41	51	40
Streaming	56	70	59	70

Table 3: Unit test counts per conversion category and provider. All tests pass in both strip and preserve metadata modes.

Payload	OpenAI Chat	Anthropic	Google	Responses
Simple text (req)	21	24	24	22
Multi-turn (req)	71	75	77	73
Tool calls (req)	44	46	46	44
Simple text (resp)	30	29	31	31
Tool calls (resp)	55	47	48	57

Table 4: Round-trip conversion overhead in microseconds (median over 1,000 iterations). “Req” denotes request round-trip (provider

\to

\to

provider); “Resp” denotes response round-trip.

Payload	LiteLLM (µs)	LLM-Rosetta (µs)	Ratio
Simple text	28	24	0.8 $\times$
Multi-turn	34	76	2.2 $\times$
Tool calls	29	46	1.6 $\times$

Table 5: Cross-provider conversion latency: LiteLLM (one-directional, OpenAI Chat

\to

Anthropic) vs. LLM-Rosetta (two-hop, OpenAI Chat

\to

\to

Anthropic). Median over 1,000 iterations; LiteLLM v1.83.

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 207 papers

May 11, 2026

SOMA: Efficient Multi-turn LLM Serving via Small Language Model

Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models' responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://github.com/LabRAI/SOMA.

Xueqi Cheng, Qiong Wu, Zhengyi Zhou +3

Florida State University · AT&T Chief Data Office · Louisiana State University +1

LLM Serving Small Language Models Language Model Distillation Efficient Language Model Inference Multi-Turn Dialogue Management

Ask PDF

Sep 1, 2026

Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives

Large language models (LLMs) augmented with external tools have demonstrated remarkable capability in solving complex real-world tasks. However, existing approaches suffer from two key challenges: brittle multi-step and multi-turn reasoning caused by incompatible tool output types and API schemas, and performance degradation under large tool catalogues. To address these, we introduce Tool Primitives, a design that replaces rigid API schema-based invocation with natural language as the interface for tool calling, where each tool is wrapped with an LLM interface that handles schema resolution and execution internally, enabling natural inter-tool communication for nested and multi-turn tool calling. Building on Tool Primitives, we host ToolFace, a centralized repository of 25,519 functions from which LLMs dynamically retrieve only the relevant tools at inference time, eliminating the need to enumerate raw API schemas in context. To orchestrate Tool Primitives and ToolFace reliably in complex settings, we further propose HEART, a Harness Engineering framework via Agent-native, Reusable Tool Primitives, comprising a Planner, Router, and Verifier that jointly support dynamic tool invocation planning, multi-step execution, and feedback-driven recovery. Experiments on five benchmarks demonstrate that HEART outperforms SFT-based models by

10\%

on average and surpasses GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro by

6\%

on average while reducing API cost by up to

85\%

. On 50 real-world tasks, HEART achieves

84\%

task completion,

3.8\times

the average of three frontier commercial models (

22\%

Haibo Jin, Suijin Wang, Xucheng Yu +2

School of Information Sciences University of Illinois at Urbana-Champaign Champaign, IL 61820 · Independent Researcher Starc Institute

Tool-Augmented Language Model Agents LLM Agent Orchestration LLM Tool Use Tool-Using Language Agents LLM Agent Harnesses

Ask PDF

Jun 22, 2026

RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving

We present RLM-Cascade, a proxy-layer system that applies speculative decoding at the response level to reduce LLM API costs without requiring model architecture access or a shared vocabulary. A fast, inexpensive draft model generates a candidate response; a capable verify model accepts, enhances, or is bypassed entirely depending on a lightweight complexity router. On a real-world agentic coding workload (Claude Code), RLM-Cascade achieves a draft-use rate of 88.8% across 125 production requests, reducing API cost by 45.8% relative to a direct Opus baseline. Counter-intuitively, the proxy also reduces end-to-end latency: median response time is 2,026 ms versus 3,698 ms for Native Opus -- a 1.83X speedup at p50 -- because the SKIPPED path (DeepSeek only, no Opus call) dominates the workload distribution. Quality matches or exceeds the Opus baseline: 100% pass rate on a 20-task Code/Math/Instruct benchmark versus 95% for Native Opus. We further describe a rule-based complexity router that selects the SKIPPED path for simple agentic turns and a hybrid tool-call strategy that bypasses the speculative pipeline for schema-critical tool-selection turns. RLM-Cascade is deployed in production as an enterprise AI infrastructure component and published as open source with a live metrics dashboard and Prometheus endpoint.

Haifeng Wu, Srinivasan Manoharan, Fangbo Tu +2

PayPal

LLM Inference Serving Speculative Decoding LLM Routing Efficient Language Model Inference Language Model Cascading

Ask PDF

Abstract

LLM Serving

Explore similar work

SOMA: Efficient Multi-turn LLM Serving via Small Language Model

Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives

RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving