MOSAIC: More LLMs, Less Cost through Collaborative Reasoning for Compiler Optimizations

Annabelle Sujun Tang; Christopher Priebe; Lianhui Qin; Hadi Esmaeilzadeh

cs.LGFeb 2, 2026

MOSAIC: More LLMs, Less Cost through Collaborative Reasoning for Compiler Optimizations

Authors: Annabelle Sujun Tang, Christopher Priebe, Lianhui Qin, Hadi Esmaeilzadeh

Organizations: Alternative Computing Technologies (ACT) Lab, University of California San Diego

Abstract

LLM-guided compiler optimization has recently shown promise. However, current LLM-guided optimizers query a single LLM at every search step, so their cost grows with both that model's price and the search budget. A smaller model is cheaper but degrades the search: in our experiments, a recent single-LLM optimizer reaches 10.2% (6.0%) lower speedup on GPU (CPU) with gpt-5-mini than with GPT-5.2. Agentic multi-LLM frameworks add controllers, concurrent calls, and shared memory, whose overhead can offset the savings of cheaper models. This paper shows that the optimization search tree itself can coordinate heterogeneous LLMs: the tree already records which proposals improved the program, and we can augment the tree to record which model made them. That evidence can then be used to decide which model should propose next. To this end, we introduce MOSAIC, in which each Monte Carlo tree search (MCTS) node records a program and the LLM that expands it, and the active LLM proposes both a compiler transformation and the LLM for the next step. We also contribute an LLM-aware UCT that favors smaller models and a course-alteration rule that invokes the largest model only after repeated small-model regressions. Against that single-LLM optimizer run with GPT-5.2 alone and the same number of search iterations and tree parameters, on an NVIDIA GPU (Intel CPU), MOSAIC with eight LLMs improves the geometric-mean speedup on five kernels by 7.2% (14.4%) and the end-to-end Llama-3-8B speedup by 1.61x (1.41x), while reducing kernel compilation time by 1.95x (1.74x) and API cost by 4.47x (4.32x), with GPT-5.2 making only 23.1% (23.9%) of LLM calls. With the same eight models, random or round-robin model choice improves over the single-LLM baseline by only 2.5% on CPU, showing that the gains come from letting the LLMs choose the next model rather than from mixing the pool at random.

Figures & tables

Figure 1: MOSAIC ’s LLMs collaborate only through one MCTS tree. (a) Reasoning Compiler uses one large LLM. (b) In MOSAIC , small LLMs and GPT-5.2 take turns (color = expanding LLM, green = active): the active LLM reads nodes others wrote (green band), LA-UCT favors small LLMs, GPT-5.2 proposes a replacement child after two consecutive small-LLM regressions (dashed), and

R

credits every LLM on the path. (c) One call to

\text{llm}_{t}

returns

\langle o_{t},\text{llm}_{t+1}\rangle

, picking the next LLM (black) from the pool.

Panel of Figure 2 — Figure 2: Speedup over unoptimized code as a function of the number of searched samples on 2 GPU and 2 CPU. RC is Reasoning Compiler ( Tang et al., 2025 ) run with the named LLM alone; MOSAIC uses GPT-5.2 as its largest model. Both panels share the legend of 2 ; Appendix D lists the final values.

Benchmark	Metric	GPT-5.2			Llama-3.3-70B-Instruct
Benchmark	Metric	MOSAIC (8 LLMs)	MOSAIC (4 LLMs)	MOSAIC (2 LLMs)	MOSAIC (8 LLMs)	MOSAIC (4 LLMs)	MOSAIC (2 LLMs)
Llama-3-8B Attention Layer	Comp. Time $\downarrow$ ( $\times$ )	1.85 /1.48	1.46 /1.09	1.01 /1.00	1.93	1.68	1.31
Llama-3-8B Attention Layer	API Cost $\downarrow$ ( $\times$ )	4.35 /3.06	3.30 /2.57	1.87 /2.64	2.04	1.38	1.14
DeepSeek-R1 MoE Layer	Comp. Time $\downarrow$ ( $\times$ )	1.79 /1.68	1.20 /1.49	1.05 /1.10	1.87	1.75	1.35
DeepSeek-R1 MoE Layer	API Cost $\downarrow$ ( $\times$ )	5.18 /4.18	2.88 /3.21	2.55 /2.42	2.35	1.59	1.15
FLUX Attention Layer	Comp. Time $\downarrow$ ( $\times$ )	2.01 /2.03	1.66 /1.18	1.04 /1.10	1.84	1.62	1.35

Table 1: Compilation time and API cost reduction of MOSAIC relative to Reasoning Compiler run with the largest model alone. Column groups indicate the largest model. GPU /CPU results are reported; Llama-3.3-70B-Instruct results are CPU only.

Largest Model: GPT-5.2 GPU /CPU				Largest Model: Llama-3.3-70B-Instruct (CPU)
Model	MOSAIC (8 LLMs)	MOSAIC (4 LLMs)	MOSAIC (2 LLMs)	Model	MOSAIC (8 LLMs)	MOSAIC (4 LLMs)	MOSAIC (2 LLMs)
GPT-5.2 (Regular)	9.8 /10.7	16.1 /17.8	26.2 /24.5	Llama-3.3-70B-Inst (Reg.)	18.6	30.5	45.5
GPT-5.2 (C.A.)	13.3 /13.2	12.2 /12.0	10.6 /10.5	Llama-3.3-70B-Inst (C.A.)	12.1	10.1	7.9
GPT-5.2 (Total)	23.1 /23.9	28.3 /29.8	36.7 /35.1	Llama-3.3-70B-Inst (Total)	30.7	40.6	53.4
gpt-5-mini	1.8 /7.8	33.3 /36.5	63.3 /64.9	gpt-5-mini	3.0	27.9	46.6
DS-R1-Qwen-7B	25.4 /12.6	—	—	DS-R1-Qwen-7B	8.9	—	—

Table 2: Invocation rates (%) of each model, averaged across the five benchmarks, for 2-, 4-, and 8-LLM configurations of MOSAIC on GPU /CPU (Llama-3.3-70B-Instruct: CPU only). C.A. denotes course-alteration calls.

Largest Model: GPT-5.2				Largest Model: Llama-3.3-70B-Instruct
MOSAIC Configuration	Speedup over RC $\uparrow$ ( $\times$ )	Comp. Time $\downarrow$ ( $\times$ )	API Cost $\downarrow$ ( $\times$ )	Speedup over RC $\uparrow$ ( $\times$ )	Comp. Time $\downarrow$ ( $\times$ )	API Cost $\downarrow$ ( $\times$ )
MOSAIC(8 LLMs)	1.61 /1.41	1.52 /1.71	3.94 /4.44	1.34	1.45	1.96
MOSAIC(4 LLMs)	1.86 /1.19	1.32 /1.27	2.81 /2.79	1.27	1.39	1.55
MOSAIC(2 LLMs)	1.11 /1.28	1.22 /1.10	2.19 /2.21	1.23	1.28	1.36

Table 3: End-to-end Llama-3-8B : final speedup improvement, compilation time reduction, and API cost reduction of MOSAIC over Reasoning Compiler run with the largest model alone. GPU /CPU results are reported; Llama-3.3-70B-Instruct results are CPU only.

Variant	L3 Attn.	MoE	FLUX Attn.	FLUX Conv.	L4 MLP	Geo- mean	vs. RC	Time	Cost
RC (GPT-5.2 only)	12.40	12.35	7.35	4.64	10.73	8.91	—	1.74 $\times$	4.32 $\times$
Random next LLM	12.82	12.54	7.37	4.93	10.83	9.12	+2.5%	1.37 $\times$	1.27 $\times$
Round-robin next LLM	12.96	12.69	7.42	4.82	10.80	9.13	+2.5%	1.38 $\times$	1.26 $\times$
MOSAIC(8 LLMs)	14.98	13.87	8.12	5.51	11.82	10.19	+14.4%	1.00 $\times$	1.00 $\times$
w/o size term ( $\lambda{=}0$ )	14.62	13.65	7.85	5.44	11.58	9.97	+12.0%	—	—
w/o course alteration	14.05	12.91	7.61	5.33	11.36	9.65	+8.3%	—	—

Table 4: Where the gains come from, on CPU at 1000 samples. All MOSAIC variants use the same eight-LLM pool and the same shared tree; only the listed component changes, and indented rows modify MOSAIC(8 LLMs) . RC is Reasoning Compiler. The last two columns give compilation time and API cost relative to MOSAIC (higher is more expensive).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Platform	Method	Llama-3 Attn.	DS-R1 MoE	FLUX Attn.	FLUX Conv.	L4-Scout MLP	Geomean
GPU	RC (gpt-5-mini)	29.49	29.56	25.05	15.60	27.90	24.86
	RC (GPT-5.2)	30.50	31.25	29.17	19.09	30.62	27.68
	MOSAIC(2 LLMs)	31.07	32.40	30.50	21.68	30.76	28.99
	MOSAIC(4 LLMs)	31.35	32.13	31.28	21.95	31.51	29.35
	MOSAIC(8 LLMs)	32.70	33.77	30.80	21.37	31.64	29.67
CPU	RC (gpt-5-mini)	11.84	11.10	7.31	4.17	10.26	8.37

Appendix

Table 5: Speedup over unoptimized code at 1000 samples. RC is Reasoning Compiler run with the named LLM alone. CPU (70B) uses Llama-3.3-70B-Instruct as the largest model.

Figure 3: CPU speedup over unoptimized code as a function of the number of searched samples, using Llama-3.3-70B-Instruct as the largest model. RC is Reasoning Compiler run with the named LLM alone.

Llama-3-8B Attention Layer	Setting	Runtime (sec)	Runtime Imp.	API Cost	API Cost Imp.
	Reasoning Compiler (GPT-5.2)	2236	—	$13.49	—
	2 Models	2227	1.00	$5.11	2.64
	4 Models	2051	1.09	$5.25	2.57
	8 Models	1514	1.48	$4.41	3.06
DeepSeek-R1 MoE Layer	Setting	Runtime (sec)	Runtime Imp.	API Cost	API Cost Imp.
	Reasoning Compiler (GPT-5.2)	2493	—	$13.75	—

Appendix

Table 6: Absolute compilation time and API cost on CPU for Reasoning Compiler with GPT-5.2 and for MOSAIC with 2, 4, and 8 LLMs.

Benchmark	$\lambda$ Value / Number of Samples	50	100	250	500	750	1000
Llama-3-8B Attention Layer	0.00	7.36	10.22	10.93	12.27	12.50	14.62
	0.25	7.49	10.49	10.82	12.51	12.70	14.85
	0.50	7.56	10.61	10.99	12.65	12.81	14.98
	0.75	7.30	10.19	10.61	12.16	12.40	14.52
	1.00	7.09	9.87	10.78	11.77	12.09	14.15
DeepSeek-R1 MoE Layer	0.00	7.22	9.01	11.64	12.05	12.37	13.65

Appendix

Table 7: Speedup over unoptimized code across varying numbers of samples for different choices of

\lambda

values.

Benchmark	$\lambda$ / Models	DeepSeek-R1- Distill-Qwen-7B	Llama-3.1- 8B-Instruct	Qwen3-8B	Qwen3-14B	Devstral- Small-2505	gpt-5-mini	DeepSeek-R1- Distill-Qwen-32B	gpt-5.2	gpt-5.2 Course Alteration
Llama-3-8B Attention Layer	0.00	16.2%	18.7%	10.0%	7.4%	1.3%	5.8%	4.8%	23.1%	12.7%
	0.25	13.2%	6.5%	28.0%	7.2%	3.6%	7.2%	1.0%	19.8%	13.5%
	0.50	11.6%	7.6%	34.1%	5.1%	1.0%	5.6%	2.5%	18.3%	14.3%
	0.75	15.4%	22.4%	16.8%	7.8%	1.0%	5.3%	1.8%	14.4%	15.1%
	1.00	14.4%	9.2%	32.1%	3.8%	4.5%	6.1%	1.3%	12.8%	16.0%
DeepSeek-R1 MoE Layer	0.00	10.3%	20.9%	7.3%	11.1%	3.0%	21.7%	1.9%	12.4%	11.4%

Appendix

Table 8: Invocation rates (%) of different models for different choices of

\lambda

values.

Benchmark	Course Alteration / Number of Samples	50	100	250	500	750	1000
Llama-3-8B Attention Layer	No Course Alteration	6.91	10.02	10.19	11.78	12.01	14.05
	Every 1 Small Model Regression	7.60	10.68	11.16	12.88	13.07	15.26
	Every 2 Small Model Regressions	7.56	10.61	10.99	12.65	12.81	14.98
DeepSeek-R1 MoE Layer	No Course Alteration	6.92	8.57	10.88	11.85	11.91	12.91
	Every 1 Small Model Regression	7.56	9.29	12.09	12.31	12.40	14.24
	Every 2 Small Model Regressions	6.99	8.66	11.84	12.20	12.28	13.87

Appendix

Table 9: Speedup over unoptimized code across varying numbers of samples for different course alteration settings.

Benchmark	Course Alteration / Models	DeepSeek-R1- Distill-Qwen-7B	Llama-3.1- 8B-Instruct	Qwen3-8B	Qwen3-14B	Devstral- Small-2505	gpt-5-mini	DeepSeek-R1- Distill-Qwen-32B	gpt-5.2	gpt-5.2 Course Alteration
Llama-3-8B Attention Layer	No Course Alteration	13.8%	17.5%	30.2%	5.3%	1.4%	7.8%	2.6%	21.4%	0.0%
	Every 1 Small Model Reg.	13.8%	12.5%	14.8%	4.0%	2.2%	8.7%	1.5%	11.7%	30.7%
	Every 2 Small Model Reg.	11.6%	7.6%	34.1%	5.1%	1.0%	5.6%	2.5%	18.3%	14.3%
DeepSeek-R1 MoE Layer	No Course Alteration	18.9%	23.7%	10.5%	1.1%	2.7%	30.9%	1.4%	10.8%	0.0%
	Every 1 Small Model Reg.	11.7%	15.8%	6.5%	3.4%	2.9%	21.2%	1.2%	8.7%	28.4%
	Every 2 Small Model Reg.	12.4%	23.5%	8.9%	1.8%	2.7%	27.3%	1.3%	9.6%	12.4%

Appendix

Table 10: Invocation rates (%) of different models for different course alteration settings.

	Llama-3-8B Attention Layer	DeepSeek-R1 MoE Layer	FLUX Attention Layer	FLUX Convolution Layer	Llama-4-Scout MLP Layer
Comp. Time $\downarrow$ ( $\times$ )	1.29	1.34	1.53	1.37	1.38
API Cost $\downarrow$ ( $\times$ )	1.30	1.57	1.69	1.80	1.58

Appendix

Table 11: Compilation time and API cost reduction of performing course alteration every two small model regression against performing course alteration every one small model regression.

Benchmark	LLM Selection / Number of Samples	50	100	250	500	750	1000
Llama-3-8B Attention Layer	MOSAIC(8 LLMs)	7.56	10.61	10.99	12.65	12.81	14.98
	Random	6.20	8.55	10.05	10.60	11.10	12.82
	Round-Robin	6.30	8.70	10.18	10.55	11.25	12.96
DeepSeek-R1 MoE Layer	MOSAIC(8 LLMs)	6.99	8.66	11.84	12.20	12.28	13.87
	Random	5.28	7.62	10.72	11.90	11.98	12.54
	Round-Robin	5.06	6.60	10.89	11.96	11.93	12.69

Appendix

Table 12: Speedup over unoptimized code across varying numbers of samples for different LLM selection settings.

Benchmark	LLM Selection / Models	DeepSeek-R1- Distill-Qwen-7B	Llama-3.1- 8B-Instruct	Qwen3-8B	Qwen3-14B	Devstral- Small-2505	gpt-5-mini	DeepSeek-R1- Distill-Qwen-32B	gpt-5.2	gpt-5.2 Course Alteration
Llama-3-8B Attention Layer	MOSAIC(8 LLMs)	11.6%	7.6%	34.1%	5.1%	1.0%	5.6%	2.5%	18.3%	14.3%
	Random	10.5%	10.4%	11.5%	9.9%	11.2%	10.7%	10.6%	10.1%	15.0%
	Round-Robin	10.7%	10.7%	10.7%	10.7%	10.7%	10.7%	10.7%	10.7%	14.4%
DeepSeek-R1 MoE Layer	MOSAIC(8 LLMs)	12.4%	23.5%	8.9%	1.8%	2.7%	27.3%	1.3%	9.6%	12.4%
	Random	11.3%	10.2%	11.5%	10.0%	10.7%	11.0%	10.3%	11.1%	13.9%
	Round-Robin	10.9%	10.9%	10.9%	10.9%	10.9%	10.9%	10.9%	10.9%	12.9%

Appendix

Table 13: Invocation rates (%) of different models for different LLM selection settings.

	Llama-3-8B Attention Layer	DeepSeek-R1 MoE Layer	FLUX Attention Layer	FLUX Convolution Layer	Llama-4-Scout MLP Layer
Comp. Time $\downarrow$ ( $\times$ )	1.17 / 1.18	1.21 / 1.24	1.39 / 1.45	1.49 / 1.44	1.64 / 1.62
API Cost $\downarrow$ ( $\times$ )	1.05 / 1.03	1.13 / 1.15	1.29 / 1.31	1.55 / 1.53	1.38 / 1.36

Appendix

Table 14: Compilation time and API cost reduction of MOSAIC(8 LLMs) against random / round-robin LLM selection settings.

		Number of Regular Model Calls								Number of Course Alterations
Benchmark	MOSAIC Combination	DeepSeek- R1-Distill- Qwen-7B	Llama-3.1- 8B- Instruct	Qwen3- 8B	Qwen3- 14B	Devstral- Small- 2505	gpt-5- mini	DeepSeek- R1-Distill- Qwen-32B	GPT-5.2	GPT-5.2
Llama-3-8B Attention Layer	MOSAIC(8 LLMs)	183	21	411	94	43	60	79	109	158
	MOSAIC(4 LLMs)	—	358	—	—	—	377	99	166	142
	MOSAIC(2 LLMs)	—	—	—	—	—	588	—	412	103
DeepSeek-R1 MoE Layer	MOSAIC(8 LLMs)	257	129	86	417	0	14	18	79	162
	MOSAIC(4 LLMs)	—	198	—	—	—	576	30	196	146

Appendix

Table 15: Model call counts for 2-, 4-, and 8-LLM configurations of MOSAIC across five benchmarks on GPU when GPT-5.2 is the largest model.

		Number of Regular Model Calls								Number of Course Alterations
Layer	Experiment	DeepSeek- R1-Distill- Qwen-7B	Llama-3.1- 8B- Instruct	Qwen3- 8B	Qwen3- 14B	Devstral- Small- 2505	gpt-5- mini	DeepSeek- R1-Distill- Qwen-32B	GPT-5.2	GPT-5.2
Llama-3-8B Attention Layer	MOSAIC(8 LLMs)	135	89	398	59	12	65	29	213	167
	MOSAIC(4 LLMs)	—	163	—	—	—	229	365	243	128
	MOSAIC(2 LLMs)	—	—	—	—	—	776	—	224	110
DeepSeek-R1 MoE Layer	MOSAIC(8 LLMs)	142	268	102	21	31	312	15	109	141
	MOSAIC(4 LLMs)	—	346	—	—	—	181	301	172	137

Appendix

Table 16: Model call counts for 2-, 4-, and 8-LLM configurations of MOSAIC across five benchmarks on CPU when GPT-5.2 is the largest model.

		Number of Regular Model Calls								Number of Course Alterations
Layer	Experiment	DeepSeek- R1-Distill- Qwen-7B	Llama-3.1- 8B- Instruct	Qwen3- 8B	Qwen3- 14B	Devstral- Small- 2505	gpt-5- mini	DeepSeek- R1-Distill- Qwen-32B	Llama-3.3- 70B- Instruct	Llama-3.3- 70B- Instruct
Llama-3-8B Attention Layer	MOSAIC(8 LLMs)	51	54	372	108	88	92	6	229	144
	MOSAIC(4 LLMs)	—	214	—	—	—	329	122	335	94
	MOSAIC(2 LLMs)	—	—	—	—	—	495	—	505	87
DeepSeek-R1 MoE Layer	MOSAIC(8 LLMs)	98	88	213	368	20	5	2	206	139
	MOSAIC(4 LLMs)	—	416	—	—	—	159	69	356	122

Appendix

Table 17: Model call counts for 2-, 4-, and 8-LLM configurations of MOSAIC across five benchmarks on CPU when Llama-3.3-70B-Instruct is the largest model.

Large Language Model-Guided Optimization

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 587 papers

Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization
May 19, 2026Dmitry Redko, Albert Fazlyev, Konstantin Sozykin +3GPU Kernel Optimization Large Language Model-Guided Optimization
Applied AI Institute · AI Talent Hub, ITMO University · YSDA +1
LLM Program Optimization via Retrieval Augmented Search
Jan 31, 2025Sagnik Anupam, Alexander Shypula, Osbert BastaniLarge Language Model-Guided Optimization In-Context Learning
University of Pennsylvania
Optimizing Prompt Sequences using Monte Carlo Tree Search for LLM-Based Optimization
Aug 8, 2025Fei Xu Yu, Gina Adam, Nathaniel D. Bastian +1Natural Language to Optimization Modeling Large Language Model-Guided Optimization
The George Washington University · United States Military Academy

May 19, 2026J/K move · Enter open · S save

Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization

Dmitry Redko, Albert Fazlyev, Konstantin Sozykin, Maria Ivanova+2

Applied AI Institute · AI Talent Hub, ITMO University · YSDA+1

GPU Kernel Optimization Large Language Model-Guided Optimization GPU Kernel Generation Code Optimization

Open paper AskPDF arXiv

LLM discovery and optimization systems are increasingly applied across domains, implementing a common propose-evaluate-revise loop. Such optimization or discovery progresses via context conditioning on received feedback from an environment. However, as modern LLM agents are increasingly complex in their structure, it is difficult to evaluate which components contribute the most, and when and how this exploration may fail. We answer these questions through three controlled experiments. Our findings: (1) In pure black-box optimization, LLMs act as greedy optimizers. (2) In zero-shot kernel generation, providing explicit input-size information has no measurable effect, models converge to the same kernel parameters regardless of size or temperature, as though the size instruction were invisible. Moreover, when tasked to perform kernel optimization for uncommon kernel sizes, performance sharply degrades regardless of the language used. (3) In feedback-loop kernel optimization, CUDA improves monotonically under iterative feedback, while TVM IR actively degrades, which demonstrates that kernel optimization degrades when models operate with low-density language. Our results conclude that LLMs in code optimization tasks highly depend on pretrained priors rather than provided feedback or agentic structure.

Abstract

Large Language Model-Guided Optimization

Explore similar work

Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization

LLM Program Optimization via Retrieval Augmented Search

Optimizing Prompt Sequences using Monte Carlo Tree Search for LLM-Based Optimization