CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models

Tim Lukas Adam; Phongsakon Mark Konrad; Riccardo Terrenzi; Florian Girardo Lukas; Rahime Yilmaz; Krzysztof Sierszecki; Serkan Ayvaz

cs.SEApr 7, 2026

CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models

Authors: Tim Lukas Adam, Phongsakon Mark Konrad, Riccardo Terrenzi, Florian Girardo Lukas, Rahime Yilmaz, Krzysztof Sierszecki, Serkan Ayvaz

Organizations: Centre for Industrial Software, University of Southern Denmark, Alsion 2, Sønderborg, 6400, Denmark

Abstract

In today's software architecture, large language models (LLMs) serve as software architecture co-pilots. However, no benchmark currently exists to evaluate large language models' actual understanding of cloud-native software architecture. For this reason we present a benchmark called CAKE, which consists of 188 expert-validated questions covering four cognitive levels of Bloom's revised taxonomy -- recall, analyze, design, and implement -- and five cloud-native topics. Evaluation is conducted on 22 model configurations (0.5B--70B parameters) across four LLM families, using three-run majority voting for multiple-choice questions (MCQs) and LLM-as-a-judge scoring for free-responses (FR). Based on this evaluation, four notable findings were identified. First, MCQ accuracy plateaus above 3B parameters, with the best model reaching 99.2%. Second, free-response scores scale steadily across all cognitive levels. Third, the two formats capture different facets of knowledge, as the MCQ accuracy approaches a ceiling while free-responses continue to differentiate models. Finally, reasoning augmentation (+think) improves free-response quality, while tool augmentation (+tool) degrades performance for small models. These results suggest that the evaluation format fundamentally shapes how we measure architectural knowledge in LLMs.

Figures & tables

Family Sizes Base +think +tool Total Qwen 0.5–30B 6 – – 6 Llama 1–70B 4 3 3 10 Mistral 3–14B 3 1 – 4 GPT 4o-Mini, 5-Mini 2 – – 2 Total 15 4 3 22

Count of configurations per family. Configurations consist of base (direct question answering), +think (structured reasoning with Chain-of-Thought), and +tool (agentic tool use enabled).

TABLE I: Evaluated Model Families and Configurations.

Fig. 1: Question distribution across five cloud-native topics and four cognitive levels. (A) Full Bench (188 evaluated questions; 12 implement-level MCQs excluded due to a formatting defect). (B) CAKE-Core subset (116 questions passing mean correctness $\geq$ 4.0 and clarity $\geq$ 4 filters, with no flags). Cell values show question counts; color intensity indicates density.

Fig. 2: Distribution of expert ratings across clarity, correctness, and difficulty accuracy. Violin plots show density; box overlays show median and IQR. High means (4.72, 4.63) with narrow spread confirm benchmark quality.

Model	Params	Cfg	MCQ	FR	Time	VRAM
			(%) $\uparrow$	Overall $\uparrow$	(s) $\downarrow$	(GB)
Qwen
Qwen 2.5	0.5B	base	47.7	1.99	1.2	30.5
Qwen 2.5	1.5B	base	91.5	2.71	1.0	31.8
Qwen 2.5	3B	base	95.4	3.06	0.9	28.7
Qwen 2.5	7B	base	96.9	3.02	1.1	28.7

TABLE II: Performance and resources for all 22 configurations, grouped by family (

\uparrow

higher is better). GPT models show “—” for Params and VRAM as they run via OpenRouter.

Fig. 3: Free-response judge scores (0–5) for all 22 configurations, ranked per panel, (A) Analyze, (B) Design, (C) Implement and (D) Overall. Each panel shows one cognitive level; color indicates model family. Mistral models consistently rank highest among local models.

Fig. 4: Augmentation effects on MCQ and Free-response performance. (A) MCQ accuracy delta. (B) FR overall delta for +think and +tool variants relative to their base models. Color indicates model family, hatching indicates +tool.

Fig. 5: Full Bench vs. CAKE-Core MCQ accuracy for all 22 configurations. Darker bars show Full Bench; lighter bars show CAKE-Core. Delta values at bar ends indicate the accuracy change after quality filtering. Ranking order is largely preserved across model families.

Fig. 6: Free-response judge scores (0–5) vs. model parameters across cognitive levels. (A) Analyze, (B) Design, (C) Implement, and (D) Overall. Scores scale consistently with size for all families; GPT model baselines shown as dashed lines. Unlike MCQ (which saturates above 3B), free-response continues differentiating across the full parameter range.

Fig. 7: Per-topic performance across all 22 configurations. (A) MCQ accuracy (%) aggregated across recall, analyze, and design levels. (B) free-response judge scores (0–5) aggregated across analyze, design, and implement levels. Color intensity indicates performance; white separators mark family boundaries.

Fig. 8: Expert–model alignment analysis. (A) Expert difficulty ratings show no significant correlation with mean model accuracy across 22 configurations. (B) Questions flagged for ambiguity or typos show no accuracy difference from unflagged questions.

Software Architecture Design

See all 22 papers

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models
Jun 28, 2026Tiziano Santilli, Francesco Daghero, Mayhar Tourchi MoghaddamSoftware Engineering LLM Evaluation
University of Southern Denmark, Odense 5300, DK
Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment
Jun 12, 2026Kirill Vasilevski, Ximing Dong, Benjamin Rombaut +8Software Engineering Agents LLM-as-a-Judge
Justina · Centre for Software Excellence, Huawei Canada · School of Computing, Queen’s University, Canada +1
ArchitectureIQ: On the Measure of Training Intuition
Sep 30, 2026Zirui Ren, Shaoyang Guo, Chencheng Tang +9Model Selection LLM Evaluation
MetaCircle (元环智能) · Tsinghua University · Peking University +4

Jun 28, 2026J/K move · Enter open · S save

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

Tiziano Santilli, Francesco Daghero, Mayhar Tourchi Moghaddam

University of Southern Denmark, Odense 5300, DK

Software Engineering LLM Evaluation Software Engineering Benchmarks Software Architecture Design

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software architecture remains largely unmeasured. Architectural decision-making depends on quality attribute trade-offs, design patterns, and system-level constraints, none of which are exercised by benchmarks that target syntactic or algorithmic tasks. We introduce SAKE (Software Architectural Knowledge Evaluation), a standardized and reproducible benchmark for assessing software architectural knowledge in LLMs. SAKE comprises 2154 expert-curated multiple-choice questions, each with four options, stratified across eight architectural categories and four context-length levels. We evaluate 11 proprietary and open-weight models in zero-shot and five-shot settings. Overall accuracy is high, but performance varies markedly across categories, revealing competency gaps in areas central to professional practice. SAKE, its evaluation scripts, and all results are released as open source to give the community a baseline for tracking architectural reasoning in LLMs.

Abstract

Software Architecture Design

Explore similar work

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment

ArchitectureIQ: On the Measure of Training Intuition