FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications

Yunfan Zhang; Yijie Bei; Jetashree Ravi; Pawel Garbacki

cs.CLMar 5, 2026

FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications

Authors: Yunfan Zhang, Yijie Bei, Jetashree Ravi, Pawel Garbacki

Organizations: Columbia University · Fireworks AI

Abstract

Instruction following is critical for LLMs deployed in enterprise and API-driven settings, where strict adherence to output formats, content constraints, and procedural requirements is essential for enabling reliable LLM-assisted workflows. However, existing instruction following benchmarks predominantly evaluate natural language generation constraints that reflect the needs of chat assistants rather than enterprise users. To bridge this gap, we introduce FireBench, an LLM instruction following benchmark grounded in real-world enterprise and API usage patterns. FireBench evaluates six core capability dimensions across diverse applications including information extraction, customer support, and coding agents, comprising over 2,400 samples. We evaluate 11 LLMs and present key findings on their instruction following behavior in enterprise scenarios. We open-source FireBench at fire-bench.com to help users assess model suitability, support model developers in diagnosing performance, and invite community contributions.

Figures & tables

Figure 1: Overview of FireBench . The benchmark evaluates six instruction-following capabilities critical to enterprise and API settings, each grounded in real-world application domains.

Category	Task Design	#
Output Format (§ 3.1 )	Reasoning and long-document QA under 21 format constraints; coding problems under 3 structured formats. Verified programmatically.	1,300
Ordered Responses (§ 3.2 )	Simulated multi-turn customer support sessions in which the agent must collect 10–15 information fields in strict sequential order. Results are verified programmatically.	200
Item Ranking (§ 3.3 )	Table sorting operations over realistic records; models must return top- $N$ rows verbatim according to specified sorting criteria. Results are verified programmatically.	200
Overconfidence (§ 3.4 )	Challenging tasks : paired normal and uncertainty-aware prompts from challenging reasoning benchmarks. Insufficient information : unanswerable questions over long-form documents requiring explicit refusal. Models are evaluated on whether they appropriately decline questions they cannot solve or for which sufficient information is unavailable.	370
Positive Content (§ 3.5 )	Selected Arena Hard 2.0 prompts augmented with mandatory content requirements. Evaluated via LLM judge.	200
Negative Content (§ 3.6 )	Same prompts augmented with prohibitive constraints; models must avoid forbidden elements. Evaluated via LLM judge.	200

Table 1: Composition of FireBench . Each category targets a distinct instruction-following capability critical to production deployments.

Model	Format	Ordered Responses	Ranking	Overconfidence	Positive Content Requirements	Negative Content Requirements	Overall
DeepSeek V3.1 Terminus	54.3	65.5	92.5	86.0	84.0	81.4	74.0
GPT-5.1 Medium Thinking	62.2	53.5	93.0	67.7	86.5	84.0	72.7
GPT-4.1	86.9	76.3	32.5	38.6	94.5	94.5	70.5
Qwen3 235B Thinking 2507	39.9	72.0	76.0	42.0	85.5	84.2	66.6
Qwen3 235B Instruct 2507	65.6	69.8	38.0	44.2	87.0	93.5	66.3
Claude Sonnet 4.5	64.0	59.5	63.0	53.4	79.0	74.5	65.3

Table 2: Evaluation results on FireBench . Numbers denote the percentage of samples in which the model correctly performed the expected action in each category, as well as the overall score, computed as the average across categories. Higher is better. For each category and the overall score, the best result is shown in bold , and the second best is underlined . Results indicate that instruction following in enterprise and API settings remains challenging, with no model achieving more than 75% overall.

Figure 2: (a) Overall score and standard deviation across categories for the top-6 models by overall performance. All models exhibit high variance, indicating inconsistent instruction-following ability across categories. (b) Relative ranking of the same six models within each category. Model rankings shift substantially from one category to another, demonstrating that no single model dominates uniformly.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

IFHierBench: Hierarchical Instruction Following for Large Language Models
Jul 30, 2026Yuetian Mao, Chunyang ChenLLM Evaluation Language Model Generation Evaluation
Technical University of Munich
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
Mar 5, 2026Bosi Wen, Yilin Niu, Cunxiang Wang +5LLM-as-a-Judge Instruction Following
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University · Zhipu AI · University of Electronic Science and Technology of China
CCR-Bench: A Comprehensive Benchmark for Evaluating LLMs on Complex Constraints, Control Flows, and Real-World Cases
Mar 9, 2026Xiaona Xue, Yiqiao Huang, Jiacheng Li +9LLM Evaluation Instruction Following
Jiutian Artificial Intelligence Research Institute, China Mobile, Beijing, China

Jul 30, 2026J/K move · Enter open · S save

IFHierBench: Hierarchical Instruction Following for Large Language Models

Yuetian Mao, Chunyang Chen

Technical University of Munich

LLM Evaluation Language Model Generation Evaluation Instruction Following

Open paper AskPDF arXiv

Instruction-following ability is critical for deploying large language models in real-world applications, where downstream components depend on the output satisfying specific constraints. Modern deployments increasingly handle the full task in a single LLM call, with one prompt specifying a layered output whose overall artifact, structural sections, and nested fields must each satisfy concrete constraints. Existing instruction-following benchmarks treat the constraint set as a flat list applied uniformly to the response, so they cannot scope a check to a particular section of the output. We introduce IFHierBench, a hierarchical instruction-following benchmark of 600 prompts stratified across four constraint-tree depths and 35 distinct constraints, each prompt paired with a deterministic checker that verifies satisfaction at every scope. Evaluating seven leading proprietary and open-weight models, we find that even the strongest model only marginally exceeds 50% prompt-level accuracy and that accuracy degrades sharply as constraint depth grows. Reliably following nested constraints remains a substantial gap for current LLMs, motivating future training methods that consider constraint adherence at finer granularity to achieve better instruction-following ability.

Abstract

LLM Evaluation

Explore similar work

IFHierBench: Hierarchical Instruction Following for Large Language Models

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

CCR-Bench: A Comprehensive Benchmark for Evaluating LLMs on Complex Constraints, Control Flows, and Real-World Cases