cs.AIOct 9, 2026

Unconstrained CTI-REALM with Open-Weights

Authors: Sahil Garg, Ruhi Gottumukkala, Vishal Agarwal, Shubham Vishwakarma

Organizations: Averlon, Redmond, WA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and far more expensive closed frontier models. We study that choice on CTI-REALM (arXiv:2603.13517), a benchmark in which an agent turns a threat-intelligence report into a working detection by querying a live telemetry store. We evaluate twenty-six models, nineteen open-weight and seven closed, under a common harness, scorer and panel of LLM judges. Our main finding is that an open-weight model can match the strongest closed models at less than a tenth of their cost, provided it is given enough turns. The benchmark's published limit on agent messages is a confound: a task that exhausts the limit scores zero on every checkpoint, including those it had already passed, so results at that limit partly measure which models finish in time. At the published limit the leading open-weight and closed models cannot be distinguished. With the limit relaxed, the models it was truncating recover, and the highest rewards belong to open-weight models. Among the leading models, price is a poor guide to capability: models with similar rewards differ in cost by more than an order of magnitude. Because the benchmark's checkpoint scores largely record what an agent did or mentioned, we also introduce stage-level metrics that score what the agent submits, by executing its detection queries in the telemetry store and judging its outputs against the answer key. Rather than combining them into one more overall reward, we show that different models lead different stages, which suggests routing each stage of the workflow to the model that performs it best rather than choosing a single model by its overall score. We also report how many tokens each model generates, and find that the share of output spent on thinking does not track the reward.

Figures & tables

DeveloperModels evaluated
Open-weight (19 models)
Zhipu AIGLM-5.3-flash, GLM-5.3, GLM-5.2
DeepSeekDeepSeek-v4.1-flash, DeepSeek-v4-flash, DeepSeek-v4-pro
AlibabaQwen3.8-2.4T, Qwen3.8-27B, Qwen3.6-35B-A3B
Moonshot AIKimi-K3
MiniMaxMiniMax-M3
Table 1: The twenty-six models evaluated, grouped by whether their weights are published. The closed models are available only through their developers’ APIs. Mistral-Large-4 is counted as open-weight: its developer released it on 6 October 2026 with its weights announced for publication by the end of that month.
Figure 1: Overall reward against the cost of the run at the unconstrained budget of 300 agent messages. This figure and Figure 2 share axes so that they can be compared. Open-weight models are drawn as blue circles and closed models as grey triangles. A hollow marker is a carried result: no sample of that model reached the cap at a lower budget, so the point is its result at that budget and not a run at 300. The highest reward belongs to GLM-5.3-flash, an open-weight model, at $2. Note that the least expensive open-weight run shown is GPT-OSS-20B at 51¢.
Figure 2: Overall reward against the cost of the run at the benchmark’s published limit of 70 agent messages. This figure and Figure 1 share axes so that they can be compared. Open-weight models are drawn as blue circles and closed models as grey triangles. The highest reward belongs to Kimi-K3, an open-weight model, at $27. Note that the least expensive open-weight run shown is GPT-OSS-20B at 41¢.
ModelC0C1C2C3C4overallcost
 judgedJaccardJaccardran ≥\geq 2query F1reward 
GLM-5.3-flash0.880.730.870.980.620.74$2
Kimi-K30.860.720.850.980.580.72$32
Grok-4.60.570.490.870.980.680.71$28
GLM-5.30.900.700.860.960.540.70$20
Claude-Opus-50.960.710.851.000.500.69$50
Table 2: Results for all 26 models at the unconstrained budget of 300 agent messages, ordered by overall reward. Columns C0 to C4 are the benchmark’s own five checkpoint scores, each on its own 0–1 scale: C0 is the judged score for reading the threat-intelligence report, C1 and C2 are the Jaccard scores for the ATT&CK techniques and the telemetry tables, C3 is the share of tasks in which at least two queries returned rows, and C4 is the F1 of the detection query. Overall is the weighted reward of Eq. ( 1 ); it also counts the judged quality of the Sigma rule, which has no column here. No sample reaches the message cap at this budget. Cost is the total cost of the run. In each score column the three best values are marked first , second and third , ranked on the precision shown so that equal printed values share a rank; the cost column, for which a lower value is better, is left unranked.
Figure 3: Tokens generated per run and per task, at the unconstrained budget. (a) output tokens summed over the 50 tasks, split into thinking (dark) and the answer and tool calls (light); the number at the end of each bar is the thinking share, the fraction of output tokens spent on thinking. Blue marks open-weight models and grey closed ones, with the marker shapes of Figure 1 . (b) the same output spread over the 50 tasks: the box spans the middle half of the tasks, the line is the median and the whiskers reach the 10th and 90th percentiles. The carried models are those drawn with hollow markers in Figure 1 . † marks a thinking share that is a lower bound. Scorer and judge-panel calls are excluded.
ModelBenchmark score and diagnostics  
 score  
(judged)opened any reportreports per task 
GLM-5.3-flash0.881.0010.8
Kimi-K30.861.008.9
Grok-4.60.571.0016.0
GLM-5.30.901.0012.0
Table 3: C0, threat-intelligence research, at the unconstrained budget. Benchmark: the score of the agent’s research, graded with the benchmark’s judge prompt by the reward’s panel (Section 2 ), and two diagnostics we compute, the share of tasks in which the agent opened at least one report and the reports it opened per task. The answer key holds no reference for C0, so there is no metric of ours. Rows follow the benchmark’s overall reward (Table 2 ); colours mark the first , second and third best values of the score, equal values sharing a colour.
Figure 4: The three columns of Table 3 at the unconstrained budget, for the five models with the highest overall reward, in the colours of the figures for C1 to C4; reports per task are divided by the largest value any model reaches. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label.
ModelBenchmark score and diagnostics  Sigma-rule tags, judged (ours)  
 Jaccard     
(key recall)exact matchmacro F1precisionrecallF1 
(Sigma rule’s techniques)      
GLM-5.3-flash0.730.660.580.720.830.74
Kimi-K30.720.620.630.790.790.76
Grok-4.60.490.440.390.710.780.71
Table 4: C1, ATT&CK technique mapping, at the unconstrained budget. Benchmark: the benchmark’s Jaccard index between the key and the key’s technique identifiers found anywhere in the agent’s output, which therefore equals the recall of the key, and two diagnostics we compute, the share of tasks matched exactly and the macro F1 (F1 per technique class, averaged with equal weight). Sigma-rule technique tags (ours): F1 scores the technique mapping the agent commits to, the ATT&CK techniques tagged in its submitted Sigma rule, against the key. In each task, precision is the share of the rule’s tags that the judges match to a key technique, whatever the identifier; recall is the share of the key’s techniques that some tag matches; F1 is their harmonic mean. A tag counts in proportion to the trials, three LLM judges in three trials each, that match it; a tag the judges accept but the key omits counts in neither; an untagged rule scores zero. Per-task values are averaged over the 50 tasks, so F1 can lie below both precision and recall. Rows follow the benchmark’s overall reward (Table 2 ); colours mark the first , second and third best values of the Jaccard, the exact match, the macro F1 and our F1, equal values sharing a colour.
Figure 5: Key metrics of C1 at the unconstrained budget, the benchmark’s and ours (marked “ours”), for the five models with the highest overall reward, the same five in the same colours in every checkpoint’s figure. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label. Table 4 gives every metric for every model.
ModelBenchmark score and diagnostics  Tables queried, judged (ours)  
 Jaccard     
(key recall)exact matchmacro F1precisionrecallF1 
(submitted query’s tables)      
GLM-5.3-flash0.870.720.831.000.940.96
Kimi-K30.850.700.831.000.910.94
Grok-4.60.870.720.830.980.900.92
Table 5: C2, telemetry table discovery, at the unconstrained budget. Benchmark: the benchmark’s Jaccard index between the key and the key tables the agent passed to its two exploration tools, which therefore equals the recall of the key, and two diagnostics we compute, the share of tasks matched exactly and the macro F1 (F1 per table, averaged with equal weight). Tables read by the submitted query (ours): precision, recall and F1 of the tables the submitted query reads, against the tables that record the attack in the store: key tables whose reference queries return events or in which three LLM judges, in three trials each, find the attack, with copies of their events standing in for them, and tables outside the key in proportion to the judge trials that find the attack in them; per-task values averaged over the 50 tasks, so F1 can lie below both precision and recall. Rows follow the benchmark’s overall reward (Table 2 ); colours mark the first , second and third best values of the Jaccard, the exact match, the macro F1 and our F1, equal values sharing a colour.
Figure 6: Key metrics of C2 at the unconstrained budget, the benchmark’s and ours (marked “ours”), for the five models with the highest overall reward, the same five in the same colours in every checkpoint’s figure. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label. Table 5 gives every metric for every model.
ModelBenchmark score and diagnostics  Submitted query (ours)
 two distinct queries returned rowsqueries per taskno query returned rowssubmitted query tested
GLM-5.3-flash0.9811.70.000.88
Kimi-K30.986.60.000.98
Grok-4.60.988.50.000.96
GLM-5.30.9611.00.000.92
Claude-Opus-51.006.30.000.90
Table 6: C3, query execution, at the unconstrained budget. Benchmark: the share of tasks in which at least two distinct queries returned rows, and two diagnostics we compute, the queries executed per task and the share of tasks in which no query returned rows. Submitted query tested (ours): the share of the 50 tasks in which a query the agent executed, with the selection logic of its submitted KQL query, returned rows, a near miss counting when it returns the same rows. Rows follow the benchmark’s overall reward (Table 2 ); colours mark the first , second and third best values of the benchmark’s score and of ours, equal values sharing a colour.
Figure 7: Key metrics of C3 at the unconstrained budget, the benchmark’s and ours (marked “ours”), for the five models with the highest overall reward, the same five in the same colours in every checkpoint’s figure. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label. Table 6 gives every metric for every model. Queries per task are divided by the largest value any model reaches.
ModelBenchmark scores and diagnostic  Events and Sigma grade (ours)   
 query F1zero query F1Sigma    
(judged)event precisionevent recallevent F1Sigma grade   
(judged)       
GLM-5.3-flash0.620.180.880.680.850.700.88
Kimi-K30.580.240.870.720.800.700.90
Grok-4.60.680.120.840.780.860.780.88
Table 7: C4, detection, at the unconstrained budget. Benchmark: the F1 over the key’s field patterns of the rows the agent pasted into its answer, without running the query; a diagnostic we compute, the share of tasks whose query F1 is zero; and the quality of the Sigma rule, graded with the benchmark’s judge prompt by the reward’s panel (Section 2 ). Executed submission and Sigma grade (ours): precision, recall and F1 of the events the submitted query returns when executed in full, against those of the key’s reference queries, with submissions returning aggregates, about one in ten, graded by three LLM judges; per-task values averaged over the 36 tasks in which every reference query returns events, so F1 can lie below both precision and recall; and the grade of the Sigma rule against the key by the same judges, averaged over the 50 tasks. Rows follow the benchmark’s overall reward (Table 2 ); colours mark the first , second and third best values of the query F1, both Sigma columns and the event F1, equal values sharing a colour.
Figure 8: Key metrics of C4 at the unconstrained budget, the benchmark’s and ours (marked “ours”), for the five models with the highest overall reward, the same five in the same colours in every checkpoint’s figure. The shaded hull joins the best value that any of the 26 models reaches on each axis, which is printed under the axis label. Table 7 gives every metric for every model.
More in this topic

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 516 papers

Explore similar work

CardsList
May 1, 2026

AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?

Production agentic systems make many model calls per user request, and most of those calls are short, structured, and routine. This raises a practical routing question that existing evaluations do not directly answer: which parts of an agent workflow truly require large frontier intelligence, and which can be handled by smaller models? We introduce AgentFloor, a deterministic 30-task benchmark organized as a six-tier capability ladder, spanning instruction following, tool use, multi-step coordination, and long-horizon planning under persistent constraints. We evaluate 16 open-weight models, from 0.27B to 32B parameters, alongside GPT-5 across 16,542 scored runs. Our results reveal a clear boundary of model necessity. Small and mid-sized open-weight models are already sufficient for much of the short-horizon, structured tool use work that dominates real agent pipelines, and in aggregate, the strongest open-weight model matches GPT-5 on our benchmark while being substantially cheaper and faster to run. The gap appears most clearly on long-horizon planning tasks that require sustained coordination and reliable constraint tracking over many steps, where frontier models still hold an advantage, though neither side reaches strong reliability. We also find that this boundary is not explained by scale alone: some failures respond to targeted interventions, but the effects are model-specific rather than universal. These findings suggest a practical design principle for agentic systems: use smaller open-weight models for the broad base of routine actions, and reserve large frontier models for the narrower class of tasks that truly demand deeper planning and control. We release the benchmark, harness, sweep configurations, and full run corpus.

Ranit Karmakar, Jayita Chatterjee

Harvard University, Boston, MA 02115 · Boston, MA 02115

Computer-Use Agent BenchmarksLLM Agent EvaluationLong-Horizon Agent EvaluationAI Agent BenchmarksTool-Using Language AgentsLLM Agent Planning
AskPDF
Mar 10, 2026

CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?

Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports. This process usually follows a three-stage workflow -- triage, deep search and TI drafting. While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations. These benchmarks often consist of tasks that do not reflect real-world analyst workflows. For example, human analysts rarely receive tasks in the form of multiple-choice questions. Also, existing benchmarks often rely on model-centric metrics that emphasize lexical overlap rather than actionable, detailed insights essential for security analysts. Moreover, they typically fail to cover the complete three-stage workflow. To address these issues, we introduce CyberThreat-Eval, which is collected from the daily CTI workflow of a world-leading company. This expert-annotated benchmark assesses LLMs on practical tasks across all three stages as mentioned above. It utilizes analyst-centric metrics that measure factual accuracy, content quality, and operational costs. Our evaluation using this benchmark reveals important insights into the limitations of current LLMs. For example, LLMs often lack the nuanced expertise required to handle complex details and struggle to distinguish between correct and incorrect information. To address these challenges, the CTI workflow incorporates both external ground-truth databases and human expert knowledge. TRA allows human experts to iteratively provide feedback for continuous improvement. The code is available at GitHub and HuggingFace.

Xiangsen Chen, Xuan Feng, Shuo Chen +5

Microsoft Research · Hong Kong University of Science and Techonology (Guangzhou) · Microsoft +1

LLM EvaluationCyber Threat IntelligenceCybersecurity
AskPDF
Jun 19, 2026

Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs

Open-weight Large Language Models (LLMs) enable scientific progress and broad deployment. However, they make it difficult to control access to sensitive capabilities. Current practice either suppresses dangerous capabilities before release or mediates access through closed services that use specialized model variants, input/output monitors, and API permissions. The former is susceptible to jailbreaks while sacrificing capability for all users to mitigate the risks posed by a few, and the latter is fundamentally incompatible with open-weight release. In this paper, we propose Tiered Language Models (TLMs), where a single set of released weights supports multiple capability levels. In its default public configuration, a TLM behaves as a conventional LLM. A compact secret key specifies a permutation over a small parameter subset, inducing an alternative computation graph over the same weights that exposes additional capabilities. We develop a training protocol that jointly pretrains both configurations from scratch, then fine-tunes the keyed configuration on private data with regularization to preserve the public model's behavior. We pretrain 180M- and 650M-parameter TLMs and demonstrate that the keyed configuration can acquire a new language, gain instruction-following ability, and memorize private factual knowledge, whereas the public configuration exhibits none of these capabilities. Moreover, we show that our approach extends naturally to multiple hierarchical tiers. Because authorization operates on the model's weight structure rather than in the input space, the mechanism resists fine-tuning-based extraction and partial key compromise. In general, TLMs take a step toward reconciling open-weight release with selective capability control.

Charbel El Feghali, Arkil Patel, Nicholas Meade +3

McGill-NLP/tiered-language-models

Open-Weight Language ModelsLLM Security
AskPDF