SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations

Ivo Brett

cs.SEMar 16, 2026

SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations

Authors: Ivo Brett

Organizations: independent.academia.edu/ivobrett

Abstract

As telecommunications operators accelerate adoption of AI-enabled automation, a practical question remains unresolved: can general-purpose large language model (LLM) agents reliably execute telecom operations workflows through real API interfaces, or do they require structured domain guidance? We introduce SKILLS (Structured Knowledge Injection for LLM-driven Service Lifecycle operations), a benchmark framework comprising 37 telecom operations scenarios spanning 8 TM Forum Open API domains (TMF620, TMF621, TMF622, TMF628, TMF629, TMF637, TMF639, TMF724). Each scenario is grounded in live mock API servers with seeded production-representative data, MCP tool interfaces, and deterministic evaluation rubrics combining response content checks, tool-call verification, and database state assertions. We evaluate open-weight models under two conditions: baseline (generic agent with tool access but no domain guidance) and with-skill (agent augmented with a portable SKILL.md document encoding workflow logic, API patterns, and business rules). Results across 5 open-weight model conditions and 185 scenario-runs show consistent skill lift across all models. MiniMax M2.5 leads (81.1% with-skill, +13.5pp), followed by Nemotron 120B (78.4%, +18.9pp), GLM-5 Turbo (78.4%, +5.4pp), and Seed 2.0 Lite (75.7%, +18.9pp).

Tool-Augmented Language Model Agents

Tool-augmented language model agents use external tools, APIs, and environments to solve tasks beyond text generation alone.

See all 167 papers

TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?
May 18, 2026Jieting Xiao, Yun Lin, Huizhen Qiu +10LLM Evaluation AI Agent Benchmarks
ZTE Corporation China
TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents
Mar 16, 2026Lina Bariah, Brahim Mefgouda, Farbod Tavakkoli +3AI Agent Evaluation AI Agent Benchmarks
Research Institute for Digital Future, Khalifa University · AT&T CDO · GSMA
TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Tasks
Sep 21, 2026Bohao Wang, Chenwei Wu, Hang Zou +7LLM Grounding RL for Language Model Reasoning
College of Information Science and Electronic Engineering, Zhejiang University, 310027 Hangzhou, China · Research Institute for Digital Future, Khalifa University, P O Box 127788, Abu Dhabi, UAE · Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109-2122, USA +1

May 18, 2026J/K move · Enter open · S save

TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?

Jieting Xiao, Yun Lin, Huizhen Qiu, Rui Ma+9

ZTE Corporation China

LLM Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

While Large Language Models have achieved remarkable integration in various vertical scenarios, their deployment in the telecommunications domain remains exploratory due to the lack of a standardized evaluation framework. Current telecom benchmarks primarily focus on static, foundational knowledge and isolated atomic skills, neglecting the equipment-specific documentation and end-to-end industrial workflows essential for real-world production systems. To bridge this gap, we present TeleCom-Bench, a comprehensive benchmark comprising 12 evaluation sets with 22,678 curated samples, which evaluates LLMs across a synergistic hierarchy: (1) Multi-dimensional Knowledge Comprehension, which integrates telecommunication fundamentals, 3GPP protocols, and 5G network architecture with proprietary product knowledge across wired, core, and wireless networks via knowledge graph-driven synthesis; and (2)End-to-End Knowledge Application, which formalizes six core tasks on authentic trajectories from live network agent workflows, including intent recognition, entity extraction, event verification, tool invocation, root cause analysis, and solution generation-across network optimization and fault maintenance scenarios. Evaluations of eight state-of-the-art LLMs reveal a universal Execution Wall: while models achieve 90% accuracy in linguistic interface tasks such as intent recognition and entity extraction, performance collapses to approximately 30% in procedural execution tasks like solution generation. This capability gap demonstrates that current LLMs function competently as diagnosticians but fail as field engineers. TeleCom-Bench provides standardized diagnostics to precisely pinpoint this deficit, offering actionable guidance for domain-specific alignment toward production-ready telecom agents. The dataset and evaluation code have been released at https://github.com/ZTE-AICloud/TeleCom-Bench.

Abstract

Tool-Augmented Language Model Agents

Explore similar work

TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?

TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents

TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Tasks