RFCLLM: Evaluating LLMs' Reasoning Ability of Network Protocol State Machines

Anqi Chen; Dan Goldwasser; Cristina Nita-Rotaru

cs.CLSep 11, 2026

RFCLLM: Evaluating LLMs' Reasoning Ability of Network Protocol State Machines

Authors: Anqi Chen, Dan Goldwasser, Cristina Nita-Rotaru

Organizations: Northeastern University · Purdue University

Abstract

Mapping textual specifications into formal representations is essential for ensuring the correctness of protocol designs and implementations. LLM-generated mappings, used for networking security or testing, are assumed to capture a perfect understanding of the specification, which may not hold in practice. The goal of this paper is to assess the extent to which LLMs can interpret the specification correctly. We examine the degree to which an LLM's implicit representation of a finite-state transition system-defined via natural language descriptions-aligns with a manually generated ground-truth model. We designed 4 tasks and 1482 task queries for 16 protocols. We evaluated different judge biases, observed the inherent difficulty gaps between tasks, looked into the effect of 4 context types, and the influence of protocol characteristics. Our work contributes to a step toward verifying whether LLMs can really be trusted in FSM (Finite State Machine) reasoning of protocol specifications.

Figures & tables

Figure 1: SCTP Ambiguity Ginesin et al. (2024) .

Figure 2: RFCLLM Workflow

Protocol	#States	#Trans (Task1 #Q)	#Valid paths	Avg. reachable path length	Avg. reachable paths set size	FSM presentation in RFC	Task2 #Q	Task3 #Q	Task4 #Q	Avg. Score (task1-4)
RTSP	3	33	36	4.28	6.00	FSM table	8	22	6	0.25
MQTT	12	17	115	8.93	1.53	No state terms	17	17	132	0.33
PPP	10	34	3001	12.49	33.34	Harder FSM table	22	28	90	0.34
TCP_nr	4	15	26	5.00	2.17	Partial state terms	12	11	12	0.37
FTP	10	24	182	6.66	3.57	No state terms	19	17	90	0.37
BGP	6	24	563	8.16	18.77	Clear text FSM	15	17	30	0.39

Table 1: Protocol characteristics and task workload summary. #States and #Trans describe the extracted FSM size. #Q is the number of queries evaluated per protocol for each task. Ranked from low to high by the average score of tasks 1,2,3,4: a smaller number means harder.

Judge	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5
Sim-0.6	0.23	0.37	0.43	0.53	0.56	0.63	0.60	0.58
Llama3.3-70b	0.17	0.28	0.17	0.44	0.46	0.52	0.49	0.49

Table 2: Judge score on Task 3 across answering LLMs.

Metric	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5	Avg.
$s^{\prime}$ (Task1 next state)	0.74	0.71	0.78	0.81	0.82	0.84	0.85	0.87	0.80
$A^{\prime}$ (Task1 actions)	0.38	0.41	0.44	0.48	0.48	0.49	0.50	0.51	0.46
Both (Task1 both)	0.34	0.37	0.41	0.45	0.44	0.46	0.48	0.50	0.43
$C^{\prime}$ (Task2 condition(s))	0.30	0.33	0.32	0.34	0.40	0.35	0.42	0.42	0.36
$S_{i}$ (Task3 previous state(s))	0.20	0.32	0.38	0.48	0.51	0.58	0.54	0.54	0.44
$P^{\prime}$ (Task4 path(s) set)	0.06	0.33	0.27	0.39	0.38	0.41	0.34	0.37	0.32

Table 3: Task difficulty on Task 1, 2, 3, 4 (smaller number is harder).

Context Type	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5
Easy	0.69	0.90	0.87	0.76	0.92	0.89	0.92	0.92
Manual-extracted-RFC	0.23	0.19	0.19	0.25	0.26	0.23	0.29	0.28
LLM-extracted-RFC	0.19	0.12	0.15	0.17	0.20	0.15	0.25	0.22
No-context	0.11	0.09	0.07	0.20	0.21	0.14	0.22	0.24

Table 4: Effect of context type on Task 2 (smaller number is harder).

Figure 3: Per-protocol-llm Performance on Task 4.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: IMAP ASCII FSM

Panel of Figure 5 — Figure 5: RTSP FSM table for state Init.

Judge	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5
Sim-0.6	0.48	0.48	0.52	0.56	0.54	0.55	0.58	0.59
Llama3.3-70b	0.50	0.52	0.57	0.60	0.62	0.65	0.64	0.66

Appendix

Table 5: Judge score on Task 1 across answering LLMs.

Judge	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5
Sim-0.6	0.29	0.30	0.31	0.31	0.35	0.29	0.41	0.40
Llama3.3-70b	0.31	0.36	0.33	0.38	0.44	0.42	0.43	0.43

Appendix

Table 6: Judge score on Task 2 across answering LLMs.

Judge	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5
Llama3.3-70b	0.06	0.33	0.27	0.38	0.38	0.41	0.34	0.35
Gemini-3-f-p	0.06	0.33	0.27	0.40	0.39	0.42	0.35	0.38

Appendix

Table 7: Judge score on Task 4 across answering LLMs.

Context Type	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5
Easy	0.94	0.93	0.95	0.94	0.95	0.95	0.95	0.95
Manual-extraced-RFC	0.43	0.41	0.50	0.55	0.55	0.56	0.55	0.59
LLM-extracted-RFC	0.33	0.34	0.39	0.42	0.42	0.46	0.49	0.49
No-context	0.25	0.30	0.33	0.41	0.40	0.43	0.45	0.48

Appendix

Table 8: Effect of context type on Task 1 (smaller number is harder).

Context Type	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5
Easy	0.46	0.64	0.71	0.74	0.86	0.86	0.86	0.83
Manual-extracted-RFC	0.14	0.30	0.32	0.46	0.45	0.55	0.51	0.52
LLM-extracted-RFC	0.10	0.21	0.28	0.38	0.38	0.47	0.43	0.40
No-context	0.09	0.14	0.20	0.36	0.35	0.41	0.36	0.39

Appendix

Table 9: Effect of context type on Task 3 (smaller number is harder).

Context Type	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5
Easy	0.14	0.60	0.57	0.81	0.82	0.78	0.70	0.79
Manual-extracted-RFC	0.03	0.28	0.19	0.27	0.25	0.32	0.26	0.26
LLM-extracted-RFC	0.05	0.25	0.19	0.23	0.26	0.29	0.24	0.23
No-context	0.04	0.19	0.15	0.25	0.19	0.26	0.19	0.19

Appendix

Table 10: Effect of context type on Task 4 (smaller number is harder).

Figure 6: Protocol-level difficulty across LLMs (Task 1).

Figure 7: Protocol-level difficulty across LLMs (Task 2).

Figure 8: Protocol-level difficulty across LLMs (Task 3).

Protocol	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5	Avg.	std.
RTSP	0.26	0.27	0.30	0.30	0.31	0.32	0.30	0.35	0.30	0.03
TCP_nr	0.41	0.40	0.41	0.43	0.43	0.46	0.47	0.46	0.43	0.02
MQTT	0.45	0.45	0.53	0.46	0.45	0.49	0.48	0.49	0.48	0.03
FTP	0.41	0.47	0.48	0.52	0.52	0.49	0.50	0.54	0.49	0.04
POP3	0.43	0.49	0.55	0.52	0.52	0.59	0.52	0.55	0.52	0.04
PPTP	0.51	0.49	0.51	0.53	0.53	0.60	0.57	0.60	0.54	0.04

Appendix

Table 11: Protocol-level difficulty on Task 1(ranked by Avg., smaller number is harder).

Protocol	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5	Avg.	std.
RTSP	0.06	0.09	0.09	0.09	0.09	0.11	0.11	0.09	0.09	0.01
MQTT	0.27	0.28	0.27	0.28	0.26	0.29	0.28	0.29	0.28	0.01
NNTP	0.31	0.28	0.26	0.28	0.31	0.26	0.28	0.27	0.28	0.02
SMTP_s	0.21	0.38	0.30	0.32	0.32	0.29	0.38	0.35	0.32	0.05
SCTP	0.26	0.22	0.26	0.36	0.47	0.31	0.44	0.43	0.34	0.09
BGP	0.26	0.30	0.28	0.28	0.38	0.40	0.41	0.45	0.34	0.07

Appendix

Table 12: Protocol-level difficulty on Task 2 (ranked by Avg., smaller number is harder).

Protocol	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5	Avg.	std.
NNTP	0.20	0.21	0.28	0.34	0.22	0.25	0.24	0.22	0.25	0.04
MQTT	0.21	0.24	0.26	0.27	0.32	0.30	0.32	0.30	0.28	0.04
DHCP	0.15	0.23	0.21	0.36	0.38	0.44	0.44	0.38	0.32	0.10
DCCP	0.17	0.30	0.31	0.32	0.40	0.38	0.36	0.40	0.33	0.07
PPP	0.07	0.26	0.17	0.46	0.46	0.45	0.39	0.46	0.34	0.14
FTP	0.06	0.29	0.35	0.44	0.38	0.51	0.38	0.38	0.35	0.12

Appendix

Table 13: Protocol-level difficulty on Task 3 (ranked by Avg., smaller number is harder).

Protocol	gpt-4o-mini	qwen32B	llama3.3-70B	deepseek-r1	grok-3-mini	gpt-5.2	gemini-3-f-p	claude-s-4.5	Avg.	std.
PPP	0.02	0.04	0.06	0.11	0.10	0.08	0.11	0.12	0.08	0.03
BGP	0.05	0.08	0.09	0.13	0.20	0.14	0.10	0.15	0.12	0.04
PPTP	0.05	0.07	0.10	0.20	0.19	0.16	0.13	0.18	0.14	0.05
SCTP	0.03	0.07	0.12	0.19	0.23	0.19	0.23	0.20	0.16	0.07
RTSP	0.00	0.21	0.15	0.25	0.19	0.21	0.23	0.23	0.18	0.08
DCCP	0.10	0.17	0.20	0.26	0.29	0.23	0.26	0.25	0.22	0.06

Appendix

Table 14: Protocol-level difficulty on Task 4 (ranked by Avg., smaller number is harder).

Rank	Protocol	Task 1	Task 2	Task 3	Task 4	Avg.	Std.
1	RTSP	0.30	0.09	0.41	0.18	0.25	0.12
2	MQTT	0.48	0.28	0.28	0.27	0.33	0.08
3	PPP	0.56	0.36	0.34	0.08	0.34	0.17
4	TCP_nr	0.43	0.44	0.39	0.22	0.37	0.09
5	FTP	0.49	0.35	0.35	0.28	0.37	0.08
6	BGP	0.60	0.34	0.48	0.12	0.39	0.18

Appendix

Table 15: Overall protocol score across tasks. Smaller values indicate harder protocols. Avg. is the average score over Tasks 1, 2, 3, and 4, and Std. measures difficulty variability across tasks.

Protocol	qwen32B (P/R/F1)	llama3.3-70B (P/R/F1)	deepseek-r1 (P/R/F1)	grok-3-mini(P/R/F1)	gpt-5.2 (P/R/F1)	gemini-3-f-p (P/R/F1)	claude-s-4.5 (P/R/F1)
PPP	0.10/0.00/0.01	0.17/0.02/0.03	0.26/0.05/0.09	0.22/0.04/0.07	0.18/0.01/0.02	0.26/0.05/0.08	0.32/0.04/0.07
BGP	0.29/0.03/0.06	0.25/0.06/0.10	0.29/0.11/0.15	0.21/0.10/0.13	0.33/0.06/0.10	0.25/0.11/0.15	0.26/0.08/0.12
PPTP	0.19/0.05/0.08	0.28/0.14/0.18	0.29/0.22/0.24	0.31/0.22/0.25	0.28/0.16/0.20	0.29/0.18/0.22	0.30/0.23/0.26
SCTP	0.27/0.04/0.08	0.45/0.10/0.16	0.50/0.18/0.25	0.57/0.24/0.32	0.46/0.13/0.20	0.57/0.21/0.30	0.54/0.23/0.32
RTSP	0.31/0.11/0.16	0.31/0.15/0.20	0.21/0.22/0.21	0.19/0.19/0.19	0.27/0.13/0.17	0.27/0.24/0.26	0.29/0.31/0.30
DCCP	0.32/0.09/0.14	0.35/0.14/0.20	0.41/0.24/0.30	0.44/0.26/0.32	0.40/0.18/0.24	0.48/0.25/0.32	0.40/0.26/0.31

Appendix

Table 16: Protocol-level Precision, Recall, F1 on Task 4: P/R/F1 represents Precision/Recall/F1. The ranking is the same as in Table 14 , i.e., ranked by avg. score in task 4 from low to high. We exclude the column for gpt-4o-mini to save space because its performance on task 4 is very low.

Protocol	Context type	#Queries	#Correct paths set (perfect match) (judges: 0.6-sim, 0.7-sim, 0.8-sim, llama3.3-70b , gemini3fp)	Micro Precision, Micro Recall, F1 (judge: gemini3fp)
TCP	Easy table Manually-extracted rfc LLM-extracted No context	110	86, 86, 86, 66, 67 67, 66, 59, 45, 42 65, 62, 58, 39, 38 60, 58, 56, 36, 31	0.61, 0.81, 0.69 0.31, 0.54, 0.40 0.26, 0. 52, 0.35 0.26, 0.43, 0.32
TCP new reno	Easy table Manually-extracted rfc LLM-extracted No context	12	12, 12, 12, 11, 11 11, 11, 7, 2, 0 10, 9, 4, 2, 0 11, 10, 8, 4, 0	0.93, 1.0, 0.96 0.16, 0.15, 0.16 0.10, 0.08, 0.09 0.14, 0.12, 0.12
SCTP	Easy table Manually-extracted rfc LLM-extracted No context	56	47, 46, 42, 26, 25 44, 40, 33, 17, 11 44, 40, 35, 10, 7 45, 43, 37, 8, 8	0.73, 0.39, 0.51 0.79, 0.26, 0.39 0.60, 0.14, 0.23 0.50, 0.15, 0.23
DCCP	Easy table Manually-extracted rfc LLM-extracted No context	72	64, 60, 53, 36, 37 58, 45, 35, 23, 19 53, 36, 23, 25, 13 60, 53, 42, 10, 10	0.92, 0.50, 0.70 0.56, 0.28, 0.37 0.48, 0.21, 0.29 0.24, 0.14, 0.18
PPP	Easy table Manually-extracted rfc LLM-extracted No context	90	86, 77, 62, 20, 21 64, 39, 8, 11, 10 46, 18, 4, 7, 5 77, 54, 27, 8, 6	0.89, 0.17, 0.29 0.31, 0.07, 0.11 0.29, 0.06, 0.10 0.19, 0.03, 0.06
PPTP	Easy table Manually-extracted rfc LLM-extracted No context	72	72, 72, 64, 34, 38 67, 55, 28, 4, 1 41, 31, 17, 3, 1 71, 67, 41, 3, 1	0.96, 0.62, 0.75 0.12, 0.07, 0.09 0.13, 0.04, 0.07 0.09, 0.07, 0.08

Appendix

Table 17: Task 4 results (Gemini-3-flash-preview).

Implicit Reasoning in Language Models

Implicit reasoning in language models studies how models solve problems through internal computation rather than, or alongside, written chains of thought.

See all 95 papers

Designing FSMs Specifications from Requirements with GPT 4.0
Mar 31, 2026Omer Nguena Timo, Paul-Alexis Rodriguez, Florent AvellanedaFinite-State Machine Model-Based Systems Engineering
Universit´e du Qu´ebec en Outaouais, Quebec-Canada · Universit´e Paris-Saclay, Paris-France · Universit´e du Qu´ebec `a Montr´eal, Qu´ebec-Canada
Evaluating Large Language Models for Symbolic Security Protocol Analysis
Jul 22, 2026Paolo Modesti, Syed Ahmed, Ioannis Sfyrakis +1LLM Evaluation Cybersecurity
Teesside University, Middlesbrough, United Kingdom
PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces
Sep 17, 2026Pyrros Koussios, Benjamin Jäger, John Hua Yao +3LLM Evaluation Temporal Reasoning in Language Models
ETH Zürich · Stanford University

Mar 31, 2026J/K move · Enter open · S save

Designing FSMs Specifications from Requirements with GPT 4.0

Omer Nguena Timo, Paul-Alexis Rodriguez, Florent Avellaneda

Universit´e du Qu´ebec en Outaouais, Quebec-Canada · Universit´e Paris-Saclay, Paris-France · Universit´e du Qu´ebec `a Montr´eal, Qu´ebec-Canada

Finite-State Machine Model-Based Systems Engineering Automated Program Repair

Open paper AskPDF arXiv

Finite state machines (FSM) are executable formal specifications of reactive systems. These machines are designed based on systems' requirements. The requirements are often recorded in textual documents written in natural languages. FSMs play a crucial role in different phases of the model-driven system engineering (MDE). For example, they serve to automate testing activities. FSM quality is critical: the lower the quality of FSM, the higher the number of faults surviving the testing phase and the higher the risk of failure of the systems in production, which could lead to catastrophic scenarios. Therefore, this paper leverages recent advances in the domain of LLM to propose an LLM-based framework for designing FSMs from requirements. The framework also suggests an expert-centric approach based on FSM mutation and test generation for repairing the FSMs produced by LLMs. This paper also provides an experimental analysis and evaluation of LLM's capacities in performing the tasks presented in the framework and FSM repair via various methods. The paper presents experimental results with simulated data. These results and methods bring a new analysis and vision of LLMs that are useful for further development of machine learning technology and its applications to MDE.

Abstract

Implicit Reasoning in Language Models

Explore similar work

Designing FSMs Specifications from Requirements with GPT 4.0

Evaluating Large Language Models for Symbolic Security Protocol Analysis

PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces