DeEscalWild: A Real-World Benchmark for Automated De-Escalation Training with SLMs

Md Hasebul Hasan; Krity Haque Charu; Eshwara Prasad Sridhar; Shuchisnigdha Deb; Mohammad A. Islam

cs.CLMar 20, 2026

DeEscalWild: A Real-World Benchmark for Automated De-Escalation Training with SLMs

Authors: Md Hasebul Hasan, Krity Haque Charu, Eshwara Prasad Sridhar, Shuchisnigdha Deb, Mohammad A. Islam

Organizations: Department of Computer Science and Engineering, University of Texas at Arlington, USA · Department of Industrial, Manufacturing, and Systems Engineering, University of Texas at Arlington, USA

Abstract

Effective de-escalation is critical for law enforcement safety and community trust, yet traditional training methods lack scalability and realism. While Large Language Models (LLMs) enable dynamic, open-ended simulations, their substantial computational footprint renders them impractical for deployment on the lightweight, portable hardware required for immersive field training. Small Language Models (SLMs) offer a viable real-time alternative but suffer from a critical scarcity of high-quality, domain-specific training data. To bridge this gap, we present DeEscalWild, a novel benchmark dataset curated from a multi-stage pipeline of in-the-wild police-civilian interactions extracted from publicly available video repositories. Starting with 5,000 raw inputs, we employed a rigorous hybrid filtering process combining human-in-the-loop verification with LLM-as-a-Judge evaluation to distill 1,500 high-fidelity scenarios. The resulting corpus comprises 285,887 dialogue turns, totaling approximately 4.7 million tokens. Extensive experiments demonstrate that SLMs fine-tuned on this data significantly outperform their base counterparts across ROUGE-L, BLEU-4, METEOR, BERTScore, Realism Score, and human evaluation metrics. Notably, our fine-tuned Qwen 2.5 (3B-Instruct) surpasses the general-purpose Gemini 2.5 Flash model when evaluated under equivalent conditions, demonstrating that domain-optimized SLMs can achieve superior performance with a fraction of the computational cost. This work establishes the foundational infrastructure for accessible, low-latency, and privacy-preserving officer training systems at the edge. We publicly release our code(https://github.com/Hasebul/DeEscalWild-Benchmark-Framework) and dataset(https://doi.org/10.7910/DVN/CWMCZI).

Figures & tables

Figure 1: DeEscalWild end-to-end data curation pipeline. Starting from 5,000 videos across 23 social media channels, a human-verified 30-signal LLM filter retains 1,500 police-civilian interactions satisfying three validity conditions: no off-domain noise, confirmed police presence, and sufficient escalation depth. Gemini 2.5 Flash performs native-audio transcription and speaker diarization. Transcripts undergo structural cleaning and privacy-preserving anonymization via NER and LLM-based parsing; no raw audio or video is released. The final corpus comprises 1,350 training interactions and a strictly disjoint 150-scenario benchmark totalling 285,887 dialogue turns and approximately 4.7 million tokens, released under a non-commercial research license. Solid arrows indicate the main curation pipeline; dashed arrows indicate the dual-annotator validation branch applied to the same N=100 subsample at two checkpoints: filtering precision and transcript fidelity.

Metric	Count
Total Scenarios	1,500
Total Dialogue Turns	285,887
Total Word Count	3,626,196
Total Token Count (Est.)	4,714,067

Table 1: Summary statistics of the DeEscalWild dataset.

Model	Params	Context Window	Disk Size (FP16)	Key Architecture
Gemma 2 (2B-Instruct)	2.6B	8k	${\sim}$ 2.6 GB	Distillation, SWA
Granite 3.0 (2B-Instruct)	2.5B	4k	${\sim}$ 2.5 GB	Safety aligned
Qwen 2.5 (3B-Instruct)	3.1B	32k	${\sim}$ 3.8 GB	SwiGLU, RoPE
Falcon 3 (3B-Instruct)	3.2B	32k	${\sim}$ 3.5 GB	GQA, reasoning
Llama 3.2 (3B-Instruct)	3.2B	128k	${\sim}$ 3.4 GB	GQA, pruned

Table 2: Technical specifications of the five Small Language Models (SLMs) evaluated in this study. Memory estimates represent FP16 weights. All models contain fewer than 4 billion parameters and were released in late 2024 to early 2025.

Model	ROUGE-L		BLEU-4		METEOR
Model	Base	FT Δ	Base	FT Δ	Base	FT Δ
Qwen 2.5 (3B-Instruct)	11.2 $\pm$ 0.2	15.7 $\pm$ 0.3 ↑4.5	0.9 $\pm$ 0.1	3.7 $\pm$ 0.2 ↑2.8	15.7 $\pm$ 0.2	19.4 $\pm$ 0.4 ↑3.7
Llama 3.2 (3B-Instruct)	11.3 $\pm$ 0.2	14.9 $\pm$ 0.4 ↑3.6	0.6 $\pm$ 0.1	2.4 $\pm$ 0.2 ↑1.8	15.7 $\pm$ 0.2	17.7 $\pm$ 0.3 ↑2.0
Gemma 2 (2B-Instruct)	7.2 $\pm$ 0.2	14.7 $\pm$ 0.3 ↑7.5	0.4 $\pm$ 0.1	1.9 $\pm$ 0.2 ↑1.5	11.7 $\pm$ 0.2	16.3 $\pm$ 0.3 ↑4.6
Granite 3.0 (2B-Instruct)	10.2 $\pm$ 0.1	15.6 $\pm$ 0.4 ↑5.4	1.2 $\pm$ 0.1	2.3 $\pm$ 0.2 ↑1.1	13.5 $\pm$ 0.2	16.7 $\pm$ 0.3 ↑3.2
Falcon 3 (3B-Instruct)	9.2 $\pm$ 0.2	11.2 $\pm$ 0.3 ↑2.0	0.5 $\pm$ 0.1	1.2 $\pm$ 0.1 ↑0.7	12.4 $\pm$ 0.2	15.6 $\pm$ 0.2 ↑3.2

Table 3: Standard NLP metrics for base versus fine-tuned (FT) models (mean

\pm

SD across 3 independent runs). Subscripts indicate the absolute gain (

\uparrow

) from the base model. Best results per metric in bold . All metrics scaled to 0–100.

Model	BERTScore (F1)		Realism: Gemini 3.1 Pro		Realism: GPT-5.4
Model	Base	FT Δ	Base	FT Δ	Base	FT Δ
Qwen 2.5 (3B-Instruct)	0.86 $\pm$ 0.01	0.88 $\pm$ 0.01 ↑0.02	35.4 $\pm$ 0.4	62.1 $\pm$ 0.5 ↑26.7	34.8 $\pm$ 0.5	60.8 $\pm$ 0.6 ↑26.0
Llama 3.2 (3B-Instruct)	0.86 $\pm$ 0.01	0.87 $\pm$ 0.01 ↑0.01	36.1 $\pm$ 0.3	58.4 $\pm$ 0.6 ↑22.3	35.5 $\pm$ 0.4	57.2 $\pm$ 0.6 ↑21.7
Gemma 2 (2B-Instruct)	0.81 $\pm$ 0.01	0.87 $\pm$ 0.01 ↑0.06	28.5 $\pm$ 0.4	55.2 $\pm$ 0.5 ↑26.7	28.1 $\pm$ 0.5	54.0 $\pm$ 0.6 ↑25.9
Granite 3.0 (2B-Instruct)	0.83 $\pm$ 0.01	0.87 $\pm$ 0.01 ↑0.04	32.4 $\pm$ 0.3	60.5 $\pm$ 0.4 ↑28.1	31.8 $\pm$ 0.4	59.0 $\pm$ 0.5 ↑27.2
Falcon 3 (3B-Instruct)	0.83 $\pm$ 0.01	0.85 $\pm$ 0.01 ↑0.02	30.2 $\pm$ 0.5	48.7 $\pm$ 0.6 ↑18.5	29.6 $\pm$ 0.5	47.5 $\pm$ 0.7 ↑17.9

Table 4: Semantic fidelity and cross-judge realism evaluation for base versus fine-tuned (FT) models. BERTScore is reported on a 0–1 scale. Realism is reported on a 0–100 scale using two independent LLM judges from different model families: Gemini 3.1 Pro and GPT-5.4. Subscripts indicate the absolute gain (

\uparrow

) from the base model. Best results per metric are shown in bold . GPT-5.4 scores are included as a cross-judge robustness check to reduce single-model preference bias.

Model	ROUGE-L	BLEU-4	METEOR	BERTScore (F1)	Realism	Realism	Latency (s)
					Gemini 3.1 Pro	GPT-5.4
Qwen 2.5 (3B) FT	15.7 $\pm$ 0.3 ∗	3.7 $\pm$ 0.2 ∗	19.4 $\pm$ 0.4 ∗	0.88 $\pm$ 0.01 ∗	62.1 $\pm$ 0.5 ∗	60.8 $\pm$ 0.6 ∗	0.38 $\pm$ 0.02
Llama 3.2 (3B) FT	14.9 $\pm$ 0.4 ∗	2.4 $\pm$ 0.2 ∗	17.7 $\pm$ 0.3 ∗	0.87 $\pm$ 0.01 ∗	58.4 $\pm$ 0.6 ∗	57.2 $\pm$ 0.6 ∗	0.44 $\pm$ 0.03
Gemma 2 (2B) FT	14.7 $\pm$ 0.3 ∗	1.9 $\pm$ 0.2 ∗	16.3 $\pm$ 0.3 ∗	0.87 $\pm$ 0.01 ∗	55.2 $\pm$ 0.5 ∗	54.0 $\pm$ 0.6 ∗	0.33 $\pm$ 0.02
Granite 3.0 (2B) FT	15.6 $\pm$ 0.4 ∗	2.3 $\pm$ 0.2 ∗	16.7 $\pm$ 0.3 ∗	0.87 $\pm$ 0.01 ∗	60.5 $\pm$ 0.4 ∗	59.0 $\pm$ 0.5 ∗	0.30 $\pm$ 0.02
Falcon 3 (3B) FT	11.2 $\pm$ 0.3	1.2 $\pm$ 0.1	15.6 $\pm$ 0.2 ∗	0.85 $\pm$ 0.01	48.7 $\pm$ 0.6 ∗	47.5 $\pm$ 0.7 ∗	0.41 $\pm$ 0.03

Table 5: Final performance, cross-judge realism, and latency comparison of fine-tuned SLMs versus the Gemini 2.5 Flash generalist baseline (mean

\pm

SD across 3 seeds). Realism is evaluated using two independent LLM judges from different model families, Gemini 3.1 Pro and GPT-5.4, to reduce single-judge preference bias. An asterisk ( ∗ ) denotes statistical significance versus the Gemini 2.5 Flash baseline under the corresponding metric or judge setting (paired

t

-test,

p<0.05

). Best results per metric in bold . All fine-tuned SLMs operate at sub-second latency, corresponding to an approximately 8

\times

to 12

\times

speedup over the API-based baseline.

Officer Input	Base Model Qwen 2.5 (3B)	Gemini 2.5 Flash (Few-Shot)	Fine-Tuned Qwen 2.5 (3B, Ours)
“Step out of the vehicle and keep your hands where I can see them!”	I am afraid I cannot comply with that request. I am staying inside.	No! What the hell is going on? I haven’t done anything! Get away from my car!	I ain’t getting out! You tell me why you stopped me first! I didn’t do nothing!
“Stop resisting! Put your hands behind your back now!”	Please calm down. I am not resisting.	I am not doing anything! Get off me!	Get off me! I didn’t do nothing! Let go! Why are you grabbing me?

Table 6: Qualitative comparison of generated civilian responses across model configurations.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Training Dynamics and Convergence Analysis. We report the training (blue) and validation (red) loss trajectories for (a) Qwen 2.5, (b) Llama 3.2, (c) Gemma 2, (d) Granite 3, and (e) Falcon 3 during fine-tuning on the DeEscalWild dataset. All models exhibit rapid initial convergence within the first 50 steps, followed by a stabilization phase. The close alignment between training and validation curves across all architectures indicates that the models are effectively learning domain-specific features without succumbing to overfitting.

Figure 3: Fine-tuning gains across evaluation metrics. Heatmap showing the absolute improvement from the base model to the fine-tuned model across automatic lexical metrics, semantic similarity, realism, and de-escalation. Rows correspond to open-weight models and columns correspond to evaluation metrics. Darker cells indicate larger absolute gains. Fine-tuning improves all models across all reported metrics, with especially large gains in realism and de-escalation for Qwen 2.5.

Figure 4: Quality–latency trade-off between fine-tuned SLMs and the Gemini 2.5 Flash baseline. Each point represents one model configuration. The $x$ -axis shows inference latency in seconds, while the $y$ -axis shows average realism, computed as the mean of the Gemini 3.1 Pro and GPT-5.4 realism scores. Fine-tuned SLMs achieve sub-second latency while remaining competitive with, or outperforming, the Gemini 2.5 Flash baseline in realism. Qwen 2.5 provides the strongest overall realism, whereas Granite 3.0 provides the lowest latency.

Panel of Figure 5 — Figure 5: Cross-judge agreement between Gemini 3.1 Pro and GPT-5.4. Scatter plots compare realism and de-escalation scores assigned by two independent LLM judges. Each point corresponds to either a fine-tuned open-weight model or the Gemini 2.5 Flash baseline. The dashed diagonal line indicates perfect agreement between judges. Points close to the diagonal show that both judges produce consistent score trends, suggesting that the observed ranking is not driven by a single evaluator. The fine-tuned Qwen 2.5 model achieves the strongest overall performance, while Gemini 2.5 Flash provides a proprietary baseline reference.

Time	Speaker	Transcript
0:00	Deputy Warney	I’m trying to walk up this long driveway to the house now.
0:02	[Dispatcher]	He advised that the woman was in the pool and he is coming to let y’all in the gate now.
0:07	Deputy Warney	That’s 10-4. I’m at the house now.
0:09	[Law enforcement officer]	Is this that lady that was out here a while back that says she was pregnant with his baby or she
0:13	[Law enforcement officer]	pulling me freaking amaze.
0:15	[Law enforcement officer]	Well, that used to be a baseball field.

Appendix

Table 7: Full Transcript of Scenario 188. An officer interacts with a trespassing subject experiencing delusions. Colors indicate speaker roles: Police , Subject , Dispatch .

Time	Speaker	Transcript
00:30	[Police Officer]	How you doing, sir?
00:31	Subject	I’m good, right here, sir.
00:33	[Police Officer]	Oh, okay. How’s your day going?
00:35	Subject	Not bad. It’s good. Welcome to Publix.
00:37	[Police Officer]	Understand. But you know I’m pulling you over, right? You’ve come to a police stop on that stop sign.
00:42	Subject	I thought I paused enough, but if I didn’t, I just kept seeing you roll.

Appendix

Table 8: Full Transcript of Scenario 371. A traffic stop for a stop sign violation involving complex insurance verification. The middle portion is omitted for brevity. Colors indicate speaker roles: Police , Subject , Dispatch , Other .

Time	Speaker	Transcript
00:00	Officer	Can be honest with me. Okay? Honesty, honesty, honesty’s going to go a long way.
00:04	Subject	I know I can’t get in trouble.
00:06	Officer	Honesty’s going to go a long way.
00:08	Subject	I, right? I, I really don’t want to get in trouble. I will rip anything out.
00:12	Officer	Why are you crying?
00:13	Third Parties	For months, officers have kept an eye on this teen’s suspicious looking car, a near perfect fake squad vehicle that’s fooled plenty. But tonight, things take a sharp turn when he’s finally caught in the act, and there’s no way out this time.

Appendix

Table 9: Full Transcript of Scenario 803. A subject (Subject) is stopped for impersonating a police officer. He initially claims the emergency lights are disconnected, but the interrogation reveals otherwise. Colors indicate speaker roles: Police , Subject , Third Parties .

Figure 6: Conceptual architecture of a multimodal virtual de-escalation system. The pipeline operates as a closed-loop system in which the police officer’s multimodal input is processed by a perception layer , reasoned over by the specialized SLM core , and rendered through a synthesis layer to control a virtual avatar in real time.

Panel of Figure 8 — Figure 8: Zero-shot prompt used for LLM-based feature extraction.

Annotation Category	Agreement	Kappa ( $\kappa$ )	Observations and Methodology Notes
Human Context Valid	100.0%	Undefined ∗	Perfect raw agreement. Undefined $\kappa$ due to zero label variance from unanimous inclusion. Indicates highly effective automated pre-filtering.
Human Relevance Valid	100.0%	Undefined ∗	Perfect raw agreement on the presence of police-civilian role separation. Undefined $\kappa$ due to unanimous positive labels.
Final Keep Decision	90.9%	1.000	Perfect chance-adjusted agreement. Reflects unambiguous inclusion criteria and consistent annotator judgment.
False Positive Category	100.0%	1.000	Perfect agreement on classifying failure modes of excluded videos, confirming clear categorical definitions.
Human Intensity Valid	90.9%	0.621	Substantial agreement. Confirms that annotators reliably recognize genuine escalation and de-escalation dynamics in uncontrolled footage.
Audio Quality Score	63.6%	0.718	Substantial agreement on a subjective 1-to-5 ordinal scale. Reflects a well-calibrated shared threshold for acoustic viability.

Appendix

Table 10: Inter-annotator agreement (IAA) statistics for the human verification study (

N=100

sampled videos). Raw proportion agreement and Cohen’s Kappa (

\kappa

) are reported for each annotation dimension. Undefined

\kappa

values ( ∗ ) arise from unanimous annotator agreement, which produces zero label variance and renders chance correction inapplicable. All disagreements on Scenario Category and Final Keep Decision were resolved through consensus adjudication with a senior domain expert.

Panel of Figure 9 — Figure 9: Dual-task prompt template used for automated diarization and speaker profiling via Gemini 2.5 Flash. The prompt is dynamically populated with a timecode specification matched to the recording duration. Task 1 extracts a verbatim, turn-attributed transcript; Task 2 extracts structured speaker metadata for each identified voice. Key design choices include explicit verbatim transcription rules to prevent paraphrasing, overlap handling instructions for chaotic multi-party scenes, and a strict JSON-only output policy with automatic re-querying of malformed responses.

Field Name	Description
Video_ID	Unique identifier or filename of the source video.
Human_Context_Valid	Binary indicator: genuine real-world police interaction (0 = No, 1 = Yes).
Human_Relevance_Valid	Binary indicator: clear officer and civilian role separation present (0 = No, 1 = Yes).
Human_Intensity_Valid	Binary indicator: genuine escalation or substantive de-escalation attempt present (0 = No, 1 = Yes).
False_Positive_Category	Categorical failure label for videos marked invalid, such as news broadcast or wind noise.
Audio_Quality_Score	Ordinal rating of dialogue clarity on a 1-to-5 scale (1 = completely unintelligible, 5 = broadcast quality).

Appendix

Table 11: Schema of the structured per-video annotation record used for quality assessment and metric aggregation. Each row corresponds to one sampled video. Symbolic notation in parentheses links each field directly to its corresponding variable in the WER and DER formulae (Equations 2 and 3 ).

		Annotator 1		Annotator 2
Evaluation Metric	Total Units	Total Errors	Global Rate	Total Errors	Global Rate
Global WER	313,737 words	2,881	0.91%	2,410	0.77%
Global DER	107,873 seconds	8,810	8.16%	6,754	6.26%

Appendix

Table 12: Corpus-level global error rates for Gemini 2.5 Flash, evaluated independently by two annotators across the

N=100

video subset.

\text{WER}_{\text{global}}

and

\text{DER}_{\text{global}}

are computed by summing raw error counts across the full subset rather than averaging per-video rates, consistent with Equations 4 and 5 . Inter-annotator variance in both metrics reflects the inherent subjectivity of human reference annotation in noisy, multi-party acoustic environments.

Macro-Category	Fine-Grained Class	Description
Low / No Tension	1. Full Compliance	Immediate adherence to instructions without hesitation or argument.
	2. Neutral Interaction	Normal, respectful conversation with no signs of stress or hostility.
	3. Clarification / Questioning	Respectful questioning by the civilian regarding officer actions or rationale.
Mild Tension	4. Reluctant Compliance	Slow, unwilling, or visibly hesitant adherence to orders.
	5. Emotional Distress	Strong non-aggressive emotional responses, including crying, panic, or confusion.
De-escalation	6. Successful De-escalation	Measurable reduction in tension following active calming attempts by the officer.

Appendix

Table 13: Taxonomy of police-civilian interaction dynamics. Videos are classified into one of 10 fine-grained classes, which hierarchically map to 4 broader macro-categories representing the overall tension trajectory of the interaction.

Macro-Category	Fine-Grained Scenario	Count ( $N=100$ )	Percentage
Low / No Tension	1. Full Compliance	15	15.0%
	2. Neutral Interaction	10	10.0%
	3. Clarification	8	8.0%
	Subtotal	33	33.0%
Mild Tension	4. Reluctant Compliance	14	14.0%
	5. Emotional Distress	9	9.0%

Appendix

Table 14: Frequency distribution of interaction scenarios across the

N=100

manually adjudicated video subset. The balanced spread across all four macro-categories confirms that the curated dataset captures the full spectrum of real-world policing dynamics, from routine compliance to physical aggression, without over- representing any single tension trajectory.

Category	Description
Dataset name	DeEscalWild
Modality	Anonymized, diarized text transcripts
Source	Publicly available police-civilian interaction videos from YouTube, TikTok, and Facebook
Raw media released	No. Raw video, audio, thumbnails, and biometric modalities are never redistributed
Scenarios	1,500 interactions (285,887 dialogue turns, ${\sim}$ 4.7M tokens)
Held-out benchmark	150 interactions, isolated from all training data

Appendix

Table 15: Dataset release card summarizing the scope, access conditions, intended uses, prohibited uses, and known risks of DeEscalWild . The dataset is released under a restricted-use, non-commercial research license.

Item	Released	Notes
Raw video	No	Never redistributed.
Raw audio	No	Never redistributed.
Thumbnails	No	Never redistributed.
Creator metadata	No	Usernames and creator identifiers excluded.
Platform identifiers	No	Retained internally for audit and takedown only.
Source URLs	No	Retained internally for audit and takedown only.

Appendix

Table 16: Release policy for source material, derived artifacts, and supporting resources. Items marked No are withheld entirely from the public release; gated items are available under restricted access upon request.

Panel of Figure 10 — Figure 10: Comprehensive diversity analysis of the DeEscalWild dataset. Panels (a) and (b) show demographic composition by race/ethnicity and gender; panels (c) and (d) illustrate age group and dialectal diversity; panels (e) and (f) present the distribution of incident types and severity levels. For the full 1,500-scenario corpus, demographic attributes are LLM-inferred from transcript content and platform metadata and carry inherent uncertainty. For the $N{=}100$ manually audited subset, race/ethnicity and gender distributions are additionally grounded in direct human observation from source video, providing a calibration anchor for corpus-level estimates. The intentional concentration of high-severity interactions (90.6% classified as high tension) confirms the dataset’s focus on operational conditions where de-escalation capability is most consequential.

Panel of Figure 11 — Figure 11: Prompt structure used for the Gemini 2.5 Flash few-shot baseline in civilian response generation. The prompt comprises three components: a system instruction establishing the civilian persona and output constraints (top); eight static few-shot input-output examples grounding the desired response style (middle); and a dynamically populated conversation history providing full interaction context up to the current turn (bottom). At each evaluation turn, the ground-truth officer utterance is appended to the history and the model generates the next civilian response autoregressively.

Panel of Figure 27 — (Continued in Figure 13 )

Panel of Figure 13 — Figure 13: LLM-as-a-Judge realism evaluation prompt (Part 2 of 2): input data specification, task requirements, and required JSON output schema. The expanded schema captures observations across all four penalty dimensions, enforces an evidence-only deduction policy, and applies a hard score floor of zero to prevent underflow. The prompt is applied identically across all model configurations to ensure comparability.

Score	Label
1	Very unrealistic / bot-like
2	Somewhat unrealistic
3	Mixed / acceptable but noticeably flawed
4	Mostly realistic
5	Highly realistic / human-like

Appendix

Table 17: Interpretation of the 1–5 realism rating scale.

#	Criterion	Key evaluator question
Group 1: Emotional authenticity
1 ∗	Emotional realism	Does the victim’s emotional reaction feel believable for this moment?
2 ∗	Response to escalation	When the officer escalates, does the victim’s behavior change in a believable way?
3	Response to de-escalation	Does the victim respond naturally when the officer tries to calm the situation?
4	Trauma-aware behavior	Does the victim’s behavior reflect distress in a believable and respectful way?
Group 2: Linguistic naturalism

Appendix

Table 18: Human evaluation rubric: 15 criteria organized by conceptual group. Primary criteria (marked ∗ ) are used to compute the weighted primary score. Each criterion is scored 1–5 (1 = very unrealistic; 5 = highly realistic / human-like).

Criterion	$\kappa_{w}$	Interpretation
1. Emotional realism ∗	0.74	Substantial
2. Response to esc./de-esc. ∗	0.76	Substantial
3. Trauma-aware behavior	0.63	Substantial
4. Natural spoken language ∗	0.82	Near-perfect
5. Context awareness	0.79	Substantial
6. Character consistency ∗	0.77	Substantial

Appendix

Table 19: Inter-annotator agreement by criterion using quadratic weighted Cohen’s kappa

\kappa_{w}

	Qwen 2.5 (3B)		Llama 3.2 (3B)		Gemini 2.5 Flash
Criterion	Base	FT	Base	FT
Emotional realism ∗	2.48	4.28	2.10	3.88	4.05
Response to escalation ∗	2.12	4.20	2.17	3.58	4.13
Response to de-escalation	2.20	4.20	2.10	3.65	3.83
Trauma-aware behavior	2.05	4.10	2.00	3.45	3.83
Natural spoken language ∗	2.00	4.50	1.80	3.65	4.57

Appendix

Table 20: Human evaluation results: mean criterion scores on a 1–5 scale by model, averaged across 20 scenarios and two evaluators. Primary criteria are marked with ∗ .

Model	Realism: Gemini 3.1 Pro		Realism: GPT-5.4
Model	Base	FT Δ	Base	FT Δ
Qwen 2.5 (3B-Instruct)	61.3 $\pm$ 7.5	75.2 $\pm$ 6.7 ↑13.9	60.0 $\pm$ 7.6	73.6 $\pm$ 6.8 ↑13.6
Llama 3.2 (3B-Instruct)	62.5 $\pm$ 7.2	68.8 $\pm$ 6.5 ↑6.3	61.2 $\pm$ 7.3	67.4 $\pm$ 6.6 ↑6.2
Gemma 2 (2B-Instruct)	49.3 $\pm$ 7.4	54.2 $\pm$ 6.8 ↑4.9	48.3 $\pm$ 7.5	53.1 $\pm$ 6.9 ↑4.8
Granite 3.0 (2B-Instruct)	56.1 $\pm$ 7.0	61.7 $\pm$ 6.4 ↑5.6	55.0 $\pm$ 7.1	60.4 $\pm$ 6.5 ↑5.4
Falcon 3 (3B-Instruct)	52.3 $\pm$ 7.8	57.5 $\pm$ 7.1 ↑5.2	51.2 $\pm$ 7.9	56.3 $\pm$ 7.2 ↑5.1

Appendix

Table 21: Cross-judge realism evaluation. Base versus fine-tuned (FT) realism scores across held-out benchmark scenarios (mean

\pm

SD). Realism is evaluated using two independent LLM judges, Gemini 3.1 Pro and GPT-5.4, to reduce single-judge preference bias. The subscript indicates the absolute increase (

\uparrow

) from the base model. Best results are shown in bold .

Model	De-Escalation: Gemini 3.1 Pro		De-Escalation: GPT-5.4
Model	Base	FT Δ	Base	FT Δ
Qwen 2.5 (3B-Instruct)	56.8 $\pm$ 10.7	76.8 $\pm$ 14.7 ↑20.0	55.6 $\pm$ 10.8	75.2 $\pm$ 14.8 ↑19.6
Llama 3.2 (3B-Instruct)	57.9 $\pm$ 10.5	63.7 $\pm$ 9.8 ↑5.8	56.7 $\pm$ 10.6	62.4 $\pm$ 9.9 ↑5.7
Gemma 2 (2B-Instruct)	45.7 $\pm$ 11.2	50.3 $\pm$ 10.1 ↑4.6	44.8 $\pm$ 11.3	49.3 $\pm$ 10.2 ↑4.5
Granite 3.0 (2B-Instruct)	52.0 $\pm$ 10.8	57.2 $\pm$ 9.5 ↑5.2	51.0 $\pm$ 10.9	56.0 $\pm$ 9.6 ↑5.0
Falcon 3 (3B-Instruct)	48.4 $\pm$ 11.5	53.2 $\pm$ 10.4 ↑4.8	47.4 $\pm$ 11.6	52.1 $\pm$ 10.5 ↑4.7

Appendix

Table 22: Cross-judge de-escalation evaluation. Base versus fine-tuned (FT) de-escalation rates across held-out benchmark scenarios (mean

\pm

SD). De-escalation is evaluated using two independent LLM judges, Gemini 3.1 Pro and GPT-5.4, to reduce single-judge preference bias. The subscript indicates the absolute increase (

\uparrow

) from the base model. Best results are shown in bold .

Panel of Figure 35 — (Continued in Figure 15 )

Panel of Figure 15 — Figure 15: LLM-as-a-Judge de-escalation scoring prompt (Part 2 of 2): penalty rubric, scoring instructions, input specification, and required JSON output schema. The outcome-first rubric evaluates the net behavioral trajectory of the interaction rather than aggregating individual skill scores, ensuring that high-quality de-escalation is recognized regardless of the specific verbal strategies employed. The evidence-only penalty policy prevents unjustified deductions, and the structured JSON output enforces reproducibility across all evaluated model configurations.

Platform	Channel / Page Name	URL
YouTube	Law & Crime Network	https://www.youtube.com/@LawAndCrime
YouTube	COPSCAM - Criminal Chase	https://www.youtube.com/@COPSCAM-CriminalChase
YouTube	Police Insider	https://www.youtube.com/channel/UCGBZFFufYgUJ56MCvfyAXfA
YouTube	Police Watch	https://www.youtube.com/channel/UCW8Oog9pSV5i4pC4aEB50dw
YouTube	Arkansas Police Activity	https://www.youtube.com/@arkansaspoliceactivity
YouTube	Police Files	https://www.youtube.com/channel/UC-1hQ_xRBVjEx80vzZSYuDA

Appendix

Table 23: Complete list of social media data sources used in the construction of DeEscalWild . Sources span 15 YouTube channels, 5 TikTok channels, and 3 Facebook pages, selected for their focus on law enforcement interactions and body-worn camera footage. URLs are provided for independent verification and reproducibility.

Small Language Models

Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.

See all 277 papers

Iterate on an LLM, Serve on an SLM
Oct 10, 2026Keegan WangLLM Agent Training Small Language Models
Carnegie Mellon University · Phoebe
Ignore All Previous Instructions: Jailbreaking as a de-escalatory peace building practise to resist LLM social media bots
Mar 2, 2026Huw Day, Adrianna Jezierska, Jessica WoodgateJailbreak Attacks Computational Social Science
School of Computer Science, University of Bristol
Towards Reliable Local Security Agents: Verifiable Post-Training for Linux Privilege Escalation
Mar 18, 2026Philipp Normann, Andreas Happe, Jürgen Cito +1LLM Agent Security LLM Post-Training
TU Wien, Vienna, Austria

Oct 10, 2026J/K move · Enter open · S save

Iterate on an LLM, Serve on an SLM

Keegan Wang

Carnegie Mellon University · Phoebe

LLM Agent Training Small Language Models Efficient Language Model Inference Language Model Distillation

Open paper AskPDF arXiv

As frontier LLMs gain more capabilities, the cost to run them also goes up. At the same time, small language models (SLMs) also gain more capabilities but have largely been able to keep their cost stable, offering 5-40 times cheaper inference for the narrow, repetitive subtasks that agents decompose work into. However, the standing limiting factor is getting the correct behavior out of a small model in a fast, efficient, and economical way. Unfortunately, in most cases, zero-shot small models fail narrow production tasks. Furthermore, retraining one every time the behavior changes is too slow to iterate on. We propose doing both in sequence. First, develop the desired behavior in prompt space, on a frontier model, with immediate feedback from live traffic. Then, once the behavior converges, distill the recorded production behavior (not synthetic imitations of it) into an SLM through supervised fine-tuning (SFT). We show a proper use case of this method through a scheduling agent for home-care agencies that fans out a single yes/no safety decision per caregiver-shift pair on GPT-5.4-nano. A LoRA-tuned Qwen3-8B trained on 11.6k recorded production decisions reaches 0.913 agreement, while the teacher (GPT-5.4-nano), re-run on its own recorded prompts, agrees with itself only 72.7% of the time, pointing to the fact that the SLM may also be more consistent. Furthermore, when the two disagreed, a blinded professional scheduler sided with the student 48-29. Finally, we find that the SLM has approximately 2.5 times lower median latency and is a fraction of the cost. We believe this lifecycle of quick building, live feedback, and cheap inference is a blueprint for others building similar systems.

Abstract

Small Language Models

Explore similar work

Iterate on an LLM, Serve on an SLM

Ignore All Previous Instructions: Jailbreaking as a de-escalatory peace building practise to resist LLM social media bots

Towards Reliable Local Security Agents: Verifiable Post-Training for Linux Privilege Escalation