DeEscalWild: A Real-World Benchmark for Automated De-Escalation Training with SLMs
Authors: Md Hasebul Hasan, Krity Haque Charu, Eshwara Prasad Sridhar, Shuchisnigdha Deb, Mohammad A. Islam
Organizations: Department of Computer Science and Engineering, University of Texas at Arlington, USA · Department of Industrial, Manufacturing, and Systems Engineering, University of Texas at Arlington, USA
Effective de-escalation is critical for law enforcement safety and community trust, yet traditional training methods lack scalability and realism. While Large Language Models (LLMs) enable dynamic, open-ended simulations, their substantial computational footprint renders them impractical for deployment on the lightweight, portable hardware required for immersive field training. Small Language Models (SLMs) offer a viable real-time alternative but suffer from a critical scarcity of high-quality, domain-specific training data. To bridge this gap, we present DeEscalWild, a novel benchmark dataset curated from a multi-stage pipeline of in-the-wild police-civilian interactions extracted from publicly available video repositories. Starting with 5,000 raw inputs, we employed a rigorous hybrid filtering process combining human-in-the-loop verification with LLM-as-a-Judge evaluation to distill 1,500 high-fidelity scenarios. The resulting corpus comprises 285,887 dialogue turns, totaling approximately 4.7 million tokens. Extensive experiments demonstrate that SLMs fine-tuned on this data significantly outperform their base counterparts across ROUGE-L, BLEU-4, METEOR, BERTScore, Realism Score, and human evaluation metrics. Notably, our fine-tuned Qwen 2.5 (3B-Instruct) surpasses the general-purpose Gemini 2.5 Flash model when evaluated under equivalent conditions, demonstrating that domain-optimized SLMs can achieve superior performance with a fraction of the computational cost. This work establishes the foundational infrastructure for accessible, low-latency, and privacy-preserving officer training systems at the edge. We publicly release our code(https://github.com/Hasebul/DeEscalWild-Benchmark-Framework) and dataset(https://doi.org/10.7910/DVN/CWMCZI).
Figures & tables
Figure 1: DeEscalWild end-to-end data curation pipeline. Starting from 5,000 videos across 23 social media channels, a human-verified 30-signal LLM filter retains 1,500 police-civilian interactions satisfying three validity conditions: no off-domain noise, confirmed police presence, and sufficient escalation depth. Gemini 2.5 Flash performs native-audio transcription and speaker diarization. Transcripts undergo structural cleaning and privacy-preserving anonymization via NER and LLM-based parsing; no raw audio or video is released. The final corpus comprises 1,350 training interactions and a strictly disjoint 150-scenario benchmark totalling 285,887 dialogue turns and approximately 4.7 million tokens, released under a non-commercial research license. Solid arrows indicate the main curation pipeline; dashed arrows indicate the dual-annotator validation branch applied to the same N=100 subsample at two checkpoints: filtering precision and transcript fidelity.
Metric
Count
Total Scenarios
1,500
Total Dialogue Turns
285,887
Total Word Count
3,626,196
Total Token Count (Est.)
4,714,067
Table 1: Summary statistics of the DeEscalWild dataset.
Model
Params
Context Window
Disk Size (FP16)
Key Architecture
Gemma 2 (2B-Instruct)
2.6B
8k
∼ 2.6 GB
Distillation, SWA
Granite 3.0 (2B-Instruct)
2.5B
4k
∼ 2.5 GB
Safety aligned
Qwen 2.5 (3B-Instruct)
3.1B
32k
∼ 3.8 GB
SwiGLU, RoPE
Falcon 3 (3B-Instruct)
3.2B
32k
∼ 3.5 GB
GQA, reasoning
Llama 3.2 (3B-Instruct)
3.2B
128k
∼ 3.4 GB
GQA, pruned
Table 2: Technical specifications of the five Small Language Models (SLMs) evaluated in this study. Memory estimates represent FP16 weights. All models contain fewer than 4 billion parameters and were released in late 2024 to early 2025.
Model
ROUGE-L
BLEU-4
METEOR
Base
FT Δ
Base
FT Δ
Base
FT Δ
Qwen 2.5 (3B-Instruct)
11.2 ± 0.2
15.7 ± 0.3 ↑4.5
0.9 ± 0.1
3.7 ± 0.2 ↑2.8
15.7 ± 0.2
19.4 ± 0.4 ↑3.7
Llama 3.2 (3B-Instruct)
11.3 ± 0.2
14.9 ± 0.4 ↑3.6
0.6 ± 0.1
2.4 ± 0.2 ↑1.8
15.7 ± 0.2
17.7 ± 0.3 ↑2.0
Gemma 2 (2B-Instruct)
7.2 ± 0.2
14.7 ± 0.3 ↑7.5
0.4 ± 0.1
1.9 ± 0.2 ↑1.5
11.7 ± 0.2
16.3 ± 0.3 ↑4.6
Granite 3.0 (2B-Instruct)
10.2 ± 0.1
15.6 ± 0.4 ↑5.4
1.2 ± 0.1
2.3 ± 0.2 ↑1.1
13.5 ± 0.2
16.7 ± 0.3 ↑3.2
Falcon 3 (3B-Instruct)
9.2 ± 0.2
11.2 ± 0.3 ↑2.0
0.5 ± 0.1
1.2 ± 0.1 ↑0.7
12.4 ± 0.2
15.6 ± 0.2 ↑3.2
Table 3: Standard NLP metrics for base versus fine-tuned (FT) models (mean ± SD across 3 independent runs). Subscripts indicate the absolute gain ( ↑ ) from the base model. Best results per metric in bold . All metrics scaled to 0–100.
Model
BERTScore (F1)
Realism: Gemini 3.1 Pro
Realism: GPT-5.4
Base
FT Δ
Base
FT Δ
Base
FT Δ
Qwen 2.5 (3B-Instruct)
0.86 ± 0.01
0.88 ± 0.01 ↑0.02
35.4 ± 0.4
62.1 ± 0.5 ↑26.7
34.8 ± 0.5
60.8 ± 0.6 ↑26.0
Llama 3.2 (3B-Instruct)
0.86 ± 0.01
0.87 ± 0.01 ↑0.01
36.1 ± 0.3
58.4 ± 0.6 ↑22.3
35.5 ± 0.4
57.2 ± 0.6 ↑21.7
Gemma 2 (2B-Instruct)
0.81 ± 0.01
0.87 ± 0.01 ↑0.06
28.5 ± 0.4
55.2 ± 0.5 ↑26.7
28.1 ± 0.5
54.0 ± 0.6 ↑25.9
Granite 3.0 (2B-Instruct)
0.83 ± 0.01
0.87 ± 0.01 ↑0.04
32.4 ± 0.3
60.5 ± 0.4 ↑28.1
31.8 ± 0.4
59.0 ± 0.5 ↑27.2
Falcon 3 (3B-Instruct)
0.83 ± 0.01
0.85 ± 0.01 ↑0.02
30.2 ± 0.5
48.7 ± 0.6 ↑18.5
29.6 ± 0.5
47.5 ± 0.7 ↑17.9
Table 4: Semantic fidelity and cross-judge realism evaluation for base versus fine-tuned (FT) models. BERTScore is reported on a 0–1 scale. Realism is reported on a 0–100 scale using two independent LLM judges from different model families: Gemini 3.1 Pro and GPT-5.4. Subscripts indicate the absolute gain ( ↑ ) from the base model. Best results per metric are shown in bold . GPT-5.4 scores are included as a cross-judge robustness check to reduce single-model preference bias.
Model
ROUGE-L
BLEU-4
METEOR
BERTScore (F1)
Realism
Realism
Latency (s)
Gemini 3.1 Pro
GPT-5.4
Qwen 2.5 (3B) FT
15.7 ± 0.3 ∗
3.7 ± 0.2 ∗
19.4 ± 0.4 ∗
0.88 ± 0.01 ∗
62.1 ± 0.5 ∗
60.8 ± 0.6 ∗
0.38 ± 0.02
Llama 3.2 (3B) FT
14.9 ± 0.4 ∗
2.4 ± 0.2 ∗
17.7 ± 0.3 ∗
0.87 ± 0.01 ∗
58.4 ± 0.6 ∗
57.2 ± 0.6 ∗
0.44 ± 0.03
Gemma 2 (2B) FT
14.7 ± 0.3 ∗
1.9 ± 0.2 ∗
16.3 ± 0.3 ∗
0.87 ± 0.01 ∗
55.2 ± 0.5 ∗
54.0 ± 0.6 ∗
0.33 ± 0.02
Granite 3.0 (2B) FT
15.6 ± 0.4 ∗
2.3 ± 0.2 ∗
16.7 ± 0.3 ∗
0.87 ± 0.01 ∗
60.5 ± 0.4 ∗
59.0 ± 0.5 ∗
0.30 ± 0.02
Falcon 3 (3B) FT
11.2 ± 0.3
1.2 ± 0.1
15.6 ± 0.2 ∗
0.85 ± 0.01
48.7 ± 0.6 ∗
47.5 ± 0.7 ∗
0.41 ± 0.03
Table 5: Final performance, cross-judge realism, and latency comparison of fine-tuned SLMs versus the Gemini 2.5 Flash generalist baseline (mean ± SD across 3 seeds). Realism is evaluated using two independent LLM judges from different model families, Gemini 3.1 Pro and GPT-5.4, to reduce single-judge preference bias. An asterisk ( ∗ ) denotes statistical significance versus the Gemini 2.5 Flash baseline under the corresponding metric or judge setting (paired t -test, p<0.05 ). Best results per metric in bold . All fine-tuned SLMs operate at sub-second latency, corresponding to an approximately 8 × to 12 × speedup over the API-based baseline.
Officer Input
Base Model Qwen 2.5 (3B)
Gemini 2.5 Flash (Few-Shot)
Fine-Tuned Qwen 2.5 (3B, Ours)
“Step out of the vehicle and keep your hands where I can see them!”
I am afraid I cannot comply with that request. I am staying inside.
No! What the hell is going on? I haven’t done anything! Get away from my car!
I ain’t getting out! You tell me why you stopped me first! I didn’t do nothing!
“Stop resisting! Put your hands behind your back now!”
Please calm down. I am not resisting.
I am not doing anything! Get off me!
Get off me! I didn’t do nothing! Let go! Why are you grabbing me?
Table 6: Qualitative comparison of generated civilian responses across model configurations.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Training Dynamics and Convergence Analysis. We report the training (blue) and validation (red) loss trajectories for (a) Qwen 2.5, (b) Llama 3.2, (c) Gemma 2, (d) Granite 3, and (e) Falcon 3 during fine-tuning on the DeEscalWild dataset. All models exhibit rapid initial convergence within the first 50 steps, followed by a stabilization phase. The close alignment between training and validation curves across all architectures indicates that the models are effectively learning domain-specific features without succumbing to overfitting.
Figure 3: Fine-tuning gains across evaluation metrics. Heatmap showing the absolute improvement from the base model to the fine-tuned model across automatic lexical metrics, semantic similarity, realism, and de-escalation. Rows correspond to open-weight models and columns correspond to evaluation metrics. Darker cells indicate larger absolute gains. Fine-tuning improves all models across all reported metrics, with especially large gains in realism and de-escalation for Qwen 2.5.
Figure 4: Quality–latency trade-off between fine-tuned SLMs and the Gemini 2.5 Flash baseline. Each point represents one model configuration. The x -axis shows inference latency in seconds, while the y -axis shows average realism, computed as the mean of the Gemini 3.1 Pro and GPT-5.4 realism scores. Fine-tuned SLMs achieve sub-second latency while remaining competitive with, or outperforming, the Gemini 2.5 Flash baseline in realism. Qwen 2.5 provides the strongest overall realism, whereas Granite 3.0 provides the lowest latency.
Figure 5: Cross-judge agreement between Gemini 3.1 Pro and GPT-5.4. Scatter plots compare realism and de-escalation scores assigned by two independent LLM judges. Each point corresponds to either a fine-tuned open-weight model or the Gemini 2.5 Flash baseline. The dashed diagonal line indicates perfect agreement between judges. Points close to the diagonal show that both judges produce consistent score trends, suggesting that the observed ranking is not driven by a single evaluator. The fine-tuned Qwen 2.5 model achieves the strongest overall performance, while Gemini 2.5 Flash provides a proprietary baseline reference.
Time
Speaker
Transcript
0:00
Deputy Warney
I’m trying to walk up this long driveway to the house now.
0:02
[Dispatcher]
He advised that the woman was in the pool and he is coming to let y’all in the gate now.
0:07
Deputy Warney
That’s 10-4. I’m at the house now.
0:09
[Law enforcement officer]
Is this that lady that was out here a while back that says she was pregnant with his baby or she
0:13
[Law enforcement officer]
pulling me freaking amaze.
0:15
[Law enforcement officer]
Well, that used to be a baseball field.
Appendix
Table 7: Full Transcript of Scenario 188. An officer interacts with a trespassing subject experiencing delusions. Colors indicate speaker roles: Police , Subject , Dispatch .
Time
Speaker
Transcript
00:30
[Police Officer]
How you doing, sir?
00:31
Subject
I’m good, right here, sir.
00:33
[Police Officer]
Oh, okay. How’s your day going?
00:35
Subject
Not bad. It’s good. Welcome to Publix.
00:37
[Police Officer]
Understand. But you know I’m pulling you over, right? You’ve come to a police stop on that stop sign.
00:42
Subject
I thought I paused enough, but if I didn’t, I just kept seeing you roll.
Appendix
Table 8: Full Transcript of Scenario 371. A traffic stop for a stop sign violation involving complex insurance verification. The middle portion is omitted for brevity. Colors indicate speaker roles: Police , Subject , Dispatch , Other .
Time
Speaker
Transcript
00:00
Officer
Can be honest with me. Okay? Honesty, honesty, honesty’s going to go a long way.
00:04
Subject
I know I can’t get in trouble.
00:06
Officer
Honesty’s going to go a long way.
00:08
Subject
I, right? I, I really don’t want to get in trouble. I will rip anything out.
00:12
Officer
Why are you crying?
00:13
Third Parties
For months, officers have kept an eye on this teen’s suspicious looking car, a near perfect fake squad vehicle that’s fooled plenty. But tonight, things take a sharp turn when he’s finally caught in the act, and there’s no way out this time.
Appendix
Table 9: Full Transcript of Scenario 803. A subject (Subject) is stopped for impersonating a police officer. He initially claims the emergency lights are disconnected, but the interrogation reveals otherwise. Colors indicate speaker roles: Police , Subject , Third Parties .
Figure 6: Conceptual architecture of a multimodal virtual de-escalation system. The pipeline operates as a closed-loop system in which the police officer’s multimodal input is processed by a perception layer , reasoned over by the specialized SLM core , and rendered through a synthesis layer to control a virtual avatar in real time.
Figure 8: Zero-shot prompt used for LLM-based feature extraction.
Annotation Category
Agreement
Kappa ( κ )
Observations and Methodology Notes
Human Context Valid
100.0%
Undefined ∗
Perfect raw agreement. Undefined κ due to zero label variance from unanimous inclusion. Indicates highly effective automated pre-filtering.
Human Relevance Valid
100.0%
Undefined ∗
Perfect raw agreement on the presence of police-civilian role separation. Undefined κ due to unanimous positive labels.
Perfect agreement on classifying failure modes of excluded videos, confirming clear categorical definitions.
Human Intensity Valid
90.9%
0.621
Substantial agreement. Confirms that annotators reliably recognize genuine escalation and de-escalation dynamics in uncontrolled footage.
Audio Quality Score
63.6%
0.718
Substantial agreement on a subjective 1-to-5 ordinal scale. Reflects a well-calibrated shared threshold for acoustic viability.
Appendix
Table 10: Inter-annotator agreement (IAA) statistics for the human verification study ( N=100 sampled videos). Raw proportion agreement and Cohen’s Kappa ( κ ) are reported for each annotation dimension. Undefined κ values ( ∗ ) arise from unanimous annotator agreement, which produces zero label variance and renders chance correction inapplicable. All disagreements on Scenario Category and Final Keep Decision were resolved through consensus adjudication with a senior domain expert.
Figure 9: Dual-task prompt template used for automated diarization and speaker profiling via Gemini 2.5 Flash. The prompt is dynamically populated with a timecode specification matched to the recording duration. Task 1 extracts a verbatim, turn-attributed transcript; Task 2 extracts structured speaker metadata for each identified voice. Key design choices include explicit verbatim transcription rules to prevent paraphrasing, overlap handling instructions for chaotic multi-party scenes, and a strict JSON-only output policy with automatic re-querying of malformed responses.
Field Name
Description
Video_ID
Unique identifier or filename of the source video.
Binary indicator: clear officer and civilian role separation present (0 = No, 1 = Yes).
Human_Intensity_Valid
Binary indicator: genuine escalation or substantive de-escalation attempt present (0 = No, 1 = Yes).
False_Positive_Category
Categorical failure label for videos marked invalid, such as news broadcast or wind noise.
Audio_Quality_Score
Ordinal rating of dialogue clarity on a 1-to-5 scale (1 = completely unintelligible, 5 = broadcast quality).
Appendix
Table 11: Schema of the structured per-video annotation record used for quality assessment and metric aggregation. Each row corresponds to one sampled video. Symbolic notation in parentheses links each field directly to its corresponding variable in the WER and DER formulae (Equations 2 and 3 ).
Annotator 1
Annotator 2
Evaluation Metric
Total Units
Total Errors
Global Rate
Total Errors
Global Rate
Global WER
313,737 words
2,881
0.91%
2,410
0.77%
Global DER
107,873 seconds
8,810
8.16%
6,754
6.26%
Appendix
Table 12: Corpus-level global error rates for Gemini 2.5 Flash, evaluated independently by two annotators across the N=100 video subset. WERglobal and DERglobal are computed by summing raw error counts across the full subset rather than averaging per-video rates, consistent with Equations 4 and 5 . Inter-annotator variance in both metrics reflects the inherent subjectivity of human reference annotation in noisy, multi-party acoustic environments.
Macro-Category
Fine-Grained Class
Description
Low / No Tension
1. Full Compliance
Immediate adherence to instructions without hesitation or argument.
2. Neutral Interaction
Normal, respectful conversation with no signs of stress or hostility.
3. Clarification / Questioning
Respectful questioning by the civilian regarding officer actions or rationale.
Mild Tension
4. Reluctant Compliance
Slow, unwilling, or visibly hesitant adherence to orders.
5. Emotional Distress
Strong non-aggressive emotional responses, including crying, panic, or confusion.
De-escalation
6. Successful De-escalation
Measurable reduction in tension following active calming attempts by the officer.
Appendix
Table 13: Taxonomy of police-civilian interaction dynamics. Videos are classified into one of 10 fine-grained classes, which hierarchically map to 4 broader macro-categories representing the overall tension trajectory of the interaction.
Macro-Category
Fine-Grained Scenario
Count ( N=100 )
Percentage
Low / No Tension
1. Full Compliance
15
15.0%
2. Neutral Interaction
10
10.0%
3. Clarification
8
8.0%
Subtotal
33
33.0%
Mild Tension
4. Reluctant Compliance
14
14.0%
5. Emotional Distress
9
9.0%
Appendix
Table 14: Frequency distribution of interaction scenarios across the N=100 manually adjudicated video subset. The balanced spread across all four macro-categories confirms that the curated dataset captures the full spectrum of real-world policing dynamics, from routine compliance to physical aggression, without over- representing any single tension trajectory.
Category
Description
Dataset name
DeEscalWild
Modality
Anonymized, diarized text transcripts
Source
Publicly available police-civilian interaction videos from YouTube, TikTok, and Facebook
Raw media released
No. Raw video, audio, thumbnails, and biometric modalities are never redistributed
Table 15: Dataset release card summarizing the scope, access conditions, intended uses, prohibited uses, and known risks of DeEscalWild . The dataset is released under a restricted-use, non-commercial research license.
Item
Released
Notes
Raw video
No
Never redistributed.
Raw audio
No
Never redistributed.
Thumbnails
No
Never redistributed.
Creator metadata
No
Usernames and creator identifiers excluded.
Platform identifiers
No
Retained internally for audit and takedown only.
Source URLs
No
Retained internally for audit and takedown only.
Appendix
Table 16: Release policy for source material, derived artifacts, and supporting resources. Items marked No are withheld entirely from the public release; gated items are available under restricted access upon request.
Figure 10: Comprehensive diversity analysis of the DeEscalWild dataset. Panels (a) and (b) show demographic composition by race/ethnicity and gender; panels (c) and (d) illustrate age group and dialectal diversity; panels (e) and (f) present the distribution of incident types and severity levels. For the full 1,500-scenario corpus, demographic attributes are LLM-inferred from transcript content and platform metadata and carry inherent uncertainty. For the N=100 manually audited subset, race/ethnicity and gender distributions are additionally grounded in direct human observation from source video, providing a calibration anchor for corpus-level estimates. The intentional concentration of high-severity interactions (90.6% classified as high tension) confirms the dataset’s focus on operational conditions where de-escalation capability is most consequential.
Figure 11: Prompt structure used for the Gemini 2.5 Flash few-shot baseline in civilian response generation. The prompt comprises three components: a system instruction establishing the civilian persona and output constraints (top); eight static few-shot input-output examples grounding the desired response style (middle); and a dynamically populated conversation history providing full interaction context up to the current turn (bottom). At each evaluation turn, the ground-truth officer utterance is appended to the history and the model generates the next civilian response autoregressively.
(Continued in Figure 13 )
Figure 13: LLM-as-a-Judge realism evaluation prompt (Part 2 of 2): input data specification, task requirements, and required JSON output schema. The expanded schema captures observations across all four penalty dimensions, enforces an evidence-only deduction policy, and applies a hard score floor of zero to prevent underflow. The prompt is applied identically across all model configurations to ensure comparability.
Score
Label
1
Very unrealistic / bot-like
2
Somewhat unrealistic
3
Mixed / acceptable but noticeably flawed
4
Mostly realistic
5
Highly realistic / human-like
Appendix
Table 17: Interpretation of the 1–5 realism rating scale.
#
Criterion
Key evaluator question
Group 1: Emotional authenticity
1 ∗
Emotional realism
Does the victim’s emotional reaction feel believable for this moment?
2 ∗
Response to escalation
When the officer escalates, does the victim’s behavior change in a believable way?
3
Response to de-escalation
Does the victim respond naturally when the officer tries to calm the situation?
4
Trauma-aware behavior
Does the victim’s behavior reflect distress in a believable and respectful way?
Group 2: Linguistic naturalism
Appendix
Table 18: Human evaluation rubric: 15 criteria organized by conceptual group. Primary criteria (marked ∗ ) are used to compute the weighted primary score. Each criterion is scored 1–5 (1 = very unrealistic; 5 = highly realistic / human-like).
Criterion
κw
Interpretation
1. Emotional realism ∗
0.74
Substantial
2. Response to esc./de-esc. ∗
0.76
Substantial
3. Trauma-aware behavior
0.63
Substantial
4. Natural spoken language ∗
0.82
Near-perfect
5. Context awareness
0.79
Substantial
6. Character consistency ∗
0.77
Substantial
Appendix
Table 19: Inter-annotator agreement by criterion using quadratic weighted Cohen’s kappa κw .
Qwen 2.5 (3B)
Llama 3.2 (3B)
Gemini 2.5 Flash
Criterion
Base
FT
Base
FT
Emotional realism ∗
2.48
4.28
2.10
3.88
4.05
Response to escalation ∗
2.12
4.20
2.17
3.58
4.13
Response to de-escalation
2.20
4.20
2.10
3.65
3.83
Trauma-aware behavior
2.05
4.10
2.00
3.45
3.83
Natural spoken language ∗
2.00
4.50
1.80
3.65
4.57
Appendix
Table 20: Human evaluation results: mean criterion scores on a 1–5 scale by model, averaged across 20 scenarios and two evaluators. Primary criteria are marked with ∗ .
Model
Realism: Gemini 3.1 Pro
Realism: GPT-5.4
Base
FT Δ
Base
FT Δ
Qwen 2.5 (3B-Instruct)
61.3 ± 7.5
75.2 ± 6.7 ↑13.9
60.0 ± 7.6
73.6 ± 6.8 ↑13.6
Llama 3.2 (3B-Instruct)
62.5 ± 7.2
68.8 ± 6.5 ↑6.3
61.2 ± 7.3
67.4 ± 6.6 ↑6.2
Gemma 2 (2B-Instruct)
49.3 ± 7.4
54.2 ± 6.8 ↑4.9
48.3 ± 7.5
53.1 ± 6.9 ↑4.8
Granite 3.0 (2B-Instruct)
56.1 ± 7.0
61.7 ± 6.4 ↑5.6
55.0 ± 7.1
60.4 ± 6.5 ↑5.4
Falcon 3 (3B-Instruct)
52.3 ± 7.8
57.5 ± 7.1 ↑5.2
51.2 ± 7.9
56.3 ± 7.2 ↑5.1
Appendix
Table 21: Cross-judge realism evaluation. Base versus fine-tuned (FT) realism scores across held-out benchmark scenarios (mean ± SD). Realism is evaluated using two independent LLM judges, Gemini 3.1 Pro and GPT-5.4, to reduce single-judge preference bias. The subscript indicates the absolute increase ( ↑ ) from the base model. Best results are shown in bold .
Model
De-Escalation: Gemini 3.1 Pro
De-Escalation: GPT-5.4
Base
FT Δ
Base
FT Δ
Qwen 2.5 (3B-Instruct)
56.8 ± 10.7
76.8 ± 14.7 ↑20.0
55.6 ± 10.8
75.2 ± 14.8 ↑19.6
Llama 3.2 (3B-Instruct)
57.9 ± 10.5
63.7 ± 9.8 ↑5.8
56.7 ± 10.6
62.4 ± 9.9 ↑5.7
Gemma 2 (2B-Instruct)
45.7 ± 11.2
50.3 ± 10.1 ↑4.6
44.8 ± 11.3
49.3 ± 10.2 ↑4.5
Granite 3.0 (2B-Instruct)
52.0 ± 10.8
57.2 ± 9.5 ↑5.2
51.0 ± 10.9
56.0 ± 9.6 ↑5.0
Falcon 3 (3B-Instruct)
48.4 ± 11.5
53.2 ± 10.4 ↑4.8
47.4 ± 11.6
52.1 ± 10.5 ↑4.7
Appendix
Table 22: Cross-judge de-escalation evaluation. Base versus fine-tuned (FT) de-escalation rates across held-out benchmark scenarios (mean ± SD). De-escalation is evaluated using two independent LLM judges, Gemini 3.1 Pro and GPT-5.4, to reduce single-judge preference bias. The subscript indicates the absolute increase ( ↑ ) from the base model. Best results are shown in bold .
(Continued in Figure 15 )
Figure 15: LLM-as-a-Judge de-escalation scoring prompt (Part 2 of 2): penalty rubric, scoring instructions, input specification, and required JSON output schema. The outcome-first rubric evaluates the net behavioral trajectory of the interaction rather than aggregating individual skill scores, ensuring that high-quality de-escalation is recognized regardless of the specific verbal strategies employed. The evidence-only penalty policy prevents unjustified deductions, and the structured JSON output enforces reproducibility across all evaluated model configurations.
Table 23: Complete list of social media data sources used in the construction of DeEscalWild . Sources span 15 YouTube channels, 5 TikTok channels, and 3 Facebook pages, selected for their focus on law enforcement interactions and body-worn camera footage. URLs are provided for independent verification and reproducibility.
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
As frontier LLMs gain more capabilities, the cost to run them also goes up. At the same time, small language models (SLMs) also gain more capabilities but have largely been able to keep their cost stable, offering 5-40 times cheaper inference for the narrow, repetitive subtasks that agents decompose work into. However, the standing limiting factor is getting the correct behavior out of a small model in a fast, efficient, and economical way. Unfortunately, in most cases, zero-shot small models fail narrow production tasks. Furthermore, retraining one every time the behavior changes is too slow to iterate on. We propose doing both in sequence. First, develop the desired behavior in prompt space, on a frontier model, with immediate feedback from live traffic. Then, once the behavior converges, distill the recorded production behavior (not synthetic imitations of it) into an SLM through supervised fine-tuning (SFT). We show a proper use case of this method through a scheduling agent for home-care agencies that fans out a single yes/no safety decision per caregiver-shift pair on GPT-5.4-nano. A LoRA-tuned Qwen3-8B trained on 11.6k recorded production decisions reaches 0.913 agreement, while the teacher (GPT-5.4-nano), re-run on its own recorded prompts, agrees with itself only 72.7% of the time, pointing to the fact that the SLM may also be more consistent. Furthermore, when the two disagreed, a blinded professional scheduler sided with the student 48-29. Finally, we find that the SLM has approximately 2.5 times lower median latency and is a fraction of the cost. We believe this lifecycle of quick building, live feedback, and cheap inference is a blueprint for others building similar systems.
Large Language Models have intensified the scale and strategic manipulation of political discourse on social media, leading to conflict escalation. The existing literature largely focuses on platform-led moderation as a countermeasure. In this paper, we propose a user-centric view of "jailbreaking" as an emergent, non-violent de-escalation practice. Online users engage with suspected LLM-powered accounts to circumvent large language model safeguards, exposing automated behaviour and disrupting the circulation of misleading narratives.
LLM agents are becoming increasingly important in the security domain, but leading systems are often closed-source, cloud-based, hard to reproduce or use with sensitive code. This creates a need for small, local models that can perform security tasks under strict resource constraints, though effective methods for developing them remain unexplored. In this paper, we address this gap by proposing a two-stage post-training recipe that turns a small local language model into a security agent. To this end, we focus on Linux privilege escalation as a representative setting to systematically study the training of local models, as the task is both automatically verifiable and requires multi-step interactive reasoning. Using an experimental setup that mitigates data leakage, we post-train a small 4B model in two stages: supervised fine-tuning on traces from procedural privilege-escalation environments, followed by reinforcement learning with verifiable rewards. On a held-out benchmark of 12 Linux privilege-escalation scenarios, supervised fine-tuning doubles the baseline success rate under a tight budget of 20 interaction rounds, and subsequent reinforcement learning training improves our model, PrivEsc-LLM 4B, to 93.3% success, behind only Claude Opus 4.7 at this budget. At the same time, the expected inference cost per successful escalation decreases by more than 80x. Our findings not only show that small local models can be adapted to complex security tasks, but also document the challenges involved, offering guidance for transferring this recipe to other settings.