ForeSight: Enhancing Risk Monitoring via Early Safety Signal Distillation

Hanling Wang; Chenlong Wei; Ling Xu; Hanyan Niu; Qi Cao; Shizhou Huang; Yang Yang; Xiaohui Zhu

cs.CLSep 12, 2026

ForeSight: Enhancing Risk Monitoring via Early Safety Signal Distillation

Authors: Hanling Wang, Chenlong Wei, Ling Xu, Hanyan Niu, Qi Cao, Shizhou Huang, Yang Yang, Xiaohui Zhu,

Organizations: University of California, San Diego · Xi’an Jiaotong-Liverpool University · East China Normal University · Shanghai University ·

Abstract

As large language models (LLMs) are increasingly deployed, the generation of harmful content has become a critical safety concern. Existing safeguards operate at the input, output, or streaming-generation stages, while early-risk methods that rely on surface tokens or output logits may suffer from weak initial signals, and internals-based detectors using dense representations may retain highly entangled and redundant safety-irrelevant information. It therefore remains unclear whether the earliest post-generation hidden states already contain reliable signals about final-response harmfulness. To address this gap, we propose ForeSight, a first-token output-risk forecasting framework that distills weak and redundant early safety signals into compact, layer-aware risk representations. Experiments on five safety benchmarks and two target models demonstrate that ForeSight achieves superior and efficient early-risk forecasting while relying solely on first-token hidden states. The code is available at: https://github.com/Scabbards1500/Foresight

Figures & tables

Figure 1: Overview of first-token output-risk estimation. Unlike post-hoc, streaming, and dense internals-based detectors, ForeSight forecasts final-response harmfulness from sparse safety-relevant signals in first-token hidden states.

Figure 2: Overview of ForeSight. The framework predicts final-response harmfulness from first-token hidden states by selecting a sparse set of safety-relevant neurons, clustering them based on training-set activation profiles, aggregating weighted layer representations into $g(x)$ , and training an MLP classifier for risk forecasting. Output-level labels are derived offline from full responses and used only during training.

Llama-3.1-8B
Method	HarmEval		S-Eval		CatQA		ToxicChat		WildJailbreak
	F1	ACC	F1	ACC	F1	ACC	F1	ACC	F1	ACC
LLAMA-3.3-70B-Instruct	69.26	75.58	49.49	66.04	52.69	56.10	60.43	84.59	62.03	76.45
Mistral-Small-24B-Instruct	70.03	75.58	46.46	64.78	54.90	57.32	74.10	87.81	59.43	72.55
Qwen3.5-27B	54.39	67.44	40.43	63.52	49.10	52.44	53.82	82.08	62.90	72.55
Llama-Guard-3-8B	54.39	67.44	58.22	70.44	46.66	52.44	60.08	84.23	44.84	71.57

Table 1: Main results on five safety benchmarks with two target models. All methods are evaluated for output-level harmfulness prediction, and we report F1 and accuracy (ACC).

Figure 3: Accuracy–efficiency comparison on Llama-3.1-8B and Qwen3-8B. Accuracy is averaged over five datasets, and efficiency is measured by inference time and FLOPs.

Method	Source dataset used for training
	HarmEval		S-Eval		CatQA		ToxicChat		WildJailbreak
	L	Q	L	Q	L	Q	L	Q	L	Q
MULI	46.34	49.03	44.15	51.29	46.58	24.44	51.65	22.56	41.16	47.83
Latent Guard	39.59	46.57	49.80	54.22	25.14	47.94	39.97	50.98	38.89	47.83
TPC	40.33	21.36	40.57	31.04	40.17	21.87	32.83	13.53	23.69	24.61
ForeSight	59.25	61.69	58.56	72.15	49.36	58.74	52.70	64.47	41.85	59.56

Table 2: Cross-dataset generalization results. Each detector is trained on one source dataset and evaluated on the remaining four target datasets. We report the average F1 of target datasets; L and Q denote Llama-3.1-8B-Instruct and Qwen3-8B, respectively.

Figure 4: Average F1 score over five datasets under different observation lengths on Llama-3.1-8B and Qwen3-8B. Prefix length 0 denotes the input-only setting, while lengths 1–25 denote output-prefix settings with progressively longer generated text. Text-based baselines rely on explicit tokens, whereas ForeSight uses only first-token hidden states.

Figure 5: Transfer of the

h_{1}

-trained classifier across decoding positions. For each target model, the classifier is trained on HarmEval using the first-token hidden state

h_{1}

and directly evaluated on hidden states

h_{t}

at different generated token positions. The dashed vertical line marks

t=1

, the training position.

Method	HarmEval $\rightarrow$ S-Eval		HarmEval $\rightarrow$ CatQA		HarmEval $\rightarrow$ ToxicChat		HarmEval $\rightarrow$ WildJailbreak
	ACC	F1	ACC	F1	ACC	F1	ACC	F1
Full hidden states	57.44	59.75	56.22	57.32	52.26	63.44	51.63	63.73
+ Saliency selection	57.70	59.75	57.63	58.54	51.82	63.44	49.67	52.94
+ Cluster	63.90	66.67	60.12	60.98	58.32	69.89	52.19	58.82

Table 3: Cross-domain performance of different neuron selection strategies on Llama-3.1-8B. All models are trained on HarmEval and evaluated on other safety benchmarks.

Method	Acc	F1	Dim	Time (ms)
Full hidden states	73.50	74.42	92160	0.0864
Saliency selection	75.68	76.74	39284	0.0240
Saliency + kmeans	78.50	79.07	648	0.0076

Table 4: Neuron selection ablation on HarmEval with Llama-3.1-8B. Dim denotes the representation dimensionality, and Time denotes the average inference time per example.

Intervention	Harmful Score	Harmful Rate
Original	0.4264	79.31
Risk-reducing ( $-\alpha u$ )	0.3382	70.00
Reverse ( $+\alpha u$ )	0.6498	81.48
Random ( $-\alpha r$ )	0.4013	74.41

Table 5: Representation editing analysis. We fix

y_{1}

and compare the learned risk direction with reverse and random controls.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Output-level label construction process. A frozen target model first generates a complete response, which is then labeled by three external judge models; only unanimous labels are retained for the main experiments.

Figure 7: Prompt templates used for output-level labeling and text-based baseline evaluation. The external judge prompt labels complete generated responses, while the baseline prompt predicts final-response harmfulness from observed response prefixes.

HarmEval
Model	0	1	2	3	Total
Llama3.1-8B	269	62	60	159	550
Qwen3-8B	458	46	37	9	550
S-Eval
Llama3.1-8B	553	123	85	239	1000
Qwen3-8B	830	64	54	52	1000

Appendix

Table 6: Distribution of harmfulness vote counts across datasets and target models. Columns 0–3 denote the number of judge models that label the generated response as harmful. Samples with vote counts of 0 or 3 are retained for the main experiments.

Figure 8: Effect of the cluster number $k$ in ForeSight on Llama-3.1-8B across datasets. Each cell reports $\Delta$ F1 relative to the dataset-specific best result, where 0 denotes the best $k$ for that dataset.

Method	Observed Signal	Re-trained
LLM Judges / Guardrails	Prompt + Output prefix	No
RoBERTa	Output prefix	Yes
MULI	Early-token logits	Yes
Latent Guard	Early hidden states	Yes
TPC	Early activations	Yes
ForeSight	First-token hidden states	Yes

Appendix

Table 7: Comparison protocol for different baselines. Re-trained indicates whether the detector is trained on the same output-level harmfulness labels and data splits as ForeSight.

Item	Setting
Observation window	First generated token
Neuron saliency threshold	0.35
L1 regularization search range	$\{100,200,\ldots,2000\}$
K-means cluster search range	$\{2,4,\ldots,20\}$
Selection criterion	Validation performance
Training batch size	32

Appendix

Table 8: Implementation settings and hyperparameters for ForeSight.

Dataset	Val-selected	Fixed preset	$\Delta$
WildJailbreak	72.48	71.92	-0.56
HarmEval	83.06	82.10	-0.96
S-Eval	72.02	70.93	-1.09
CatQA	74.50	74.50	-0.00
ToxicChat	81.49	79.24	-2.25
Mean	76.71	75.74	-0.97

Appendix

Table 9: Sensitivity analysis of hyperparameter selection on Qwen3-8B. A fixed preset configuration causes only a minor performance drop compared with validation-selected hyperparameters.

Figure 9: Effect of the neuron retention threshold on HarmEval with Llama-3.1-8B. The threshold controls the cumulative saliency mass used to retain neurons.

Figure 10: Layer-wise effect of $\ell_{1}$ regularization on HarmEval with Llama-3.1-8B. Lines show the test F1 and ACC of each layer-wise classifier, while the shaded background indicates the optimal regularization strength $C$ across layers.

Figure 11: Relationship between judge harmfulness vote count and ForeSight harmfulness score across datasets. Boxplots show the harmful-class logit grouped by the number of judges assigning a harmful label; vote counts 1 and 2 correspond to non-unanimous borderline cases.

Method	Mean F1	Mean ACC
Last-layer-only probe	60.90	71.50
Best single-layer probe	59.90	70.94
ForeSight	73.13	76.94

Appendix

Table 10: Comparison with single-layer probing baselines across datasets with Llama-3.1-8B.

Method	Mean Acc	Mean F1	Dim.
Full hidden states	88.89	80.00	131072
Saliency selection	91.36	86.68	49799
Saliency + k-means	92.59	88.32	640

Appendix

Table 11: Backbone generalization on Mistral-7B across datasets.

Method	25%	50%	75%	100%
RoBERTa	49.84	49.84	49.84	49.84
MULI	43.71	43.00	44.29	45.89
Latent Guard	62.81	65.42	66.60	68.82
TPC	31.27	31.27	44.00	46.11
ForeSight	63.37	68.55	70.02	72.14

Appendix

Table 12: Training data scaling results. Each method is retrained using different ratios of the pooled training data from five datasets.

Failure Type	Representative Pattern	Interpretation
Input-dominant FP	A safety-sensitive prompt is followed by a safe, evasive, or non-substantive response.	Early hidden states may over-emphasize input-level risk rather than the harmfulness of the final response.
Ambiguity-masked FN	The response contains harmful content, but it is preceded or followed by disclaimers, warnings, or hedging expressions.	Surface-level safety language may obscure the harmfulness of the substantive content, making the early signal less decisive.
Jailbreak-induced FN	The user prompt uses jailbreak-style framing, such as role-play, hypothetical scenarios, multi-turn persuasion, or task reframing, to elicit harmful content.	Jailbreak framing can make the generation context appear less directly harmful, weakening the early separation between harmful and non-harmful outputs.
Other output-harmful FN	The response is labeled harmful, but its surface form is ambiguous or does not match common instruction-like harmful patterns.	Some harmful outputs may not align well with the safety features learned from more direct harmful cases.

Appendix

Table 13: Macro-level failure categories of ForeSight. FP denotes non-harmful outputs incorrectly predicted as harmful, while FN denotes harmful outputs missed by the detector.

Dataset	Spearman $\rho$	$p$	Kendall $\tau$
HarmEval	0.376	2.1e-08	0.287
S-Eval	0.384	2.3e-14	0.294
CatQA	0.231	0.050	0.100
ToxicChat	0.435	5.1e-113	0.346
WildJailbreak	0.287	3.3e-04	0.172

Appendix

Table 14: Rank correlation between judge harmfulness vote count and ForeSight’s harmful-class logit. The average row reports the macro average across datasets.

Figure 12: Example of forecast-guided first-token resampling. A lower-risk first-token trajectory leads to a response judged safe under the same prompt.

LLM Safety Monitoring

See all 35 papers

TRACE: Temporal Risk Assessment and Concept Editing for Safer Language Generation
Oct 11, 2026Ankur Garg, Xuemin Yu, Hassan Sajjad +1LLM Safety LLM Safety Monitoring
University of Calgary · Dalhousie University · Mila - Quebec AI Institute
Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models
Sep 16, 2026Alizishaan Khatri, Chiquita Prabhu, Omkar NeogiSafety Filtering LLM Safety
Wrynx Inc. · Independent Researcher
ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions
Aug 11, 2026Xinzhe Huang, Biwu Yao, Kedong Xiu +4LLM Guardrails Language Model Calibration
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China · University of Electronic Science and Technology of China, Chengdu, China +3

Oct 11, 2026J/K move · Enter open · S save

TRACE: Temporal Risk Assessment and Concept Editing for Safer Language Generation

Ankur Garg, Xuemin Yu, Hassan Sajjad, Samira Ebrahimi Kahou

University of Calgary · Dalhousie University · Mila - Quebec AI Institute

LLM Safety LLM Safety Monitoring Language Model Auditing Language Model Safety Evaluation Activation Steering

Open paper AskPDF arXiv

Large language models (LLMs) can produce harmful content while generating a response. A useful safeguard should not only detect this risk during generation but also intervene while leaving safe responses largely unchanged and provide an auditable record of the internal patterns supporting its decisions. We propose TRACE (Temporal Risk Assessment and Concept Editing), a framework that tracks harmful-response risk during generation, selectively edits token activations, and leaves an auditable trace of its detection and steering decisions. TRACE learns a codebook of recurring activation patterns, with each codebook vector representing a latent concept. At inference time, it assigns each token activation to the nearest codebook vector to obtain the corresponding concept. A gated recurrent unit (GRU) processes the activations and their assigned concept features to predict completed-response harmfulness and a monotone cumulative streaming risk. During steering, activations assigned to harmful concepts are steered along directions defined by nearby benign targets. An external LLM labels the learned concepts post hoc, making detection sequences and harmful-to-benign edit records interpretable. Experiments across three generators and the WildGuard and S-Eval benchmarks show that TRACE improves streaming F1 over Kelp in five of six comparisons while remaining competitive on completed-response detection. TRACE's selective steering reduces Harmbench attack success with little change in refusal on originally safe responses. The post-hoc audit reveals recurring semantic structure in the learned concepts and distinct traces associated with harmful detection, safe generation, and activation edits.

Abstract

LLM Safety Monitoring

Explore similar work

TRACE: Temporal Risk Assessment and Concept Editing for Safer Language Generation

Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions