Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach

Che Shen; Junwei Su; Lingpeng Kong; Chuan Wu

cs.LGAug 15, 2026

Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach

Authors: Che Shen, Junwei Su, Lingpeng Kong, Chuan Wu

Organizations: Department of Computer Science The University of Hong Kong, Hong Kong, China · School of Artificial Intelligence and Data Science University of Science and Technology of China, Hefei, Anhui, China

Abstract

Reliable post-hoc evaluation asks whether already generated text satisfies a target criterion after generation. In this paper we study a focused frozen-embedding setting using principle-evaluation proxy tasks: toxicity detection, fine-grained emotion categorization, and ordinal review rating. General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples in overlapping regions of the representation space. We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings. The module combines a semantic stream, a prototype-anchor attention stream, supervised contrastive learning, offset-based prototype-margin regularization, and stream regularization to produce a compact task-adapted representation without updating the base encoder. Controlled experiments show that PGCL improves over raw frozen embeddings on all three datasets and gives the clearest direct-baseline margin on AmazonReviews, while remaining competitive with strong direct frozen metric-learning baselines on GoEmotions and ToxicComment. We also add supervised residual-adapter, encoder-LoRA, full fine-tuning, objective ablation, sensitivity, and fully logged few-shot LLM protocol diagnostics to define the boundary of the claim. The theoretical analysis is revised as a sufficient-condition account for prototype-margin behavior under explicit assumptions in the prototype-mapping space, rather than as an unconditional training or final-embedding separation guarantee.

Figures & tables

Dataset	Train	Validation	Test
GoEmotions	5126	906	881
AmazonReviews	2271	487	487
ToxicComment	44256	15073	22609

Table 1: Fixed split sizes used in the revised leakage-control audit.

Metric	Emb. Type	SVM	RF	LR	XGBoost	Transformer
Precision	Raw Emb.	0.748 $\pm$ 0.049	0.733 $\pm$ 0.059	0.737 $\pm$ 0.031	0.747 $\pm$ 0.020	0.785 $\pm$ 0.031
Precision	Opt. Emb.	0.787 $\pm$ 0.035	0.789 $\pm$ 0.036	0.791 $\pm$ 0.033	0.773 $\pm$ 0.028	0.787 $\pm$ 0.029
Recall	Raw Emb.	0.721 $\pm$ 0.045	0.737 $\pm$ 0.031	0.722 $\pm$ 0.031	0.741 $\pm$ 0.014	0.763 $\pm$ 0.024
Recall	Opt. Emb.	0.764 $\pm$ 0.034	0.769 $\pm$ 0.039	0.772 $\pm$ 0.033	0.765 $\pm$ 0.039	0.769 $\pm$ 0.031
F1	Raw Emb.	0.729 $\pm$ 0.046	0.722 $\pm$ 0.035	0.726 $\pm$ 0.031	0.737 $\pm$ 0.018	0.764 $\pm$ 0.036
F1	Opt. Emb.	0.770 $\pm$ 0.033	0.767 $\pm$ 0.036	0.776 $\pm$ 0.032	0.764 $\pm$ 0.026	0.770 $\pm$ 0.030

Table 2: Overall (Avg. Principle) Performance on GoEmotions Five-Principle Set (Mean

\pm

Std. Dev.)

Metric	Emb. Type	SVM	RF	LR	XGBoost	Transformer
MSE	Raw Emb.	0.668 $\pm$ 0.135	0.506 $\pm$ 0.120	0.635 $\pm$ 0.097	0.546 $\pm$ 0.163	0.602 $\pm$ 0.173
MSE	Opt. Emb.	0.365 $\pm$ 0.158	0.392 $\pm$ 0.143	0.394 $\pm$ 0.149	0.377 $\pm$ 0.097	0.359 $\pm$ 0.086
RMSE	Raw Emb.	0.813 $\pm$ 0.083	0.706 $\pm$ 0.083	0.795 $\pm$ 0.059	0.731 $\pm$ 0.111	0.768 $\pm$ 0.110
RMSE	Opt. Emb.	0.593 $\pm$ 0.119	0.617 $\pm$ 0.107	0.618 $\pm$ 0.112	0.609 $\pm$ 0.080	0.595 $\pm$ 0.071
R 2	Raw Emb.	0.604 $\pm$ 0.086	0.700 $\pm$ 0.074	0.624 $\pm$ 0.060	0.677 $\pm$ 0.095	0.643 $\pm$ 0.103
R 2	Opt. Emb.	0.785 $\pm$ 0.089	0.770 $\pm$ 0.080	0.768 $\pm$ 0.083	0.777 $\pm$ 0.055	0.788 $\pm$ 0.047

Table 3: Overall Ordinal Regression Performance on Amazon Reviews (Mean

\pm

Std. Dev.)

Method	Weighted F1	MAE ( $\downarrow$ )	QWK ( $\uparrow$ )	Severe error ( $\downarrow$ )
Raw embeddings	0.589	0.4784	0.8175	0.0637
SupCon projection	0.628	0.4271	0.8319	0.0472
PGCL	0.727	0.3039	0.8904	0.0205

Table 4: AmazonReviews ordinal diagnostics on the held-out test split.

Metric	Emb. Type	SVM	RF	LR	XGBoost	Transformer
Avg. F1	Raw Emb.	0.932 $\pm$ 0.004	0.949 $\pm$ 0.003	0.897 $\pm$ 0.004	0.918 $\pm$ 0.004	0.956 $\pm$ 0.004
Avg. F1	Opt. Emb.	0.938 $\pm$ 0.004	0.949 $\pm$ 0.004	0.936 $\pm$ 0.003	0.943 $\pm$ 0.004	0.959 $\pm$ 0.004
Minority F1	Raw Emb.	0.497 $\pm$ 0.025	0.405 $\pm$ 0.044	0.396 $\pm$ 0.018	0.433 $\pm$ 0.024	0.574 $\pm$ 0.027
Minority F1	Opt. Emb.	0.507 $\pm$ 0.024	0.537 $\pm$ 0.035	0.493 $\pm$ 0.023	0.518 $\pm$ 0.028	0.589 $\pm$ 0.023

Table 5: Performance on Toxic Comment Classification Challenge (Mean

\pm

Std. Dev.)

Dataset	RAW	PGCL	SupCon	Center	Triplet	Prototype	ArcFace	CosFace
AmazonReviews	0.589	0.727	0.628	0.597	0.592	0.599	0.588	0.566
GoEmotions	0.726	0.772	0.757	0.755	0.759	0.764	0.747	0.752
ToxicComment	0.915	0.953	0.949	0.951	0.943	0.947	0.948	0.949

Table 6: Direct metric-learning and prototype baselines under the same frozen-embedding linear-probe protocol. Values are test weighted F1.

Dataset	Comparison	Difference	95% CI
AmazonReviews	PGCL $-$ best direct SupCon	0.0904	[0.0443, 0.1377]
GoEmotions	PGCL $-$ best direct center loss	0.0256	[0.0011, 0.0473]
ToxicComment	PGCL $-$ best direct center loss	0.0009	[-0.0012, 0.0029]

Table 7: Paired bootstrap differences in weighted F1. Positive CIs support a positive PGCL difference; ToxicComment is reported as a small competitive margin.

Dataset	PGCL ref.	Frozen residual adapter	Encoder-LoRA	Full fine-tuning
AmazonReviews	0.727	0.614	0.573	0.589
GoEmotions	0.772	0.786	0.831	0.820
ToxicComment	0.953	0.925	0.959	0.965

Table 8: Separate supervised adaptation diagnostics. Values are test weighted F1. Encoder-LoRA and full fine-tuning update an MPNet encoder and are not part of the frozen-embedding direct-baseline table.

Embedding	SVM	RF	LR	XGB	Trans.	Avg. F1
Unsupervised SimCSE	0.851 / 0.568	0.823 / 0.539	0.843 / 0.572	0.840 / 0.567	0.730 / 0.407	0.531
Standard Supervised	0.921 / 0.697	0.916 / 0.718	0.918 / 0.687	0.922 / 0.687	0.921 / 0.687	0.695
PGCL (Ours)	0.934 / 0.709	0.933 / 0.749	0.928 / 0.719	0.930 / 0.724	0.927 / 0.726	0.725

Table 9: Downstream Classification Performance on Amazon Reviews: Contrastive Baselines Comparison (AUC / Overall F1-Score)

Task	Calls	Avg latency	P95 latency	LLM F1	PGCL F1
ToxicComment	520/520	1.182s	1.310s	0.940	0.947
GoEmotions	881/881	0.793s	0.859s	0.680	0.772
AmazonReviews	500/500	1.310s	1.576s	0.648	0.727

Table 10: Audited local qwen3.6:27b few-shot LLM protocol telemetry.

Embedding	W/B ratio ( $\downarrow$ )	Silhouette ( $\uparrow$ )	Class overlap ( $\downarrow$ )	GQI ( $\uparrow$ )
Raw Embeddings	8.760	0.018	0.563	< 0.000
Unsupervised SimCSE	1.010	0.010	0.491	-0.0005
Standard Supervised	0.458	0.158	0.286	0.0614
PGCL (Ours)	0.358	0.203	0.253	0.0975

Table 11: Quantitative Geometric Quality and GQI Comparison on Amazon Reviews

Panel of Figure 2 — Figure 2: t-SNE comparison of embedding spaces. Raw embeddings (a, c) show overlap among related labels. PGCL optimized embeddings (b, d) show more organized task-specific structure and a visible ordinal pattern for Amazon Reviews.

Configuration	Disappt.	Sad.	Disapprv.	Grat.	Apprv.	Average
Only Contrastive Loss	0.37	0.75	0.72	0.92	0.74	0.75
Only Offset Loss	0.40	0.75	0.72	0.94	0.77	0.77
Without Contrastive Loss	0.42	0.77	0.72	0.94	0.77	0.77
Without Offset Loss	0.44	0.71	0.73	0.93	0.76	0.77
Raw Embeddings	0.36	0.64	0.66	0.93	0.72	0.72
PGCL (Full Model)	0.49	0.77	0.72	0.94	0.76	0.78

Table 12: Ablation study on GoEmotions (F1 score Mean). Disappt.-Disappointment, Sad.-Sadness, Disapprv.-Disapproval, Grat.-Gratitude, Apprv.-Approval.

Dataset	Full	w/o offset	w/o contrast.	w/o orth.	Magnitude
AmazonReviews	0.569	0.541	0.643	0.571	0.577
GoEmotions	0.748	0.753	0.748	0.756	N/A
ToxicComment	0.944	0.936	0.944	0.946	N/A

Table 13: Same-provenance objective ablation across datasets. Values are test weighted F1.

Dataset	Full	$w_{o}{=}0.5$	$w_{o}{=}2.0$	$w_{r}{=}1.0$	$w_{r}{=}10.0$	Best interaction
AmazonReviews	0.584	0.525	0.664	0.578	0.577	Class+offset: 0.637
GoEmotions	0.745	0.757	0.758	0.756	0.744	Class+contrastive(+offset): 0.754
ToxicComment	0.945	0.941	0.947	0.949	0.945	Class+contrastive+offset: 0.946

Table 14: Objective sensitivity and interaction diagnostics. Values are test weighted F1.

Dataset	Full ref.	Attn. only	Sem./no-attn.	Fixed proto.	Fixed $\alpha$	w/o orth.
AmazonReviews	0.6217	0.5073	0.6225	0.6170	0.6253	0.6148
GoEmotions	0.7519	0.7718	0.7371	0.7544	0.7481	0.7507
ToxicComment	0.9494	0.9146	0.9483	0.9469	0.9472	0.9500

Table 15: Architecture-level diagnostics. Values are weighted F1 means across available seeds unless noted.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Principle	Emb. Type	SVM	RF	LR	XGBoost	Transformer
Disappt.	Raw Emb.	0.387 $\pm$ 0.090	0.237 $\pm$ 0.202	0.375 $\pm$ 0.054	0.359 $\pm$ 0.144	0.315 $\pm$ 0.073
Disappt.	Opt. Emb.	0.482 $\pm$ 0.102	0.410 $\pm$ 0.087	0.479 $\pm$ 0.106	0.439 $\pm$ 0.099	0.386 $\pm$ 0.117
Sad.	Raw Emb.	0.643 $\pm$ 0.059	0.728 $\pm$ 0.069	0.672 $\pm$ 0.081	0.711 $\pm$ 0.082	0.711 $\pm$ 0.087
Sad.	Opt. Emb.	0.687 $\pm$ 0.048	0.734 $\pm$ 0.031	0.721 $\pm$ 0.054	0.698 $\pm$ 0.031	0.714 $\pm$ 0.034
Disapprv.	Raw Emb.	0.663 $\pm$ 0.074	0.691 $\pm$ 0.050	0.652 $\pm$ 0.070	0.703 $\pm$ 0.032	0.677 $\pm$ 0.055
Disapprv.	Opt. Emb.	0.720 $\pm$ 0.074	0.740 $\pm$ 0.064	0.728 $\pm$ 0.063	0.724 $\pm$ 0.082	0.733 $\pm$ 0.059

Appendix

Table 16: Per-Principle F1 Performance on GoEmotions Five-Principle Set (Mean

\pm

Std. Dev.). Principles are abbreviated as Disappt., Sad., Disapprv., Grat., Apprv.

Ratings	Emb. Type	SVM	RF	LR	XGBoost	Transformer
1 - S	Raw Emb.	0.712 $\pm$ 0.219	0.772 $\pm$ 0.166	0.713 $\pm$ 0.208	0.744 $\pm$ 0.235	0.731 $\pm$ 0.209
1 - S	Opt. Emb.	0.869 $\pm$ 0.112	0.874 $\pm$ 0.085	0.875 $\pm$ 0.084	0.894 $\pm$ 0.093	0.888 $\pm$ 0.090
2 - S	Raw Emb.	0.277 $\pm$ 0.118	0.204 $\pm$ 0.213	0.297 $\pm$ 0.168	0.288 $\pm$ 0.221	0.432 $\pm$ 0.163
2 - S	Opt. Emb.	0.691 $\pm$ 0.187	0.708 $\pm$ 0.315	0.667 $\pm$ 0.176	0.760 $\pm$ 0.170	0.711 $\pm$ 0.141
3 - S	Raw Emb.	0.433 $\pm$ 0.158	0.556 $\pm$ 0.176	0.503 $\pm$ 0.081	0.520 $\pm$ 0.141	0.584 $\pm$ 0.085
3 - S	Opt. Emb.	0.669 $\pm$ 0.106	0.697 $\pm$ 0.073	0.662 $\pm$ 0.112	0.657 $\pm$ 0.076	0.696 $\pm$ 0.143

Appendix

Table 17: Classification F1 Performance per Rating on Amazon Reviews (Mean

\pm

Std. Dev.)

Metric	Emb. Type	SVM	RF	LR	XGBoost	Transformer
1-S MSE	Raw Emb.	0.483 $\pm$ 0.531	0.750 $\pm$ 1.207	0.567 $\pm$ 0.533	0.957 $\pm$ 1.145	0.350 $\pm$ 0.449
1-S MSE	Opt. Emb.	0.177 $\pm$ 0.260	0.360 $\pm$ 0.543	0.140 $\pm$ 0.254	0.420 $\pm$ 0.555	0.157 $\pm$ 0.250
2-S MSE	Raw Emb.	1.080 $\pm$ 0.867	1.415 $\pm$ 0.880	1.250 $\pm$ 0.972	1.465 $\pm$ 0.912	1.025 $\pm$ 1.072
2-S MSE	Opt. Emb.	0.290 $\pm$ 0.386	0.435 $\pm$ 0.547	0.405 $\pm$ 0.334	0.335 $\pm$ 0.338	0.265 $\pm$ 0.363
3-S MSE	Raw Emb.	0.726 $\pm$ 0.431	0.623 $\pm$ 0.235	0.804 $\pm$ 0.378	0.712 $\pm$ 0.269	0.539 $\pm$ 0.236
3-S MSE	Opt. Emb.	0.386 $\pm$ 0.260	0.442 $\pm$ 0.312	0.442 $\pm$ 0.367	0.509 $\pm$ 0.361	0.376 $\pm$ 0.261

Appendix

Table 18: Ordinal Regression MSE Performance per Rating on Amazon Reviews (Mean

\pm

Std. Dev.)

Metric	Emb. Type	SVM	RF	LR	XGBoost	Transformer
Precision	Raw Emb.	0.541 $\pm$ 0.038	0.627 $\pm$ 0.093	0.595 $\pm$ 0.058	0.630 $\pm$ 0.073	0.639 $\pm$ 0.067
Precision	Opt. Emb.	0.728 $\pm$ 0.077	0.726 $\pm$ 0.094	0.716 $\pm$ 0.084	0.718 $\pm$ 0.077	0.725 $\pm$ 0.062
Recall	Raw Emb.	0.522 $\pm$ 0.043	0.628 $\pm$ 0.083	0.583 $\pm$ 0.055	0.634 $\pm$ 0.060	0.628 $\pm$ 0.064
Recall	Opt. Emb.	0.721 $\pm$ 0.073	0.729 $\pm$ 0.080	0.715 $\pm$ 0.078	0.713 $\pm$ 0.069	0.723 $\pm$ 0.056
Avg. F1	Raw Emb.	0.521 $\pm$ 0.041	0.609 $\pm$ 0.082	0.582 $\pm$ 0.052	0.619 $\pm$ 0.059	0.622 $\pm$ 0.061
Avg. F1	Opt. Emb.	0.717 $\pm$ 0.074	0.721 $\pm$ 0.085	0.710 $\pm$ 0.081	0.708 $\pm$ 0.069	0.718 $\pm$ 0.058

Appendix

Table 19: Overall (Avg. Rating) Classification Performance on Amazon Reviews (Mean

\pm

Std. Dev.)

Ratings	Emb. Type	SVM	RF	LR	XGBoost	Transformer
1 - S	Raw Emb.	0.712 $\pm$ 0.219	0.772 $\pm$ 0.166	0.713 $\pm$ 0.208	0.744 $\pm$ 0.235	0.731 $\pm$ 0.209
1 - S	Opt. Emb.	0.869 $\pm$ 0.112	0.874 $\pm$ 0.085	0.875 $\pm$ 0.084	0.894 $\pm$ 0.093	0.888 $\pm$ 0.090
2 - S	Raw Emb.	0.277 $\pm$ 0.118	0.204 $\pm$ 0.213	0.297 $\pm$ 0.168	0.288 $\pm$ 0.221	0.432 $\pm$ 0.163
2 - S	Opt. Emb.	0.691 $\pm$ 0.187	0.708 $\pm$ 0.315	0.667 $\pm$ 0.176	0.760 $\pm$ 0.170	0.711 $\pm$ 0.141
3 - S	Raw Emb.	0.433 $\pm$ 0.158	0.556 $\pm$ 0.176	0.503 $\pm$ 0.081	0.520 $\pm$ 0.141	0.584 $\pm$ 0.085
3 - S	Opt. Emb.	0.669 $\pm$ 0.106	0.697 $\pm$ 0.073	0.662 $\pm$ 0.112	0.657 $\pm$ 0.076	0.696 $\pm$ 0.143

Appendix

Table 20: Classification F1 Performance per Rating on Amazon Reviews (Mean

\pm

Std. Dev.)

Task	Allowed output	Demonstrations	Evaluation subset
ToxicComment	Integer 0/1 or toxic/non-toxic, parsed to binary label	3 toxic and 10 non-toxic examples, shuffled after class-stratified sampling	520 examples: 20 toxic, 500 non-toxic
GoEmotions	One index from 0-4 for Disappointment, Sadness, Disapproval, Gratitude, Approval	5 examples per emotion class from single-label training examples	881 eligible test examples
AmazonReviews	One integer rating from 1-5	5 examples per rating class from the training split	500 examples: 50/50/100/150/150 by ratings 1-5

Appendix

Table 21: Few-shot LLM prompt protocol. Demonstrations are selected from the training split only.

Task	Calls	Avg lat.	P95 lat.	Prompt tok.	LLM F1	PGCL F1
ToxicComment	520/520	1.182s	1.310s	1618.4	0.940	0.947
GoEmotions	881/881	0.793s	0.859s	842.6	0.680	0.772
AmazonReviews	500/500	1.310s	1.576s	7732.0	0.648	0.727

Appendix

Table 22: Few-shot local LLM protocol telemetry. F1 denotes weighted F1 under the fixed parsing protocol.

Dataset	Method	Within/Between	Silhouette	Class Overlap	GQI
AmazonReviews	Raw	12.4482	-0.0009	0.5823	0.0045
AmazonReviews	PGCL	0.4287	0.1894	0.3366	0.0718
AmazonReviews	Best direct SupCon	0.5666	0.0489	0.4897	0.0108
GoEmotions	Raw	14.6009	0.0221	0.4692	-0.1598
GoEmotions	PGCL	0.5202	0.2615	0.3138	0.0861
GoEmotions	Best direct center	0.7185	0.2266	0.3008	0.0446

Appendix

Table 23: Detailed all-dataset GQI point estimates.

Text Embedding Models

Text embedding models map words, sentences, or documents to vectors for semantic search, retrieval, clustering, and downstream prediction.

See all 126 papers

LLM2Vec-Gen: Generative Embeddings from Large Language Models
Mar 11, 2026Parishad BehnamGhader, Vaibhav Adlakha, Fabian David Schmidt +3Text Embedding Models Self-Supervised Pre-Training
McGill University · Mila–Quebec AI Institute · ServiceNow Research +3
Semantically Rich, Metrically Affect-Blind: A Geometric Blind Spot in Modern Text Embedders
Oct 23, 2026Mahdi Naser MoghadasiSentence Embeddings Representation Geometry in Language Models
BrightMind AI Seattle, WA, USA
Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs
Sep 26, 2026Zifeng Cheng, Lingyun Qian, Zhiwei Jiang +5Text Embeddings Text Embedding Models
State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore

Mar 11, 2026J/K move · Enter open · S save

LLM2Vec-Gen: Generative Embeddings from Large Language Models

Parishad BehnamGhader, Vaibhav Adlakha, Fabian David Schmidt, Nicolas Chapados+2

McGill University · Mila–Quebec AI Institute · ServiceNow Research+3

Text Embedding Models Self-Supervised Pre-Training

Open paper AskPDF arXiv

Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-Gen, a self-supervised alternative that instead produces embeddings directly in the LLM's output space by learning to represent the model's potential response. Specifically, trainable special tokens are appended to the input and optimized to compress the LLM's own response into a fixed-length embedding, guided by an unsupervised embedding teacher and a reconstruction objective. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% over the unsupervised embedding teacher. Since the embeddings preserve the LLM's response-space semantics, they inherit capabilities such as safety alignment (up to 22.6% reduction in harmful content retrieval) and reasoning (up to 35.6% improvement on reasoning-intensive retrieval). Finally, the learned embeddings are also interpretable: they can be decoded back into text to reveal their semantic content. Our code, data, and pretrained models are publicly available at https://github.com/McGill-NLP/llm2vec-gen.

Abstract

Text Embedding Models

Explore similar work

LLM2Vec-Gen: Generative Embeddings from Large Language Models

Semantically Rich, Metrically Affect-Blind: A Geometric Blind Spot in Modern Text Embedders

Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs