PULSE: Identifying Demonstration-Utility Features with Sparse Autoencoders

Chenduo Hao; Chuanbao Gao; Pinjun Zeng; Jingze Zhu; Chonghan Liu; Zidong Liu; Xu Yang

cs.AISep 26, 2026

PULSE: Identifying Demonstration-Utility Features with Sparse Autoencoders

Authors: Chenduo Hao, Chuanbao Gao, Pinjun Zeng, Jingze Zhu, Chonghan Liu, Zidong Liu, Xu Yang

Organizations: Southeast University · University of California, Los Angeles · School of Architecture, The University of Texas at Austin

Abstract

In-context learning is highly sensitive to demonstration choice, yet most methods select demonstrations using external query-demonstration similarity. Such criteria can miss model-specific signals: Similar demonstrations may activate different internal features and downstream behaviors. We introduce PULSE (Paired Utility Localization over Sparse Encodings), an SAE-based framework for identifying model-internal features associated with demonstration utility and using them for demonstration selection. Using a small labeled discovery set, PULSE samples candidate demonstration sets, measures their zero-shot-relative utility under the target model, and scores SAE features by how their activation differences align with utility differences. The top positive and negative coordinates form a sparse utility-localization vector. We use this vector in two complementary ways: as a signed score for controlled complete-set ranking, and as PULSE-Retriever, which converts its magnitude into a feature-relevance mask for scalable pool-scale retrieval. Across classification, generation, and reasoning benchmarks, PULSE-Retriever improves over the strongest baseline by 2-3 accuracy points, 0.6-0.9 BLEU-4, and 3.2 exact-match points, respectively, while controlled ranking validates the identified features encode a predictive set-level utility signal. Feature inspection and cross-dataset experiments suggest that the identified features capture task-relevant, dataset-conditioned patterns, yet retain utility signals that partially transfer across datasets. Our code is available at https://github.com/aohenuo/PULSE.

Figures & tables

Figure 1: Overview of the motivation and workflow of PULSE. Traditional methods still study these signals mainly at the level of query–demonstration matching. In contrast, PULSE identifies utility-related internal features and uses them for demonstration selection.

Figure 2: Overview of PULSE. Here, $k$ -shot ICD denotes a $k$ -example in-context demonstration set. (a) PULSE estimates a utility-localization vector by associating utility differences $\Delta\mathcal{U}$ with SAE activation differences $\Delta A$ across paired demonstration sets. (b) PULSE validates the identified features by ranking pre-sampled complete demonstration sets in the PULSE-Ranking experiment. (c) PULSE selects demonstrations using PULSE-Retriever, which uses the utility-localization vector as a feature-relevance mask for scalable pool-scale retrieval.

Backbone	Method	AGNews				REST14				LAP14				EMOC				Avg.
		1	2	4	8	1	2	4	8	1	2	4	8	1	2	4	8	1	2	4	8
Gemma2-2B	Random	40.62	66.99	75.00	77.93	60.94	65.82	73.83	73.05	54.21	65.44	76.03	73.87	44.92	53.52	54.69	60.16	50.17	62.94	69.89	71.25
	Lex-Sim	43.75	68.75	77.34	82.03	66.41	72.46	78.71	75.39	64.79	70.84	78.40	77.75	52.93	54.88	56.64	60.94	56.97	66.73	72.77	74.03
	SBERT	55.47	77.15	78.12	82.23	60.74	69.14	76.17	73.83	66.52	71.49	75.81	77.11	51.56	55.86	56.05	61.33	58.57	68.41	71.54	73.62
	CEIL	55.47	71.48	78.12	78.12	60.74	69.73	76.76	75.59	66.52	70.41	79.70	78.19	51.56	56.64	56.25	60.94	58.57	67.06	72.71	73.21
	KNN-SAE	54.69	77.54	76.95	81.84	63.67	69.53	78.12	76.17	65.87	72.35	79.05	76.24	48.63	52.93	58.01	60.55	58.22	68.09	73.03	73.70

Table 1: Candidate-set scoring accuracy (%) on Gemma2-2B and Llama3.1-8B. Downstream accuracy after using each score to select the top-1 candidate demonstration set. The Avg. columns report the average across four datasets under each shot setting.

Backbone	Method	AGNews				REST14				LAP14				EMOC				Avg.
		1	2	4	8	1	2	4	8	1	2	4	8	1	2	4	8	1	2	4	8
Gemma2-2B	Random	41.80	66.99	72.66	78.52	57.62	64.65	70.90	72.46	56.59	60.91	68.25	71.71	41.60	54.69	56.25	60.94	49.40	61.81	67.02	70.91
	Lex-Sim	68.16	78.32	79.88	86.13	64.45	66.80	73.63	76.76	70.63	75.16	74.30	77.54	62.70	62.50	65.82	71.68	66.48	70.69	73.41	78.03
	SBERT	76.76	78.52	79.49	84.18	65.62	66.99	71.68	74.22	68.47	73.87	75.81	78.62	63.48	64.65	66.80	72.46	68.58	71.01	73.45	77.37
	CEIL	76.76	78.91	81.25	87.11	65.62	67.97	75.39	76.95	68.47	76.03	76.03	78.83	63.48	65.62	66.02	71.09	68.58	72.13	74.67	78.50
	KNN-SAE	79.69	78.71	81.05	87.50	66.99	65.43	75.59	75.59	70.63	76.46	75.16	77.75	50.20	60.74	66.80	72.46	66.88	70.33	74.65	78.32

Table 2: Pool-scale retrieval accuracy (%) on Gemma2-2B and Llama3.1-8B. Downstream accuracy after retrieving individual examples from the support set and composing the final

k

-shot context. The Avg. columns report the average across four datasets under each shot setting.

Method	CommonGen / Gemma2-2B / BLEU-4					CommonGen / Llama3.1-8B / BLEU-4					GSM8K / Llama3.1-8B/ EM
Method	1	2	4	8	Avg.	1	2	4	8	Avg.	1	2	4	8	Avg.
Random	8.42	8.35	8.10	8.86	8.43	9.02	9.22	9.64	9.44	9.33	42.61	47.92	49.20	50.72	47.61
Lex-Sim	8.22	7.74	9.20	10.56	8.93	9.68	8.91	9.19	9.16	9.24	38.29	48.67	50.04	52.31	47.33
SBERT	8.41	8.36	9.08	9.82	8.92	9.41	9.84	10.21	9.91	9.84	39.73	49.20	43.82	52.92	46.42
KNN-SAE	9.00	8.34	8.93	7.55	8.46	8.91	9.64	9.96	9.71	9.56	39.12	46.10	46.93	53.53	46.42
CEIL	8.41	8.62	10.07	9.24	9.09	9.41	9.68	10.01	9.47	9.64	39.73	50.42	50.80	52.08	48.26

Table 3: Generation and reasoning results. Scores are BLEU-4 for CommonGen and exact match accuracy for GSM8K. The Avg. columns report the average over 1-, 2-, 4-, and 8-shot settings.

Dataset	SAE	PCA	Raw
AGNews	85.94	81.25	80.47
REST14	77.73	75.78	76.76
LAP14	77.32	73.22	74.95
EMOC	68.36	65.23	61.33
Cls. Avg.	77.34	73.87	73.38
CommonGen	11.09	10.23	10.34

Table 4: SAE-basis ablation on Gemma2-2B in the 4-shot pool-scale retrieval setting.

Dataset	PULSE-Mask	Anti-Retrieval	Sparse-Random	Bottom-Feats	PULSE-NoTopK
AGNews	83.01	51.95	82.81	82.03	81.64
REST14	76.17	63.67	75.39	75.00	78.12
LAP14	74.51	48.38	72.14	70.41	74.29
EMOC	60.16	30.08	58.20	57.81	59.38
Avg.	73.46	48.52	72.14	71.31	73.36

Table 5: Controls for the PULSE feature mask on Gemma2-2B in the 4-shot pool-scale retrieval setting. PULSE-Mask uses only the magnitude of the PULSE-identified feature vector, without the standard SAE cosine component used in PULSE-Retriever. Anti-Retrieval uses the same mask but selects the lowest-scoring pool items. Sparse-Random uses randomly selected sparse features, Bottom-Feats uses features with the smallest absolute identification scores, and PULSE-NoTopK removes the top-feature restriction. Accuracy is reported in percentage points.

Dataset	Feature	Rank	Top-activating snippets	Interpretation / Role
AGNews	#8186	P1	“Open Source: Balancing Innovation and Risk” “Citigroup … institutional investors” “MFS Institutional Emerging Equities Fund”	Institutional and technology-industry framing; task-conditioned topic anchor.
REST14	#7875	P1	“Aspect: desserts; got favorable reviews” “Aspect: service; good things to say” “Aspect: staff; good things to say”	Aspect-specific evaluative expression; aspect–opinion binding.
LAP14	#15777	N1	“Aspect: works; Toshiba works great” “Aspect: power; power, memory and speed” “Aspect: memory; power, memory and speed”	Generic capability or off-aspect framing; lower-utility pattern.
EMOC	#7063	P1	“make others cry / i know whats wrong now” “give love to others … happy always” “especially yours / mine is so so / amazing”	Self/other dialogue with affective reasoning or topical pivot; task-conditioned emotion cue.

Table 6: Representative high-

|w|

PULSE features. P1/N1 denotes the top positive or negative feature after activation filtering; the sign reflects association with paired utility differences.

	Single-source transfer				LOO transfer vs. baseline			Target PULSE
Target	AGNews	REST14	LAP14	EMOC	LOO ∪	Base max	$\Delta_{\mathrm{Base}}$	Self	$\Delta_{\mathrm{Self}}$
AGNews	–	84.77	83.59	85.55	83.59	81.25	$+2.34$	85.94	$-2.35$
REST14	76.56	–	79.69	78.52	77.73	75.59	$+2.14$	77.73	$+$ 0.00
LAP14	78.19	74.73	–	76.46	77.54	76.03	$+1.51$	77.32	$+0.22$
EMOC	60.94	66.80	66.80	–	67.97	66.80	$+1.17$	68.36	$-0.39$
Avg.	71.90	75.43	76.69	80.18	76.71	74.92	$+1.79$	77.34	$-0.63$

Table 7: Cross-dataset transfer of PULSE utility-localization vectors in the 4-shot retrieval setting on Gemma2-2B. Scores are accuracy (%). Single-source : a utility-localization vector learned on the source dataset is applied directly to the target dataset. LOO ∪ : for each target dataset, the target excludes the dataset used for feature identification, and the PULSE-identified features from the three non-target datasets are combined by union. Base max : best non-PULSE baseline per target.

\Delta_{\mathrm{Base}}

= LOO ∪

-

Base max ;

\Delta_{\mathrm{Self}}

= LOO ∪

-

Self.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	SAE release/checkpoint	Layer(s)	$d_{\mathrm{SAE}}$	Inference and decoding settings
Gemma2-2B	gemma-scope-2b-pt-res-canonical layer_12/width_16k/average_l0_82	12	16,384	Classification: teacher-forced label likelihood, no sampling. CommonGen: greedy generation, do_sample=False , max_new_tokens =32; temperature/top- $p$ disabled.
Llama3.1-8B	llama_scope_lxr_32x fnlp/Llama3_1-8B-Base-LXR-32x He et al. (2024)	12 (classification) 16 (CommonGen) 24 (GSM8K)	131,072	Classification: teacher-forced label likelihood, no sampling. CommonGen: greedy generation, do_sample=False , max_new_tokens =32; temperature/top- $p$ disabled. GSM8K: greedy generation, do_sample=False , max_new_tokens =220; temperature/top- $p$ disabled.

Appendix

Table 8: Backbone, SAE, and decoding details. All SAE features are taken from residual-stream SAEs at the listed layer. The Gemma-Scope checkpoint is the layer-12 width-16k average-

L_{0}

-82 dictionary; the Llama-Scope LXR-32x checkpoints use

d_{\mathrm{model}}=4096

and

d_{\mathrm{SAE}}=131{,}072

. For classification, downstream decisions are made by teacher-forced label likelihood rather than free-form generation.

Dataset	Task	Train	Test	Avg. len.	Pool	Label distribution
AGNews	Topic cls.	120,000	7,600	37.8	2,000	train: Business=30,000, Sci/Tech=30,000 train: Sports=30,000, World=30,000 test: Business=1,900, Sci/Tech=1,900 test: Sports=1,900, World=1,900
REST14	Aspect sentiment	2,881	721	21.9	2,000	train: Negative=633, Neutral=513, Positive=1,735 test: Negative=172, Neutral=120, Positive=429
LAP14	Aspect sentiment	1,850	463	23.8	1,850	train: Negative=686, Neutral=384, Positive=780 test: Negative=180, Neutral=76, Positive=207
EMOC	Emotion cls.	38,424	5,509	17.6	2,000	train: angry=5,954, happy=4,669 train: others=21,963, sad=5,838 test: others=4,677, happy=298 test: angry=284, sad=250
CommonGen	Constrained gen.	67,389	4,018	3.3	2,000	–
GSM8K	Math reasoning	7,473	1,319	45.3	2,000	–

Appendix

Table 9: Dataset and pool statistics. Train/test sizes and support-pool sizes are read from the experiment loaders. Average input length is computed on the input side using whitespace tokenization (concept-list length for CommonGen). Generation and reasoning experiments use smaller pools because each selected context requires autoregressive evaluation.

Symbol	Meaning	Default value
$K^{+}$	Positive-weight SAE features retained	Gemma2-2B: 512 Llama3.1-8B: 2,048
$K^{-}$	Negative-weight SAE features retained	Gemma2-2B: 512 Llama3.1-8B: 2,048
$\beta$	Sparse–dense blend weight in the retrieval score	0.3
$L$	Retrieval shortlist size	50
$\lambda_{r}$	SAE-space redundancy penalty	0.3
$\|\mathcal{D}_{\mathrm{disc}}\|$	Discovery queries	64

Appendix

Table 10: Hyperparameter summary. Defaults are the values used in the main experiments unless a task-specific exception is shown.

Method	Labeled discovery data	Validation tuning	Model-internal SAE features	Greedy redundancy
Random	No	No	No	No
Lex-Sim	No	Yes	No	Yes
SBERT	No	Yes	No	Yes
KNN-SAE	No	Yes	Yes	Yes
CEIL	No	Yes	No	Built-in diversity objective
PULSE-Retriever	Yes, 64 queries	Yes	Yes	Yes

Appendix

Table 11: Supervision and tuning resources used by each retrieval method. PULSE uses a small labeled discovery split to estimate a utility-localization vector; the non-PULSE retrievers do not learn task-specific utility weights from labeled discovery queries. For pool-scale retrieval, pointwise methods use the same shortlist and greedy redundancy procedure unless otherwise stated.

Backbone	Variant	AGNews	REST14	LAP14	EMOC	CommonGen	GSM8K
Gemma2-2B	$\|\mathbf{w}\|$ mask	85.94	77.73	77.32	68.36	11.09	–
	signed score	84.38	76.56	77.34	65.23	10.25	–
	$\Delta$	+1.56	+1.17	$-0.02$	+3.13	+0.84	–
Llama3.1-8B	$\|\mathbf{w}\|$ mask	88.09	81.84	82.07	71.09	10.90	54.66
	signed score	85.16	82.03	82.03	72.66	10.64	51.53
	$\Delta$	+2.93	$-0.19$	$+0.04$	$-1.57$	$+0.26$	+3.13

Appendix

Table 12: Empirical comparison of the magnitude mask and a sign-sensitive retrieval score in PULSE-Retriever. Results are from the 4-shot pool-scale retrieval setting. The metric is accuracy for classification, BLEU-4 for CommonGen, and exact match for GSM8K.

\Delta=|\mathbf{w}|

mask

-

signed score. Bold marks

|\Delta|\geq 0.8

. “–” indicates GSM8K is not evaluated on Gemma2-2B.

Sign	Feature	Weight	Max-activating snippet	Interpretation
$+$	8186	7.48	Open Source: Balancing Innovation and Risk	tech-industry commentary
$+$	11980	6.16	Climate fears on sharp CO 2 rise	scientific finding / climate framing
$+$	14321	5.75	Women Voters Enter San Diego Mayoral Fray	civic / institutional reference
$+$	4297	4.35	British-Irish national kidnapped in Afghanistan	named geopolitical event
$+$	1398	3.70	Australia struggling against New Zealand	sports-news event
$-$	12684	-3.65	Bond Prices Edge Higher at Close of Day	market-summary phrasing

Appendix

Table 13: Top PULSE-identified features on AGNews.

Sign	Feature	Weight	Max-activating snippet	Interpretation
$+$	7063	11.19	no / make others cry / i know whats wrong now	dialogue with topical pivot
$+$	15322	11.18	shut up / my exam is on tuesday	direct-address with factual follow-up
$+$	9565	10.81	you are very bad / sorry / because…	confrontation with stated reason
$+$	5466	10.68	what else do you know about me?	introspective Q–A small talk
$+$	15273	9.54	what happened / net problems on my phone	factual problem report and topic shift
$-$	7657	-8.59	me happy / happy happy warm warm	repetitive affect tokens

Appendix

Table 14: Top PULSE-identified features on EMOC.

Sign	Feature	Weight	Max-activating snippet	Interpretation
$+$	11485	3.13	Aspect: screen; big screen allows you to enjoy movies	aspect to enjoyment
$+$	5828	1.68	Aspect: Pages; design tools make it easier	software aspect to ease of use
$+$	13435	1.61	Aspect: PRICE; LOVE THIS LAPTOP WONDERFUL PRICE	aspect to strong positive judgment
$+$	16061	1.56	Aspect: battery; why not give you a better battery?	aspect to complaint
$+$	9749	1.46	Aspect: price; iWork is not worth the price	aspect to negative judgment
$-$	15777	-3.82	Aspect: works; my new Toshiba works great on both	generic verb / off-aspect framing

Appendix

Table 15: Top PULSE-identified features on LAP14.

Sign	Feature	Weight	Max-activating snippet	Interpretation
$+$	7875	0.61	Aspect: desserts; got favorable reviews	committed positive judgment
$+$	5435	0.56	Aspect: owner; always has a smile	warm person-description
$+$	1512	0.53	Aspect: food; good food and ok service	committed mixed judgment
$+$	3486	0.44	Aspect: Decor; leaves something to be desired	committed negative judgment
$+$	8374	0.43	Aspect: comfort food; hot and large portions	simple positive description
$-$	9781	-0.57	Aspect: ambience; I care more about the food and ambience	meta-commentary

Appendix

Table 16: Top PULSE-identified features on REST14.

Dataset pair	Intersection	Union	Jaccard
AGNews–EMOC	94	1954	0.048
AGNews–REST14	90	1958	0.046
AGNews–LAP14	87	1961	0.044
EMOC–REST14	140	1908	0.073
EMOC–LAP14	95	1953	0.049
REST14–LAP14	207	1841	0.112

Appendix

Table 17: Pairwise overlap of active PULSE features across datasets. The two aspect-based sentiment datasets, REST14 and LAP14, have the largest overlap, but the overall Jaccard values remain small.

# active datasets	# features	Consistent $+$	Consistent $-$	Mixed	% mixed
1	2894	1401	1493	–	–
2	497	145	133	219	44.1
3	64	19	6	39	60.9
4	4	1	0	3	75.0

Appendix

Table 18: Sign consistency among active PULSE features across datasets. A mixed-sign feature is utility-positive on at least one dataset and utility-negative on at least one other dataset.

		Classification (Acc.)						Generation & Reasoning
Backbone	Method	AGNews	REST14	LAP14	EMOC	Cls. Avg.	Query budget	CommonGen	GSM8K
Gemma2-2B	EPR-style learned retriever	87.50	75.00	80.35	66.21	77.27	128 queries	9.78	–
Gemma2-2B	PULSE-Retriever	85.94	77.73	77.32	68.36	77.34	64 queries	11.09	–
Llama3.1-8B	EPR-style learned retriever	89.26	77.73	83.00	70.89	80.22	128 queries	10.19	44.53
Llama3.1-8B	PULSE-Retriever	88.09	81.84	82.07	71.09	80.77	64 queries	10.90	54.66

Appendix

Table 19: Comparison with a supervised learned retriever in the 4-shot pool-scale retrieval setting. Classification metrics are accuracy (%); CommonGen is BLEU-4; GSM8K is exact match (%). Cls. Avg. is the mean over four classification datasets. The learned retriever uses 128 training queries expanded into

2{,}048

singleton feedback labels and gradient-based retriever training; PULSE-Retriever uses 64 discovery queries with closed-form feature scoring and no retriever training.

Dataset	K=256	K=512	K=1024	K=2048	K=4096
AGNews	85.16	85.94	86.13	85.94	84.38
REST14	78.12	77.73	79.30	80.08	79.69
LAP14	77.97	77.32	76.67	78.19	78.62
EMOC	65.23	68.36	68.16	66.41	66.80
Avg.	76.62	77.34	77.57	77.65	77.37

Appendix

Table 20: Retriever accuracy as a function of selected feature count

K

in the 4-shot pool-scale retrieval setting. Bold indicates the best result in each row.

Dataset	16	32	64	128
AGNews	82.42	83.98	85.94	86.72
REST14	77.73	78.52	77.73	77.34
LAP14	79.48	78.19	77.32	78.40
EMOC	62.50	64.45	68.36	66.80
Avg.	75.53	76.28	77.34	77.32

Appendix

Table 21: Retriever accuracy as a function of discovery query count

|\mathcal{D}_{\mathrm{disc}}|

in the 4-shot pool-scale retrieval setting. Bold indicates the best result in each row.

Dataset	$\beta=0.0$	$\beta=0.1$	$\beta=0.2$	$\beta=0.3$	$\beta=0.5$	$\beta=0.7$	$\beta=1.0$
AGNews	83.01	83.20	85.94	85.94	86.72	86.52	81.05
LAP14	74.51	75.59	74.30	77.32	76.03	75.16	75.16
REST14	76.17	75.59	76.95	77.73	77.93	76.76	75.59
EMOC	60.16	63.87	62.30	68.36	67.77	66.41	66.80
Avg.	73.46	74.56	74.87	77.34	77.11	76.21	74.65

Appendix

Table 22: Retriever accuracy as a function of the sparse–dense blend weight

\beta

in the 4-shot pool-scale retrieval setting.

\beta=0

denotes pure PULSE-weighted cosine similarity, while

\beta=1

denotes pure SAE-space cosine similarity. Bold indicates the best result in each row.

Shot	Best Layer	Best Avg. (%)	Layer-12 Avg. (%)	Gap (pp)	Layer Range (pp)
1-shot	12	70.64	70.64	0.00	4.86
2-shot	21	75.02	74.31	0.71	2.57
4-shot	5	78.48	77.34	1.14	3.22
8-shot	24	80.10	79.61	0.49	2.40

Appendix

Table 23: Summary of SAE layer sensitivity for PULSE-Retriever on Gemma2-2B in the pool-scale retrieval setting. Best Avg. is the highest average accuracy across layers. Layer-12 Avg. denotes the default layer used in the main experiments. Gap is the difference between Best Avg. and Layer-12 Avg.; Layer Range is the difference between the best and worst layer averages. All numeric entries except layer indices are reported in percentages or percentage points.

Layer	0	1	2	3	4	5	6	7	8	9	10	11	12 †
AGNews	73.24	79.49	77.54	76.37	74.80	73.63	75.00	77.54	73.63	79.49	78.91	72.66	81.25
LAP14	67.39	65.44	69.55	67.60	67.39	70.41	68.68	68.25	66.95	68.68	71.92	69.33	72.79
REST14	64.26	60.94	65.43	65.82	67.58	65.82	66.80	65.62	66.02	65.62	64.65	66.60	64.65
EMOC	62.11	62.50	63.28	62.70	65.23	64.06	60.74	63.28	65.43	58.40	61.52	62.30	63.87
Avg.	66.75	67.09	68.95	68.12	68.75	68.48	67.81	68.67	68.01	68.05	69.25	67.72	70.64
Layer	13	14	15	16	17	18	19	20	21	22	23	24	25

Appendix

Table 24: Full SAE layer sweep for PULSE-Retriever on Gemma2-2B in the 1-shot pool-scale retrieval setting. Each entry reports downstream accuracy (%). Avg. denotes the average across AGNews, LAP14, REST14, and EMOC. † denotes the default layer used in the main experiments. Bold indicates the best value in each row.

Layer	0	1	2	3	4	5	6	7	8	9	10	11	12 †
AGNews	80.86	76.95	79.49	79.30	82.23	80.66	80.86	83.20	83.98	81.64	81.84	83.01	82.62
LAP14	75.59	74.30	74.95	74.51	74.51	75.16	72.57	73.65	73.43	73.43	76.46	75.38	77.11
REST14	73.05	68.95	69.53	72.07	73.44	73.44	74.22	70.70	73.24	73.83	71.29	75.00	71.68
EMOC	69.14	72.07	69.34	70.51	65.82	68.36	68.95	65.82	68.36	67.38	67.77	66.02	65.82
Avg.	74.66	73.07	73.33	74.10	74.00	74.41	74.15	73.34	74.76	74.07	74.34	74.85	74.31
Layer	13	14	15	16	17	18	19	20	21	22	23	24	25

Appendix

Table 25: Full SAE layer sweep for PULSE-Retriever on Gemma2-2B in the 2-shot pool-scale retrieval setting. Each entry reports downstream accuracy (%). Avg. denotes the average across AGNews, LAP14, REST14, and EMOC. † denotes the default layer used in the main experiments. Bold indicates the best value in each row.

Layer	0	1	2	3	4	5	6	7	8	9	10	11	12 †
AGNews	84.77	84.18	83.20	85.55	83.40	86.52	86.33	85.16	86.33	88.67	84.77	85.55	85.94
LAP14	75.59	77.97	77.32	75.81	77.54	77.97	79.70	77.75	77.75	77.54	77.54	77.32	77.32
REST14	75.59	73.63	75.78	75.39	73.44	77.93	74.80	75.39	75.98	75.39	74.41	75.00	77.73
EMOC	70.51	71.29	69.34	71.09	71.48	71.48	67.58	69.53	72.07	68.95	69.92	70.12	68.36
Avg.	76.61	76.77	76.41	76.96	76.46	78.48	77.10	76.96	78.03	77.64	76.66	77.00	77.34
Layer	13	14	15	16	17	18	19	20	21	22	23	24	25

Appendix

Table 26: Full SAE layer sweep for PULSE-Retriever on Gemma2-2B in the 4-shot pool-scale retrieval setting. Each entry reports downstream accuracy (%). Avg. denotes the average across AGNews, LAP14, REST14, and EMOC. † denotes the default layer used in the main experiments. Bold indicates the best value in each row.

Layer	0	1	2	3	4	5	6	7	8	9	10	11	12 †
AGNews	86.52	85.74	85.55	83.59	90.04	86.13	85.55	88.28	88.67	88.48	86.91	87.50	88.09
LAP14	78.40	77.97	78.62	79.91	79.70	79.27	79.48	80.35	79.27	80.78	79.48	79.48	80.35
REST14	78.32	78.91	76.95	77.34	76.37	77.15	77.73	77.73	78.32	77.73	76.17	79.49	77.73
EMOC	73.83	73.44	74.80	73.83	73.05	72.85	75.00	73.44	72.85	70.31	71.68	70.31	72.27
Avg.	79.27	79.01	78.98	78.67	79.79	78.85	79.44	79.95	79.78	79.33	78.56	79.20	79.61
Layer	13	14	15	16	17	18	19	20	21	22	23	24	25

Appendix

Table 27: Full SAE layer sweep for PULSE-Retriever on Gemma2-2B in the 8-shot pool-scale retrieval setting. Each entry reports downstream accuracy (%). Avg. denotes the average across AGNews, LAP14, REST14, and EMOC. † denotes the default layer used in the main experiments. Bold indicates the best value in each row.

Backbone	Dataset (Metric)	Mean $\pm$ Std	Per-seed values
Gemma2-2B	AGNews (Acc)	$87.76\pm 1.62$	89.06, 88.28, 85.94
	REST14 (Acc)	$76.43\pm 2.60$	77.73, 73.44, 78.12
	LAP14 (Acc)	$79.41\pm 2.43$	82.07, 77.32, 78.83
	EMOC (Acc)	$69.46\pm 1.07$	68.36, 70.51, 69.53
	CommonGen (BLEU-4)	$10.47\pm 0.62$	11.09, 10.47, 9.85
Llama3.1-8B	AGNews (Acc)	$87.44\pm 0.69$	88.09, 86.72, 87.50

Appendix

Table 28: Seed stability of PULSE-Retriever in the 4-shot pool-scale retrieval setting. We report accuracy (%) for classification, BLEU-4 for CommonGen, and exact match (%) for GSM8K. Each row reports the mean

\pm

standard deviation over three seeds, along with the individual seed values. GSM8K is not evaluated on Gemma2-2B.

In-Context Example Selection

In-context example selection chooses demonstrations that help language and multimodal models perform a query-specific task without changing model weights.

See all 29 papers

Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection
May 18, 2026Haochun Wang, Chaofen Yang, Jiatong Liu +5LLM Inference Efficiency In-Context Example Selection
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China.
Data Efficient Sample Selection for In-Context Learning
Sep 6, 2026V Venktesh, Cem Levi, Avishek AnandIn-Context Example Selection In-Context Learning
Stockholm University · SKIM Group B.V. · TU Delft
UCS: Estimating Unseen Coverage for Improved In-Context Learning
Apr 13, 2026Jiayi Xin, Xiang Li, Evan Qiang +4In-Context Example Selection In-Context Learning
University of Pennsylvania

May 18, 2026J/K move · Enter open · S save

Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection

Haochun Wang, Chaofen Yang, Jiatong Liu, Jingbo Wang+4

Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China.

LLM Inference Efficiency In-Context Example Selection Few-Shot Learning In-Context Learning

Open paper AskPDF arXiv

In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because the space of possible demonstration contexts and combinations is enormous. We argue that demonstration selection is easier to judge than to find: predicting whether a specific query--context pair

(q,D)

will succeed is cheaper and more general than searching for an optimal

D^\star

. Based on this insight, we propose DiSP, a sample-and-judge framework that stratifies queries by difficulty. DiSP runs random demonstration trials to estimate success rate of each training query, trains a lightweight router to predict difficulty from the query, and trains level-specific judges for sampled demonstrations. At inference, DiSP performs stop-on-acceptance judging under an explicit budget, emitting diagnostic risk tags when no suitable context is found. Across five classification datasets with Llama~~3--8B and Qwen~~2.5--7B, DiSP achieves the best average accuracy, improving over strong learned selection baselines by up to 3.4%, while achieving up to

23\times

end-to-end wall-clock speedup.

Abstract

In-Context Example Selection

Explore similar work

Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection

Data Efficient Sample Selection for In-Context Learning

UCS: Estimating Unseen Coverage for Improved In-Context Learning