In-context learning is highly sensitive to demonstration choice, yet most methods select demonstrations using external query-demonstration similarity. Such criteria can miss model-specific signals: Similar demonstrations may activate different internal features and downstream behaviors. We introduce PULSE (Paired Utility Localization over Sparse Encodings), an SAE-based framework for identifying model-internal features associated with demonstration utility and using them for demonstration selection. Using a small labeled discovery set, PULSE samples candidate demonstration sets, measures their zero-shot-relative utility under the target model, and scores SAE features by how their activation differences align with utility differences. The top positive and negative coordinates form a sparse utility-localization vector. We use this vector in two complementary ways: as a signed score for controlled complete-set ranking, and as PULSE-Retriever, which converts its magnitude into a feature-relevance mask for scalable pool-scale retrieval. Across classification, generation, and reasoning benchmarks, PULSE-Retriever improves over the strongest baseline by 2-3 accuracy points, 0.6-0.9 BLEU-4, and 3.2 exact-match points, respectively, while controlled ranking validates the identified features encode a predictive set-level utility signal. Feature inspection and cross-dataset experiments suggest that the identified features capture task-relevant, dataset-conditioned patterns, yet retain utility signals that partially transfer across datasets. Our code is available at https://github.com/aohenuo/PULSE.
Figures & tables
Figure 1: Overview of the motivation and workflow of PULSE. Traditional methods still study these signals mainly at the level of query–demonstration matching. In contrast, PULSE identifies utility-related internal features and uses them for demonstration selection.
Figure 2: Overview of PULSE. Here, k -shot ICD denotes a k -example in-context demonstration set. (a) PULSE estimates a utility-localization vector by associating utility differences ΔU with SAE activation differences ΔA across paired demonstration sets. (b) PULSE validates the identified features by ranking pre-sampled complete demonstration sets in the PULSE-Ranking experiment. (c) PULSE selects demonstrations using PULSE-Retriever, which uses the utility-localization vector as a feature-relevance mask for scalable pool-scale retrieval.
Backbone
Method
AGNews
REST14
LAP14
EMOC
Avg.
1
2
4
8
1
2
4
8
1
2
4
8
1
2
4
8
1
2
4
8
Gemma2-2B
Random
40.62
66.99
75.00
77.93
60.94
65.82
73.83
73.05
54.21
65.44
76.03
73.87
44.92
53.52
54.69
60.16
50.17
62.94
69.89
71.25
Lex-Sim
43.75
68.75
77.34
82.03
66.41
72.46
78.71
75.39
64.79
70.84
78.40
77.75
52.93
54.88
56.64
60.94
56.97
66.73
72.77
74.03
SBERT
55.47
77.15
78.12
82.23
60.74
69.14
76.17
73.83
66.52
71.49
75.81
77.11
51.56
55.86
56.05
61.33
58.57
68.41
71.54
73.62
CEIL
55.47
71.48
78.12
78.12
60.74
69.73
76.76
75.59
66.52
70.41
79.70
78.19
51.56
56.64
56.25
60.94
58.57
67.06
72.71
73.21
KNN-SAE
54.69
77.54
76.95
81.84
63.67
69.53
78.12
76.17
65.87
72.35
79.05
76.24
48.63
52.93
58.01
60.55
58.22
68.09
73.03
73.70
Table 1: Candidate-set scoring accuracy (%) on Gemma2-2B and Llama3.1-8B. Downstream accuracy after using each score to select the top-1 candidate demonstration set. The Avg. columns report the average across four datasets under each shot setting.
Backbone
Method
AGNews
REST14
LAP14
EMOC
Avg.
1
2
4
8
1
2
4
8
1
2
4
8
1
2
4
8
1
2
4
8
Gemma2-2B
Random
41.80
66.99
72.66
78.52
57.62
64.65
70.90
72.46
56.59
60.91
68.25
71.71
41.60
54.69
56.25
60.94
49.40
61.81
67.02
70.91
Lex-Sim
68.16
78.32
79.88
86.13
64.45
66.80
73.63
76.76
70.63
75.16
74.30
77.54
62.70
62.50
65.82
71.68
66.48
70.69
73.41
78.03
SBERT
76.76
78.52
79.49
84.18
65.62
66.99
71.68
74.22
68.47
73.87
75.81
78.62
63.48
64.65
66.80
72.46
68.58
71.01
73.45
77.37
CEIL
76.76
78.91
81.25
87.11
65.62
67.97
75.39
76.95
68.47
76.03
76.03
78.83
63.48
65.62
66.02
71.09
68.58
72.13
74.67
78.50
KNN-SAE
79.69
78.71
81.05
87.50
66.99
65.43
75.59
75.59
70.63
76.46
75.16
77.75
50.20
60.74
66.80
72.46
66.88
70.33
74.65
78.32
Table 2: Pool-scale retrieval accuracy (%) on Gemma2-2B and Llama3.1-8B. Downstream accuracy after retrieving individual examples from the support set and composing the final k -shot context. The Avg. columns report the average across four datasets under each shot setting.
Method
CommonGen / Gemma2-2B / BLEU-4
CommonGen / Llama3.1-8B / BLEU-4
GSM8K / Llama3.1-8B/ EM
1
2
4
8
Avg.
1
2
4
8
Avg.
1
2
4
8
Avg.
Random
8.42
8.35
8.10
8.86
8.43
9.02
9.22
9.64
9.44
9.33
42.61
47.92
49.20
50.72
47.61
Lex-Sim
8.22
7.74
9.20
10.56
8.93
9.68
8.91
9.19
9.16
9.24
38.29
48.67
50.04
52.31
47.33
SBERT
8.41
8.36
9.08
9.82
8.92
9.41
9.84
10.21
9.91
9.84
39.73
49.20
43.82
52.92
46.42
KNN-SAE
9.00
8.34
8.93
7.55
8.46
8.91
9.64
9.96
9.71
9.56
39.12
46.10
46.93
53.53
46.42
CEIL
8.41
8.62
10.07
9.24
9.09
9.41
9.68
10.01
9.47
9.64
39.73
50.42
50.80
52.08
48.26
Table 3: Generation and reasoning results. Scores are BLEU-4 for CommonGen and exact match accuracy for GSM8K. The Avg. columns report the average over 1-, 2-, 4-, and 8-shot settings.
Dataset
SAE
PCA
Raw
AGNews
85.94
81.25
80.47
REST14
77.73
75.78
76.76
LAP14
77.32
73.22
74.95
EMOC
68.36
65.23
61.33
Cls. Avg.
77.34
73.87
73.38
CommonGen
11.09
10.23
10.34
Table 4: SAE-basis ablation on Gemma2-2B in the 4-shot pool-scale retrieval setting.
Dataset
PULSE-Mask
Anti-Retrieval
Sparse-Random
Bottom-Feats
PULSE-NoTopK
AGNews
83.01
51.95
82.81
82.03
81.64
REST14
76.17
63.67
75.39
75.00
78.12
LAP14
74.51
48.38
72.14
70.41
74.29
EMOC
60.16
30.08
58.20
57.81
59.38
Avg.
73.46
48.52
72.14
71.31
73.36
Table 5: Controls for the PULSE feature mask on Gemma2-2B in the 4-shot pool-scale retrieval setting. PULSE-Mask uses only the magnitude of the PULSE-identified feature vector, without the standard SAE cosine component used in PULSE-Retriever. Anti-Retrieval uses the same mask but selects the lowest-scoring pool items. Sparse-Random uses randomly selected sparse features, Bottom-Feats uses features with the smallest absolute identification scores, and PULSE-NoTopK removes the top-feature restriction. Accuracy is reported in percentage points.
“Aspect: works; Toshiba works great” “Aspect: power; power, memory and speed” “Aspect: memory; power, memory and speed”
Generic capability or off-aspect framing; lower-utility pattern.
EMOC
#7063
P1
“make others cry / i know whats wrong now” “give love to others … happy always” “especially yours / mine is so so / amazing”
Self/other dialogue with affective reasoning or topical pivot; task-conditioned emotion cue.
Table 6: Representative high- ∣w∣ PULSE features. P1/N1 denotes the top positive or negative feature after activation filtering; the sign reflects association with paired utility differences.
Single-source transfer
LOO transfer vs. baseline
Target PULSE
Target
AGNews
REST14
LAP14
EMOC
LOO ∪
Base max
ΔBase
Self
ΔSelf
AGNews
–
84.77
83.59
85.55
83.59
81.25
+2.34
85.94
−2.35
REST14
76.56
–
79.69
78.52
77.73
75.59
+2.14
77.73
+ 0.00
LAP14
78.19
74.73
–
76.46
77.54
76.03
+1.51
77.32
+0.22
EMOC
60.94
66.80
66.80
–
67.97
66.80
+1.17
68.36
−0.39
Avg.
71.90
75.43
76.69
80.18
76.71
74.92
+1.79
77.34
−0.63
Table 7: Cross-dataset transfer of PULSE utility-localization vectors in the 4-shot retrieval setting on Gemma2-2B. Scores are accuracy (%). Single-source : a utility-localization vector learned on the source dataset is applied directly to the target dataset. LOO ∪ : for each target dataset, the target excludes the dataset used for feature identification, and the PULSE-identified features from the three non-target datasets are combined by union. Base max : best non-PULSE baseline per target. ΔBase = LOO ∪ − Base max ; ΔSelf = LOO ∪ − Self.
Classification: teacher-forced label likelihood, no sampling. CommonGen: greedy generation, do_sample=False , max_new_tokens =32; temperature/top- p disabled.
Llama3.1-8B
llama_scope_lxr_32x fnlp/Llama3_1-8B-Base-LXR-32x He et al. (2024)
12 (classification) 16 (CommonGen) 24 (GSM8K)
131,072
Classification: teacher-forced label likelihood, no sampling. CommonGen: greedy generation, do_sample=False , max_new_tokens =32; temperature/top- p disabled. GSM8K: greedy generation, do_sample=False , max_new_tokens =220; temperature/top- p disabled.
Appendix
Table 8: Backbone, SAE, and decoding details. All SAE features are taken from residual-stream SAEs at the listed layer. The Gemma-Scope checkpoint is the layer-12 width-16k average- L0 -82 dictionary; the Llama-Scope LXR-32x checkpoints use dmodel=4096 and dSAE=131,072 . For classification, downstream decisions are made by teacher-forced label likelihood rather than free-form generation.
Table 9: Dataset and pool statistics. Train/test sizes and support-pool sizes are read from the experiment loaders. Average input length is computed on the input side using whitespace tokenization (concept-list length for CommonGen). Generation and reasoning experiments use smaller pools because each selected context requires autoregressive evaluation.
Symbol
Meaning
Default value
K+
Positive-weight SAE features retained
Gemma2-2B: 512 Llama3.1-8B: 2,048
K−
Negative-weight SAE features retained
Gemma2-2B: 512 Llama3.1-8B: 2,048
β
Sparse–dense blend weight in the retrieval score
0.3
L
Retrieval shortlist size
50
λr
SAE-space redundancy penalty
0.3
∣Ddisc∣
Discovery queries
64
Appendix
Table 10: Hyperparameter summary. Defaults are the values used in the main experiments unless a task-specific exception is shown.
Method
Labeled discovery data
Validation tuning
Model-internal SAE features
Greedy redundancy
Random
No
No
No
No
Lex-Sim
No
Yes
No
Yes
SBERT
No
Yes
No
Yes
KNN-SAE
No
Yes
Yes
Yes
CEIL
No
Yes
No
Built-in diversity objective
PULSE-Retriever
Yes, 64 queries
Yes
Yes
Yes
Appendix
Table 11: Supervision and tuning resources used by each retrieval method. PULSE uses a small labeled discovery split to estimate a utility-localization vector; the non-PULSE retrievers do not learn task-specific utility weights from labeled discovery queries. For pool-scale retrieval, pointwise methods use the same shortlist and greedy redundancy procedure unless otherwise stated.
Backbone
Variant
AGNews
REST14
LAP14
EMOC
CommonGen
GSM8K
Gemma2-2B
∣w∣ mask
85.94
77.73
77.32
68.36
11.09
–
signed score
84.38
76.56
77.34
65.23
10.25
–
Δ
+1.56
+1.17
−0.02
+3.13
+0.84
–
Llama3.1-8B
∣w∣ mask
88.09
81.84
82.07
71.09
10.90
54.66
signed score
85.16
82.03
82.03
72.66
10.64
51.53
Δ
+2.93
−0.19
+0.04
−1.57
+0.26
+3.13
Appendix
Table 12: Empirical comparison of the magnitude mask and a sign-sensitive retrieval score in PULSE-Retriever. Results are from the 4-shot pool-scale retrieval setting. The metric is accuracy for classification, BLEU-4 for CommonGen, and exact match for GSM8K. Δ=∣w∣ mask − signed score. Bold marks ∣Δ∣≥0.8 . “–” indicates GSM8K is not evaluated on Gemma2-2B.
Sign
Feature
Weight
Max-activating snippet
Interpretation
+
8186
7.48
Open Source: Balancing Innovation and Risk
tech-industry commentary
+
11980
6.16
Climate fears on sharp CO 2 rise
scientific finding / climate framing
+
14321
5.75
Women Voters Enter San Diego Mayoral Fray
civic / institutional reference
+
4297
4.35
British-Irish national kidnapped in Afghanistan
named geopolitical event
+
1398
3.70
Australia struggling against New Zealand
sports-news event
−
12684
-3.65
Bond Prices Edge Higher at Close of Day
market-summary phrasing
Appendix
Table 13: Top PULSE-identified features on AGNews.
Sign
Feature
Weight
Max-activating snippet
Interpretation
+
7063
11.19
no / make others cry / i know whats wrong now
dialogue with topical pivot
+
15322
11.18
shut up / my exam is on tuesday
direct-address with factual follow-up
+
9565
10.81
you are very bad / sorry / because…
confrontation with stated reason
+
5466
10.68
what else do you know about me?
introspective Q–A small talk
+
15273
9.54
what happened / net problems on my phone
factual problem report and topic shift
−
7657
-8.59
me happy / happy happy warm warm
repetitive affect tokens
Appendix
Table 14: Top PULSE-identified features on EMOC.
Sign
Feature
Weight
Max-activating snippet
Interpretation
+
11485
3.13
Aspect: screen; big screen allows you to enjoy movies
aspect to enjoyment
+
5828
1.68
Aspect: Pages; design tools make it easier
software aspect to ease of use
+
13435
1.61
Aspect: PRICE; LOVE THIS LAPTOP WONDERFUL PRICE
aspect to strong positive judgment
+
16061
1.56
Aspect: battery; why not give you a better battery?
aspect to complaint
+
9749
1.46
Aspect: price; iWork is not worth the price
aspect to negative judgment
−
15777
-3.82
Aspect: works; my new Toshiba works great on both
generic verb / off-aspect framing
Appendix
Table 15: Top PULSE-identified features on LAP14.
Sign
Feature
Weight
Max-activating snippet
Interpretation
+
7875
0.61
Aspect: desserts; got favorable reviews
committed positive judgment
+
5435
0.56
Aspect: owner; always has a smile
warm person-description
+
1512
0.53
Aspect: food; good food and ok service
committed mixed judgment
+
3486
0.44
Aspect: Decor; leaves something to be desired
committed negative judgment
+
8374
0.43
Aspect: comfort food; hot and large portions
simple positive description
−
9781
-0.57
Aspect: ambience; I care more about the food and ambience
meta-commentary
Appendix
Table 16: Top PULSE-identified features on REST14.
Dataset pair
Intersection
Union
Jaccard
AGNews–EMOC
94
1954
0.048
AGNews–REST14
90
1958
0.046
AGNews–LAP14
87
1961
0.044
EMOC–REST14
140
1908
0.073
EMOC–LAP14
95
1953
0.049
REST14–LAP14
207
1841
0.112
Appendix
Table 17: Pairwise overlap of active PULSE features across datasets. The two aspect-based sentiment datasets, REST14 and LAP14, have the largest overlap, but the overall Jaccard values remain small.
# active datasets
# features
Consistent +
Consistent −
Mixed
% mixed
1
2894
1401
1493
–
–
2
497
145
133
219
44.1
3
64
19
6
39
60.9
4
4
1
0
3
75.0
Appendix
Table 18: Sign consistency among active PULSE features across datasets. A mixed-sign feature is utility-positive on at least one dataset and utility-negative on at least one other dataset.
Classification (Acc.)
Generation & Reasoning
Backbone
Method
AGNews
REST14
LAP14
EMOC
Cls. Avg.
Query budget
CommonGen
GSM8K
Gemma2-2B
EPR-style learned retriever
87.50
75.00
80.35
66.21
77.27
128 queries
9.78
–
PULSE-Retriever
85.94
77.73
77.32
68.36
77.34
64 queries
11.09
–
Llama3.1-8B
EPR-style learned retriever
89.26
77.73
83.00
70.89
80.22
128 queries
10.19
44.53
PULSE-Retriever
88.09
81.84
82.07
71.09
80.77
64 queries
10.90
54.66
Appendix
Table 19: Comparison with a supervised learned retriever in the 4-shot pool-scale retrieval setting. Classification metrics are accuracy (%); CommonGen is BLEU-4; GSM8K is exact match (%). Cls. Avg. is the mean over four classification datasets. The learned retriever uses 128 training queries expanded into 2,048 singleton feedback labels and gradient-based retriever training; PULSE-Retriever uses 64 discovery queries with closed-form feature scoring and no retriever training.
Dataset
K=256
K=512
K=1024
K=2048
K=4096
AGNews
85.16
85.94
86.13
85.94
84.38
REST14
78.12
77.73
79.30
80.08
79.69
LAP14
77.97
77.32
76.67
78.19
78.62
EMOC
65.23
68.36
68.16
66.41
66.80
Avg.
76.62
77.34
77.57
77.65
77.37
Appendix
Table 20: Retriever accuracy as a function of selected feature count K in the 4-shot pool-scale retrieval setting. Bold indicates the best result in each row.
Dataset
16
32
64
128
AGNews
82.42
83.98
85.94
86.72
REST14
77.73
78.52
77.73
77.34
LAP14
79.48
78.19
77.32
78.40
EMOC
62.50
64.45
68.36
66.80
Avg.
75.53
76.28
77.34
77.32
Appendix
Table 21: Retriever accuracy as a function of discovery query count ∣Ddisc∣ in the 4-shot pool-scale retrieval setting. Bold indicates the best result in each row.
Dataset
β=0.0
β=0.1
β=0.2
β=0.3
β=0.5
β=0.7
β=1.0
AGNews
83.01
83.20
85.94
85.94
86.72
86.52
81.05
LAP14
74.51
75.59
74.30
77.32
76.03
75.16
75.16
REST14
76.17
75.59
76.95
77.73
77.93
76.76
75.59
EMOC
60.16
63.87
62.30
68.36
67.77
66.41
66.80
Avg.
73.46
74.56
74.87
77.34
77.11
76.21
74.65
Appendix
Table 22: Retriever accuracy as a function of the sparse–dense blend weight β in the 4-shot pool-scale retrieval setting. β=0 denotes pure PULSE-weighted cosine similarity, while β=1 denotes pure SAE-space cosine similarity. Bold indicates the best result in each row.
Shot
Best Layer
Best Avg. (%)
Layer-12 Avg. (%)
Gap (pp)
Layer Range (pp)
1-shot
12
70.64
70.64
0.00
4.86
2-shot
21
75.02
74.31
0.71
2.57
4-shot
5
78.48
77.34
1.14
3.22
8-shot
24
80.10
79.61
0.49
2.40
Appendix
Table 23: Summary of SAE layer sensitivity for PULSE-Retriever on Gemma2-2B in the pool-scale retrieval setting. Best Avg. is the highest average accuracy across layers. Layer-12 Avg. denotes the default layer used in the main experiments. Gap is the difference between Best Avg. and Layer-12 Avg.; Layer Range is the difference between the best and worst layer averages. All numeric entries except layer indices are reported in percentages or percentage points.
Layer
0
1
2
3
4
5
6
7
8
9
10
11
12 †
AGNews
73.24
79.49
77.54
76.37
74.80
73.63
75.00
77.54
73.63
79.49
78.91
72.66
81.25
LAP14
67.39
65.44
69.55
67.60
67.39
70.41
68.68
68.25
66.95
68.68
71.92
69.33
72.79
REST14
64.26
60.94
65.43
65.82
67.58
65.82
66.80
65.62
66.02
65.62
64.65
66.60
64.65
EMOC
62.11
62.50
63.28
62.70
65.23
64.06
60.74
63.28
65.43
58.40
61.52
62.30
63.87
Avg.
66.75
67.09
68.95
68.12
68.75
68.48
67.81
68.67
68.01
68.05
69.25
67.72
70.64
Layer
13
14
15
16
17
18
19
20
21
22
23
24
25
Appendix
Table 24: Full SAE layer sweep for PULSE-Retriever on Gemma2-2B in the 1-shot pool-scale retrieval setting. Each entry reports downstream accuracy (%). Avg. denotes the average across AGNews, LAP14, REST14, and EMOC. † denotes the default layer used in the main experiments. Bold indicates the best value in each row.
Layer
0
1
2
3
4
5
6
7
8
9
10
11
12 †
AGNews
80.86
76.95
79.49
79.30
82.23
80.66
80.86
83.20
83.98
81.64
81.84
83.01
82.62
LAP14
75.59
74.30
74.95
74.51
74.51
75.16
72.57
73.65
73.43
73.43
76.46
75.38
77.11
REST14
73.05
68.95
69.53
72.07
73.44
73.44
74.22
70.70
73.24
73.83
71.29
75.00
71.68
EMOC
69.14
72.07
69.34
70.51
65.82
68.36
68.95
65.82
68.36
67.38
67.77
66.02
65.82
Avg.
74.66
73.07
73.33
74.10
74.00
74.41
74.15
73.34
74.76
74.07
74.34
74.85
74.31
Layer
13
14
15
16
17
18
19
20
21
22
23
24
25
Appendix
Table 25: Full SAE layer sweep for PULSE-Retriever on Gemma2-2B in the 2-shot pool-scale retrieval setting. Each entry reports downstream accuracy (%). Avg. denotes the average across AGNews, LAP14, REST14, and EMOC. † denotes the default layer used in the main experiments. Bold indicates the best value in each row.
Layer
0
1
2
3
4
5
6
7
8
9
10
11
12 †
AGNews
84.77
84.18
83.20
85.55
83.40
86.52
86.33
85.16
86.33
88.67
84.77
85.55
85.94
LAP14
75.59
77.97
77.32
75.81
77.54
77.97
79.70
77.75
77.75
77.54
77.54
77.32
77.32
REST14
75.59
73.63
75.78
75.39
73.44
77.93
74.80
75.39
75.98
75.39
74.41
75.00
77.73
EMOC
70.51
71.29
69.34
71.09
71.48
71.48
67.58
69.53
72.07
68.95
69.92
70.12
68.36
Avg.
76.61
76.77
76.41
76.96
76.46
78.48
77.10
76.96
78.03
77.64
76.66
77.00
77.34
Layer
13
14
15
16
17
18
19
20
21
22
23
24
25
Appendix
Table 26: Full SAE layer sweep for PULSE-Retriever on Gemma2-2B in the 4-shot pool-scale retrieval setting. Each entry reports downstream accuracy (%). Avg. denotes the average across AGNews, LAP14, REST14, and EMOC. † denotes the default layer used in the main experiments. Bold indicates the best value in each row.
Layer
0
1
2
3
4
5
6
7
8
9
10
11
12 †
AGNews
86.52
85.74
85.55
83.59
90.04
86.13
85.55
88.28
88.67
88.48
86.91
87.50
88.09
LAP14
78.40
77.97
78.62
79.91
79.70
79.27
79.48
80.35
79.27
80.78
79.48
79.48
80.35
REST14
78.32
78.91
76.95
77.34
76.37
77.15
77.73
77.73
78.32
77.73
76.17
79.49
77.73
EMOC
73.83
73.44
74.80
73.83
73.05
72.85
75.00
73.44
72.85
70.31
71.68
70.31
72.27
Avg.
79.27
79.01
78.98
78.67
79.79
78.85
79.44
79.95
79.78
79.33
78.56
79.20
79.61
Layer
13
14
15
16
17
18
19
20
21
22
23
24
25
Appendix
Table 27: Full SAE layer sweep for PULSE-Retriever on Gemma2-2B in the 8-shot pool-scale retrieval setting. Each entry reports downstream accuracy (%). Avg. denotes the average across AGNews, LAP14, REST14, and EMOC. † denotes the default layer used in the main experiments. Bold indicates the best value in each row.
Backbone
Dataset (Metric)
Mean ± Std
Per-seed values
Gemma2-2B
AGNews (Acc)
87.76±1.62
89.06, 88.28, 85.94
REST14 (Acc)
76.43±2.60
77.73, 73.44, 78.12
LAP14 (Acc)
79.41±2.43
82.07, 77.32, 78.83
EMOC (Acc)
69.46±1.07
68.36, 70.51, 69.53
CommonGen (BLEU-4)
10.47±0.62
11.09, 10.47, 9.85
Llama3.1-8B
AGNews (Acc)
87.44±0.69
88.09, 86.72, 87.50
Appendix
Table 28: Seed stability of PULSE-Retriever in the 4-shot pool-scale retrieval setting. We report accuracy (%) for classification, BLEU-4 for CommonGen, and exact match (%) for GSM8K. Each row reports the mean ± standard deviation over three seeds, along with the individual seed values. GSM8K is not evaluated on Gemma2-2B.
In-context example selection chooses demonstrations that help language and multimodal models perform a query-specific task without changing model weights.
In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because the space of possible demonstration contexts and combinations is enormous. We argue that demonstration selection is easier to judge than to find: predicting whether a specific query--context pair (q,D) will succeed is cheaper and more general than searching for an optimal D⋆. Based on this insight, we propose DiSP, a sample-and-judge framework that stratifies queries by difficulty. DiSP runs random demonstration trials to estimate success rate of each training query, trains a lightweight router to predict difficulty from the query, and trains level-specific judges for sampled demonstrations. At inference, DiSP performs stop-on-acceptance judging under an explicit budget, emitting diagnostic risk tags when no suitable context is found. Across five classification datasets with Llama3--8B and Qwen2.5--7B, DiSP achieves the best average accuracy, improving over strong learned selection baselines by up to 3.4%, while achieving up to 23× end-to-end wall-clock speedup.
Haochun Wang, Chaofen Yang, Jiatong Liu +5
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China.
The In-context learning (ICL) paradigm aids large language models (LLMs) to adapt to new tasks without need for fine-tuning. However, selecting an optimal combination of demonstration examples from a large pool of example subsets is a challenging problem. Existing approaches for selection do not model the complex relationship between ICL samples and downstream LLM performance. They typically perform static task-level selection, choosing subsets once offline, which can fail to generalize to unseen queries. We introduce DearICL (Data Efficient Algorithm for Ranking) ICL samples, a new framework that models demonstration example selection as a subset ranking problem. DearICL employs a non-linear surrogate employing a differentiable sorting objective within a gap-index bandit algorithm. The gap-index based approach enables fine-grained separation of good arms and borderline arms, which is used as an auxiliary objective to train the non-linear surrogate through sufficient sampling of borderline arms, supporting instance-level subset ranking. On exemplar selection benchmarks with open-source LLMs, DearICL achieves 8.08-15.9% accuracy gains over strong linear bandit baselines, with low sample complexity. Code and data: https://github.com/VenkteshV/DearICL.
In-context learning (ICL) performance depends critically on which demonstrations are placed in the prompt, yet most existing selectors prioritize heuristic notions of relevance or diversity and provide limited insight into the coverage of a demonstration set. We propose Unseen Coverage Selection (UKS), a training-free, subset-level coverage prior motivated by the principle that a good demonstration set should expose the model to latent cluster unrevealed by the currently selected subset. UCS operationalizes this idea by (1) inducing discrete latent clusters from model-consistent embeddings and (2) estimating the number of unrevealed clusters within a candidate subset via a Smoothed Good--Turing estimator from its empirical frequency spectrum. Unlike previous selection methods, UCS is coverage-based and training-free, and can be seamlessly combined with both query-dependent and query-independent selection baselines via a simple regularized objective. Experiments on multiple intent-classification and reasoning benchmarks with frontier Large Language Models show that augmenting strong baselines with UCS consistently improves ICL accuracy by up to 2-6% under the same selection budget, while also yielding insights into task- and model-level latent cluster distributions. Code is available at https://github.com/Raina-Xin/UCS.