In-context learning (ICL) adapts large language models by conditioning on a small set of ICL examples, avoiding costly parameter updates. Among other factors, performance is often highly sensitive to the ordering of the examples. However, exhaustive search over the n! possible orderings is infeasible. Therefore more efficient ordering methods use model confidence measures (e.g., label-probability entropy) over label sets or take a direct approach to finding the best ordering. We propose PLR, a probabilistic approach to in-context example ordering that replaces discrete ordering search with learning a probability distribution over orderings with the Plackett-Luce model. PLR models orderings using a Plackett-Luce distribution and iteratively updates its parameters to concentrate probability mass on high-performing orderings under a task-level metric. Candidate orderings are sampled efficiently via a Gumbel perturb-and-sort procedure. Experiments on multiple classification benchmarks show that PLR consistently improves few-shot accuracy for k∈{4,8,16,32} examples, and we further demonstrate gains on mathematical reasoning tasks where label-based ordering methods are not applicable. Our code is available at https://github.com/Batorskq/PLR.
Figures & tables
Figure 2: Illustration of PLR. Given a Plackett–Luce distribution, we sample high-probability permutations with the Gumbel trick. Each permutation is scored and the top-K are retained and used for fitting an improved Plackett-Luce distribution. This is iterated until high quality ICL ordering is found.
Method
Model
MR
NEWS
SST5
TREC
SUBJ
Avg
k=4
k=8
k=16
k=32
k=4
k=8
k=16
k=32
k=4
k=8
k=16
k=32
k=4
k=8
k=16
k=32
k=4
k=8
k=16
k=32
k=4
k=8
k=16
k=32
Static
Qwen2.5-7B-Instruct
90.22
91.10
91.75
91.73
83.37
82.47
83.71
83.79
54.78
54.12
54.78
56.52
61.95
63.90
60.26
59.99
65.88
71.50
76.55
79.23
71.24
72.62
73.41
74.25
Top-K
90.90
91.14
91.95
91.96
84.18
84.40
85.07
85.35
55.55
55.36
55.85
56.85
65.79
65.45
61.00
63.29
71.54
77.43
79.69
82.32
73.59
74.76
74.71
75.95
LocalE
90.72
91.20
91.36
91.90
82.59
83.53
83.69
82.76
54.72
54.55
54.67
56.86
63.87
65.86
61.14
61.74
69.07
73.70
75.96
80.05
72.19
73.77
73.36
74.66
GlobalE
90.66
91.11
91.69
91.80
82.79
83.60
83.81
82.97
54.50
54.47
55.07
56.53
59.31
63.02
61.67
61.07
68.48
70.93
74.64
77.82
71.15
72.63
73.38
74.04
PDO
90.56
91.08
91.39
91.73
84.11
83.50
82.64
84.23
54.87
54.34
55.31
56.29
59.01
64.10
62.21
61.81
65.69
70.12
75.26
77.00
70.85
72.63
73.36
74.21
Table 1: Test accuracy (%) averaged over seeds. best score is colored in red, second best in orange and third best in yellow. Avg is macro-average over MR/NEWS/SST5/TREC/SUBJ. Top block: qwen2.5-7b-instruct . Bottom block: Llama3.1-8B-instruct .
Method
GSM8K
DeepMath
Math500
Avg
k=4
k=8
k=16
k=32
k=4
k=8
k=16
k=32
k=8
k=4
k=32
k=16
k=4
k=8
k=16
k=32
Static
35.45
36.33
39.70
39.05
36.03
38.70
41.85
43.27
28.52
30.83
29.60
34.98
34.10
34.52
38.84
37.31
Top-K
36.05
38.86
39.35
40.40
36.49
38.74
42.20
45.13
31.40
31.13
29.80
38.60
34.56
36.33
40.05
38.44
PLR-EMA
40.88
39.50
39.85
41.60
36.64
41.52
42.52
46.16
32.32
33.73
31.80
39.00
37.08
37.78
40.46
39.85
PLR-1
39.63
39.65
39.85
42.85
36.64
39.93
42.69
45.62
31.43
32.70
32.00
39.00
36.32
37.00
40.51
40.16
PLR-4
40.88
39.77
41.40
42.80
36.64
39.17
42.46
46.36
31.90
33.73
33.40
38.78
37.08
36.95
40.88
40.85
Table 2: Test accuracy (%) averaged over seeds for Qwen2.5-7B-instruct on math benchmarks. best, second best, third best (per column). Avg is macro-average over GSM8K/DeepMath/Math500.
K
k=4
k=8
k=16
k=32
4
73.09
80.94
85.17
85.98
8
73.04
80.10
84.17
86.75
16
72.83
79.78
84.57
86.06
32
72.52
80.48
83.95
86.25
64
72.65
79.51
84.24
86.40
Table 3: Results for SUBJ across different number of K of Mixture PL distribution and number of ICL examples k .
Method
Weighting
k=4
k=8
k=16
k=32
PLR-1
Unweighted
72.80
80.33
83.44
85.90
Weighted
72.78
80.88
83.85
87.48
PLR-4
Unweighted
73.09
80.94
85.17
85.98
Weighted
73.09
80.70
84.22
86.97
Table 4: SUBJ results for PLR variants with weighted vs. unweighted MLE.
Figure 3: Top : Ablation probability-test accuracy results for PLR-EMA and Bottom : for PLR-1 .
ρ
PLR-EMA
PLR-1
PLR-4
k=4
k=8
k=16
k=32
k=4
k=8
k=16
k=32
k=4
k=8
k=16
k=32
0.1
73.09
80.59
83.16
85.31
73.04
80.81
83.78
86.46
73.09
80.96
83.95
85.94
0.2
72.78
80.64
83.98
86.75
72.80
80.33
83.44
85.90
73.09
80.94
85.17
85.98
0.3
73.04
80.58
83.05
86.02
73.04
80.97
82.37
85.47
72.78
80.62
84.43
85.07
0.5
72.75
79.91
82.65
84.90
72.91
79.83
82.06
86.15
72.91
79.81
82.00
83.88
Table 5: Results for SUBJ for varying parameters ρ of choosing the elite set in line 17 in Algorithm 1 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Symbol
Value
Meaning
Experimental protocol
# demonstrations
k
{4,8,16,32}
Shots used in the prompt.
# seeds
—
5
Runs per dataset/ k /method.
Validation budget
∣Dval∣
1000
Total labeled pool used for ordering selection (if using a subset; otherwise full ).
Inner/outer split
—
80/20
Inner set used for updates ( Dttrain ); outer set used for final selection ( Dval ).
Scoring batch size
—
16
Batch size for label-scoring / evaluation.
Appendix
Table 6: Hyperparameters used for PLR (unless stated otherwise). We use the same notation as in the main body/Alg. 1 : T (CE iterations), B (samples/iter), ρ (elite fraction), K (final draws), α (EMA smoothing), and τ (rank temperature). For the mixture model, K also denotes the number of mixture components (as in the mixture definition in Sec. 3 ).