IJev: Order-Invariant Decisions via Iterated Cyclic Reordering
Organizations: Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences
Abstract
System One decision models answer a choice question in one forward pass, but their answer depends on the order in which options are listed, and their probabilities carry a prior over answer labels. An invariant ordering operator is any fixed function that returns the same ordered option list for every input order of a fixed option set. Starting from any such order and averaging a listwise readout over its K cyclic shifts removes the input order, but the decision still depends on the chosen cyclic class: those shifts are only one of the (K-1)! classes, and different invariant operators can select different classes. We propose IJev, which repeatedly reorders the options by the cyclic-averaged, label-prior-corrected distribution and runs another cyclic average. Every iterate is exactly input-order-invariant. When the iteration does not converge, we average the round distributions rather than return the last round. On five benchmarks and five models, IJev has flip rate zero in every cell. The default uniform average over T=20 rounds substantially improves calibration over a single cyclic average, and its accuracy is comparable. A descriptive choice of horizon and averaging rule is best or tied-best for accuracy in all 25 settings.
Figures & tables
| Banking20 | Newsgroups | MMLU-Pro | Typed Dec. | JevBench | Avg | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Method | Acc | Flip | Acc | Flip | Acc | Flip | Acc | Flip | Acc | Flip | Acc | Flip |
| Qwen3-1.7B | Listwise (raw) | 62.1 | 33.7 | 59.0 | 28.7 | 27.6 | 59.7 | 47.3 | 39.0 | 54.7 | 28.1 | 50.1 | 37.8 |
| AnyJev-L0 | 69.5 | 11.8 | 61.3 | 19.7 | 33.2 | 31.2 | 49.0 | 25.7 | 59.0 | 9.4 | 54.4 | 19.5 | |
| IJev (uniform, ) | 70.3 | 0.0 | 62.0 | 0.0 | 33.2 | 0.0 | 57.7 | 0.0 | 57.6 | 0.0 | 56.2 | 0.0 | |
| IJev (log-avg, ) | 70.5 | 0.0 | 63.0 | 0.0 | 33.3 | 0.0 | 57.5 | 0.0 | 58.3 | 0.0 | 56.5 | 0.0 | |
| IJev (best) | 72.0 | 0.0 | 65.3 | 0.0 | 34.6 | 0.0 | 57.7 | 0.0 | 61.9 | 0.0 | 58.3 | 0.0 | |
| Banking20 | Newsgroups | MMLU-Pro | Typed Dec. | JevBench | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Method | ECE | Brier | NLL | ECE | Brier | NLL | ECE | Brier | NLL | ECE | Brier | NLL | ECE | Brier | NLL |
| Qwen3-1.7B | Listwise (raw) | 0.36 | 0.73 | 6.56 | 0.39 | 0.79 | 7.03 | 0.63 | 1.31 | 7.59 | 0.51 | 1.02 | 7.29 | 0.40 | 0.81 | 4.14 |
| AnyJev-L0 | 0.27 | 0.57 | 3.38 | 0.35 | 0.72 | 4.72 | 0.46 | 1.08 | 4.70 | 0.37 | 0.83 | 3.03 | 0.36 | 0.76 | 3.71 | |
| IJev (uniform) | 0.19 | 0.47 | 1.84 | 0.28 | 0.64 | 2.98 | 0.31 | 0.94 | 3.32 | 0.25 | 0.72 | 2.50 | 0.28 | 0.64 | 2.13 | |
| IJev (log-avg) | 0.25 | 0.53 | 2.62 | 0.32 | 0.69 | 4.12 | 0.37 | 1.00 | 3.77 | 0.34 | 0.75 | 2.86 | 0.33 | 0.72 | 2.63 | |
| IJev (best) | 0.19 | 0.49 | 2.28 | 0.31 | 0.66 | 4.11 | 0.33 | 0.97 | 3.60 | 0.25 | 0.72 | 2.50 | 0.31 | 0.69 | 3.05 | |
| IJev uniform | IJev log-avg | |||||||||||
| Model | ||||||||||||
| Banking20 | ||||||||||||
| Qwen3-1.7B | 69.5 | 70.5 | 70.6 | 70.8 | 70.2 | 70.3 | 69.5 | 70.8 | 70.7 | 70.8 | 70.1 | 70.5 |
| Qwen3-4B | 76.0 | 75.8 | 76.8 | 76.2 | 77.3 | 77.1 | 76.0 | 76.2 | 77.0 | 76.6 | 77.2 | 77.2 |
| Qwen3-8B | 80.4 | 81.4 | 82.0 | 81.4 | 81.5 | 81.7 | 80.4 | 81.6 | 81.7 | 81.5 | 81.5 | 81.9 |
| Mistral-7B | 73.3 | 75.1 | 76.7 | 76.8 | 77.0 | 76.8 | 73.3 | 76.8 | 77.0 | 77.0 | 77.6 | 77.5 |
| Model | Task | Listwise | AnyJev-L0 | ||||||
|---|---|---|---|---|---|---|---|---|---|
| 1.7B | Banking20 | 0.008 | 0.141 | 0.211 | 0.363 | 0.819 | 1.578 | 2.337 | 3.096 |
| 4B | Banking20 | 0.017 | 0.333 | 0.483 | 0.825 | 1.850 | 3.557 | 5.265 | 6.972 |
| 8B | Banking20 | 0.030 | 0.589 | 0.821 | 1.413 | 3.193 | 6.146 | 9.096 | 12.045 |
| 1.7B | Newsgroups | 0.014 | 0.244 | 0.445 | 0.697 | 1.454 | 2.715 | 3.978 | 5.239 |
| 4B | Newsgroups | 0.031 | 0.557 | 1.003 | 1.571 | 3.273 | 6.110 | 8.947 | 11.785 |
| 8B | Newsgroups | 0.049 | 0.918 | 1.617 | 2.534 | 5.284 | 9.869 | 14.455 | 19.041 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Prior type | batch position prior |
| Reference set | evaluation batch |
| Per-item grouping | (group, , shift), states |
| Fallback | -pooled; none if states |
| Content-free probes | available, not used |
| Prior strength | 1.0 (AnyJev-L0 and IJev) |
| Model | Method | Acc | Flip | ECE | Brier | NLL |
|---|---|---|---|---|---|---|
| Qwen3-1.7B, Banking20, full 799 | ||||||
| Listwise (raw) | 62.1 | 33.7 | 0.355 | 0.728 | 6.560 | |
| AnyJev-L0 | 69.5 | 11.8 | 0.270 | 0.571 | 3.382 | |
| IJev (last round) | 69.1 | 0.0 | 0.271 | 0.568 | 2.979 | |
| IJev (uniform average) | 70.3 | 0.0 | 0.190 | 0.471 | 1.839 | |
| IJev (log-average) | 70.5 | 0.0 | 0.254 | 0.532 | 2.619 | |
| Method | Acc | Flip | ECE | Brier | NLL |
|---|---|---|---|---|---|
| Qwen3-1.7B, Newsgroups, 300 | |||||
| Listwise (raw) | 59.0 | 28.7 | 0.387 | 0.790 | 7.026 |
| AnyJev-L0 | 61.3 | 19.7 | 0.349 | 0.720 | 4.719 |
| IJev (last round) | 60.0 | 0.0 | 0.358 | 0.732 | 4.484 |
| IJev (uniform average) | 62.0 | 0.0 | 0.275 | 0.635 | 2.983 |
| IJev (log-average) | 63.0 | 0.0 | 0.324 | 0.689 | 4.124 |
| Model | Uniform avg Acc / ECE / NLL | Log-average Acc / ECE / NLL | s/dec | |
|---|---|---|---|---|
| 1.7B | 1 | 69.5 / 0.262 / 3.128 | 69.5 / 0.262 / 3.128 | 0.211 |
| 2 | 70.5 / 0.205 / 2.427 | 70.8 / 0.249 / 2.779 | 0.363 | |
| 5 | 70.6 / 0.193 / 2.107 | 70.7 / 0.249 / 2.690 | 0.819 | |
| 10 | 70.8 / 0.189 / 1.951 | 70.8 / 0.249 / 2.678 | 1.578 | |
| 15 | 70.2 / 0.194 / 1.911 | 70.1 / 0.257 / 2.668 | 2.337 | |
| 20 | 70.3 / 0.190 / 1.839 | 70.5 / 0.254 / 2.619 | 3.096 |
| Model | Uniform avg Acc / ECE / NLL | Log-average Acc / ECE / NLL | s/dec | |
|---|---|---|---|---|
| 1.7B | 1 | 65.0 / 0.317 / 4.795 | 65.0 / 0.317 / 4.795 | 0.445 |
| 2 | 64.3 / 0.257 / 3.635 | 65.3 / 0.312 / 4.108 | 0.697 | |
| 5 | 63.0 / 0.267 / 3.254 | 63.0 / 0.328 / 4.105 | 1.454 | |
| 10 | 61.3 / 0.278 / 3.083 | 62.7 / 0.323 / 4.083 | 2.715 | |
| 15 | 61.7 / 0.277 / 3.045 | 62.0 / 0.332 / 4.118 | 3.978 | |
| 20 | 62.0 / 0.275 / 2.983 | 63.0 / 0.324 / 4.124 | 5.239 |
| Uniform Acc / ECE / NLL | Exp. Acc / ECE / NLL | Best at | s/dec | |
|---|---|---|---|---|
| 1 | 69.5 / 0.262 / 3.128 | 69.5 / 0.262 / 3.128 | uniform 69.5 / 3.128 | 0.211 |
| 2 | 70.5 / 0.205 / 2.427 | 70.1 / 0.209 / 2.426 | logavg 70.8 / 2.779 | 0.363 |
| 3 | 69.8 / 0.213 / 2.274 | 69.8 / 0.214 / 2.275 | logavg 70.7 / 2.778 | 0.515 |
| 4 | 71.1 / 0.187 / 2.143 | 71.2 / 0.185 / 2.143 | exp10 71.2 / 2.143 | 0.667 |
| 5 | 70.6 / 0.193 / 2.107 | 70.3 / 0.195 / 2.110 | logavg 70.7 / 2.690 | 0.819 |
| 6 | 70.8 / 0.188 / 2.064 | 70.8 / 0.188 / 2.068 | avg_last3 71.1 / 2.324 | 0.971 |
| Uniform Acc / ECE / NLL | Exp. Acc / ECE / NLL | Best at | s/dec | |
|---|---|---|---|---|
| 1 | 76.0 / 0.223 / 3.611 | 76.0 / 0.223 / 3.611 | uniform 76.0 / 3.611 | 0.483 |
| 2 | 75.8 / 0.179 / 2.705 | 74.7 / 0.190 / 2.700 | logavg 76.2 / 3.098 | 0.825 |
| 3 | 76.0 / 0.182 / 2.524 | 76.1 / 0.181 / 2.521 | logavg 76.8 / 3.043 | 1.166 |
| 4 | 76.3 / 0.170 / 2.437 | 76.0 / 0.173 / 2.436 | logavg 76.8 / 2.972 | 1.508 |
| 5 | 76.8 / 0.168 / 2.426 | 77.1 / 0.174 / 2.430 | exp10 77.1 / 2.430 | 1.850 |
| 6 | 76.8 / 0.166 / 2.345 | 76.6 / 0.163 / 2.346 | logavg 77.1 / 2.931 | 2.191 |
| Uniform Acc / ECE / NLL | Exp. Acc / ECE / NLL | Best at | s/dec | |
|---|---|---|---|---|
| 1 | 80.4 / 0.181 / 2.908 | 80.4 / 0.181 / 2.908 | uniform 80.4 / 2.908 | 0.821 |
| 2 | 81.4 / 0.149 / 2.276 | 81.4 / 0.149 / 2.273 | logavg 81.6 / 2.588 | 1.413 |
| 3 | 81.2 / 0.146 / 2.170 | 81.1 / 0.147 / 2.167 | logavg 81.7 / 2.579 | 2.005 |
| 4 | 81.6 / 0.137 / 2.056 | 81.9 / 0.135 / 2.051 | exp10 81.9 / 2.051 | 2.598 |
| 5 | 82.0 / 0.135 / 2.014 | 81.6 / 0.136 / 2.012 | uniform 82.0 / 2.014 | 3.193 |
| 6 | 82.0 / 0.134 / 1.937 | 81.5 / 0.135 / 1.934 | uniform 82.0 / 1.937 | 3.786 |
| Uniform Acc / ECE / NLL | Exp. Acc / ECE / NLL | Best at | s/dec | |
|---|---|---|---|---|
| 1 | 65.0 / 0.317 / 4.795 | 65.0 / 0.317 / 4.795 | uniform 65.0 / 4.795 | 0.445 |
| 2 | 64.3 / 0.257 / 3.635 | 62.3 / 0.277 / 3.632 | logavg 65.3 / 4.108 | 0.697 |
| 3 | 62.3 / 0.277 / 3.418 | 62.3 / 0.278 / 3.415 | logavg 64.0 / 4.067 | 0.949 |
| 4 | 62.7 / 0.268 / 3.319 | 62.3 / 0.272 / 3.319 | logavg 63.3 / 4.032 | 1.202 |
| 5 | 63.0 / 0.267 / 3.254 | 63.0 / 0.269 / 3.258 | uniform 63.0 / 3.254 | 1.454 |
| 6 | 62.7 / 0.265 / 3.198 | 62.3 / 0.269 / 3.205 | avg_last5 63.3 / 3.294 | 1.706 |
| Uniform Acc / ECE / NLL | Exp. Acc / ECE / NLL | Best at | s/dec | |
|---|---|---|---|---|
| 1 | 65.7 / 0.297 / 4.540 | 65.7 / 0.297 / 4.540 | uniform 65.7 / 4.540 | 1.003 |
| 2 | 68.0 / 0.233 / 3.434 | 69.0 / 0.224 / 3.431 | exp10 69.0 / 3.431 | 1.571 |
| 3 | 67.0 / 0.247 / 3.162 | 67.0 / 0.249 / 3.158 | logavg 68.0 / 3.854 | 2.138 |
| 4 | 68.7 / 0.222 / 3.066 | 68.7 / 0.222 / 3.064 | exp10 68.7 / 3.064 | 2.706 |
| 5 | 67.0 / 0.242 / 2.974 | 66.7 / 0.246 / 2.973 | logavg 67.7 / 3.849 | 3.273 |
| 6 | 67.3 / 0.232 / 2.890 | 67.3 / 0.232 / 2.891 | avg_last5 68.0 / 2.920 | 3.840 |
| Uniform Acc / ECE / NLL | Exp. Acc / ECE / NLL | Best at | s/dec | |
|---|---|---|---|---|
| 1 | 64.7 / 0.318 / 4.553 | 64.7 / 0.318 / 4.553 | uniform 64.7 / 4.553 | 1.617 |
| 2 | 65.0 / 0.261 / 3.626 | 64.0 / 0.271 / 3.623 | logavg 65.3 / 4.150 | 2.534 |
| 3 | 65.0 / 0.270 / 3.557 | 65.0 / 0.271 / 3.561 | logavg 65.3 / 4.242 | 3.450 |
| 4 | 65.0 / 0.262 / 3.329 | 64.7 / 0.266 / 3.328 | logavg 65.3 / 4.123 | 4.367 |
| 5 | 65.7 / 0.254 / 3.286 | 65.7 / 0.255 / 3.287 | avg_last3 66.3 / 3.509 | 5.284 |
| 6 | 65.7 / 0.250 / 3.250 | 66.0 / 0.247 / 3.254 | exp10 66.0 / 3.254 | 6.201 |