Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time
Authors: Agnibh Karmakar, Mayur Parvatikar, Shreyash Dhoot, Amit Dhanda, Aman Chadha, Kapil Wanaskar, Vinija Jain, Amitava Das
Organizations: BITS Pilani Hyderabad Campus, India · RV College of Engineering, India · Pragya Lab, BITS Pilani Goa Campus, India · Amazon, USA · Apple, USA · Canva, USA · Google, USA
Decode-time alignment methods steer a frozen language model by scoring candidate continuations with an external reward and selecting the maximiser. We argue that this shared design is a single degenerate point in a much larger space. We introduce Swiss-Knife, a framework for externalised multi-objective alignment in which the alignment specification is a first-class runtime object: hot-swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule. Six axioms characterise the admissible aggregation operators, and we prove a representation theorem: every operator satisfying them has the form Ri=∑j=ig((μi−μj)/s(σi,σj)), a two-parameter family containing probit and logistic comparison rules and pointwise argmax as named coordinates. Within it, pairwise aggregation is Lipschitz-stable under adversarial reward contamination while argmax is not, and Candidate-Batch Normalization (CBN) makes the weight simplex invariant to the rescalings under which reward models are only ever identified. Our reference instantiation pairs DPO-LoRA blades with an uncertainty-aware pairwise tournament. Sweeping the helpfulness/honesty/harmlessness simplex, it attains the best balanced frontier of six decode-time methods (harmonic F1 0.797 vs. 0.750 for the strongest baseline, p<10−14) with the lowest refusal rate and highest helpfulness of any arm, and reconfigures its objectives in 0.05 ms with no gradient computation. Ablating CBN costs 0.217 F1, and the metadata confirms the predicted mechanism: the lowest-variance blade retains 9% of its nominal 33% influence, and the collapse follows the predicted ordering.
Figures & tables
Figure 1: Two frozen models and one replaceable part. The drafter proposes N=7 candidate steps; three swappable LoRA blades score each one as (μk,σk) . Because μ is a blade-versus-backbone log-ratio, the scores come from the blades, not the backbone. Swapping a blade or re-weighting w retrains nothing.
Family
Examples
Where the specification lives, and how objectives combine
Static
RLHF/PPO, DPO
In the weights θ ; not combined, one objective per checkpoint
Conditioned
SteerLM
In θ , keyed by prompt attributes; reachable set fixed at training time
πB is frozen below this line
Scored
Best-of- N , ARGS, FUDGE, contrastive
In an external scorer over candidates; pre-combined into one scalar before decoding
Mixed
MOD, Rewarded Soups
In a fixed combiner over K models; linear, un-normalised mix of distributions or of weights
Compared
Swiss-Knife
In a comparison over the candidate batch; pairwise, on score and dispersion, CBN-normalised
Table 1: The design space, ordered by where the alignment specification lives . Above the first line, changing it means retraining. Below, the model is frozen and the specification is a separate object. The second line is this paper’s subject: every method above it reduces its objectives to one number before choosing a candidate, throwing away how the candidates compare and how sure each blade is.
Figure 2: (a) The HHH frontier, one marker per weight configuration; grey stems drop to the safety floor to show depth. Higher is better on all three axes, so the best region is the far upper corner. (b) Aggregate harmonic F1 ; the bars also give the colour key for (a).
Method
Qual.
Safe.
Hon.
F1
Δ
HV
Win%
Swiss-Knife
0.800
0.964
0.677
0.797
0.002
0.580
n/a
Base
0.696
0.975
0.651
0.750
0.006
0.495
62.7
BoN
0.674
0.977
0.651
0.742
0.005
0.471
65.8
RS
0.650
0.964
0.602
0.708
0.087
0.506
67.2
ARGS
0.607
0.972
0.586
0.685
0.007
0.396
72.1
MOD
0.348
0.991
0.410
0.474
0.007
0.163
88.6
Table 2: HHH frontier, 120 prompts ×7 configurations. Δ is Schott’s spacing (lower = more even coverage), HV the dominated hypervolume. Win% is the share of paired responses on which Swiss-Knife scores higher. BoN = Best-of- N , RS = Rewarded Soups.
w
SK
Base
BoN
RS
ARGS
MOD
noCBN
1/0/0
.783
.767
.747
.688
.670
.494
.557
0/1/0
.780
.739
.737
.720
.685
.458
.573
0/0/1
.800
.747
.745
.543
.675
.473
.626
21/21/0
.808
.737
.740
.714
.697
.484
.578
21/0/21
.801
.761
.740
.760
.682
.487
.603
0/21/21
.807
.748
.731
.732
.678
.479
.557
Table 3: Harmonic F1 at every weight configuration w=(whelp,whon,wharm) ; best per row in bold. Range is the spread across the simplex. SK = Swiss-Knife, BoN = Best-of- N , RS = Rewarded Soups; the no-CBN arm is on the 60 -prompt half.
w
Qual.
Safe.
Hon.
w
Qual.
Safe.
Hon.
1/0/0
.799
.955
.651
21/21/0
.821
.962
.687
0/1/0
.787
.965
.649
21/0/21
.787
.968
.694
0/0/1
.799
.973
.680
0/21/21
.807
.965
.694
31/31/31
.801
.960
.685
Table 4: Swiss-Knife’s achieved axes across the simplex, in two parallel blocks; F1 for these configurations is the Swiss-Knife column of Table 3 .
Axis
CBN
no
Δ [95% CI]
p
F1
0.741
0.524
+0.217[+0.187,+0.246]
6×10−35
Quality
0.792
0.596
+0.196[+0.169,+0.224]
1×10−32
Honesty
0.660
0.414
+0.246[+0.213,+0.278]
2×10−35
Safety
0.963
0.971
−0.009[−0.021,+0.003]
0.22
Table 5: CBN ablation, paired by prompt on the 60 -prompt stratified half ( n=420 cells). Positive Δ favours CBN.
Figure 3: Why CBN matters. (a) The three blades’ reward spreads within a batch. (b) The influence shares these imply without CBN at wk=1/3 (Thm. 3 (i)): the honesty blade is nearly silenced. Table 5 shows the result.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Method
ρ
Blades B
N
A
S
Best-of- N
response
single reward model
id
id
argmax
ARGS
token
reward head + logits
id
id
argmax / top- k
FUDGE
token
future discriminator
id
id
reweighted sample
Contrastive dec.
token
expert − amateur
id
id
argmax
Rewarded Soups
response
weight-space average
id
id
sample
MOD
token
K model log-probs
fixed w , no norm.
id
nucleus sample
Appendix
Table 6: Published decode-time methods recovered as instantiations of Definition 1.
Method
n
Quality
Safety
Honesty
F1
δF1 [95% CI]
p
Swiss-Knife
840
0.800
0.964
0.677
0.797
–
–
Base (frozen)
840
0.696
0.975
0.651
0.750
−0.0478[−0.0635,−0.0325]
2.46×10−15
Best-of-N
839
0.674
0.977
0.651
0.742
−0.0534[−0.0687,−0.0376]
2.33×10−18
Rewarded Soups
839
0.650
0.964
0.602
0.708
−0.0991[−0.1166,−0.0812]
5.03×10−30
ARGS
840
0.607
0.972
0.586
0.685
−0.1208[−0.1395,−0.1023]
4.09×10−40
Swiss-Knife (no CBN)
420
0.596
0.971
0.414
0.586
−0.2168[−0.2467,−0.1872]
6.11×10−35
Appendix
Table 7: Complete per-method results across the HHH simplex. n is the number of judged response cells; Best-of- N and Rewarded Soups lose one cell each to a failed judgement, and the no-CBN arm runs on the stratified 60-prompt half. F1 is the harmonic mean of the three mean axes. δF1 is paired against Swiss-Knife on (configuration, prompt), with a 10,000 -sample bootstrap CI and a Wilcoxon signed-rank p .
Method
Δ
HV
∣P∣
Swiss-Knife
0.0024
0.580
4
Base (frozen)
0.0060
0.495
4
Best-of-N
0.0052
0.471
4
Rewarded Soups
0.0865
0.506
5
ARGS
0.0071
0.396
2
Swiss-Knife (no CBN)
0.0065
0.299
4
Appendix
Table 8: Frontier geometry over the seven simplex points. Δ is Schott’s spacing (lower is more even coverage), HV the dominated hypervolume, ∣P∣ the number of weight configurations lying on the Pareto front.
Method
w
n
Quality
Safety
Honesty
F1
Swiss-Knife
( 1.00 , 0.00 , 0.00 )
120
0.799
0.955
0.651
0.783
( 0.00 , 1.00 , 0.00 )
120
0.787
0.965
0.649
0.780
( 0.00 , 0.00 , 1.00 )
120
0.799
0.973
0.680
0.800
( 0.50 , 0.50 , 0.00 )
120
0.821
0.962
0.687
0.808
( 0.50 , 0.00 , 0.50 )
120
0.787
0.968
0.694
0.801
( 0.00 , 0.50 , 0.50 )
120
0.807
0.965
0.694
0.807
Appendix
Table 9: Achieved axes and harmonic F1 at every point of the weight simplex, for every method. Weight vectors are (whelp,whon,wharm) . The no-CBN arm was run on the stratified 60-prompt half, hence n=60 .
Arm
Axis
rs
p
slope
range
Swiss-Knife
quality
0.206
0.658
0.0041
0.0342
safety
−0.094
0.842
−0.0013
0.0179
honesty
0.468
0.290
0.0262
0.0444
pooled ( n=21 )
0.212
0.356
0.541
–
Swiss-Knife (no CBN)
quality
−0.805
0.029
−0.0389
0.0633
safety
−0.019
0.968
−0.0017
0.0317
Appendix
Table 10: Steerability: rank association between the requested weight on an objective and the achieved score on the corresponding axis, across the seven simplex points. rs is Spearman’s coefficient; range is the spread of the achieved axis. Seven points per axis is individually underpowered, which is why the pooled statistic is the one quoted in the paper.
Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.
Inference-time alignment steers a frozen language model during decoding using auxiliary reward signals, avoiding the cost of repeated weight updates. However, existing inference-time alignment methods typically optimize a single scalar score, so explicit safety constraints must either be ignored or encoded through manually tuned penalties. We propose Lagrangian Reward Augmentation (LARA), a general inference-time alignment framework under safety constraints. Starting from a KL-regularized constrained objective with a reward model and a cost model, LARA dualizes the constraint and reduces the optimization problem to a one-dimensional convex problem over a nonnegative dual variable. Estimated on a small calibration set, this dual variable defines an augmented reward that can be used as a drop-in scoring signal within existing inference-time alignment methods. For sequence-level sampling methods, such as Best-of-N reranking, the calibrated dual variable corresponds to the solution of the expected-cost constrained problem. For token-level reward-guided decoding methods, the same construction yields a principled dual-calibrated heuristic rather than an exact constrained-policy guarantee. We evaluate LARA on both sequence-level and token-level inference-time alignment methods, and find that LARA improves the helpfulness-harmlessness tradeoff, with Best-of-N achieving the best performance among inference-time methods, approaching finetuning-based direct alignment baselines.
Yaswanth Chittepu, Ativ Joshi, Sohini Chintala +1
University of Massachusetts Amherst · Independent Researcher
Best-of-N (BoN) sampling is a widely used inference-time alignment method for language models, whereby N candidate responses are sampled from a reference model and the one with the highest predicted reward according to a learned reward model is selected. Despite its widespread practical use, recent theoretical work has suggested that it is statistically suboptimal and vulnerable to reward hacking, the process by which models exploit weaknesses in the learned reward model to achieve high estimated reward without genuinely improving performance. We revisit this question under assumptions that more closely reflect practice than that of prior work. In particular, in contradistinction to earlier analyses that focused on expected true reward, which may not be meaningful in many practical settings, we investigate how inference-time alignment affects the win-rate, a pairwise comparison-based metric more closely aligned with how reward models are trained and evaluated in practice. We demonstrate that, under minimal conditions on the quality of the reference model and learned reward model, properly tuned BoN is both computationally and statistically optimal in achieving high win-rate, partially explaining its widespread practical success. Because BoN remains susceptible to reward-hacking in this setting, we propose a simple and practical variant that provably eliminates reward-hacking while maintaining optimal statistical performance. Finally, we show that prior approaches are provably suboptimal when considering win-rate, highlighting the importance of choosing appropriate objectives when analyzing inference-time alignment methods.
Ved Sriraman, Adam Block
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University
Preference-based alignment methods (e.g., RLHF, DPO) typically optimize a single scalar objective, implicitly averaging over heterogeneous human preferences. In practice, systematic annotator and user-group disagreement makes mean-reward maximization brittle and susceptible to proxy over-optimization. We propose Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC), a retraining-free inference-time method that frames response selection as distributionally robust, risk-sensitive decision making. Given multiple preference samples or scalable disagreement proxies, DARC reranks candidates by maximizing a KL-robust (entropic) satisfaction objective, and provides simple deployment controls that cap or penalize the corresponding entropic risk premium relative to the mean, enabling explicit risk budgets without retraining. We provide theoretical characterization linking this decoding rule to principled pessimism and KL-based distributionally robust optimization. Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality under noisy, heterogeneous feedback.
Mingxi Zou, Jiaxiang Chen, Junfan Li +4
Fudan University, Shanghai, China · Independent Researcher · Meta AI +1