Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

Agnibh Karmakar; Mayur Parvatikar; Shreyash Dhoot; Amit Dhanda; Aman Chadha; Kapil Wanaskar; Vinija Jain; Amitava Das

cs.CLSep 3, 2026

Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

Authors: Agnibh Karmakar, Mayur Parvatikar, Shreyash Dhoot, Amit Dhanda, Aman Chadha, Kapil Wanaskar, Vinija Jain, Amitava Das

Organizations: BITS Pilani Hyderabad Campus, India · RV College of Engineering, India · Pragya Lab, BITS Pilani Goa Campus, India · Amazon, USA ·

Abstract

Decode-time alignment methods steer a frozen language model by scoring candidate continuations with an external reward and selecting the maximiser. We argue that this shared design is a single degenerate point in a much larger space. We introduce Swiss-Knife, a framework for externalised multi-objective alignment in which the alignment specification is a first-class runtime object: hot-swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule. Six axioms characterise the admissible aggregation operators, and we prove a representation theorem: every operator satisfying them has the form $R_i = \sum_{j \neq i} g((μ_i - μ_j)/s(σ_i,σ_j))$ , a two-parameter family containing probit and logistic comparison rules and pointwise argmax as named coordinates. Within it, pairwise aggregation is Lipschitz-stable under adversarial reward contamination while argmax is not, and Candidate-Batch Normalization (CBN) makes the weight simplex invariant to the rescalings under which reward models are only ever identified. Our reference instantiation pairs DPO-LoRA blades with an uncertainty-aware pairwise tournament. Sweeping the helpfulness/honesty/harmlessness simplex, it attains the best balanced frontier of six decode-time methods (harmonic $F_1$ 0.797 vs. 0.750 for the strongest baseline, $p < 10^{-14}$ ) with the lowest refusal rate and highest helpfulness of any arm, and reconfigures its objectives in 0.05 ms with no gradient computation. Ablating CBN costs 0.217 $F_1$ , and the metadata confirms the predicted mechanism: the lowest-variance blade retains 9% of its nominal 33% influence, and the collapse follows the predicted ordering.

Figures & tables

Panel of Figure 1 — Figure 1: Two frozen models and one replaceable part. The drafter proposes $N{=}7$ candidate steps; three swappable LoRA blades score each one as $(\mu_{k},\sigma_{k})$ . Because $\mu$ is a blade-versus-backbone log-ratio, the scores come from the blades, not the backbone. Swapping a blade or re-weighting $\mathbf{w}$ retrains nothing.

Family	Examples	Where the specification lives, and how objectives combine
Static	RLHF/PPO, DPO	In the weights $\theta$ ; not combined, one objective per checkpoint
Conditioned	SteerLM	In $\theta$ , keyed by prompt attributes; reachable set fixed at training time
$\pi_{B}$ is frozen below this line
Scored	Best-of- $N$ , ARGS, FUDGE, contrastive	In an external scorer over candidates; pre-combined into one scalar before decoding
Mixed	MOD, Rewarded Soups	In a fixed combiner over $K$ models; linear, un-normalised mix of distributions or of weights
Compared	Swiss-Knife	In a comparison over the candidate batch; pairwise, on score and dispersion, CBN-normalised

Table 1: The design space, ordered by where the alignment specification lives . Above the first line, changing it means retraining. Below, the model is frozen and the specification is a separate object. The second line is this paper’s subject: every method above it reduces its objectives to one number before choosing a candidate, throwing away how the candidates compare and how sure each blade is.

Figure 2: (a) The HHH frontier, one marker per weight configuration; grey stems drop to the safety floor to show depth. Higher is better on all three axes, so the best region is the far upper corner. (b) Aggregate harmonic

F_{1}

; the bars also give the colour key for (a).

Method	Qual.	Safe.	Hon.	$F_{1}$	$\Delta$	HV	Win%
Swiss-Knife	$\mathbf{0.800}$	$0.964$	$\mathbf{0.677}$	$\mathbf{0.797}$	$\mathbf{0.002}$	$\mathbf{0.580}$	n/a
Base	$0.696$	$0.975$	$0.651$	$0.750$	$0.006$	$0.495$	$62.7$
BoN	$0.674$	$\mathbf{0.977}$	$0.651$	$0.742$	$0.005$	$0.471$	$65.8$
RS	$0.650$	$0.964$	$0.602$	$0.708$	$0.087$	$0.506$	$67.2$
ARGS	$0.607$	$0.972$	$0.586$	$0.685$	$0.007$	$0.396$	$72.1$
MOD	$0.348$	$0.991$	$0.410$	$0.474$	$0.007$	$0.163$	$88.6$

Table 2: HHH frontier,

120

prompts

\times

7

configurations.

\Delta

is Schott’s spacing (lower = more even coverage), HV the dominated hypervolume. Win% is the share of paired responses on which Swiss-Knife scores higher. BoN = Best-of-

N

, RS = Rewarded Soups.

$\mathbf{w}$	SK	Base	BoN	RS	ARGS	MOD	noCBN
$1/0/0$	$\mathbf{.783}$	$.767$	$.747$	$.688$	$.670$	$.494$	$.557$
$0/1/0$	$\mathbf{.780}$	$.739$	$.737$	$.720$	$.685$	$.458$	$.573$
$0/0/1$	$\mathbf{.800}$	$.747$	$.745$	$.543$	$.675$	$.473$	$.626$
$\frac{1}{2}/\frac{1}{2}/0$	$\mathbf{.808}$	$.737$	$.740$	$.714$	$.697$	$.484$	$.578$
$\frac{1}{2}/0/\frac{1}{2}$	$\mathbf{.801}$	$.761$	$.740$	$.760$	$.682$	$.487$	$.603$
$0/\frac{1}{2}/\frac{1}{2}$	$\mathbf{.807}$	$.748$	$.731$	$.732$	$.678$	$.479$	$.557$

Table 3: Harmonic

F_{1}

at every weight configuration

\mathbf{w}=(w_{\text{help}},w_{\text{hon}},w_{\text{harm}})

; best per row in bold. Range is the spread across the simplex. SK = Swiss-Knife, BoN = Best-of-

N

, RS = Rewarded Soups; the no-CBN arm is on the

60

-prompt half.

$\mathbf{w}$	Qual.	Safe.	Hon.	$\mathbf{w}$	Qual.	Safe.	Hon.
$1/0/0$	$.799$	$.955$	$.651$	$\frac{1}{2}/\frac{1}{2}/0$	$.821$	$.962$	$.687$
$0/1/0$	$.787$	$.965$	$.649$	$\frac{1}{2}/0/\frac{1}{2}$	$.787$	$.968$	$.694$
$0/0/1$	$.799$	$.973$	$.680$	$0/\frac{1}{2}/\frac{1}{2}$	$.807$	$.965$	$.694$
				$\frac{1}{3}/\frac{1}{3}/\frac{1}{3}$	$.801$	$.960$	$.685$

Table 4: Swiss-Knife’s achieved axes across the simplex, in two parallel blocks;

F_{1}

for these configurations is the Swiss-Knife column of Table 3 .

Axis	CBN	no	$\Delta$ [95% CI]	$p$
$F_{1}$	$\mathbf{0.741}$	$0.524$	$+0.217\ [+0.187,+0.246]$	$6{\times}10^{-35}$
Quality	$\mathbf{0.792}$	$0.596$	$+0.196\ [+0.169,+0.224]$	$1{\times}10^{-32}$
Honesty	$\mathbf{0.660}$	$0.414$	$+0.246\ [+0.213,+0.278]$	$2{\times}10^{-35}$
Safety	$0.963$	$0.971$	$-0.009\ [-0.021,+0.003]$	$0.22$

Table 5: CBN ablation, paired by prompt on the

60

-prompt stratified half (

n=420

cells). Positive

\Delta

favours CBN.

Figure 3: Why CBN matters. (a) The three blades’ reward spreads within a batch. (b) The influence shares these imply without CBN at

w_{k}=1/3

(Thm. 3 (i)): the honesty blade is nearly silenced. Table 5 shows the result.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$\rho$	Blades $\mathcal{B}$	$\mathcal{N}$	$\mathcal{A}$	$\mathcal{S}$
Best-of- $N$	response	single reward model	id	id	$\arg\max$
ARGS	token	reward head $+$ logits	id	id	$\arg\max$ / top- $k$
FUDGE	token	future discriminator	id	id	reweighted sample
Contrastive dec.	token	expert $-$ amateur	id	id	$\arg\max$
Rewarded Soups	response	weight-space average	id	id	sample
MOD	token	$K$ model log-probs	fixed $\mathbf{w}$ , no norm.	id	nucleus sample

Appendix

Table 6: Published decode-time methods recovered as instantiations of Definition 1.

Method	$n$	Quality	Safety	Honesty	$F_{1}$	$\delta F_{1}$ [95% CI]	$p$
Swiss-Knife	840	$0.800$	$0.964$	$0.677$	$0.797$	–	–
Base (frozen)	840	$0.696$	$0.975$	$0.651$	$0.750$	$-0.0478$ $[-0.0635,\,-0.0325]$	$2.46{\times}10^{-15}$
Best-of-N	839	$0.674$	$0.977$	$0.651$	$0.742$	$-0.0534$ $[-0.0687,\,-0.0376]$	$2.33{\times}10^{-18}$
Rewarded Soups	839	$0.650$	$0.964$	$0.602$	$0.708$	$-0.0991$ $[-0.1166,\,-0.0812]$	$5.03{\times}10^{-30}$
ARGS	840	$0.607$	$0.972$	$0.586$	$0.685$	$-0.1208$ $[-0.1395,\,-0.1023]$	$4.09{\times}10^{-40}$
Swiss-Knife (no CBN)	420	$0.596$	$0.971$	$0.414$	$0.586$	$-0.2168$ $[-0.2467,\,-0.1872]$	$6.11{\times}10^{-35}$

Appendix

Table 7: Complete per-method results across the HHH simplex.

n

is the number of judged response cells; Best-of-

N

and Rewarded Soups lose one cell each to a failed judgement, and the no-CBN arm runs on the stratified 60-prompt half.

F_{1}

is the harmonic mean of the three mean axes.

\delta F_{1}

is paired against Swiss-Knife on (configuration, prompt), with a

10{,}000

-sample bootstrap CI and a Wilcoxon signed-rank

p

Method	$\Delta$	HV	$\|P\|$
Swiss-Knife	$0.0024$	$0.580$	4
Base (frozen)	$0.0060$	$0.495$	4
Best-of-N	$0.0052$	$0.471$	4
Rewarded Soups	$0.0865$	$0.506$	5
ARGS	$0.0071$	$0.396$	2
Swiss-Knife (no CBN)	$0.0065$	$0.299$	4

Appendix

Table 8: Frontier geometry over the seven simplex points.

\Delta

is Schott’s spacing (lower is more even coverage), HV the dominated hypervolume,

|P|

the number of weight configurations lying on the Pareto front.

Method	$\mathbf{w}$	$n$	Quality	Safety	Honesty	$F_{1}$
Swiss-Knife	( $1.00$ , $0.00$ , $0.00$ )	120	$0.799$	$0.955$	$0.651$	$0.783$
	( $0.00$ , $1.00$ , $0.00$ )	120	$0.787$	$0.965$	$0.649$	$0.780$
	( $0.00$ , $0.00$ , $1.00$ )	120	$0.799$	$0.973$	$0.680$	$0.800$
	( $0.50$ , $0.50$ , $0.00$ )	120	$0.821$	$0.962$	$0.687$	$0.808$
	( $0.50$ , $0.00$ , $0.50$ )	120	$0.787$	$0.968$	$0.694$	$0.801$
	( $0.00$ , $0.50$ , $0.50$ )	120	$0.807$	$0.965$	$0.694$	$0.807$

Appendix

Table 9: Achieved axes and harmonic

F_{1}

at every point of the weight simplex, for every method. Weight vectors are

(w_{\mathrm{help}},w_{\mathrm{hon}},w_{\mathrm{harm}})

. The no-CBN arm was run on the stratified 60-prompt half, hence

n=60

Arm	Axis	$r_{s}$	$p$	slope	range
Swiss-Knife	quality	$0.206$	$0.658$	$0.0041$	$0.0342$
	safety	$-0.094$	$0.842$	$-0.0013$	$0.0179$
	honesty	$0.468$	$0.290$	$0.0262$	$0.0444$
	pooled ( $n{=}21$ )	$0.212$	$0.356$	$0.541$	–
Swiss-Knife (no CBN)	quality	$-0.805$	$0.029$	$-0.0389$	$0.0633$
	safety	$-0.019$	$0.968$	$-0.0017$	$0.0317$

Appendix

Table 10: Steerability: rank association between the requested weight on an objective and the achieved score on the corresponding axis, across the seven simplex points.

r_{s}

is Spearman’s coefficient; range is the spread of the achieved axis. Seven points per axis is individually underpowered, which is why the pooled statistic is the one quoted in the paper.

Multi-Objective Language Model Alignment

Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.

See all 92 papers

Safe Inference-Time Alignment via Lagrangian Reward Augmentation
Jul 2, 2026Yaswanth Chittepu, Ativ Joshi, Sohini Chintala +1LLM Alignment LLM Safety Alignment
University of Massachusetts Amherst · Independent Researcher
Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment
Mar 5, 2026Ved Sriraman, Adam BlockBest-of-N Sampling Inference-Time Language Model Alignment
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University
DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding
Mar 9, 2026Mingxi Zou, Jiaxiang Chen, Junfan Li +4Preference Alignment Inference-Time Language Model Alignment
Fudan University, Shanghai, China · Independent Researcher · Meta AI +1

Jul 2, 2026J/K move · Enter open · S save

Safe Inference-Time Alignment via Lagrangian Reward Augmentation

Yaswanth Chittepu, Ativ Joshi, Sohini Chintala, Scott Niekum

University of Massachusetts Amherst · Independent Researcher

LLM Alignment LLM Safety Alignment Constrained Optimization Multi-Objective Language Model Alignment Inference-Time Language Model Alignment Inference-Time Guidance

Open paper AskPDF arXiv

Inference-time alignment steers a frozen language model during decoding using auxiliary reward signals, avoiding the cost of repeated weight updates. However, existing inference-time alignment methods typically optimize a single scalar score, so explicit safety constraints must either be ignored or encoded through manually tuned penalties. We propose Lagrangian Reward Augmentation (LARA), a general inference-time alignment framework under safety constraints. Starting from a KL-regularized constrained objective with a reward model and a cost model, LARA dualizes the constraint and reduces the optimization problem to a one-dimensional convex problem over a nonnegative dual variable. Estimated on a small calibration set, this dual variable defines an augmented reward that can be used as a drop-in scoring signal within existing inference-time alignment methods. For sequence-level sampling methods, such as Best-of-N reranking, the calibrated dual variable corresponds to the solution of the expected-cost constrained problem. For token-level reward-guided decoding methods, the same construction yields a principled dual-calibrated heuristic rather than an exact constrained-policy guarantee. We evaluate LARA on both sequence-level and token-level inference-time alignment methods, and find that LARA improves the helpfulness-harmlessness tradeoff, with Best-of-N achieving the best performance among inference-time methods, approaching finetuning-based direct alignment baselines.

Abstract

Multi-Objective Language Model Alignment

Explore similar work

Safe Inference-Time Alignment via Lagrangian Reward Augmentation

Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment

DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding