PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decoding

Baolong Bi; Yuyao Ge; Shenghua Liu; Yuchen He; Siqian Tong; Lizhe Chen; Lingrui Mei; Zehao Li

cs.AIFeb 24, 2026

PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decoding

Authors: Baolong Bi, Yuyao Ge, Shenghua Liu, Yuchen He, Siqian Tong, Lizhe Chen, Lingrui Mei, Zehao Li,

Organizations: Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · University of Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China · Shenzhen International Graduate School, Tsinghua University, Shenzhen, China ·

Abstract

Reliable AI systems require large language models (LLMs) to exhibit behaviors aligned with human preferences and values. However, most existing alignment approaches operate at training time and rely on additional high-quality data, incurring significant computational and annotation costs. While recent work has shown that contrastive decoding can leverage a model's internal distributions to improve specific capabilities, its applicability remains limited to narrow behavioral scopes and scenarios. In this work, we introduce Polarity-Prompt Contrastive Decoding (PromptCD), a test-time behavior control method that generalizes contrastive decoding to broader enhancement settings. PromptCD constructs paired positive and negative guiding prompts for a target behavior and contrasts model responses-specifically token-level probability distributions in LLMs and visual attention patterns in VLMs-to reinforce desirable outcomes. This formulation extends contrastive decoding to a wide range of enhancement objectives and is applicable to both LLMs and Vision-Language Models (VLMs) without additional training. For LLMs, experiments on the "3H" alignment objectives (helpfulness, honesty, and harmlessness) demonstrate consistent and substantial improvements, indicating that post-trained models can achieve meaningful self-enhancement purely at test time. For VLMs, we further analyze contrastive effects on visual attention, showing that PromptCD significantly improves VQA performance by reinforcing behavior-consistent visual grounding. Collectively, these results highlight PromptCD as a simple, general, and cost-efficient strategy for reliable behavior control across modalities.

Figures & tables

Fig. 1: Overview of PromptCD. A unified framework for test-time behavior enhancement. By contrasting the logits (for text) or visual attention maps (for images) induced by positive and negative prompts, PromptCD effectively steers models toward desired goals without retraining. We demonstrate this capability through faithfulness enhancement in LLMs and visual attentiveness improvement in VLMs.

Fig. 2: Comparison of contextual and parametric knowledge shifts under positive and negative prompts. The first discriminative tokens are identified to represent the corresponding knowledge, and their logit and rank distributions are tracked to illustrate the underlying distributional dynamics.

Fig. 3: Selected cases showing changes in the first token for both parametric and contextual knowledge. The cases are obtained by introducing positive prompts carrying counterfactual information into the LLaMA2-7B-chat model. ’

\rightarrow

’ denotes the knowledge shift after incorporating positive prompts. ’Logits’ and ‘rank’ refer to the first discriminative token in the knowledge answer, indicating the model’s knowledge confidence.

Fig. 4: Illustration of PromptCD enhancing context-faithfulness. By contrasting the probability distributions induced by positive and negative prompts, PromptCD amplifies contextual signals, guiding the LLM to generate faithful responses to the input question.

Fig. 5: The PromptCD framework for cross-modal attention refinement. By contrasting the task-specific attention distribution $A^{(P)}$ (derived from positive prompts) against the general attention pattern $A^{(N)}$ (derived from negative prompts), the method computes a refined attention map $\hat{A}$ . This contrastive mechanism effectively isolates and highlights request-relevant visual regions, thereby enhancing visual grounding.

Model	Method	NQ			ConFiQA			CoConflictQA
Model	Method	ConR $(\uparrow)$	ParR $(\downarrow)$	MR $(\downarrow)$	ConR $(\uparrow)$	ParR $(\downarrow)$	MR $(\downarrow)$	ConR $(\uparrow)$	ParR $(\downarrow)$	MR $(\downarrow)$
LLaMA2-7B-chat	Vanilla	43.35	43.75	50.24	69.66	28.14	28.78	62.07	18.76	23.20
	Attr	64.39	21.99	25.46	82.03	18.76	18.61	65.46	13.97	17.58
	Opin	74.59	15.39	17.11	88.62	15.38	14.79	67.26	13.57	16.79
	PromptCD	75.99	6.39	7.76	90.42	13.76	13.21	70.25	11.16	13.71
LLaMA3-8B-instruct	Vanilla	43.92	34.05	43.47	54.23	22.39	29.22	68.46	29.74	30.29

TABLE I: Performance (%) of PromptCD on LLM helpfulness evaluation.

Model	Method	TruthfulQA(MC)			FActScore
Model	Method	MC1	MC2	MC3	Score	#facts
	Vanilla	33.66	51.29	24.91	33.20	25.60
LLaMA2-	DoLa	32.68	61.12	29.86	35.60	54.10
7B-chat	SLED	37.08	63.83	32.92	36.00	32.40
7B-chat	RECITE	32.19	50.38	25.06	36.10	32.40
	PromptCD	42.59	64.05	33.89	36.60	46.40

TABLE II: Performance (%) of PromptCD on LLM honesty evaluation.

Method	Mode	Defense	Defense Generalization				Fluency
Method	Mode	Success	onlyQ	otherA	otherQ	otherAQ	Fluency
	Vanilla	40.59	76.96	22.41	41.33	21.78	7.01
LLaMA2-	FT-L	42.52	76.22	28.37	42.89	28.85	5.94
7B-chat	DINM	90.00	89.93	59.26	87.41	57.63	6.61
	PromptCD	96.67	98.67	82.30	96.59	82.37	7.29
	Vanilla	12.15	54.81	18.07	12.07	18.74	7.46

TABLE III: Performance (%) of PromptCD on LLM harmlessness evaluation.

Fig. 6: Average performance across the three “3H” dimensions on different models. PromptCD consistently yields substantial improvements over the original models, while the baselines (Opin, DoLa, and DINM) show relatively limited gains.

Model	Layers $\mathcal{L}$	A-OKVQA	POPE	V ∗	TextVQA
Qwen2.5-VL-3B	Vanilla	73.0	86.9	50.3	72.8
	[10, 15]	74.0	86.9	53.4	73.0
	[15, 20]	76.8	87.7	56.0	76.0
	[20, 25]	78.3	87.9	57.1	76.3
Qwen2.5-VL-7B	Vanilla	75.0	87.0	50.8	75.0
	[10, 15]	75.0	87.0	51.3	75.0

TABLE IV: Performance (%) of PromptCD on VLMs across different layer intervals.

Model	$\gamma=0.2$	$\gamma=0.5$	$\gamma=0.8$
Helpfulness (ConR)	77.32	78.89	70.53
Honesty (MC-AVG)	39.59	46.84	44.87
Harmlessness (DG-AVG)	83.29	89.98	85.34

TABLE V: Ablation results of the adjustment coefficient

\gamma

on LLaMA2-7B-chat models across the ‘3H’ dimensions. Performance is reported using the average values of ConR, MC-AVG, and DG-AVG, respectively.

Fig. 7: Visualization of contrastive attention refinement in VQA. As the threshold $\tau$ decreases from 0.6 to 0.1, PromptCD progressively suppresses visual noise to isolate request-relevant regions. Our PromptCD precise grounding corrects initial hallucinations (Red $\times$ ) and yields accurate answers (Green $\checkmark$ ).

Method	Original	SAM	YOLO	CLIP	ViCrop			PromptCD
Method	Original	SAM	YOLO	CLIP	rel-att	grad-att	pure-grad	PromptCD
Accuracy	47.80	49.42	48.84	48.55	55.17	56.06	51.67	58.2
GPU Time	0.17	3.33	0.35	1.07	1.16	0.89	2.36	1.34

TABLE VI: Performance comparison of PromptCD against external tool-based approaches and ViCrop on TextVQA dataset: accuracy (%) and inference time (seconds) overhead per sample.

Model	Method	Latency (ms/token)	Throughput (token/s)
LLaMA2-7B	Vanilla	36.03 ( $\times$ 1.00)	27.76 ( $\times$ 1.00)
LLaMA2-7B	PromptCD	64.85 ( $\times$ 1.79)	16.64 ( $\times$ 0.60)
LLaMA2-13B	Vanilla	51.41 ( $\times$ 1.00)	19.45 ( $\times$ 1.00)
LLaMA2-13B	PromptCD	89.45 ( $\times$ 1.74)	12.25 ( $\times$ 0.63)
LLaMA3-8B	Vanilla	42.17 ( $\times$ 1.00)	23.92 ( $\times$ 1.00)
LLaMA3-8B	PromptCD	69.77 ( $\times$ 1.65)	14.08 ( $\times$ 0.59)

TABLE VII: Comparison of inference latency and throughput for PromptCD.

Model	Vanilla	PromptCD
Model	Vanilla	w/ APC	w/o APC
LLaMA2-7B	78.9	77.8	30.7
LLaMA3-8B	83.7	85.5	53.8
Mistralv0.3-7B	92.4	91.7	49.9
Qwen2.5-7B	88.8	89.8	62.4

TABLE VIII: Ablation study on the NQ dataset evaluating the effect of the Adaptive Plausibility Constraint (APC) on generation quality (Hit Rate).

Model	Method	MQuAKE
Model	Method	3k	2002	hard
LLaMA2-7B-chat	IKE	20.7	20.6	2.3
	+ PromptCD	22.4	20.4	3.8
	MeLLo	32.6	40.8	5.1
	+ PromptCD	43.1	45.8	5.8
LLaMA2-13B-chat	IKE	19.4	18.8	2.7

TABLE IX: Experimental results (accuracy %) using LLaMA2-chat models. We conduct experiments with full-batch edit memory to evaluate the performance of memory-based Knowledge Editing.

Fig. 8: Performance comparison (%) between standard IKE and IKE enhanced with PromptCD across various models and datasets. We set the edit memory batch size to 1 to evaluate the fundamental capability of direct knowledge editing.

Fig. 9: Probability statistics of edited new knowledge on the MQuAKE-Stubborn dataset using LLaMA2-7B-chat . The probabilities are computed by applying softmax to the model’s token logits.

Inference-Time Language Model Alignment

Inference-time language model alignment steers or selects outputs during generation to better meet safety, preference, and task goals without retraining.

See all 78 papers

TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models
Jun 2, 2026Wenbo Zhang, Jianxiong Li, Shuai Yang +4Test-Time Adaptation Test-Time Adaptation of VLMs
ByteDance Seed · The University of Adelaide · Tsinghua University +3
Post-training makes large language models less human-like
May 8, 2026Marcel Binz, Elif Akata, Abdullah Almaatouq +76LLM Evaluation LLM Alignment
ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment
Jan 6, 2026Kapil Wanaskar, Gaytri Jena, Vinija Jain +2LLM Alignment Inference-Time Language Model Alignment
San José de State University, USA · Pragya Lab, BITS Pilani Goa, India · Google, USA +1

Jun 2, 2026J/K move · Enter open · S save

TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models

Wenbo Zhang, Jianxiong Li, Shuai Yang, Sijin Chen+3

ByteDance Seed · The University of Adelaide · Tsinghua University+3

Test-Time Adaptation Test-Time Adaptation of VLMs Prompt Tuning Test-Time Training Vision-Language-Action Models Test-Time Adaptation of VLA Models+1

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models trained on large-scale data have made remarkable progress, but they remain vulnerable to distribution shifts at deployment time. Recent VLA models suggest that prompts can serve as an efficient interface for steering policy behavior, but existing prompt-based steering typically relies on external guidance. This raises a natural question: can test-time training (TTT) for VLA be achieved by optimizing a prompt, so that the steering interface itself can be learned and adapted from interaction? We address this question with TTT-VLA, a test-time training framework based on Latent Prompt Optimization (LPO). During training, the latent prompt is learned with an additional proxy task, providing an extra learned conditioning signal for policy learning. At test time, TTT is performed by collecting interaction data from the current environment and optimizing only the latent prompt on those data using the proxy task's self-supervised signal, without modifying the policy itself. Experiments on SimplerEnv demonstrate that the proposed method consistently improves task success rates in both single- and multi-embodiment settings. Further analysis shows that the gains arise primarily from correcting a small number of critical decisions rather than globally altering policy behavior. These results suggest that LPO provides an effective and practical pathway for deployment-time improvement of foundation manipulation policies.

Abstract

Inference-Time Language Model Alignment

Explore similar work

TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models

Post-training makes large language models less human-like

ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment