Improving JEV Entailment Reranking with Span Readouts and Selective Adaptation

Haocheng Mei; Yuchuan Tian; Ye Yuan; Enhua Wu; Chao Xu; Allen Chen; Kai Han; Yunhe Wang

cs.CLOct 26, 2026

Improving JEV Entailment Reranking with Span Readouts and Selective Adaptation

Authors: Haocheng Mei, Yuchuan Tian, Ye Yuan, Enhua Wu, Chao Xu, Allen Chen, Kai Han, Yunhe Wang

Organizations: Peking University · SUIBE · University of Macau · Tsinghua University ·

Abstract

JEV decision models pair a language model backbone with a lightweight scoring head to select among natural-language options. Direct selection compares candidate answers within one request, whereas entailment reranking evaluates each candidate in a separate natural language inference (NLI) request over fixed relation options. We examine whether the readout and adaptation choices used for direct selection remain effective for cross-request reranking. We combine a mean-span readout with selective backbone adaptation. Mean-span pooling replaces the option-ending pointer with an average of the contextualized states inside each NLI option, without adding head parameters. Selective adaptation retains low-rank updates in attention and gated DeltaNet projections while removing feed-forward adapters. On MC11, which averages accuracy over eleven reranking configurations from seven sources, selective adaptation improves NeoHorse-1-4B accuracy over the all-module baseline while cutting trainable parameters by more than half, and mean-span pooling yields further consistent gains at the same parameter count.

Figures & tables

Panel of Figure 1 — Figure 1: The final reranker. The candidate answer enters the instruction; pooling reads the three NLI option spans. The shared head scores those spans against the decision query, and the outer comparison selects an answer using its entailment probability.

Population and training	Pointer	Pooling	Pointer	Pooling	Direct	NLI
	Direct accuracy (%)		NLI accuracy (%)		Pooling minus pointer (pp)
PIQA, higher-rate models	79.33	79.20	65.15	73.22	$-0.12\!\pm\!1.41$	$+8.07\!\pm\!3.76$
SocialIQA, one epoch	73.76	71.43	61.12	61.40	$-2.33\!\pm\!0.64$	$+0.28\!\pm\!0.72$
SocialIQA, two epochs	73.67	71.81	56.82	58.84	$-1.86\!\pm\!0.23$	$+2.02\!\pm\!1.20$

Table 1: The request interface changes the readout comparison on the same questions and fixed checkpoints. Accuracies are four-setting means; differences include descriptive SEs. Neither source enters MC11.

	Adapter scope
Readout	All modules	No-FFN
Pointer	$50.28\!\pm\!1.02$	$52.49\!\pm\!0.46$
Mean-span	$51.00\!\pm\!0.21$	$\mathbf{53.39}\pm 0.24$
Pool $-$ pointer (pp)	$+0.73\!\pm\!1.07$	$+0.91\!\pm\!0.35$
Trainable parameters (M)	131.17	58.82

Table 2: MC11 accuracy (%) and trainable parameter count for the core readout-by-scope comparison. All four models use one epoch, lower head and adapter rates, and rank 64. Accuracies and paired changes are four-setting means with descriptive SEs. Both readouts have the same parameter count within each scope.

Parameter group	All modules	No-FFN
Attention adapters	12.58	12.58
DeltaNet adapters	44.92	44.92
FFN adapters	72.35	0
Bilinear head	1.31	1.31
Total trainable	131.17	58.82

Table 3: Trainable parameters in millions at rank 64, rounded independently. The pretrained backbone weights are frozen. Both readouts use the same bilinear head.

	Adapter peak rate
Head peak rate	Lower	Higher
Lower	$50.28\!\pm\!1.02$	$46.81\!\pm\!2.37$
Higher	$49.66\!\pm\!1.24$	$47.46\!\pm\!1.35$
	Training budget, both rates lower
Readout	One epoch	Two epochs
Pointer	$50.28\!\pm\!1.02$	$43.96\!\pm\!0.91$

Table 4: Training-recipe diagnostics on MC11 (%, mean

\pm

descriptive SE). The upper panel contains one-epoch pointers; the lower panel compares fresh runs with budget-specific schedule horizons. Lower/higher rates are

1.25\!\times\!10^{-5}

2.5\!\times\!10^{-5}

Panel of Figure 2 — Figure 2: Accuracy loss from one to two epochs under the lower-rate all-module recipe. Both readouts favor one epoch on ARC and HellaSwag.

Panel of Figure 3 — Figure 3: Task-level effects of the two design choices. Left: mean-span pooling minus the pointer under each adapter scope. Right: removing FFN adapters for each readout. Bars are means over four paired settings under the one-epoch, lower-rate recipe. All configurations retain their original weight in MC11.

Factor	Paired comparison	PIQA (pp)	CommonsenseQA (pp)
Learning rate	Lower-rate pointer $-$ higher-rate pointer	$+12.17\!\pm\!2.49$	$+11.65\!\pm\!1.57$
Readout	Pooling $-$ pointer, both lower-rate	$+1.26\!\pm\!0.85$	$+4.01\!\pm\!1.83$
Adapter scope	No-FFN pointer $-$ all-module pointer	$+5.21\!\pm\!1.15$	$+9.56\!\pm\!2.81$
Scope comparison accuracy: all-module pointer		75.04%	54.24%
Scope comparison accuracy: no-FFN pointer		80.25%	63.80%

Table 5: Additional-source support for individual design choices. Changes are paired four-setting means with descriptive SEs. The learning-rate comparison changes head and adapter rates together; the readout comparison uses the same model group. Adapter scope is tested separately with lower-rate, one-epoch pointers and matched initializations. These groups are separate from the core readout-by-scope models in Table 2 ; the final no-FFN pooling model has not been evaluated on these sources.

Core model	JevBench (%)	Kev (%)
All-module pointer	$77.11\!\pm\!0.84$	$81.37\!\pm\!0.59$
All-module pooling	$75.37\!\pm\!0.96$	$79.91\!\pm\!0.35$
No-FFN pointer	$77.52\!\pm\!0.98$	$79.78\!\pm\!0.35$
No-FFN pooling	$73.79\!\pm\!1.18$	$78.12\!\pm\!0.26$

Table 6: Direct-decision results for the core readout-by-scope comparison. Entries are four-setting means with descriptive SEs. MC11 results for these same models appear in Table 2 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Questions	Split	External candidates and demonstrations
ARC-Easy	2,376	Test	Choices supplied with each source question.
ARC-Challenge	1,172	Test	Choices supplied with each source question.
WinoGrande	1,267	Validation	Supplied alternatives, inserted into the incomplete sentence.
GSM8K, four options	1,319	Test	Correct numerical answer and three generated distractors.
GSM8K, ten options	1,319	Test	Correct numerical answer and nine generated distractors.
GPQA	198	Diamond	Supplied answers, with the correct answer first in loaded order.

Appendix

Table 7: MC11 populations, candidate construction, and demonstration sources.

	NeoHorse				Qwen3-4B
Configuration	Pool $-$ ptr. all	Pool $-$ ptr. no-FFN	No-FFN $-$ all pooling	Pool $-$ last-content all	Pool $-$ ptr. all
MMLU, few-shot	$-0.34$	$-0.38$	$+1.32$	$+0.94$	$+0.16$
MMLU	$+0.49$	$+0.46$	$+1.42$	$+1.75$	$+0.01$
HellaSwag	$+4.23$	$+4.21$	$+9.10$	$+8.10$	$+3.91$
GSM8K, ten options	$-1.48$	$-0.66$	$-0.30$	$-0.55$	$-1.84$
ARC-Easy	$+1.60$	$+0.99$	$+1.73$	$+4.08$	$+0.03$

Appendix

Table 8: Mean within-setting task differences in percentage points. Each column uses four paired settings; all models use one epoch and the lower rates. The second NeoHorse column compares readouts with no-FFN adapters; the third compares adapter scopes with pooling fixed. The Qwen3-4B column uses a separate pointer–pooling comparison. Numbers are computed before rounding.

Kev subset	Pool $-$ ptr., all	Pool $-$ ptr., no-FFN	Ptr.: no-FFN $-$ all	Pool: no-FFN $-$ all
Decision development	$+0.20\pm 0.24$	$+0.24\pm 0.25$	$-0.34\pm 0.17$	$-0.30\pm 0.33$
Decision test	$-0.06\pm 0.24$	$-0.90\pm 0.49$	$-0.92\pm 0.03$	$-1.75\pm 0.43$
Transfer-v4 development	$-2.44\pm 0.69$	$-2.13\pm 0.65$	$-2.71\pm 1.10$	$-2.40\pm 0.27$
Transfer-v4 test	$-1.56\pm 0.20$	$-1.68\pm 0.62$	$-1.87\pm 0.70$	$-1.98\pm 0.47$
Transfer-v9 development	$-3.23\pm 0.53$	$-3.30\pm 0.13$	$-2.25\pm 0.51$	$-2.32\pm 0.37$
Transfer-v9 test	$-1.70\pm 0.06$	$-2.20\pm 0.29$	$-1.48\pm 0.33$	$-1.98\pm 0.39$

Appendix

Table 9: The six Kev selections in the core readout-by-scope comparison. Entries are paired mean changes in pp with descriptive SEs. The direct-decision cost includes changes on decision-test as well as the transfer suites.

Pointer recipe or paired change	MC11	JevBench	Kev
Joint training, higher rate (accuracy, %)	45.90	75.62	81.47
Lower-rate joint $-$ higher-rate joint (pp)	$+4.43\pm 1.03$	$+2.35\pm 0.67$	$-0.43\pm 0.53$
Head-first $-$ higher-rate joint (pp)	$+6.38\pm 2.17$	$+2.03\pm 0.84$	$-1.02\pm 0.28$
No-FFN $-$ higher-rate joint (pp)	$+3.67\pm 1.33$	$+1.80\pm 1.53$	$-0.02\pm 0.34$
Head-first + no-FFN $-$ higher-rate joint (pp)	$+7.64\pm 1.91$	$+0.96\pm 0.66$	$-1.95\pm 0.40$
Head-only tail $-$ higher-rate joint (pp)	$+3.84\pm 0.88$	$-0.31\pm 1.40$	$-0.30\pm 0.23$

Appendix

Table 10: Effects of update timing and adapter placement in a separately trained set of pointers. All recipes use the option-ending pointer. Except for the explicit lower-rate row, both peak rates are

2.5\times 10^{-5}

. Differences are means

\pm

descriptive SE across four settings. Head initialization can vary when adapter scope changes in these comparisons.

Lower-rate training comparison	MC11	JevBench	Kev
Repeated pointer $-$ reference	$+0.17\pm 0.74$	$-0.66\pm 0.52$	$-0.15\pm 0.37$
Frozen branch $-$ reference	$+0.40\pm 0.93$	$-0.13\pm 1.60$	$-0.06\pm 0.52$
Trained residual $-$ reference	$+0.05\pm 0.57$	$-0.82\pm 1.61$	$-0.11\pm 0.41$
Trained residual $-$ frozen branch	$-0.35\pm 0.59$	$-0.69\pm 0.87$	$-0.04\pm 0.22$
Trained residual $-$ repeated pointer	$-0.12\pm 0.58$	$-0.16\pm 1.82$	$+0.04\pm 0.20$

Appendix

Table 11: Lower-rate residual controls, in percentage points (mean

\pm

descriptive SE over four settings). All arms use one epoch and all-module rank-64 adapters. The repeated pointer reruns the reference recipe with the same nominal seed.

Contrast	MC11	Seven sources	Config.-weighted LOSO	Source-weighted LOSO
Head-rate main effect	$+0.02\pm 0.43$	$-0.11\pm 0.46$	$[-0.132,+0.285]$	$[-0.345,+0.314]$
Adapter-rate main effect	$-2.83\pm 0.79$	$-2.96\pm 0.65$	$[-3.393,-2.240]$	$[-3.408,-2.478]$
Pooling $-$ pointer, all	$+0.73\pm 1.07$	$+1.05\pm 1.11$	$[+0.363,+1.209]$	$[+0.524,+1.470]$
Pooling $-$ pointer, no-FFN	$+0.91\pm 0.35$	$+1.64\pm 0.57$	$[+0.264,+1.489]$	$[+0.693,+2.202]$
No-FFN $-$ all, pointer	$+2.21\pm 0.66$	$+3.10\pm 0.64$	$[+1.451,+3.015]$	$[+1.985,+3.854]$
No-FFN $-$ all, pooling	$+2.39\pm 0.35$	$+3.69\pm 0.40$	$[+1.110,+3.259]$	$[+1.773,+4.558]$

Appendix

Table 12: Sensitivity of learning-rate, readout, adapter-placement, and training-budget effects to benchmark aggregation (pp). The first two columns are paired means

\pm

descriptive SE. Each LOSO range spans seven omission-specific means. Main effects use higher minus lower rate; the budget interaction is the two-epoch pooling advantage minus the one-epoch advantage.

LLM Reranking

Large language model reranking uses LLMs to reorder retrieved documents or candidate outputs by relevance, usefulness, or task-specific criteria.

See all 132 papers

Decision-Oriented Recommendation Reranking: An Empirical Study of Its Promise and Limitations
Sep 30, 2026Hanjia Lyu, Yinglong XiaLLM Reranking Large Language Model-Based Recommendation
Singapore Management University · Meta AI
Selective State-Space Adaptation and Retrieval for Language Model Reasoning
Jul 21, 2026Atahan Dokme, Larry HeckRetrieval-Augmented Generation Selective SSMs
AI Virtual Assistant (AVA) Lab Georgia Institute of Technology
HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads
Apr 19, 2026Juyuan Wang, Chenxing Wang, Yuchen Fang +8Attention Head Analysis Self-Attention
Weixin Group, Tencent, China · 2South China University of Technology, Guangzhou, China

Sep 30, 2026J/K move · Enter open · S save

Decision-Oriented Recommendation Reranking: An Empirical Study of Its Promise and Limitations

Hanjia Lyu, Yinglong Xia

Singapore Management University · Meta AI

LLM Reranking Large Language Model-Based Recommendation LLM Inference Efficiency

Open paper AskPDF arXiv

Large language models (LLMs) have shown promise for recommendation reranking, but their use introduces an important trade-off between recommendation quality and serving efficiency. We investigate whether decision-oriented systems provide a useful alternative when the reranking task is fundamentally a structured choice among predefined candidate items. Specifically, we conduct a controlled empirical study of two decision-oriented systems, TypeSafe Jev and OpenAI Decisions, for recommendation reranking. Across three Amazon Reviews domains and four candidate-set sizes, we compare these systems against recommendation-specific models and pointwise and listwise LLM-based rerankers, systematically evaluating recommendation effectiveness, candidate-order robustness, and efficiency and scalability. Our results reveal that Jev frequently achieves a favorable quality--latency trade-off, offering competitive recommendation effectiveness at substantially lower observed latency than stronger LLM-based alternatives. Overall, our findings suggest that specialization for decision-making does not inherently guarantee superior recommendation quality or robustness. Instead, the practical value of decision-oriented inference depends on implementation-specific characteristics and application requirements. These findings motivate further investigation of decision-oriented systems for recommendation and other ranking tasks with structured output spaces.

Abstract

LLM Reranking

Explore similar work

Decision-Oriented Recommendation Reranking: An Empirical Study of Its Promise and Limitations

Selective State-Space Adaptation and Retrieval for Language Model Reasoning

HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads