RELISH: LLM REgression with a Latent Iterative State Head

Yiheng Su; Matthew Lease

cs.CLApr 1, 2026

RELISH: LLM REgression with a Latent Iterative State Head

Authors: Yiheng Su, Matthew Lease

Organizations: School of Information, The University of Texas at Austin

Abstract

We present RELISH (REgression with a Latent Iterative State Head), a novel, lightweight architecture designed for text regression with large language models. Rather than decoding numeric targets as text or aggregating multiple generated outputs, RELISH predicts scalar values directly from frozen LLM representations by iteratively refining a learned latent state through cross-attention over token-level representations, and then mapping the final state to a point estimate with a linear regressor. Across six datasets, four LLM backbones, and two LLM training regimes, RELISH consistently outperforms prior baselines from all three major LLM regression families, including autoregressive decoding, regression-aware inference, and existing predictive head methods. Despite these gains, RELISH remains highly parameter-efficient, requiring only $\sim$ 3.4-3.7M trainable parameters across frozen LLM backbones (only 0.01-0.04 $\%$ additional overhead), far less than LoRA-based alternatives that grow with model size (0.26-0.42 $\%$ ). Our code is available at https://github.com/SamSoup/RELISH.

Figures & tables

Inference family	How to make predictions	Suitable training setups
Autoregressive decoding ( Vacareanu et al., 2024 )	Parse from string: $\displaystyle\hat{y}_{\textsc{ard}}(x)=\text{float}(\arg\max_{y\in\mathcal{Y}}p(\text{str}(y)\mid x))$	PEFT/SFT: Minimize token-level CE on $\text{str}(y^{})$ . $\displaystyle\mathcal{L}_{\textsc{ce}}=-\sum_{t=1}^{\|str(y^{})\|}\log p(y_{t}\mid x,y_{<t})$
Regression-aware inference (RAIL) ( Lukasik et al., 2024 )	Bayes-optimal from $p(\cdot\mid x)$ : $\displaystyle\hat{y}_{\textsc{rail}}(x)=\arg\min_{v\in\mathbb{R}}\mathbb{E}_{y\sim p}[\ell(\text{float}(y),v)]$ When $\ell=\mathcal{L}_{\textsc{mse}}$ , same as posterior mean: $\displaystyle\hat{y}_{\textsc{rail}}(x)\approx\sum_{y\in\mathcal{Y}_{\text{grid}}}y\cdot p(\text{str}(y)\mid x)$	PEFT/SFT: (i) Minimize $\mathcal{L}_{\textsc{ce}}$ . (ii) RAFT ( Lukasik et al., 2025 ) . $\displaystyle\mathcal{L}_{\textsc{raft}}=\Big(y^{*}-\mathbb{E}_{y\sim p}[\text{float}(y)]\Big)^{2}$
Predictive Head ( Tang et al., 2025 )	Extract representations, then regress: $\displaystyle\hat{y}_{\textsc{head}}(x)=f_{\textsc{reg}}\big(\phi\big(H(x)\big)\big)$	Head-only: train $f_{\textsc{reg}}$ with $\mathcal{L}_{\textsc{mse}}$ . PEFT/SFT + head: fine-tune backbone jointly with $f_{\textsc{reg}}$ .
RELISH (ours)	Iteratively refine a latent state, then regress: $\displaystyle r^{(i)}=\mathrm{Refine}\!\left(r^{(i-1)},H(x)\right),\ i=1,\dots,L$ $\displaystyle\hat{y}_{\textsc{RELISH}}(x)=w^{\top}r^{(L)}+b$	Head-only: train $r^{(0)}$ , $w$ , $b$ , and refinement modules with $\mathcal{L}_{\textsc{MSE}}$ . PEFT/SFT + head: optionally fine-tune backbone jointly with RELISH modules.

Table 1: Taxonomy of methods leveraging LLMs for regression. We characterize LLM-based regression along two axes: (a) how the model produces a point estimate ( inference family ) and (b) which parameters are trained and how ( training setup ). Autoregressive decoding generates

\text{str}(y)

and parses it. RAIL computes a Bayes-optimal estimate under

p(\cdot\mid x)

for loss

\ell

(e.g., the posterior mean for

\mathcal{L}_{\textsc{mse}}

), where the posterior is estimated via sampling or candidate enumeration over a grid

\mathcal{Y}_{\text{grid}}

. Predictive heads regress from pooled hidden states, whereas RELISH replaces static pooling with iterative latent-state summarization over token-level representations. Typical objectives are

\mathcal{L}_{\textsc{ce}}

for autoregressive decoding,

\mathcal{L}_{\textsc{ce}}

\mathcal{L}_{\textsc{raft}}

for RAIL, and

\mathcal{L}_{\textsc{mse}}

for predictive-head methods.

Method	Family	Training	Inference
Zero / Many Shot ( Vacareanu et al., 2024 )	Autoregressive	None	Medium–High
Linear / MLP ( Tang et al., 2025 )	Predictive head	Low	Low
RAIL ( Lukasik et al., 2024 )	Regression-aware	None	High
RAFT ( Lukasik et al., 2025 )	Regression-aware	High	High
RELISH (ours)	Predictive head	Low	Low

Table 2: Qualitative operational costs of common LLM regression approaches. Training costs reflect optimization overhead. Autoregressive methods and RAIL do not update parameters (None), predictive heads and RELISH freeze the LLM backbone and only train auxiliary modules (Low), while RAFT fine-tunes the LLM backbone (High). Inference costs correspond to runtime latency. Predictive heads and RELISH are lightweight (Low), whereas autoregressive methods sequentially decode tokens, so latency scales with the number of in-context demonstrations (Medium–High). RAIL and RAFT approximate model posterior via multiple samples or scoring grid candidates (High).

	Unsupervised			Supervised							Ours
Metric	Zero	Many	RAIL	Lin.	MLP	L+Zero	L+Lin.	L+MLP	L+RAIL	RAFT	RELISH
$r$ $\uparrow$	$55.0_{\scriptscriptstyle\pm 0.3}$	$52.3_{\scriptscriptstyle\pm 0.8}$	$56.0_{\scriptscriptstyle\pm 0.1}$	$44.4_{\scriptscriptstyle\pm 0.0}$	$56.3_{\scriptscriptstyle\pm 0.4}$	$57.2_{\scriptscriptstyle\pm 0.4}$	$44.9_{\scriptscriptstyle\pm 0.0}$	$55.9_{\scriptscriptstyle\pm 0.5}$	$61.8_{\scriptscriptstyle\pm 0.5}$	$66.1_{\scriptscriptstyle\pm 0.3}$	$\mathbf{72.9_{\scriptscriptstyle\pm 0.1}}$
$\rho$ $\uparrow$	$55.5_{\scriptscriptstyle\pm 0.2}$	$52.6_{\scriptscriptstyle\pm 1.0}$	$58.8_{\scriptscriptstyle\pm 0.1}$	$42.5_{\scriptscriptstyle\pm 0.0}$	$53.7_{\scriptscriptstyle\pm 0.4}$	$56.1_{\scriptscriptstyle\pm 0.5}$	$43.0_{\scriptscriptstyle\pm 0.0}$	$53.4_{\scriptscriptstyle\pm 0.3}$	$62.9_{\scriptscriptstyle\pm 0.6}$	$65.3_{\scriptscriptstyle\pm 0.5}$	$\mathbf{71.4_{\scriptscriptstyle\pm 0.3}}$
NRMSE $\downarrow$	$25.1_{\scriptscriptstyle\pm 0.0}$	$22.2_{\scriptscriptstyle\pm 0.5}$	$25.5_{\scriptscriptstyle\pm 0.0}$	$23.1_{\scriptscriptstyle\pm 0.0}$	$17.8_{\scriptscriptstyle\pm 0.3}$	$18.9_{\scriptscriptstyle\pm 0.1}$	$23.5_{\scriptscriptstyle\pm 0.2}$	$17.7_{\scriptscriptstyle\pm 0.1}$	$17.9_{\scriptscriptstyle\pm 0.1}$	$15.7_{\scriptscriptstyle\pm 0.2}$	$\mathbf{12.6_{\scriptscriptstyle\pm 0.2}}$

Table 3: Predictive performance macro-averaged across six datasets, four LLMs, and three runs for each baseline. We report Pearson (

r

) and Spearman (

\rho

) correlations (

\uparrow

higher is better) and range-Normalized Root Mean Squared Error (NRMSE) (

\downarrow

lower is better), all scaled to %. L+ denotes LoRA-based fine-tuning. We also report standard deviation (

\pm

Figure 1: Detailed results across datasets (columns) and LLMs (rows). The rightmost column averages over all datasets, and the bottom row averages over LLMs. Bars show absolute improvement in Pearson ( $r$ ) correlation over the zero-shot baseline (the ”ground zero” line). Error bars denote variability across 3 independently-seeded runs. All metrics are computed using gold test labels. RELISH consistently performs best across all settings.

Method	Llama 3.1 8B	Qwen 3 8B	Gemma 3 27B	Qwen 3 32B
LoRA / RAFT	21.0M (0.26%)	21.8M (0.27%)	116.5M (0.42%)	134.2M (0.41%)
RELISH	3.4M (0.04%)	3.4M (0.04%)	3.7M (0.01%)	3.7M (0.01%)

Table 4: How trainable parameter sizes vary for different LLM backbones. We report the total trainable parameters and their % relative to the underlying LLM. RAFT applies a regression-aware loss to LoRA adapters, sharing the same trainable footprint as LoRA with cross-entropy loss. While LoRA’s budgets scale with model size, RELISH remains essentially invariant to the backbone, scaling instead with the hidden dimension (Table 7 ).

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset Name	Train	Validation	Test	Score Range
STS-B	5,749	1,500	1,380	[0, 5]
SICKR-STS	6,948	1,985	994	[1, 5]
WMT_EN_ZH	7,000	1,000	1,000	[0, 100]
WMT_RU_EN	7,000	1,000	1,000	[0, 100]
WMT_SI_EN	7,000	1,000	1,000	[0, 100]
StatLLM	321	100	200	[0, 50]

Appendix

Table 5: Dataset split statistics.

Hyperparameter	Value
Shared
Seed Values	42, 1234, 2026
Effective Batch Size	32
Early Stopping Patience	2
Warmup Ratio	0.1
Weight Decay	0.01

Appendix

Table 6: Hyperparameter configurations.

Figure 2: Spearman ( $\rho$ ) correlation across datasets (columns) and LLMs (rows). The rightmost column averages over all datasets, and the bottom row averages over LLMs. Whereas earlier Figure 1 showed Pearson correlation, this figure shows absolute improvement in Spearman correlation over the zero-shot baseline (the ”ground zero” line). Error bars denote variability across 3 independently-seeded runs. All metrics are computed using gold test labels. RELISH consistently performs best across all settings.

Figure 3: NRMSE results across datasets (columns) and LLMs (rows). The rightmost column averages over all datasets, and the bottom row averages over LLMs. Whereas earlier Figure 1 showed Pearson correlation, this figure shows absolute improvement in range-normalized root mean squared error (NRMSE) over the zero-shot baseline (the ”ground zero” line). Error bars denote variability across 3 independently-seeded runs. All metrics are computed using gold test labels. RELISH consistently performs best across all settings.

Method	Trainable ( $n$ )	% of Model
Llama 3.1 8B Instruct ( $d=4096,N=8,030,261,248$ )
LoRA/RAFT	20,971,520	0.2612%
RELISH	3,418,625	0.0426%
Qwen 3 8B ( $d=4096,N=8,190,735,360$ )
LoRA/RAFT	21,823,488	0.2664%
RELISH	3,418,625	0.0417%

Appendix

Table 7: Detailed parameter footprints across architectures.

N

represents the total number of backbone parameters and

d

is the model hidden dimension. Trainable (

n

) is the number of parameters updated during training, and % of Model denotes

n/N

. Note that while baseline footprints scale with the total parameter count

N

, RELISH’s footprint remains largely invariant to backbone size, scaling primarily as a function of

d

	Fine-tuned Baselines		RELISH Variants
Metric	LoRA+MLP	RAFT	RELISH (MSE)	RELISH (Huber)
$r$ $\uparrow$	$64.8_{\scriptscriptstyle\pm 0.1}$	$71.5_{\scriptscriptstyle\pm 0.4}$	$74.2_{\scriptscriptstyle\pm 0.4}$	$\mathbf{76.3_{\scriptscriptstyle\pm 0.2}}$
$\rho$ $\uparrow$	$61.7_{\scriptscriptstyle\pm 0.1}$	$69.0_{\scriptscriptstyle\pm 0.2}$	$71.3_{\scriptscriptstyle\pm 0.4}$	$\mathbf{74.0_{\scriptscriptstyle\pm 0.1}}$
NRMSE $\downarrow$	$16.7_{\scriptscriptstyle\pm 0.1}$	$16.6_{\scriptscriptstyle\pm 0.3}$	$13.5_{\scriptscriptstyle\pm 0.1}$	$\mathbf{13.3_{\scriptscriptstyle\pm 0.3}}$

Appendix

Table 8: Ablating RELISH with Huber versus MSE loss. We report macro-averaged Pearson

r

and Spearman

\rho

correlation (higher is better), as well as range-normalized RMSE (NRMSE, lower is better) in %. Metrics are averaged across the five STS/WMT datasets, excluding StatLLM, four LLM backbones, and three independently seeded runs. We also report standard deviation (

\pm

Figure 4: RELISH loss ablations . We compare LoRA + MLP, RAFT, and two RELISH variants trained with MSE or Huber across the five STS/WMT datasets, excluding StatLLM (columns) and four LLM backbones (rows). Metrics are NRMSE (lower is better), Pearson correlation (higher is better), and Spearman correlation (higher is better). Error bars show variability across 3 seeds. Overall, MSE and Huber perform similarly, but Huber is modestly better in most settings.

Figure 5: RELISH refinement depth ablations. We systematically vary the refinement depth of RELISH from 1 to 5. We report range-normalized root mean square error (NRMSE; lower is better), Pearson correlation, and Spearman correlation (higher is better) macro-averaged across four LLM backbones and the five STS/WMT datasets, excluding StatLLM. Error bars denote the standard deviation across three random seeds. The $L=1$ configuration isolates the performance of a single-step, attention-based pooling baseline. For most LLM backbones, multiple refinement iterations ( $L\geq 2$ ) substantially enhance predictive performance, with diminishing returns past $L=3$ (our default setting, shaded in grey). Notably, the optimal depth varies by LLM architecture: Qwen 3 8B achieves peak performance at $L=1$ , while the larger Gemma 3 27B continues to benefit from refinement even at higher depths.

Figure 6: How predictive performance scales with data availability. We compare RELISH against MLP and RAFT on STS-B and WMT_EN_ZH as the number of training examples increases from 64 to the full training set (5,749 for STS-B and 7,000 for WMT_EN_ZH). We report RMSE (lower is better), Pearson correlation (higher is better), and Spearman correlation (higher is better), averaged across four LLM backbones and three random seeds. Error bars denote standard error across the corresponding 12 model-seed runs. RAFT performs best under extremely low data regimes ( $<$ 256 examples), but RELISH benefits more from additional supervision ( $>$ 256–512 examples).

Figure 7: Training overhead across different data scales. We compare RELISH and RAFT on STS-B and WMT_EN_ZH as the number of training examples increases from 64 to the full training set. Training time is reported in minutes on a log scale, averaged across four LLM backbones and three random seeds. Error bars denote standard error across 12 model-seed runs, and text labels report the RAFT/RELISH runtime ratio at each training size. RELISH is consistently faster to train than RAFT.

Dataset	Method	Throughput (ex/s) $\uparrow$	Latency/RELISH $\downarrow$
STS-B	MLP	$15.58_{\scriptscriptstyle\pm 8.63}$	$0.99\times$
	Linear	$15.57_{\scriptscriptstyle\pm 8.61}$	$0.99\times$
	RELISH	$15.35_{\scriptscriptstyle\pm 8.36}$	$1.00\times$
	Zero-shot	$3.17_{\scriptscriptstyle\pm 2.53}$	$4.85\times$
	RAFT	$2.92_{\scriptscriptstyle\pm 1.70}$	$5.26\times$
	RAIL	$1.24_{\scriptscriptstyle\pm 0.78}$	$12.43\times$

Appendix

Table 9: Inference efficiency of RELISH. We report throughput as the number of examples processed per second, averaged across four LLM backbones and three random seeds. Latency/RELISH compares each baseline’s wall-clock latency against RELISH, where larger values indicate slower inference. RELISH adds negligible inference overhead compared to simple predictive heads and is faster than autoregressive and regression-aware methods.

Dataset	RELISH Variant	RMSE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$	Training Steps $\downarrow$
STS-B	Last Token	$0.677_{\scriptscriptstyle\pm 0.059}$	$0.898_{\scriptscriptstyle\pm 0.019}$	$0.897_{\scriptscriptstyle\pm 0.017}$	$1035_{\scriptscriptstyle\pm 267}$
	Mean Pool	$0.688_{\scriptscriptstyle\pm 0.051}$	$0.894_{\scriptscriptstyle\pm 0.017}$	$0.893_{\scriptscriptstyle\pm 0.016}$	$\mathbf{960_{\scriptscriptstyle\pm 310}}$
	Random (Ours)	$\mathbf{0.623_{\scriptscriptstyle\pm 0.037}}$	$\mathbf{0.917_{\scriptscriptstyle\pm 0.010}}$	$\mathbf{0.916_{\scriptscriptstyle\pm 0.009}}$	$1080_{\scriptscriptstyle\pm 188}$
WMT_EN_ZH	Last Token	$\mathbf{11.088_{\scriptscriptstyle\pm 1.092}}$	$0.501_{\scriptscriptstyle\pm 0.029}$	$0.493_{\scriptscriptstyle\pm 0.025}$	$493_{\scriptscriptstyle\pm 297}$
	Mean Pool	$11.365_{\scriptscriptstyle\pm 1.206}$	$0.498_{\scriptscriptstyle\pm 0.027}$	$0.489_{\scriptscriptstyle\pm 0.024}$	$493_{\scriptscriptstyle\pm 249}$
	Random (Ours)	$11.410_{\scriptscriptstyle\pm 1.521}$	$\mathbf{0.524_{\scriptscriptstyle\pm 0.040}}$	$\mathbf{0.521_{\scriptscriptstyle\pm 0.030}}$	$\mathbf{420_{\scriptscriptstyle\pm 174}}$

Appendix

Table 10: Warm start ablations for

r^{(0)}

. We compare the default random initialization of

r^{(0)}

against input-dependent last-token and mean-pooling warm start configurations. Metrics are averaged across four LLM backbones and three random seeds. Bold indicates the best variant within each dataset and metric. Overall, random initialization appears to be the most robust choice, matching or exceeding the predictive performance of other warm-start variants while maintaining training efficiency.

Dataset: STS-B
System Prompt	You are an expert human annotator for semantic textual similarity (STS). You will see English sentence pairs and judge how similar in MEANING they are. Scoring instructions: - Output a real-valued similarity score from 0.000 to 5.000, inclusive. - 5.000 = same meaning / paraphrases. - 2.500 = partially similar with important differences. - 0.000 = completely unrelated meaning. CRITICAL FORMAT RULE: Return ONLY the numeric score, with EXACTLY three digits after the decimal point (e.g., 3.000). Do NOT include any words, punctuation, units, labels, or explanation.
User Message	Sentence A: {sentence_a} Sentence B: {sentence_b} Score (0.000 to 5.000 only):

Appendix

Table 11: Full prompt template for the STS-Benchmark dataset.

Dataset: SICKR-STS
System Prompt	You are an expert human annotator for semantic textual similarity (STS) for Sentences Involving Compositional Knowledge. You will see sentence pairs and judge how similar in MEANING they are. Scoring instructions: - Output a real-valued similarity score from 1.000 to 5.000, inclusive. - 5.000 = same meaning / paraphrases; 1.000 = completely unrelated. CRITICAL FORMAT RULE: Return ONLY the numeric score, with EXACTLY three digits after the decimal point (e.g., 3.000). Do NOT include any words, punctuation, units, labels, or explanation.
User Message	Sentence A: {sentence_a} Sentence B: {sentence_b} Score (1.000 to 5.000 only):

Appendix

Table 12: Full prompt template for the SICKR-STS dataset.

Dataset: WMT_EN_ZH
System Prompt	You are an expert human annotator for translation quality / semantic textual similarity (STS). You will see sentence pairs where Sentence A is in English (source) and Sentence B is in Chinese (translation). You must judge how well the meaning is preserved from English to Chinese. Scoring instructions: - Output a real-valued translation quality / similarity score from 0.000 to 100.000. - 100.000 = perfect meaning preservation (excellent translation / paraphrases). - 50.000 = partially similar meaning but with important differences or missing info. - 0.000 = completely unrelated or very bad translation. CRITICAL FORMAT RULE: Return ONLY the numeric score with EXACTLY three digits after the decimal point. Do NOT include any words, punctuation, units, labels, or explanation.
User Message	Sentence A (English source): {sentence_a} Sentence B (Chinese translation): {sentence_b} Score (0.000 to 100.000 only):

Appendix

Table 13: Full prompt template for the WMT English-Chinese quality estimation task.

Dataset: WMT_RU_EN
System Prompt	You are an expert human annotator for translation quality / semantic textual similarity (STS). You will see sentence pairs where Sentence A is in Russian (source) and Sentence B is in English (translation). You must judge how well the meaning is preserved from Russian to English. Scoring instructions: - Output a real-valued translation quality / similarity score from 0.000 to 100.000. - 100.000 = perfect meaning preservation (excellent translation / paraphrases). - 50.000 = partially similar meaning but with important differences or missing info. - 0.000 = completely unrelated or very bad translation. CRITICAL FORMAT RULE: Return ONLY the numeric score with EXACTLY three digits after the decimal point. Do NOT include any words, punctuation, units, labels, or explanation.
User Message	Sentence A (Russian source): {sentence_a} Sentence B (English translation): {sentence_b} Score (0.000 to 100.000 only):

Appendix

Table 14: Full prompt template for the WMT Russian-English quality estimation task.

Dataset: WMT_SI_EN
System Prompt	You are an expert human annotator for translation quality / semantic textual similarity (STS). You will see sentence pairs where Sentence A is in Sinhala (source) and Sentence B is in English (translation). You must judge how well the meaning is preserved from Sinhala to English. Scoring instructions: - Output a real-valued translation quality / similarity score from 0.000 to 100.000. - 100.000 = perfect meaning preservation (excellent translation / paraphrases). - 50.000 = partially similar meaning but with important differences or missing info. - 0.000 = completely unrelated or very bad translation. CRITICAL FORMAT RULE: Return ONLY the numeric score with EXACTLY three digits after the decimal point. Do NOT include any words, punctuation, units, labels, or explanation.
User Message	Sentence A (Sinhala source): {sentence_a} Sentence B (English translation): {sentence_b} Score (0.000 to 100.000 only):

Appendix

Table 15: Full prompt template for the WMT Sinhala-English quality estimation task.

Dataset: StatLLM
System Prompt	You are a human expert in statistical programming using the Statistical Analysis System (SAS) language. You will see a statistical analysis task, dataset description, and SAS code. You must judge the quality of the provided SAS code within its intended context. Scoring instructions: - Output a real-valued total score from 0.000 to 50.000, inclusive. - 50.000 = excellent code: correct, executable, readable, concise, and produces the necessary clear output. - 25.000 = partially correct code: addresses some of the task but has important mistakes, omissions, or unclear output. - 0.000 = unusable code: does not address the task, is likely non-executable, or would produce wrong/missing output. CRITICAL FORMAT RULE: Return ONLY the numeric score with EXACTLY three digits after the decimal point. Do NOT include any words, punctuation, units, labels, or explanation.
User Message	Statistical analysis task: {problem_description} Dataset name: {dataset_name} Dataset description: {data_description} Generated SAS code: `sas {generated_sas_code}` Total score (0.000 to 50.000 only):

Appendix

Table 16: Full prompt template for the StatLLM SAS code quality estimation task.

Dataset: STS-Benchmark (Semantic Textual Similarity Benchmark)
LLM encoder	Inference	RMSE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$
Llama 3.1 8B Instruct	Zero Shot	$1.215_{\scriptscriptstyle\pm 0.010}$	$0.717_{\scriptscriptstyle\pm 0.006}$	$0.688_{\scriptscriptstyle\pm 0.010}$
	Many Shot	$1.235_{\scriptscriptstyle\pm 0.045}$	$0.675_{\scriptscriptstyle\pm 0.031}$	$0.635_{\scriptscriptstyle\pm 0.030}$
	Linear	$1.323_{\scriptscriptstyle\pm 0.000}$	$0.632_{\scriptscriptstyle\pm 0.000}$	$0.637_{\scriptscriptstyle\pm 0.000}$
	MLP	$1.051_{\scriptscriptstyle\pm 0.015}$	$0.732_{\scriptscriptstyle\pm 0.009}$	$0.725_{\scriptscriptstyle\pm 0.010}$
	RAIL	$1.252_{\scriptscriptstyle\pm 0.007}$	$0.743_{\scriptscriptstyle\pm 0.002}$	$0.771_{\scriptscriptstyle\pm 0.003}$

Appendix

Table 17: Comparison on STS-B (Semantic Textual Similarity Benchmark). Metrics are mean

\pm

std over 3 seeds and rounded to 3 decimals. Lower is better for RMSE (

\downarrow

); higher is better for correlations (

\uparrow

Dataset: SICKR_STS (Sentences Involving Compositional Knowledge, Revised)
LLM encoder	Inference	RMSE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$
Llama 3.1 8B Instruct	Zero Shot	$1.095_{\scriptscriptstyle\pm 0.012}$	$0.666_{\scriptscriptstyle\pm 0.015}$	$0.650_{\scriptscriptstyle\pm 0.014}$
	Many Shot	$0.918_{\scriptscriptstyle\pm 0.090}$	$0.608_{\scriptscriptstyle\pm 0.019}$	$0.477_{\scriptscriptstyle\pm 0.036}$
	Linear	$0.612_{\scriptscriptstyle\pm 0.000}$	$0.807_{\scriptscriptstyle\pm 0.000}$	$0.764_{\scriptscriptstyle\pm 0.000}$
	MLP	$0.509_{\scriptscriptstyle\pm 0.007}$	$0.858_{\scriptscriptstyle\pm 0.004}$	$0.809_{\scriptscriptstyle\pm 0.007}$
	RAIL	$1.126_{\scriptscriptstyle\pm 0.003}$	$0.686_{\scriptscriptstyle\pm 0.002}$	$0.746_{\scriptscriptstyle\pm 0.002}$

Appendix

Table 18: Comparison on SICKR_STS (Sentences Involving Compositional Knowledge, Revised). Metrics are mean

\pm

std over 3 seeds and rounded to 3 decimals. Lower is better for RMSE (

\downarrow

); higher is better for correlations (

\uparrow

Dataset: WMT_RU_EN (Russian $\rightarrow$ English)
LLM encoder	Inference	RMSE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$
Llama 3.1 8B Instruct	Zero Shot	$32.853_{\scriptscriptstyle\pm 0.160}$	$0.473_{\scriptscriptstyle\pm 0.008}$	$0.409_{\scriptscriptstyle\pm 0.012}$
	Many Shot	$31.481_{\scriptscriptstyle\pm 1.342}$	$0.198_{\scriptscriptstyle\pm 0.106}$	$0.133_{\scriptscriptstyle\pm 0.098}$
	Linear	$22.637_{\scriptscriptstyle\pm 0.000}$	$0.619_{\scriptscriptstyle\pm 0.000}$	$0.567_{\scriptscriptstyle\pm 0.000}$
	MLP	$19.161_{\scriptscriptstyle\pm 0.045}$	$0.695_{\scriptscriptstyle\pm 0.001}$	$0.646_{\scriptscriptstyle\pm 0.004}$
	RAIL	$33.237_{\scriptscriptstyle\pm 0.175}$	$0.528_{\scriptscriptstyle\pm 0.005}$	$0.530_{\scriptscriptstyle\pm 0.007}$

Appendix

Table 19: Comparison on WMT_RU_EN (Russian

\rightarrow

English). Metrics are mean

\pm

std over 3 seeds and rounded to 3 decimals. Lower is better for RMSE (

\downarrow

); higher is better for correlations (

\uparrow

Dataset: WMT_EN_ZH (English $\rightarrow$ Chinese)
LLM encoder	Inference	RMSE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$
Llama 3.1 8B Instruct	Zero Shot	$33.543_{\scriptscriptstyle\pm 0.217}$	$0.348_{\scriptscriptstyle\pm 0.013}$	$0.276_{\scriptscriptstyle\pm 0.018}$
	Many Shot	$26.312_{\scriptscriptstyle\pm 1.674}$	$0.075_{\scriptscriptstyle\pm 0.010}$	$0.042_{\scriptscriptstyle\pm 0.013}$
	Linear	$19.648_{\scriptscriptstyle\pm 0.000}$	$0.277_{\scriptscriptstyle\pm 0.000}$	$0.253_{\scriptscriptstyle\pm 0.000}$
	MLP	$14.198_{\scriptscriptstyle\pm 0.057}$	$0.331_{\scriptscriptstyle\pm 0.012}$	$0.321_{\scriptscriptstyle\pm 0.011}$
	RAIL	$38.840_{\scriptscriptstyle\pm 0.096}$	$0.345_{\scriptscriptstyle\pm 0.004}$	$0.367_{\scriptscriptstyle\pm 0.002}$

Appendix

Table 20: Comparison on WMT_EN_ZH (English

\rightarrow

Chinese). Metrics are mean

\pm

std over 3 seeds and rounded to 3 decimals. Lower is better for RMSE (

\downarrow

); higher is better for correlations (

\uparrow

Dataset: WMT_SI_EN (Sinhala $\rightarrow$ English)
LLM encoder	Inference	RMSE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$
Llama 3.1 8B Instruct	Zero Shot	$40.343_{\scriptscriptstyle\pm 0.720}$	$0.433_{\scriptscriptstyle\pm 0.028}$	$0.427_{\scriptscriptstyle\pm 0.020}$
	Many Shot	$35.311_{\scriptscriptstyle\pm 3.971}$	$0.186_{\scriptscriptstyle\pm 0.020}$	$0.149_{\scriptscriptstyle\pm 0.054}$
	Linear	$29.768_{\scriptscriptstyle\pm 0.000}$	$0.368_{\scriptscriptstyle\pm 0.000}$	$0.348_{\scriptscriptstyle\pm 0.000}$
	MLP	$19.460_{\scriptscriptstyle\pm 0.198}$	$0.521_{\scriptscriptstyle\pm 0.004}$	$0.477_{\scriptscriptstyle\pm 0.004}$
	RAIL	$44.966_{\scriptscriptstyle\pm 0.055}$	$0.438_{\scriptscriptstyle\pm 0.001}$	$0.481_{\scriptscriptstyle\pm 0.010}$

Appendix

Table 21: Comparison on WMT_SI_EN (Sinhala

\rightarrow

English). Metrics are mean

\pm

std over 3 seeds and rounded to 3 decimals. Lower is better for RMSE (

\downarrow

); higher is better for correlations (

\uparrow

Dataset: StatLLM
LLM encoder	Inference	RMSE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$
Llama 3.1 8B Instruct	Zero Shot	$26.170_{\scriptscriptstyle\pm 0.121}$	$0.060_{\scriptscriptstyle\pm 0.025}$	$0.038_{\scriptscriptstyle\pm 0.013}$
	Many Shot	$13.988_{\scriptscriptstyle\pm 0.777}$	$0.052_{\scriptscriptstyle\pm 0.072}$	$0.067_{\scriptscriptstyle\pm 0.078}$
	Linear	$20.478_{\scriptscriptstyle\pm 0.000}$	$0.022_{\scriptscriptstyle\pm 0.000}$	$0.032_{\scriptscriptstyle\pm 0.000}$
	MLP	$23.720_{\scriptscriptstyle\pm 4.167}$	$0.158_{\scriptscriptstyle\pm 0.000}$	$0.153_{\scriptscriptstyle\pm 0.000}$
	RAIL	$27.078_{\scriptscriptstyle\pm 0.112}$	$0.020_{\scriptscriptstyle\pm 0.021}$	$0.039_{\scriptscriptstyle\pm 0.019}$

Appendix

Table 22: Comparison on StatLLM (code quality estimation). Metrics are mean

\pm

std over 3 seeds and rounded to 3 decimals. Lower is better for RMSE (

\downarrow

); higher is better for correlations (

\uparrow

Oct 13, 2026

DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression

While work on LLM-based regression has largely focused on scalar prediction, multi-output text regression remains less explored. We present DualView, a parameter-efficient approach to leverage LLMs for multi-output text regression. DualView first extracts broad and output-specific representations from a frozen LLM, then uses a dual-branch architecture to produce the full score vector. Across five heterogeneous regression tasks and eight LLMs (up to 122B), DualView outperforms seven supervised regression baselines while requiring 82--1842 times fewer fitted parameters than the three strongest baselines. These improvements extend across individual output dimensions and persist even with limited training data. Notably, DualView using a smaller open-weight LLM can outperform unsupervised prediction with Gemini 3.1 Pro. Controlled ablations show that broad and output-specific evidence provide complementary predictive benefits, while lightweight shared predictors achieve strong performance with fewer trainable parameters.

Yiheng Su, Junyi Jessy Li, Matthew Lease

The University of Texas at Austin

Parameter-Efficient Fine-Tuning

Ask PDF

Mar 17, 2026

REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge

Large language models (LLMs) are increasingly deployed as automated evaluators that assign numeric scores to model outputs, a paradigm known as LLM-as-a-Judge. However, standard Reinforcement Learning (RL) methods typically rely on binary rewards (e.g., 0-1 accuracy), thereby ignoring the ordinal structure inherent in regression tasks; for instance, they fail to recognize that predicting 4 is significantly better than predicting 1 when the ground truth is 5. Conversely, existing regression-aware approaches are often confined to Supervised Fine-Tuning (SFT), limiting their ability to explore optimal reasoning paths. To bridge this gap, we propose REAL (REgression-Aware Reinforcement Learning), a principled RL framework designed to optimize regression rewards, and also proven to be optimal for correlation metrics. A key technical challenge is that the regression objective is explicitly policy-dependent, thus invalidating standard policy gradient methods. To address this, we employ the generalized policy gradient estimator, which naturally decomposes optimization into two complementary components: (1) exploration over Chain-of-Thought (CoT) trajectory, and (2) regression-aware prediction refinement of the final score. Extensive experiments across model scales (8B to 32B) demonstrate that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods, exhibiting significantly better generalization on out-of-domain benchmarks. On Qwen3-32B specifically, we achieve gains of +8.40 Pearson and +7.20 Spearman correlation over the SFT baseline, and +18.30/+11.20 over the base model. These findings highlight the critical value of integrating regression objectives into RL exploration for accurate LLM evaluation.

Yasi Zhang, Tianyu Chen, Mingyuan Zhou +3

University of California, Los Angeles · The University of Texas at Austin · Google Research

RL for Language Models Reinforcement Learning LLM-as-a-Judge LLM Evaluation

Ask PDF

Oct 14, 2026

INTACT: Inducing Latent Reasoning on a Frozen Backbone

Latent reasoning replaces natural-language chains of thought with compact continuous "soft-token" representations, reducing the number of intermediate tokens that must be decoded autoregressively. We use a small external encoder to induce latent reasoning in a frozen backbone model. The encoder iteratively maps the backbone's hidden states to new latent blocks, constructing a latent reasoning prefix, after which the frozen model writes its remaining reasoning and its answer as text. Our method, INTACT, trains this encoder through distillation to reproduce the effect of written reasoning steps while keeping the model's behavior close to that of the frozen backbone, and then reinforces it on answer correctness. Across four frozen backbones from 1B to 8B, INTACT improves both GSM8K and average out-of-domain accuracy, including GSM8K gains from 75.8% to 79.3% on Llama-3.2-3B and from 77.6% to 79.7% on Qwen2.5-3B. It also reduces end-to-end latency per problem by 19% on Llama-3.2-3B and outperforms the evaluated latent-reasoning baselines at the 3B and 8B scales. Beyond the mathematics setting, the same recipe trained on a multi-domain corpus improves the frozen backbone on twelve evaluation benchmarks spanning mathematics, science, and commonsense, without task-specific adaptation.

Kutay Tire, Atula Tejaswi, Haris Vikalo +1

The University of Texas at Austin

Efficient Language Model Reasoning Implicit Reasoning in Language Models Continuous Latent Reasoning Numerical Reasoning in Language Models

Ask PDF

Abstract

Explore similar work

DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression

REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge

INTACT: Inducing Latent Reasoning on a Frozen Backbone