Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression

Mingqing Yuan; Xiaobo Liang; Junwei Yang; Ziwei Chen; Zeren Zhang; Hejin Wang; Yubin Wang; Juntao Li

cs.CLOct 7, 2026

Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression

Authors: Mingqing Yuan, Xiaobo Liang, Junwei Yang, Ziwei Chen, Zeren Zhang, Hejin Wang, Yubin Wang, Juntao Li

Organizations: Soochow University · University of Cambridge · Chalmers University of Technology · Peking University · Tsinghua University · The Hong Kong University of Science and Technology

Abstract

Reward modeling often requires jointly representing and reasoning over multiple evaluation criteria, yet verbalizing this process token by token can incur substantial inference cost. Recent work on latent reasoning suggests that continuous states may support this computation more compactly. We introduce LatentGRM, a latent evaluation framework built on semantic chunking, compression, and reconstruction. By using the structure of rubric-guided evaluations to guide compression, LatentGRM learns compact continuous trajectories that support autonomous pairwise judgments without generating textual assessments. A separate interpreter reconstructs evaluation text from these trajectories, providing an offline view of the information retained under compression. Under matched training data and backbones, LatentGRM achieves competitive aggregate preference accuracy relative to explicit Supervised Fine-Tuning (SFT) judges at both 4B and 8B scales. Across four benchmark domains, LatentGRM-8B compresses evaluation trajectories by 8.9--9.2x and reduces total judge inference time by 6.1--7.0x at vote@5. Controlled rubric interventions show that criterion-dependent preference information is carried through the latent sequence. Together, these results demonstrate that continuous latent evaluation can substantially reduce inference cost while preserving competitive judgment quality.

Figures & tables

Figure 1: Accuracy–efficiency scaling of LatentGRM-8B (stars) and Rubric-RM-8B (circles) from vote@1 to vote@9. Points run left to right at 1, 3, 5, 7, and 9 votes.

Figure 2: LatentGRM training and inference. Stage 1 encodes semantic boundaries into vocabulary-space targets and trains suffix reconstruction; Stage 2 distills autonomous latent judging. An offline interpreter reconstructs evaluations from saved trajectories.

Panel of Figure 3 — Figure 3: Illustrative frontiers in cropped evaluations. Green and red mark the semantic and fixed-rate cuts on identical text.

White-box judge/reward LLMs (for reference only)
Model	RewardBench		IF Evaluation		RM-Bench	RewardBench2		HS3	Avg.-8
	Chat	Hard	PPE	IFBench	Chat	Prec. IF	Focus
JudgeLRM-7B	93.2	56.4	47.9	47.5	55.0	29.7	39.9	60.5	53.7
RRM-7B	88.0	70.6	52.5	55.2	59.8	29.4	63.9	63.7	60.4
RM-R1-7B (Qwen-2.5-Inst)	94.4	72.3	56.1	57.8	65.8	35.6	80.0	69.3	66.4
RM-R1-7B (DeepSeek-Dist)	86.2	67.0	52.5	56.8	62.1	23.8	56.9	63.3	58.6

Table 1: Pairwise preference evaluation (%). Avg.-8 is the mean of the eight benchmark scores. Shading mark the best score among the Rubric-RM and LatentGRM configurations.

vote@1
	RewardBench		RewardBench2
Method	Chat	Hard	Prec. IF	Focus
Fixed-Rate Chunking	90.5	69.0	37.2	83.3
Semantic Chunking	88.7	70.5	37.9	84.6
vote@5
Fixed-Rate Chunking	90.6	70.8	39.5	85.8

Table 2: Semantic Chunking ablation on Qwen3-8B (%).

Figure 4: Rubric-conditioned preference shifts on 398 pairs. (A) Each trajectory is generated and evaluated under the same rubric. (B) The evaluation prompt is fixed to the baseline rubric while the latent-trajectory source varies. Changes are measured against the baseline condition; positive values favor the response targeted by the criterion intervention. Bands show pointwise 95% question-cluster bootstrap intervals.

Domain ( $N$ )	Model	Steps	Compression	HF time (s)	HF speedup	vLLM time (s)	vLLM speedup
RB Chat (716)	Rubric-RM-8B	659.60	$1.00\times$	8196	$1.00\times$	539.70	$1.00\times$
	LatentGRM-8B	74.38	$\mathbf{8.87}\times$	1002	$\mathbf{8.17}\times$	87.76	$\mathbf{6.15}\times$
RB Chat Hard (912)	Rubric-RM-8B	582.09	$1.00\times$	11088	$1.00\times$	589.58	$1.00\times$
	LatentGRM-8B	64.42	$\mathbf{9.04}\times$	1308	$\mathbf{8.46}\times$	96.44	$\mathbf{6.11}\times$
RB2 Precise IF (960)	Rubric-RM-8B	720.89	$1.00\times$	13416	$1.00\times$	812.93	$1.00\times$
	LatentGRM-8B	78.59	$\mathbf{9.17}\times$	1524	$\mathbf{8.82}\times$	126.99	$\mathbf{6.40}\times$

Table 3: Inference cost at vote@5. Steps are mean generated positions per rollout. Compression and speedup are the Rubric-RM-8B/LatentGRM-8B ratios of generated steps and total inference time. HF denotes ordinary Transformers/PyTorch generation without vLLM. Bold marks fewer steps or less time and the corresponding compression/speedup ratios within each domain.

Readout input	State Macro-F1	Vector Exact	Rubric Semantic	ROUGE-L
Prompt-only	40.20	18.61	56.42	63.36
Replay-hidden	69.42	33.03	61.29	68.22
Latent Trace Interpreter	98.93	93.98	74.65	79.24

Table 4: LatentGRM-8B retrospective reconstruction on the 607-record criterion-rich test set (%).

Feedback interface	Repair $\uparrow$	Regression $\downarrow$	Full $\uparrow$	Criterion sat. $\uparrow$
Prompt-only	54.07	5.20	34.17	76.28
Rubric-RM trajectory	78.71	4.65	56.67	87.78
Latent only (soft top-10)	63.32	5.11	41.67	80.54
Interpreter-restored trajectory	80.49	1.86	57.50	90.12

Table 5: Frozen-model revision on 240 examples, equally split between Chat and Chat Hard (percent). Repair is computed over 897 originally unsatisfied criteria and Regression over 1,076 originally satisfied criteria. Bold marks the best result in each column.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Boundary type	Base cost $\beta(b)$
Observed sentence end	$-3$
Complete criterion-header end or predicate-bearing clause edge	$0.5$
Soft punctuation, such as a comma, colon, or semicolon	$1$
Lexical separator within code or mathematics	$1.5$
Ordinary word boundary	$2$
Internal position of a criterion header	$6$

Appendix

Table 6: Base costs for candidate internal boundaries. Lower costs favor a cut; protected expressions are excluded before scoring.

Explicit tokens per latent	Chunk count	Share (%)
4	37,514	1.08
5	223,876	6.47
6	399,275	11.55
7	572,989	16.57
8	1,053,395	30.46
9	523,161	15.13

Appendix

Table 7: Chunk lengths in the Qwen3-8B semantic training cache. All 35,612 evaluations retain their exact rate-8 latent budgets.

Panel of Figure 5 — Figure 5: A complete evaluation from the semantic training cache, with the syntax-aware frontiers used for training. All 219 explicit tokens and all 28 latent frontiers are shown inside one frame. Contiguous background regions follow the coarse semantic hierarchy, their identities are written at the right, and each uniform dark | marks one latent frontier.

Panel of Figure 6 — Figure 6: Fixed-Rate Chunking of exactly the same evaluation trace as Figure 5 . The 219 explicit tokens are divided into 27 complete groups of eight and one final group of three, yielding the same 28 latent tokens. Only the red frontier locations change: unlike Semantic Chunking, they may split a phrase, a rubric criterion, or a transition between coarse semantic regions.

Figure 7: Stage 2 loss for the LatentGRM-8B Semantic Chunking ablation. Semantic Chunking maintains a lower Stage 2 loss after the initial decline.

Setting	Value
Initialization	LatentGRM-8B Stage 1 decoder; frozen backbone
Training data	28,430 examples; 512 validation examples; seed 42
Trajectory view	Full trajectory in forward order; one view per example
LTI representation	Probability-weighted top- $K$ input embeddings
LoRA	Rank 32; $\alpha=64$ ; dropout 0.05; all attention and MLP projections
Training duration	2 epochs

Appendix

Table 8: LTI training configuration. Effective batch size includes data-parallel workers and gradient accumulation.

Component	Text from the accepted example
Request	Who created the Superman cartoon character?
Response A	Superman, the iconic comic book superhero, was created by writer Jerry Siegel and artist Joe Shuster. Superman first appeared in Action Comics #1, which was published by Detective Comics, Inc. (later DC Comics) in June 1938. The character’s immense popularity established him as one of the most enduring and recognizable figures in the superhero genre.
Response B	The Superman cartoon character was created by Jerry Siegel and Joe Shuster in 1938.
Original	3. The response should be concise and avoid unnecessary elaboration. [Principle]
Paraphrase	3. The response should be brief and avoid including extraneous information. [Principle]
Counterfactual	3. The response should provide thorough and enriching details that go beyond the minimum facts required to answer. [Principle]

Appendix

Table 9: Example 1: brevity versus contextual detail.

Component	Text from the accepted example
Request	Generate an advertisement slogan that promotes healthy eating.
Response A	“Get Fit, Fuel Up - Find Balance with Our App!”
Response B	Eat healthy, feel healthy!
Original	5. The response should employ vivid, engaging imagery or metaphors to capture attention. [Principle]
Paraphrase	5. The response should use striking, imaginative language or figurative expressions to draw attention. [Principle]
Counterfactual	5. The response should use simple, literal language that directly communicates its message without figurative embellishment. [Principle]

Appendix

Table 10: Example 2: figurative versus literal language.

		RewardBench		RewardBench2
Model	Votes	Chat	Chat Hard	Precise IF	Focus
DirectJudge	1	90.92	67.21	35.94	85.15
	5	91.62	67.43	41.56	86.57
	9	90.78	67.56	41.25	86.47
Rubric-RM	1	89.80	69.80	31.90	77.90
	5	91.80	70.50	45.00	81.10

Appendix

Table 11: Direct judgment versus explicit and latent evaluation (%).

Model	Mean prompt positions	Output positions	Total inference time (s)	Throughput (positions/s)
Rubric-RM-8B	913.8	81,400	46.469	1,751.7
LatentGRM-8B	914.8	81,400	45.714	1,780.6

Appendix

Table 12: Equal-length serving throughput. The full RewardBench Chat and Chat Hard subsets are evaluated at vote@1 with exactly 50 generated positions per sample.

Readout input	Met F1	Not-Met F1	Partial F1
Prompt-only	59.95	50.21	10.44
Replay-hidden	94.45	79.46	34.36
Latent Trace Interpreter	99.64	98.68	98.46

Appendix

Table 13: Class-wise criterion-state F1 on the 607-record test set (%).

Panel of Figure 8 — Figure 8: Case study, record 14270: teacher reference evaluation. Green highlights decisive wording in the original trace.

Panel of Figure 9 — Figure 9: Case study, record 14270: LTI reconstruction. Purple highlights semantic paraphrases of the teacher trace.

Panel of Figure 10 — Figure 10: Reconstruction failure, record 17234: teacher reference evaluation. Green marks the evidence that Response B’s Criterion 3 is not met.

Panel of Figure 11 — Figure 11: Reconstruction failure, record 17234: LTI reconstruction. Red marks the erroneous Met state for Response B’s Criterion 3.

Setting	Value
Backbones / training records	Qwen3-4B and Qwen3-8B; 35,612 records
Hardware per run	$8\times$ NVIDIA RTX 6000D (85,651 MiB per GPU)
Precision / distributed training	bfloat16; DeepSpeed ZeRO Stage 2; gradient checkpointing
Maximum sequence length	6,144 tokens
Compression rate / latent support	$\rho=8$ ; top- $K=10$ ; projection temperature $\tau=1$
Semantic Chunking constraints	hard 4–16 tokens; preferred 6–10 tokens; local search radius 4 tokens (5 for lexical rescue)

Appendix

Table 14: Training configuration for LatentGRM. Effective batch size includes eight data-parallel workers and gradient accumulation.

Panel of Figure 12 — Figure 12: Prompt used to construct task-specific rubric criteria.

Panel of Figure 13 — Figure 13: Prompt used to elicit a structured explicit evaluation trace.

Panel of Figure 14 — Figure 14: Prompt used for LatentGRM training and inference.

Explore similar work

Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling
Jun 5, 2026Xing Yue, Linjuan Wu, Daoxin Zhang +2Reward Modeling LLM Evaluation
Zhejiang University · Xiaohongshu Inc.
EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Oct 4, 2026Xuancheng Li, Beining Wang, Haitao Li +7Reward Modeling Process Reward Models
Department of Computer Science and Technology, Tsinghua University, Beijing, China · Tencent, Beijing, China
Latent Thought Credit: Multi-Answer Credit Assignment for Latent Reasoning
Aug 3, 2026Xuyang Zhao, Liting Zhang, Zichen Xu +4Numerical Reasoning in Language Models RL for Language Model Reasoning
TMCC, College of Computer Science, Nankai University, Tianjin, China · Lingxi (Beijing) Technology Co., Ltd.

Jun 5, 2026, cs.CLJ/K move · Enter open · S save

Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling

Xing Yue, Linjuan Wu, Daoxin Zhang, Yongliang Shen+1

Zhejiang University · Xiaohongshu Inc.

Reward Modeling LLM Evaluation LLM-as-a-Judge Reward Model Evaluation

Open paper AskPDF arXiv

Open-ended reward modeling requires judges that can follow subtle, domain-specific preferences when verifiable answers are unavailable. Existing rubric-based methods often address this by generating criteria online for each query, but the extra generation step can add inference overhead and produce rigid or misaligned guidance. We introduce Eval-Skill, an exploration-guided method that synthesizes reusable evaluation skills for reward modeling and reframes reward guidance as context evolution rather than parameter training or per-query rubric generation. Using only 100 cases per domain for skill evolution, Eval-Skill synthesizes reusable domain-level evaluation skills through two progressive stages, workflow generation followed by principle generation, with exploration and selection interleaved across both stages. Once generated, a skill is directly injected into the judge context. Across multiple RM benchmarks, Eval-Skill consistently improves diverse judge backbones; on RewardBench 2, it yields significant gains over vanilla judging for each main backbone (+13.44% for Qwen3-8B, and 18.51% for DeepSeek-V4-Flash). Further analyses of evolution-time scaling, generalizability, and transferability show that compact evaluation skills offer an efficient new paradigm for LLM-based evaluation. Code is available at https://github.com/xing-stellus-yue/Eval-Skill.

Abstract

Explore similar work

Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling

EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans

Latent Thought Credit: Multi-Answer Credit Assignment for Latent Reasoning