RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation

Haocheng Yang; Yuchao Zhang; Licheng Pan; Jiajun Fan; Maolin Wang; Kangning Zhang; Shuai Shao; Shijian Wang

cs.CLOct 4, 2026

RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation

Authors: Haocheng Yang, Yuchao Zhang, Licheng Pan, Jiajun Fan, Maolin Wang, Kangning Zhang, Shuai Shao, Shijian Wang, +3 more

Organizations: National University of Singapore · Beijing University of Chemical Technology · Zhejiang University · University of Illinois at Urbana-Champaign · City University of Hongkong · Shanghai Jiaotong University · Southeast University · Xiaohongshu · Peking University · MBZUAI

Abstract

Rubric-based reinforcement learning (RL) provides interpretable rewards for aligning large language models (LLMs) by evaluating responses against query-specific evaluation criteria. To construct rubrics at scale, a straightforward approach to LLM-based rubric generation is to prompt an LLM to generate a rubric directly from the query. However, rubrics directly generated by LLMs are vulnerable to reward hacking, since omitted or underspecified criteria allow the policy to obtain high rubric rewards with low-quality responses. Existing LLM-based rubric generation methods improve the granularity and coverage of the generated criteria but do not proactively guard against reward hacking. To address this limitation, we propose RubricArmor, an adversarial framework that exposes and mitigates potential reward hacking at the rubric generation stage before it occurs in subsequent RL. Specifically, RubricArmor performs adversarial evolution, in which an attack step and a repair step alternate over multiple rounds. The attack step simulates the reward hacking of the policy by constructing adversarial responses that satisfy the current rubric but fail to properly complete the task. The repair step then revises the rubric to detect the response defects exposed by the attack step while preserving other valid criteria. Extensive experiments demonstrate that RubricArmor outperforms competitive rubric generation baselines and translates into more effective downstream rubric-based RL.

Figures & tables

Panel of Figure 1 — Figure 1 : Two examples of rubric reward hacking. (a) Omitting essential criteria leaves an additional factual error unchecked. (b) Underspecified criteria allow an incorrect response to pass. The refined rubrics (Ours) capture both defects.

Figure 2 : Overview of a RubricArmor evolution round. Attack Discovery constructs adversarial responses that satisfy the current rubric and independently confirms their defects and the resulting loss in response quality. Verified Rubric Repair revises the rubric to detect the confirmed defects, verifies that the adversarial response receives a lower reward, and reviews the revised criteria. Each accepted rubric becomes the target of subsequent attacks.

LLM-based rubric generation
	JudgeBench				RM-Bench				RewardBench	Macro
Method	Knowledge	Reasoning	Math	Coding	Chat	Math	Code	Safety	Chat-Hard	Avg.
Direct	62.01	53.69	70.56	69.18	64.38	77.59	69.03	89.91	74.67	70.11
TICK	65.75	58.39	73.33	67.12	66.67	72.13	69.76	90.61	75.22	71.00
RocketEval	70.78	63.42	74.44	68.49	65.59	82.13	70.47	88.23	73.14	72.97
ChasingTail	72.40	68.12	76.67	67.12	66.45	84.58	69.62	88.38	72.37	73.97

Table 1 : Pairwise preference accuracy (%) across nine evaluation sets from three benchmark suites.

Method	Resp.	Adv.	JudgeBench				RM-Bench				RewardBench	Macro
Method	Resp.	Adv.	Knowledge	Reasoning	Math	Coding	Chat	Math	Code	Safety	Chat-Hard	Avg.
Direct	✗	✗	62.01	53.69	70.56	69.18	64.38	77.59	69.03	89.91	74.67	70.11
RubricArmor †	✗	✓	68.99	61.41	72.22	69.86	64.60	80.01	70.81	88.23	69.08	71.69
RubricArmor ‡	✓	✗	70.45	62.75	73.33	71.23	64.81	83.23	69.01	89.51	70.83	72.80
RubricArmor (Full)	✓	✓	74.03	67.11	83.33	72.60	68.48	86.23	71.71	91.58	78.73	77.09

Table 2 : Ablations of response construction and the adversarial objective.

Figure 3 : Performance under varying maximum number of evolution rounds

T

with two rubric generator backbones. For JudgeBench and RM-Bench, Acc denotes the micro accuracy, which weights each evaluation set by its number of comparisons.

Backbone: Gemini-3.1-Flash-Lite
	JudgeBench		RM-Bench		Chat-Hard
	Micro		Micro		Accuracy
Method	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$
Direct	62.10	—	78.93	—	74.67	—
RubricArmor	73.63	$18.6\%\!\uparrow$	83.69	$6.0\%\!\uparrow$	78.73	$5.4\%\!\uparrow$
Backbone: GPT-4.1

Table 3 : Generalization across rubric generator backbones.

Policy Model: Qwen2.5-3B-Instruct
Method	BiGGen-Bench		AlpacaEval 2.0				Arena-Hard
	Score		Vanilla		LC		Vanilla		SC
	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$
Original	63.18	-	16.48	-	15.62	-	26.11	-	26.55	-
Direct	63.41	0.36% $\uparrow$	19.81	20.21% $\uparrow$	18.18	16.39% $\uparrow$	26.41	1.15% $\uparrow$	28.10	5.84% $\uparrow$
TICK	63.84	1.04% $\uparrow$	18.38	11.53% $\uparrow$	17.29	10.69% $\uparrow$	26.69	2.22% $\uparrow$	28.14	5.99% $\uparrow$

Table 4 : Downstream RL results.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone: GPT-4.1
	JudgeBench		RM-Bench		Chat-Hard
	Micro		Micro		Accuracy
Method	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$
Direct	65.56	—	75.92	—	74.89	—
RubricArmor	73.06	$11.4\%\!\uparrow$	79.98	$5.3\%\!\uparrow$	77.74	$3.8\%\!\uparrow$
Backbone: Gemini-3.5-Flash

Appendix

Table 5 : Generalization across verifier backbones.

Figure 4 : Cumulative total tokens and elapsed time per query under varying maximum number of evolution rounds

T

Policy Model: Qwen2.5-3B-Instruct
Method	Accuracy		Communication Quality		Completeness		Context Awareness		Instruction Following		Average
Method	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$	Value	$\Delta$
Original	19.66	-	46.64	-	11.12	-	13.69	-	34.84	-	25.19	-
Direct	20.43	3.92% $\uparrow$	45.23	3.02% $\downarrow$	12.36	11.15% $\uparrow$	13.45	1.75% $\downarrow$	35.56	2.07% $\uparrow$	25.41	0.86% $\uparrow$
TICK	19.85	0.97% $\uparrow$	46.27	0.79% $\downarrow$	11.14	0.18% $\uparrow$	14.24	4.02% $\uparrow$	35.40	1.61% $\uparrow$	25.38	0.75% $\uparrow$
RocketEval	20.22	2.85% $\uparrow$	48.00	2.92% $\uparrow$	12.83	15.38% $\uparrow$	13.89	1.46% $\uparrow$	38.06	9.24% $\uparrow$	26.60	5.60% $\uparrow$

Appendix

Table 6 : Out-of-domain generalization of RL-trained policies on HealthBench-Hard.

Explore similar work

EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
Jun 22, 2026Hongxin Ding, Baixiang Huang, Yue Fang +6On-Policy Self-Evolution Large Language Model Reinforcement Learning
National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education +2
CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training
Sep 1, 2026Siyuan Li, Xinxin Song, Chen Ruinian +5Large Language Model Reinforcement Learning Large Language Model Evaluation
Department of Automation, Tsinghua University · Meituan
Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
May 28, 2026Zijie Wang, Eduardo BlancoRubrics Llm-As-A-Judge
University of Arizona Department of Computer Science

Jun 22, 2026, cs.LGJ/K move · Enter open · S save

EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

Hongxin Ding, Baixiang Huang, Yue Fang, Weibin Liao+5

National Engineering Research Center of Software Engineering, Peking University, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education+2

On-Policy Self-Evolution Large Language Model Reinforcement Learning Co-Evolution Adversarial Robustness

Open paper AskPDF arXiv

Rubric-based rewards offer interpretable and fine-grained optimization signals for reinforcement learning in open-ended tasks where verifiable answers are unavailable. However, pre-constructed rubrics remain static throughout training, creating a fundamental mismatch with the evolving policy: fixed criteria gradually lose discriminative power as the model improves, leading to reward saturation and potential hacking. Recent dynamic rubric methods partially address this but rely on external frontier models or ground-truth answers, and update rubrics only at coarse granularity. We propose EvoRubrics, a co-evolutionary RL framework where a Policy LLM and a Rubric Generator jointly improve through adversarial interaction within each training step. As the policy improves under the rubric generator's guidance, the rubric generator adapts its criteria to remain discriminative and informative, enabling evaluation to track the policy in real time and naturally inducing an automatic curriculum. Experiments show that EvoRubrics consistently outperforms static and dynamic rubric baselines across benchmarks. The learned Rubric Generator further generalizes as a transferable reward model. Notably, even a fully self-supervised variant without any external supervision achieves meaningful gains, suggesting that co-evolution between generation and evaluation alone can provide sufficiently rich learning signals. Our code is publicly available at https://anonymous.4open.science/r/EvoRubrics-2155/.

Abstract

Explore similar work

EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge