Policy-Adaptive Credit Assignment for Spatial Reasoning with Verifiable Rewards

Hailiang Yang; Shengchao Chen

cs.CVOct 19, 2026

Policy-Adaptive Credit Assignment for Spatial Reasoning with Verifiable Rewards

Authors: Hailiang Yang, Shengchao Chen

Organizations: School of Computer Science and Engineering, University of New South Wales · Australian AI Institute, University of Technology Sydney

Abstract

Image-based spatial reasoning requires perception, decomposition, and inference, but the stage that limits accuracy can change during training. Answer-level rewards neither localize failures to individual stages nor provide discriminative group-relative feedback when all rollouts receive the same reward. Process supervision provides intermediate feedback, but fixed weights cannot adapt reward allocation to these changing bottlenecks. To address this gap, we propose CALIPER, an adaptive process-reward framework supported by SpatialTrace, a corpus of stage-decomposed spatial reasoning traces. CALIPER jointly estimates the associations between stage-wise reference coverage and answer correctness through within-group ridge regression, using these associations as utility proxies to gradually update reward weights. The resulting process reward supervises all rollout groups, including uniformly failing ones. Because utility estimates can become unreliable when answer-discriminative groups are scarce, a closed-loop harness monitors estimator reliability and validation performance to guide checkpoint rollback, rollout diversification, and early stopping. Extensive experiments across eight benchmarks and three backbones show that CALIPER achieves the highest mean under both one-shot and harness training, with consistent gains over fixed allocation across RL optimizers. Further analyses confirm the benefits of adaptive reward allocation and harness control, with favorable scaling in model/data size.

Figures & tables

Figure 1: Outcome-only rewards (a), as in GRPO ( Shao et al., 2024 ) and DeepSeek-R1 ( Guo et al., 2025 ) , supervise final answers. Reference-assisted verification (b) improves grading but retains uniform-reward degeneracy. Process supervision (c) combines human-labeled feedback ( Lightman et al., 2024 ) or Math-Shepherd scores ( Wang et al., 2024b ) with fixed weights. CALIPER (d) adapts these weights from estimated utility.

Figure 2: SpatialTrace examples (left) and source composition (right).

Figure 3: CALIPER scores stages against reference traces, estimates utility from groups with mixed answer correctness, and updates the policy using rewards for every group. Validation and estimator diagnostics guide continuation, rollback, rollout diversification or stopping.

Method	Harness	3DSR	MMB	MME-RW	MMStar	Omni3D	RoboSp	SAT	SpatEval	Avg	$\bm{\Delta}$	1st
Qwen3-VL-4B ( Bai et al., 2025a )
Direct (No RL)	✗	55.00	89.00	49.00	61.00	38.90	43.92	75.00	56.00	58.48	✗	1
CoT (No RL)	✗	59.00	92.00	47.50	67.00	31.85	44.72	73.50	68.50	60.51	✗	0
[0.5pt/1.5pt]	✗	60.50	91.00	46.00	66.50	33.70	46.15	76.00	70.00	61.23		0
Outcome Reward	✓	58.00	91.00	50.50	64.50	33.65	46.72	76.00	70.50	61.36	$+0.13$	0
[0.5pt/1.5pt]	✗	61.00	92.00	47.50	68.00	33.95	47.13	77.00	68.00	61.82		1

Table 1: Main results (%). Avg is the mean, and 1st counts wins (ties included). Bold / underline mark best/second-best scores.

\Delta

compares protocols.

Method	BLINK	RealWorld QA	Emb Spatial	View Spatial	Avg
Direct (no RL)	80.50	73.00	76.50	37.50	66.88
CoT (no RL)	83.00	75.00	69.50	43.00	67.63
Outcome Reward	84.00	71.50	72.50	41.50	67.38
Reference-Assisted	85.00	71.50	73.00	43.00	68.13
Process Reward Model	80.50	72.00	72.00	45.00	67.38
Frozen $\alpha$	83.00	74.50	73.00	41.50	68.00

Table 2: Qwen3-VL-4B OOD accuracy (%).

Figure 5: Reward-component (a) and framework (b) ablations. Gaps are in percentage points.

Figure 6: Hyperparameter sensitivity.

Panel of Figure 7 — Figure 7: Qualitative effects of adaptive stage supervision.

Figure 8: Scaling with model size, training data, update budget and trace coverage.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Source datasets	Items	Primary reasoning demand
3D metric	3DSRBench, Omni3D-Bench, SAT-Spatial-VQA-5k	310	Depth, orientation and metric relations
General multimodal	MMBench, MME-RealWorld-Lite, MMStar	310	Object grounding and open-domain relations
Region-level spatial	RoboSpatial-Home, SpatialEval, SpatialRGPT-Bench	310	Placement and local spatial relations
Physical reasoning	SeePhys	250	Physical quantities and plausibility
Total	Ten source datasets	1,180	Single-image questions with verifiable answers

Appendix

Table 4: SpatialTrace source families and reasoning demands.

Question Validity item, binary “Given the image and the question, can an attentive reader determine a single answer from visible evidence?” positive The image contains the objects and relations the question asks about, and one answer follows from them. failure The evidence is occluded or outside the frame, the reference frame is unspecified, or several answers remain defensible.
Perception Fidelity perception, three-point ordinal “Does each observation report objects, attributes or positions that are visible in the image and needed for the question?” positive Faithful reports only visible content and covers the evidence the answer depends on. failure Partial is correct but omits one needed object or relation. Unfaithful describes absent objects, misreads a position or orientation, or states the answer in place of evidence.
Decomposition Adequacy decomposition, three-point ordinal “Would answering these subquestions in order suffice to answer the original question?” positive Adequate covers every object, quantity and reference frame the answer depends on. failure Partial misses one of them. Inadequate restates the question, targets irrelevant objects, or answers the subquestions itself.
Inference Soundness inference, five-point Likert “How well does the inference derive the reference answer from the recorded observations?” positive 5 every step follows from the observations and reaches the reference answer, 4 it is sound apart from one minor gap. failure 3 it reaches the answer through one weakly supported step, 2 it relies on facts absent from the perception stage or on unjustified numerical precision, 1 it contradicts the observations or reaches a different answer.
Alignment Agreement item, binary “Judging from the image, is the reference answer supported at its stated precision?” positive The judgment, made before the recorded answer alignment is shown, matches that record. failure The rater finds an unsupported label or unjustified precision that the record accepts, or rejects a recorded concern.

Appendix

Table 5: Rubric used by the SpatialTrace raters.

Criterion	Items	Positive ratings	Positive (%)	$\alpha$
Question validity	100	269/300	89.7	0.785
Perception fidelity	95	226/285	79.3	0.715
Decomposition adequacy	94	217/282	77.0	0.640
Inference soundness	95	212/285	74.4	0.808
Alignment agreement	100	255/300	85.0	0.765

Appendix

Table 6: Human assessment of SpatialTrace references.

Policy optimization
Hyperparameter	Symbol	Value
LoRA rank / scaling		$64$ / $64$
Rollouts per question	$n$	$8$
Rollout temperature	$T$	$1.2$
Questions / optimizer step		$512$
Gradient updates / step		$4$

Appendix

Table 7: Default hyperparameters and optimizer variants. AdamW uses bfloat16. Clipping values specify lower / upper thresholds.

Qwen3-VL-4B
Method	3DSR	MMB	MME-RW	MMStar	Omni3D	RoboSp	SAT	SpatEval	Avg
CALIPER (ours)	66.00	93.50	52.00	69.00	37.05	46.95	76.50	72.50	64.19
w/o Measured Utility	61.50	92.50	48.00	67.50	32.05	44.69	73.50	72.50	61.53
w/o Independent Judging	59.50	92.00	50.00	66.00	31.55	45.06	76.00	75.50	61.95
w/o Solved-Group Gate	62.50	91.00	49.50	67.00	35.00	43.13	74.00	75.00	62.14
w/o Length Penalty	64.00	91.00	50.50	66.50	36.70	40.48	74.50	75.00	62.34

Appendix

Table 8: Component ablations by benchmark.

Qwen3-VL-4B
Method	3DSR	MMB	MME-RW	MMStar	Omni3D	RoboSp	SAT	SpatEval	Avg
Outcome Reward	58.00	91.00	50.50	64.50	33.65	46.72	76.00	70.50	61.36
Reference-Assisted	61.50	92.00	50.00	65.00	32.55	48.03	75.00	71.50	61.95
Frozen $\alpha$	64.50	94.00	50.50	65.50	32.70	44.09	75.50	73.00	62.47
w/o Measured Utility	62.50	93.00	50.00	69.00	35.20	48.19	73.50	73.00	63.05
$\alpha$ EMA $=0.05$	60.00	92.50	52.50	65.50	35.85	49.58	77.00	73.50	63.30

Appendix

Table 9: Harness ablations by benchmark.

Qwen3-VL-4B
Method	3DSR	MMB	MME-RW	MMStar	Omni3D	RoboSp	SAT	SpatEval	Avg
GRPO, Outcome Reward	58.00	91.00	50.50	64.50	33.65	46.72	76.00	70.50	61.36
GRPO, Reference-Assisted	61.50	92.00	50.00	65.00	32.55	48.03	75.00	71.50	61.95
GRPO, Frozen $\alpha$	64.50	94.00	50.50	65.50	32.70	44.09	75.50	73.00	62.47
GRPO, CALIPER (ours)	65.50	93.00	52.50	70.00	36.35	50.37	72.50	75.00	64.40
DAPO, Outcome Reward	61.50	93.50	50.50	65.50	34.45	51.03	76.50	73.50	63.31

Appendix

Table 10: Optimizer comparisons under harness training.

Configuration	Qwen3-VL-4B	Qwen2.5-VL-7B	InternVL3-8B
Default configuration	64.19	55.97	61.84
$\eta=0.05$	62.21	54.10	60.77
$\eta=0.5$	63.27	54.48	60.68
$\lambda=0.01$	61.79	53.94	60.97
$n=4$	61.03	54.39	60.81
$n=12$	62.76	54.96	60.48

Appendix

Table 11: One-shot sensitivity averaged over eight benchmarks.

Process Reward Models

Process reward models assess intermediate reasoning steps and actions to guide learning, search, and evaluation.

See all 141 papers

Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance
Apr 25, 2026Xinzhu Chen, Wei He, Huichuan Fan +7Token-Level Credit Assignment Group Relative Policy Optimization
Beijing University of Posts and Telecommunications · Meituan · Tianjin University +3
Credit Assignment with Resets in Language Model Reasoning
May 25, 2026Ankur Samanta, Akshayaa Magesh, Ayush Jain +7RL for Language Model Reasoning Credit Assignment in RL
1Meta AI · 2Columbia University · *Work done at Meta +2
The Art of Interrogation: Consistency Amplifies Factuality in Spatial Reasoning
Jun 10, 2026Theo Uscidda, Marta Tintore Gazulla, Maks Ovsjanikov +2Reinforcement Learning 3D Spatial Reasoning
CREST, ENSAE, Institut Polytechnique de Paris · Google Zurich · Google DeepMind

Apr 25, 2026J/K move · Enter open · S save

Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance

Xinzhu Chen, Wei He, Huichuan Fan, Wenzhe Niu+6

Beijing University of Posts and Telecommunications · Meituan · Tianjin University+3

Token-Level Credit Assignment Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards Credit Assignment Fine-Grained Credit Assignment

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) performs coarse-grained credit assignment in reinforcement learning with verifiable rewards (RLVR) by assigning the same advantage to all tokens in a rollout. Process reward models can provide finer-grained supervision, but they require step-level annotation or additional reward modeling. We show that hidden-state distributions contain a useful signal for local reasoning quality that can be extracted using only outcome-level correctness labels available in RLVR. Specifically, within each GRPO group, the Wasserstein distance between span-level hidden state distributions of correct and incorrect rollouts increases around regions where their local reasoning quality diverges. This association holds both across examples and within individual trajectories, suggesting that hidden-state distributional divergence can serve as a self-supervision signal for fine-grained credit assignment. We formalize this observation with a separation theorem showing that, under mild structural assumptions, post-divergence spans have larger Wasserstein distances than pre-divergence spans whenever the population-level distributional gap exceeds finite-sample noise. Motivated by this result, we propose Span-level Hidden state Enabled Advantage Reweighting (SHEAR), which modifies GRPO by using span-level Wasserstein distances to scale token-level advantages, amplifying updates on tokens whose hidden states are more separated from the opposing group. The method requires no additional model and only minimal changes to the training pipeline. Experiments on five mathematical reasoning benchmarks and five code generation benchmarks show improvements over standard GRPO and strong performance relative to supervised process reward models, while requiring no additional annotation or reward model training.

Abstract

Process Reward Models

Explore similar work

Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance

Credit Assignment with Resets in Language Model Reasoning

The Art of Interrogation: Consistency Amplifies Factuality in Spatial Reasoning