cs.LGNov 6, 2026

Plug-and-Play Verifier for Decoupling Process Reward in Stabilizing Long-Horizon Agentic Reasoning

Authors: Mingwei Xu, Jieyuan Pei, Binghao Tu, Hao Fang

Organizations: Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, USA · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Department of Electrical Engineering, City University of Hong Kong, Hong Kong SAR, China

Abstract

Stabilizing long-horizon reasoning requires intelligent agents to sustain useful progress across successive reasoning steps and tool interactions. Such agents are typically trained with reinforcement learning (RL), where actor-critic RL uses a learnable critic network to estimate step advantages, while critic-free methods such as GRPO eliminate critic models and rely mostly on the group variants of sparse outcome-only rewards. Both reward types leave partially useful failed trajectories either implicit or indistinguishable from entirely unproductive ones, and as reasoning horizons grow, it becomes difficult to maintain consistent reasoning trajectories. In this work, we propose a plug-and-play verifier framework for stabilizing long-horizon reasoning through explicit verification of intermediate reasoning artifacts. Specifically, the framework decouples process verification through a unified interface that extracts intermediate reasoning artifacts from agent running trajectories, evaluates them using domain-specific verifiers, and aggregates the verification scores as bounded process reward. Then, we combine our dense verified process rewards and the sparse outcome rewards through a two-stage post-training schedule. We showed that with plug-and-play verifiers, we further stabilize the reasoning steps as last-mile self-reflection and self-consistency. Typical evaluation on HotpotQA using Qwen3.5-4B achieves a superior accuracy (60.18%), outperforming the standard GRPO baseline (57.73%). Our results suggest that plug-and-play verifiers provide a modular approach to learn more stable and reliable reasoning behaviors over long horizons.

Explore similar work

CardsList
  1. Verifiable Process Rewards for Agentic Reasoning

    May 11, 2026Huining Yuan, Zelai Xu, Huaijie Wang +6Process Reward ModelsCredit Assignment in RL

  2. Process Reward Agents for Steering Knowledge-Intensive Reasoning

    Apr 10, 2026Jiwoong Sohn, Tomasz Sternal, Kenneth Styppa +2LLM ReasoningProcess Reward Models

  3. AgentV-RL: Scaling Reward Modeling with Agentic Verifier

    Apr 17, 2026Jiazheng Zhang, Ziche Fu, Zhiheng Xi +13Reward ModelingRL for Language Model Reasoning