Adapting Technical-Service LLM Agents with Latent Logic Augmentation, Robust Noise Reduction, and Hybrid Reward Modeling

Junzhuo Ma; Chenghuang Shen; Yi Yu; Xingyan Liu; Jing Gu; Hangyi Sun; Guangquan Hu; Jianfeng Liu

cs.LGMar 18, 2026

Adapting Technical-Service LLM Agents with Latent Logic Augmentation, Robust Noise Reduction, and Hybrid Reward Modeling

Authors: Junzhuo Ma, Chenghuang Shen, Yi Yu, Xingyan Liu, Jing Gu, Hangyi Sun, Guangquan Hu, Jianfeng Liu,

Organizations: School of Mathematical Sciences, Fudan University, Shanghai, China · Alibaba Group, Hangzhou, China · Shanghai Center for Mathematical Sciences, Fudan University, Shanghai, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China ·

Abstract

Technical-service LLM agents are entering production workflows, where value depends on whether engineers adopt generated replies. Service tickets hide decision logic, contain noisy single-reference responses, and make reward evaluation costly, making standard post-training brittle. Existing post-training and LLM-as-a-Judge approaches improve grounding or feedback, but do not jointly model latent decision logic, response diversity, and reward cost. We address this gap by coupling latent logic augmentation, robust noise reduction, and hybrid reward modeling. The framework augments supervised fine-tuning data with Planning-Aware Trajectory Modeling and Reasoning Augmentation, builds dual-filtered Multiple Ground Truths, and trains the policy with a hybrid reward that combines a Reranker with an LLM-as-a-Judge. On real Cloud technical-service tasks, the adapted Qwen3-4B achieves the highest Multi-ECS (0.441), lower reward cost, and the highest production adoption rate (46.63%).

Figures & tables

Figure 1: Three observations that motivate our framework , each grounded in a real Cloud technical-service ticket, mapping a pain point to the component that addresses it: hidden decision logic ( RA + PATM ), multiple valid answers for one query ( Multi-GT ), and a gameable cheap Reranker vs. an accurate costly judge ( Hybrid Reward ).

Figure 2: Overview of the Proposed Framework. The framework consists of four stages: (a) Latent Logic Augmentation; (b) Multi-GT data construction; (c) SFT adaptation; (d) RL adaptation with HRM.

Method	Multi-ECS	Single-ECS	Call Tool Acc
Untrained Qwen3-4B	0.299	0.178	0.082
SFT	0.293	0.193	0.096
SFT+RA	0.319	0.224	0.139
SFT+PATM	0.326	0.233	0.149
SFT+RA+PATM	0.337	0.242	0.279

Table 1: SFT strategy analysis (RQ1). RA and PATM each improve Multi-ECS and tool-use accuracy; their combination is best.

Configuration Variant	Multi-ECS	Single-ECS
Ours (SFT+RA+PATM + Hybrid + Single-GT)	0.429	0.357
(1) Varying SFT Initialization
$\rightarrow$ replace with non-SFT	0.348	0.231
$\rightarrow$ replace with SFT	0.353	0.331
$\rightarrow$ replace with SFT+RA	0.406	0.336
$\rightarrow$ replace with SFT+PATM	0.407	0.346

Table 2: RL Adaptation and Robust Noise Reduction Analysis (RQ2–3). We establish a default configuration ( Ours ) and independently ablate its components. Upgrading the data to Multi-GT yields the final peak performance. Paired one-sided

t

-tests on Multi-ECS confirm Ours significantly exceeds the Soft-Judge reward (

p<0.05

), the Reranker-only reward, and the plain-SFT initialization (both

p<0.001

Figure 3: Expected Calibration Error (ECE) reliability diagrams for the three reward configurations. Bars show observed accuracy against predicted confidence; the dashed line denotes perfect calibration.

Model / Configuration	Adoption (%)
Untrained baselines
Qwen3-4B	32.80
Qwen3.7-Max	39.14
SFT only
SFT+RA+PATM	35.51
RL adaptation (reward ablations)

Table 3: Online adoption rate (%) from a retrospective production study. The RL-adaptation rows mirror the one-component ablations of Ours in Table 2 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Content
Context	The customer uses LATERAL to expand a JSON array. The test code runs fine in the query IDE, but after switching to the real table it fails in the data platform with “Feature not supported: LATERAL”.
Ground Truth	( inquire ) “@user Please connect the real table and run it in the query IDE, then send me the full execution log; I have not found the corresponding error record yet.”
	(Expert strategy: re-run the diagnosis in a clean query-IDE environment and collect full logs)
Baseline <think>	“No tools are available right now… the help doc says LATERAL is not supported… I should advise avoiding this syntax.”
	(Leaps from “no tools available” to a generic workaround, ignoring the environment discrepancy)
Baseline Response	“@user Your current version is 3.0, which does not support LATERAL ; I recommend using another method to process the JSON array.”

Appendix

Table 6: Case study for PATM: a LATERAL query succeeds on test data in the query IDE but fails on the real table in the data platform. Abbreviated reasoning traces ( <think> ) contrast the two decision processes; the score is the Multi-ECS against the reference set.

Component	Content
Context	In the managed-database event center the customer sees several “instance VIP migration” events and asks whether this means the instance will be migrated away. No instance ID is provided in the conversation.
Ground Truth	( inquire ) “@user To check whether this instance VIP migration will affect your instance, could you please send the instance ID?”
	(Expert strategy: obtain the instance ID before diagnosing the event)
$r_{R}$ : Reranker-trained	“@user Yes, this event is a migration of the instance’s virtual IP address; it is a normal maintenance operation and will not affect instance availability.”
	(✗ confident but premature solution from generic experience)
$r_{J}$ : Judge-trained	“@user An instance VIP migration usually means the virtual IP configuration has changed, for network optimization or maintenance; it is normal operational behavior and does not cause the instance itself to be migrated away.”

Appendix

Table 7: Case study for Hybrid Reward: the customer sees several “instance VIP migration” events in the managed-database console and asks whether the instance will be migrated away— without providing an instance ID . The reference strategy is to inquire for the instance ID before diagnosing; the two wrong answers are plausible but premature solutions. We list the responses of three policies, each trained with a different reward signal.

	$r_{R}$	$r_{J}$	$r_{H}$
	(solution)	(solution)	(inquire)
$S_{R}$ (Reranker)	0.94	0.50	0.88
$S_{J}$ (Judge)	0.30	0.82	0.60
Cascade region	Mix ↑	Mix ↓	Fast Pass
$R_{\theta}$ (Hybrid)	0.76	0.80	0.88
Multi-ECS (oracle)	0.02	0.01	1.00

Appendix

Table 8: Reward scores assigned to the three responses of Table 7 by each signal.

S_{R}

S_{J}

are the Reranker/Judge consistency scores;

R_{\theta}

is the Hybrid cascade (Eq. 2 ) with

[\tau_{a},\tau_{b}]\!=\![0.6,0.9]

w_{1}\!=\!0.05

w_{2}\!=\!0.72

; Multi-ECS is the oracle. Mix ↑ /Mix ↓ denote the upper (

S_{R}\!>\!\tau_{b}

) and lower (

S_{R}\!<\!\tau_{a}

) mixing branches. Bold marks the top-scored response per signal: the Reranker and the Judge each rank a different wrong answer highest, whereas the Hybrid and the oracle both rank

r_{H}

highest.

Component	Content
Context	The customer has multiple data sources pulling data from business databases and is unsure which tables have already been pulled; for each new extraction they are also unsure whether it was done before, and ask where this can be identified.
Ground Truth	GT-1 ( clarify ): “Are you using an offline synchronization task to pull the data?”
(3 references)	GT-2 ( solution ): “On the Data Source Management page, check the created data sources and their sync-task records to confirm the synchronized tables.”
	GT-3 ( solution ): “Go to Data Development > Project Directory > Offline Sync Nodes to view the configured sync tasks and execution records.”
Single-GT	All four samples collapse to the same inquire (workspace/project ID):
(Avg. Multi-ECS	“(1) Could you provide the specific workspace name or project ID? (2) Which data source’s table specifically? (3) Which workspace are you operating in? (4) Could you provide the workspace name?”

Appendix

Table 9: Case study for Multi-GT: the customer asks how to identify which tables have already been synchronized in the data platform. The reference set admits one clarify and two solution strategies. We draw four samples from each policy; Avg. Multi-ECS is averaged over the four samples.

Category	Hyperparameter	Value
Data	Max Sequence Length	20000
Optimization	Base Model	Qwen3-4B
	Global Batch Size	256
	Learning Rate	$2\times 10^{-6}$
	Min Learning Rate	$8\times 10^{-7}$
	LR Scheduler	Cosine

Appendix

Table 10: Hyperparameters for Supervised Fine-Tuning (SFT).

Category	Hyperparameter	Value
Actor Model	Learning Rate	$5\times 10^{-6}$
	Weight Decay	0.0001
	LR Scheduler	Constant
	Clip Ratio	0.2
	Clip Ratio(high)	0.28
	Entropy Coeff	$3.5\times 10^{-4}$

Appendix

Table 11: Hyperparameters for Reinforcement Learning (DAPO).

(a) Prompting-only models
Model	Single-ECS	Multi-ECS	Setting
Qwen3.7-Max	0.232	0.348	prompt-only
DeepSeek-V4-Pro-Preview	0.251	0.374	prompt-only
GLM-5.1	0.241	0.370	prompt-only
Kimi-K2.6	0.232	0.358	prompt-only
MiniMax-M2.5	0.237	0.364	prompt-only

Appendix

Table 12: Prompting-only comparisons, backbone transfer, and matched training baselines. Cells marked S/M report Single-/Multi-ECS. All evaluations use the same frozen 1,000-request test set and ECS protocol.

GT Category	Bare Model	Judge	Reranker	Hybrid
reply:solution	0.320	0.468	0.462	0.465
reply:inquire	0.063	0.268	0.194	0.283
reply:clarify	0.111	0.186	0.145	0.240
Action-Type Classification Accuracy (%)
Overall	19.00	62.90	61.70	64.80

Appendix

Table 13: Per-category ECS scores across reward configurations. Hybrid achieves the best performance on inquire and clarify , demonstrating superior calibration on ambiguous action types.

Figure 4: Training policy entropy comparison on the 4B model. Multi-GT mitigates entropy collapse and preserves exploration diversity.

Figure 5: Online win-rate comparison (ECS-based).

Panel of Figure 6 — Figure 6: Spearman rank correlation ( $\rho$ ) with the Ensemble-Consistency Score (oracle). (a) The Reranker alone reaches only $\rho=0.3295$ , while the Judge ( $0.5166$ ) and the Hybrid ( $0.5172$ ) are nearly identical, with the Hybrid marginally higher. (b) Sweeping the Fast-Pass budget, the Hybrid peaks at $\rho=0.5172$ and sustains a Judge-level correlation even as a sizable share of cases is offloaded to the Reranker.

Label	Embedding Score	Reranker Score	Content 1	Content 2
Topic-related, Contradictory	0.7413	0.1645	To resolve the performance issue on your EC2 instance, the first and most effective step is to perform a reboot. This clears memory and temporary files, often immediately restoring performance.	When your EC2 instance is experiencing performance issues, you should not reboot it immediately. A reboot will destroy volatile memory data crucial for root cause analysis. Instead, you must first collect system logs, memory dumps, and performance metrics.
Partially related, Partially contradictory	0.7107	0.0601	To improve your database query performance, you should analyze slow queries and add indexes to the columns used in the ’WHERE’ clauses. Having more indexes generally speeds up read operations.	While indexes can speed up read queries, be cautious. Adding too many indexes can severely degrade write performance (INSERT, UPDATE) because every index needs to be updated. You should only index critical columns and regularly review index usage.
Paraphrase	0.7382	0.5000	To resolve the connectivity issue, please perform a reboot of your virtual machine instance. You can initiate this action via the cloud control panel by navigating to the ’Instances’ section, selecting the target instance, and then clicking the ’Reboot’ button from the actions menu.	Have you tried turning it off and on again? Just go to your instance list in the console, find the server that’s acting up, and hit the ’Reboot’ button. This simple step often clears up temporary network glitches.
Unrelated	0.2293	0.0000	Your monthly invoice shows an increase in S3 storage costs. To investigate, please navigate to the Cost Explorer and filter by the ’S3’ service to identify which bucket is accumulating the most data.	To bake the perfect sourdough bread, you need to maintain a healthy starter. Feed it daily with a 1:1:1 ratio of starter, water, and flour. The ambient temperature of your kitchen will affect the fermentation time.

Appendix

Table 14: Comparison of Embedding Score and Reranker Score across Different Content Relationships

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 585 papers

AgentRM: Enhancing Agent Generalization with Reward Modeling
Feb 25, 2025Yu Xia, Jingru Fan, Weize Chen +7Reward Modeling LLM Agents
Department of Computer Science and Technology, Tsinghua University · Gaoling School of Artificial Intelligence, Renmin University of China
Training Advisors for LLM Agents from Task Outcomes
Oct 7, 2026Sergei Polezhaev, Barys Liskavets, Ori Press +1LLM Agent Training RL for LLM Agents
Nebius AI
CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments
Mar 30, 2026Yi Yu, Guangquan Hu, Chenghuang Shen +15LLM Agent Efficiency LLM Agent Evaluation
School of Mathematics and Sciences, Fudan University, Shanghai, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China · Shanghai Center for Mathematical Sciences, Fudan University, Shanghai, China +2

Feb 25, 2025J/K move · Enter open · S save

AgentRM: Enhancing Agent Generalization with Reward Modeling

Yu Xia, Jingru Fan, Weize Chen, Siyu Yan+6

Department of Computer Science and Technology, Tsinghua University · Gaoling School of Artificial Intelligence, Renmin University of China

Reward Modeling LLM Agents

Open paper AskPDF arXiv

Existing LLM-based agents have achieved strong performance on held-in tasks, but their generalizability to unseen tasks remains poor. Hence, some recent work focus on fine-tuning the policy model with more diverse tasks to improve the generalizability. In this work, we find that finetuning a reward model to guide the policy model is more robust than directly finetuning the policy model. Based on this finding, we propose AgentRM, a generalizable reward model, to guide the policy model for effective test-time search. We comprehensively investigate three approaches to construct the reward model, including explicit reward modeling, implicit reward modeling and LLM-as-a-judge. We then use AgentRM to guide the answer generation with Best-of-N sampling and step-level beam search. On four types of nine agent tasks, AgentRM enhances the base policy model by

8.8

points on average, surpassing the top general agent by

4.0

. Moreover, it demonstrates weak-to-strong generalization, yielding greater improvement of

12.6

on LLaMA-3-70B policy model. As for the specializability, AgentRM can also boost a finetuned policy model and outperform the top specialized agent by

11.4

on three held-in tasks. Further analysis verifies its effectiveness in test-time scaling. Codes will be released to facilitate the research in this area.

Abstract

LLM Agent Training

Explore similar work

AgentRM: Enhancing Agent Generalization with Reward Modeling

Training Advisors for LLM Agents from Task Outcomes

CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments