Adapting Technical-Service LLM Agents with Latent Logic Augmentation, Robust Noise Reduction, and Hybrid Reward Modeling
Authors: Junzhuo Ma, Chenghuang Shen, Yi Yu, Xingyan Liu, Jing Gu, Hangyi Sun, Guangquan Hu, Jianfeng Liu, Weiting Liu, Pu Mingyue, Wang Yu, Zhengdong Xiao, Rui Xie, Longjiu Luo, Qianrong Wang, Gurong Cui, Honglin Qiao, Wenlian Lu
Organizations: School of Mathematical Sciences, Fudan University, Shanghai, China · Alibaba Group, Hangzhou, China · Shanghai Center for Mathematical Sciences, Fudan University, Shanghai, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China · Center for Applied Mathematics & Shanghai Key Laboratory of Contemporary Applied Mathematics, Fudan University, Shanghai, China · Ji Hua Laboratory, Foshan, 528200, China
Technical-service LLM agents are entering production workflows, where value depends on whether engineers adopt generated replies. Service tickets hide decision logic, contain noisy single-reference responses, and make reward evaluation costly, making standard post-training brittle. Existing post-training and LLM-as-a-Judge approaches improve grounding or feedback, but do not jointly model latent decision logic, response diversity, and reward cost. We address this gap by coupling latent logic augmentation, robust noise reduction, and hybrid reward modeling. The framework augments supervised fine-tuning data with Planning-Aware Trajectory Modeling and Reasoning Augmentation, builds dual-filtered Multiple Ground Truths, and trains the policy with a hybrid reward that combines a Reranker with an LLM-as-a-Judge. On real Cloud technical-service tasks, the adapted Qwen3-4B achieves the highest Multi-ECS (0.441), lower reward cost, and the highest production adoption rate (46.63%).
Figures & tables
Figure 1: Three observations that motivate our framework , each grounded in a real Cloud technical-service ticket, mapping a pain point to the component that addresses it: hidden decision logic ( RA + PATM ), multiple valid answers for one query ( Multi-GT ), and a gameable cheap Reranker vs. an accurate costly judge ( Hybrid Reward ).
Figure 2: Overview of the Proposed Framework. The framework consists of four stages: (a) Latent Logic Augmentation; (b) Multi-GT data construction; (c) SFT adaptation; (d) RL adaptation with HRM.
Method
Multi-ECS
Single-ECS
Call Tool Acc
Untrained Qwen3-4B
0.299
0.178
0.082
SFT
0.293
0.193
0.096
SFT+RA
0.319
0.224
0.139
SFT+PATM
0.326
0.233
0.149
SFT+RA+PATM
0.337
0.242
0.279
Table 1: SFT strategy analysis (RQ1). RA and PATM each improve Multi-ECS and tool-use accuracy; their combination is best.
Configuration Variant
Multi-ECS
Single-ECS
Ours (SFT+RA+PATM + Hybrid + Single-GT)
0.429
0.357
(1) Varying SFT Initialization
→ replace with non-SFT
0.348
0.231
→ replace with SFT
0.353
0.331
→ replace with SFT+RA
0.406
0.336
→ replace with SFT+PATM
0.407
0.346
Table 2: RL Adaptation and Robust Noise Reduction Analysis (RQ2–3). We establish a default configuration ( Ours ) and independently ablate its components. Upgrading the data to Multi-GT yields the final peak performance. Paired one-sided t -tests on Multi-ECS confirm Ours significantly exceeds the Soft-Judge reward ( p<0.05 ), the Reranker-only reward, and the plain-SFT initialization (both p<0.001 ).
Figure 3: Expected Calibration Error (ECE) reliability diagrams for the three reward configurations. Bars show observed accuracy against predicted confidence; the dashed line denotes perfect calibration.
Model / Configuration
Adoption (%)
Untrained baselines
Qwen3-4B
32.80
Qwen3.7-Max
39.14
SFT only
SFT+RA+PATM
35.51
RL adaptation (reward ablations)
Table 3: Online adoption rate (%) from a retrospective production study. The RL-adaptation rows mirror the one-component ablations of Ours in Table 2 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Content
Context
The customer uses LATERAL to expand a JSON array. The test code runs fine in the query IDE, but after switching to the real table it fails in the data platform with “Feature not supported: LATERAL”.
Ground Truth
( inquire ) “@user Please connect the real table and run it in the query IDE, then send me the full execution log; I have not found the corresponding error record yet.”
(Expert strategy: re-run the diagnosis in a clean query-IDE environment and collect full logs)
Baseline <think>
“No tools are available right now… the help doc says LATERAL is not supported… I should advise avoiding this syntax.”
(Leaps from “no tools available” to a generic workaround, ignoring the environment discrepancy)
Baseline Response
“@user Your current version is 3.0, which does not support LATERAL ; I recommend using another method to process the JSON array.”
Appendix
Table 6: Case study for PATM: a LATERAL query succeeds on test data in the query IDE but fails on the real table in the data platform. Abbreviated reasoning traces ( <think> ) contrast the two decision processes; the score is the Multi-ECS against the reference set.
Component
Content
Context
In the managed-database event center the customer sees several “instance VIP migration” events and asks whether this means the instance will be migrated away. No instance ID is provided in the conversation.
Ground Truth
( inquire ) “@user To check whether this instance VIP migration will affect your instance, could you please send the instance ID?”
(Expert strategy: obtain the instance ID before diagnosing the event)
rR : Reranker-trained
“@user Yes, this event is a migration of the instance’s virtual IP address; it is a normal maintenance operation and will not affect instance availability.”
(✗ confident but premature solution from generic experience)
rJ : Judge-trained
“@user An instance VIP migration usually means the virtual IP configuration has changed, for network optimization or maintenance; it is normal operational behavior and does not cause the instance itself to be migrated away.”
Appendix
Table 7: Case study for Hybrid Reward: the customer sees several “instance VIP migration” events in the managed-database console and asks whether the instance will be migrated away— without providing an instance ID . The reference strategy is to inquire for the instance ID before diagnosing; the two wrong answers are plausible but premature solutions. We list the responses of three policies, each trained with a different reward signal.
rR
rJ
rH
(solution)
(solution)
(inquire)
SR (Reranker)
0.94
0.50
0.88
SJ (Judge)
0.30
0.82
0.60
Cascade region
Mix ↑
Mix ↓
Fast Pass
Rθ (Hybrid)
0.76
0.80
0.88
Multi-ECS (oracle)
0.02
0.01
1.00
Appendix
Table 8: Reward scores assigned to the three responses of Table 7 by each signal. SR / SJ are the Reranker/Judge consistency scores; Rθ is the Hybrid cascade (Eq. 2 ) with [τa,τb]=[0.6,0.9] , w1=0.05 , w2=0.72 ; Multi-ECS is the oracle. Mix ↑ /Mix ↓ denote the upper ( SR>τb ) and lower ( SR<τa ) mixing branches. Bold marks the top-scored response per signal: the Reranker and the Judge each rank a different wrong answer highest, whereas the Hybrid and the oracle both rank rH highest.
Component
Content
Context
The customer has multiple data sources pulling data from business databases and is unsure which tables have already been pulled; for each new extraction they are also unsure whether it was done before, and ask where this can be identified.
Ground Truth
GT-1 ( clarify ): “Are you using an offline synchronization task to pull the data?”
(3 references)
GT-2 ( solution ): “On the Data Source Management page, check the created data sources and their sync-task records to confirm the synchronized tables.”
GT-3 ( solution ): “Go to Data Development > Project Directory > Offline Sync Nodes to view the configured sync tasks and execution records.”
Single-GT
All four samples collapse to the same inquire (workspace/project ID):
(Avg. Multi-ECS
“(1) Could you provide the specific workspace name or project ID? (2) Which data source’s table specifically? (3) Which workspace are you operating in? (4) Could you provide the workspace name?”
Appendix
Table 9: Case study for Multi-GT: the customer asks how to identify which tables have already been synchronized in the data platform. The reference set admits one clarify and two solution strategies. We draw four samples from each policy; Avg. Multi-ECS is averaged over the four samples.
Category
Hyperparameter
Value
Data
Max Sequence Length
20000
Optimization
Base Model
Qwen3-4B
Global Batch Size
256
Learning Rate
2×10−6
Min Learning Rate
8×10−7
LR Scheduler
Cosine
Appendix
Table 10: Hyperparameters for Supervised Fine-Tuning (SFT).
Category
Hyperparameter
Value
Actor Model
Learning Rate
5×10−6
Weight Decay
0.0001
LR Scheduler
Constant
Clip Ratio
0.2
Clip Ratio(high)
0.28
Entropy Coeff
3.5×10−4
Appendix
Table 11: Hyperparameters for Reinforcement Learning (DAPO).
(a) Prompting-only models
Model
Single-ECS
Multi-ECS
Setting
Qwen3.7-Max
0.232
0.348
prompt-only
DeepSeek-V4-Pro-Preview
0.251
0.374
prompt-only
GLM-5.1
0.241
0.370
prompt-only
Kimi-K2.6
0.232
0.358
prompt-only
MiniMax-M2.5
0.237
0.364
prompt-only
Appendix
Table 12: Prompting-only comparisons, backbone transfer, and matched training baselines. Cells marked S/M report Single-/Multi-ECS. All evaluations use the same frozen 1,000-request test set and ECS protocol.
GT Category
Bare Model
Judge
Reranker
Hybrid
reply:solution
0.320
0.468
0.462
0.465
reply:inquire
0.063
0.268
0.194
0.283
reply:clarify
0.111
0.186
0.145
0.240
Action-Type Classification Accuracy (%)
Overall
19.00
62.90
61.70
64.80
Appendix
Table 13: Per-category ECS scores across reward configurations. Hybrid achieves the best performance on inquire and clarify , demonstrating superior calibration on ambiguous action types.
Figure 4: Training policy entropy comparison on the 4B model. Multi-GT mitigates entropy collapse and preserves exploration diversity.
Figure 5: Online win-rate comparison (ECS-based).
Figure 6: Spearman rank correlation ( ρ ) with the Ensemble-Consistency Score (oracle). (a) The Reranker alone reaches only ρ=0.3295 , while the Judge ( 0.5166 ) and the Hybrid ( 0.5172 ) are nearly identical, with the Hybrid marginally higher. (b) Sweeping the Fast-Pass budget, the Hybrid peaks at ρ=0.5172 and sustains a Judge-level correlation even as a sizable share of cases is offloaded to the Reranker.
Label
Embedding Score
Reranker Score
Content 1
Content 2
Topic-related, Contradictory
0.7413
0.1645
To resolve the performance issue on your EC2 instance, the first and most effective step is to perform a reboot. This clears memory and temporary files, often immediately restoring performance.
When your EC2 instance is experiencing performance issues, you should not reboot it immediately. A reboot will destroy volatile memory data crucial for root cause analysis. Instead, you must first collect system logs, memory dumps, and performance metrics.
Partially related, Partially contradictory
0.7107
0.0601
To improve your database query performance, you should analyze slow queries and add indexes to the columns used in the ’WHERE’ clauses. Having more indexes generally speeds up read operations.
While indexes can speed up read queries, be cautious. Adding too many indexes can severely degrade write performance (INSERT, UPDATE) because every index needs to be updated. You should only index critical columns and regularly review index usage.
Paraphrase
0.7382
0.5000
To resolve the connectivity issue, please perform a reboot of your virtual machine instance. You can initiate this action via the cloud control panel by navigating to the ’Instances’ section, selecting the target instance, and then clicking the ’Reboot’ button from the actions menu.
Have you tried turning it off and on again? Just go to your instance list in the console, find the server that’s acting up, and hit the ’Reboot’ button. This simple step often clears up temporary network glitches.
Unrelated
0.2293
0.0000
Your monthly invoice shows an increase in S3 storage costs. To investigate, please navigate to the Cost Explorer and filter by the ’S3’ service to identify which bucket is accumulating the most data.
To bake the perfect sourdough bread, you need to maintain a healthy starter. Feed it daily with a 1:1:1 ratio of starter, water, and flour. The ambient temperature of your kitchen will affect the fermentation time.
Appendix
Table 14: Comparison of Embedding Score and Reranker Score across Different Content Relationships
Existing LLM-based agents have achieved strong performance on held-in tasks, but their generalizability to unseen tasks remains poor. Hence, some recent work focus on fine-tuning the policy model with more diverse tasks to improve the generalizability. In this work, we find that finetuning a reward model to guide the policy model is more robust than directly finetuning the policy model. Based on this finding, we propose AgentRM, a generalizable reward model, to guide the policy model for effective test-time search. We comprehensively investigate three approaches to construct the reward model, including explicit reward modeling, implicit reward modeling and LLM-as-a-judge. We then use AgentRM to guide the answer generation with Best-of-N sampling and step-level beam search. On four types of nine agent tasks, AgentRM enhances the base policy model by 8.8 points on average, surpassing the top general agent by 4.0. Moreover, it demonstrates weak-to-strong generalization, yielding greater improvement of 12.6 on LLaMA-3-70B policy model. As for the specializability, AgentRM can also boost a finetuned policy model and outperform the top specialized agent by 11.4 on three held-in tasks. Further analysis verifies its effectiveness in test-time scaling. Codes will be released to facilitate the research in this area.
Yu Xia, Jingru Fan, Weize Chen +7
Department of Computer Science and Technology, Tsinghua University · Gaoling School of Artificial Intelligence, Renmin University of China
Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference critiques, Caddie learns from whether the agent ultimately succeeds after receiving the critic's feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including τ3 and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.
The increasing agentic capabilities of Large Language Models (LLMs) have enabled their deployment in real-world applications, such as cloud services, where customer-assistant interactions exhibit high technical complexity and long-horizon dependencies, making robustness and resolution efficiency critical for customer satisfaction. However, existing benchmarks for LLM-based agents largely rely on synthetic environments that fail to capture the diversity and unpredictability of authentic customer inputs, often ignoring the resolution efficiency essential for real-world deployment. To bridge this gap, we introduce CirrusBench, a novel evaluation framework distinguished by its foundation in real-world data from authentic cloud service tickets. CirrusBench preserves the intricate multi-turn logical chains and realistic tool dependencies inherent to technical service environments. Moving beyond execution correctness, we introduce novel Customer-Centric metrics to define agent success, quantifying service quality through metrics such as the Normalized Efficiency Index and Multi-Turn Latency to explicitly measure resolution efficiency. Experiments utilizing our framework reveal that while state-of-the-art models demonstrate strong reasoning capabilities, they frequently struggle in complex, realistic multi-turn tasks and fail to meet the high-efficiency standards required for customer service, highlighting critical directions for the future development of LLM-based agents in practical technical service applications. CirrusBench evaluation framework is released at: https://github.com/CirrusAI
Yi Yu, Guangquan Hu, Chenghuang Shen +15
School of Mathematics and Sciences, Fudan University, Shanghai, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China · Shanghai Center for Mathematical Sciences, Fudan University, Shanghai, China +2