cs.LGMar 19, 2026
SaveHISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning
Organizations: Aerospace Information Research Institute, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tencent Hunyuan · School of Computer Science and Technology, Tianjin University
Abstract
While large language models excel in diverse domains, their performance on complex longhorizon agentic decision-making tasks remains limited. Most existing methods concentrate on designing effective reward models (RMs) to advance performance via multi-turn reinforcement learning. However, they suffer from delayed propagation in sparse outcome rewards and unreliable credit assignment with potentially overly fine-grained and unfocused turnlevel process rewards. In this paper, we propose (HISR) exploiting Hindsight Information to modulate Segmental process Rewards, which closely aligns rewards with sub-goals and underscores significant segments to enhance the reliability of credit assignment. Specifically, a segment-level process RM is presented to assign rewards for each sub-goal in the task, avoiding excessively granular allocation to turns. To emphasize significant segments in the trajectory, a hindsight model is devised to reflect the preference of performing a certain action after knowing the trajectory outcome. With this characteristic, we design the ratios of sequence likelihoods between hindsight and policy model to measure action importance. The ratios are subsequently employed to aggregate segment importance scores, which in turn modulate segmental process rewards, enhancing credit assignment reliability. Extensive experimental results on three publicly benchmarks demonstrate the validity of our method.
Figures & tables
Figure 1: Comparison between mainstream reward models and our method in multi-turn reinforcement learning. , , denote the decided action in -th turn, the outcome of trajectory, and process rewards. (b) and (c) assign rewards at turn-level granularity. The unfocused process rewards in (c) denote that credit assignment fails to consider the action importance, while ours (d) aligns rewards with sub-goals and underscores significant segments.
Figure 2: An illustration of our method. The left showcases the details of behavior cloning and trajectory collection. The middle illustrates the training and inference of the segment-level process reward and hindsight model. The right describes the construction of hindsight information modulated rewards and employs them for policy optimization.
| Methods | Type | Alfworld | Virtualhome | Webshop | ||||||
| PICK | CLEAN | HEAT | COOL | LOOK | PICK2 | Avg | ||||
| Llama3.2 Dubey et al. (2024) | PE | 12.5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2.3 | 1.2 | 19.8 |
| GPT4o Hurst et al. (2024) | PE | 75.3 | 60.8 | 31.2 | 56.7 | 21.6 | 49.8 | 48.0 | 20.8 | 23.7 |
| Gemini2.5pro Team et al. (2023) | PE | 92.8 | 63.3 | 62.1 | 69.0 | 26.6 | 58.7 | 60.3 | 31.7 | 35.9 |
| SFT Ouyang et al. (2022) | BC | 79.2 | 77.4 | 73.9 | 61.9 | 83.3 | 58.8 | 73.1 | 51.8 | 62.0 |
| RFT Tang et al. (2024) | BC | 83.3 | 83.9 | 78.3 | 66.7 | 77.8 | 47.1 | 74.6 | 53.1 | 62.9 |
Table 1: Evaluation results and ablation study on three benchmarks. Avg indicates the comprehensive score in the Alfworld dataset. PE, BC, and RL indicate methods of prompt engineering, behavior cloning, and reinforcement learning. Best in bold, the runner-up with an underline. The top three parts denote baseline methods, while the last part denotes our method and its ablation variants. denotes it adopts turn-level process rewards.
| Segment (turn) | Conversations | |
| Case: Clean and put bowl Fridge | ||
| Segment1 (turn1) Task interpretation and decomposition | Task Define : You are in the middle of a room… Your task is to: clean some bowl and put it in fridge. Thought : The task is to clean a bowl and put it in the fridge. I should first locate the fridge to find a bowl that needs cleaning. Action : go to fridge 1. | |
| Segment2 (turn2-3) Opening the fridge and picking up the bowl | Observation : The fridge 1 is closed. Thought : I’m in front of the fridge, but it’s closed. I need to open it to find a bowl. Action : open fridge 1. | ({\color[rgb]{1,0,0}0.118},\ \textbf{0.392},\ {\color[rgb]{1,0,0}\textbf{0.205}\uparrow}) |
| Observation : You open the fridge 1. The fridge 1 is open. In it, you see a apple 3, a apple 2, a apple 1, a bowl 1, a bread 1, and a cup 1. Thought : The fridge is now open, and I see a bowl inside. I should take the bowl out to clean it. Action : take bowl 1 from fridge 1. | ||
| Observation : You pick up the bowl 1 from the fridge 1. Thought : With the bowl in hand, I need to find a place to clean it. The sink is the best place for this task. Action : go to sinkbasin 1. | ||
| Segment3 (turn4-5) Going to the sink and cleaning the bowl | Observation : On the sinkbasin 1, you see nothing. Thought : I’m at the sink and it’s clear, which is perfect for cleaning the bowl without any obstructions. Action : clean bowl 1 with sinkbasin 1. | |
Table 2: Case study on the Alfworld dataset. The left shows segmented results of the trajectory, where Segment (turn) denotes consecutive turns contained in the segment. The middle shows the explored trajectory, where blue texts highlight the highest importance scores segment. The right shows the tuple of raw segmental process rewards, hindsight information modulated rewards, and segmental importance ratio scores.
Figure 3: Statistical results of the number of turns in the segment of the Alfworld dataset.
Figure 4: Statistical results of the process rewards distribution in the Alfworld datasets. SPR denote the segmental process rewards.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Train | Test | Available Actions | Maximum Turns |
| ALFWorld | 2,851 | 134 | 13 | 40 |
| VirtualHome | 4,920 | 247 | 40 | 40 |
| Webshop | 1938 | 200 | - | 10 |
Table 3: The statistical results of three agentic benchmarks. Train and Test denote the number of expert trajectory sample and test sample, respectively.
| Segment (turn) | Conversations | |
| Alfworld Case: Put a clean apple Microwave | ||
| Segment1 (turn1) Initial task understanding and first action | Task Define : You are in the middle of a room. Looking quickly around you, you see a cabinet 8, a cabinet 7, a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 2, a countertop 1, a fridge 1, a garbagecan 1, a microwave 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1. Your task is to: put a clean apple in microwave. Thought : I need to put a clean apple in the microwave. First, I should locate an apple and ensure it’s clean before I can use it. Action : go to garbagecan 1. | |
| Segment2 (turn2-3) Acquiring the apple | Observation : On the garbagecan 1, you see a apple 2. Thought : I see an apple in the garbage can. I should pick it up and inspect it to see if it’s clean or if it needs to be cleaned before use. Action : take apple 2 from garbagecan 1. | ({\color[rgb]{1,0,0}0.092},\ \textbf{0.291},\ {\color[rgb]{1,0,0}\textbf{0.168}\uparrow}) |
| Observation : You pick up the apple 2 from the garbagecan 1. Thought : The apple is not clean, so I should wash it to ensure it’s suitable for eating or use in the microwave. Action : go to sinkbasin 1. | ||
| Observation : On the sinkbasin 1, you see nothing. Thought : I’m at the sink. Now I can clean the apple to make sure it’s ready to be put in the microwave. Action : clean apple 2 with sinkbasin 1. | ||
| Segment3 (turn4-5) Cleaning the apple | Observation : You clean the apple 2 using the sinkbasin 1. Thought : The apple is now clean. I need to find the microwave and open it to place the apple inside. Action : go to microwave 1. | |
Table 4: More case study on the Alfworld and Virtualhome dataset. The left shows segmented results of the trajectory, where Segment (turn) denotes consecutive turns contained in the segment. The middle shows the explored trajectory, where blue texts highlight the highest importance scores segment. The right shows the tuple of raw segmental process rewards, hindsight information modulated rewards, and segmental importance ratio scores.
| Segment (turn) | Conversations | |
| Alfworld Case: Put a clean soapbar Cabinet | ||
| Segment1 (turn1) Understanding the task and initiating a search for the soapbar | Task Define : You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1. Your task is to: put a clean soapbar in cabinet. Thought : The task is to put a clean soapbar in a cabinet. I should first look for a soapbar in the immediate vicinity. Action : go to toilet 1. | |
| Segment2 (turn2) Identifying and taking the soapbar | Observation : On the toilet 1, you see a soapbar 1, a spraybottle 1, and a toiletpaper 1. Thought : I’ve found a soapbar on the toilet. I need to pick it up to clean it before placing it in the cabinet. Action : take soapbar 1 from toilet 1. | |
| Observation : You pick up the soapbar 1 from the toilet 1. Thought : With the soapbar in hand, I should go to the sink to clean it since it’s important to ensure cleanliness before storing or using it. Action : go to sinkbasin 1. | ||
| Segment3 (turn3-4) Preparing and cleaning the soapbar | Observation : On the sinkbasin 1, you see nothing. Thought : I’m at the sink. Now I should clean the soapbar to make sure it’s clean before placing it in the cabinet. Action : clean soapbar 1 with sinkbasin 1. | ({\color[rgb]{1,0,0}0.063},\ \textbf{0.389},\ {\color[rgb]{1,0,0}\textbf{0.165}\uparrow}) |
| Segment4 (turn5-6) Locating and accessing the cabinet | Observation : You clean the soapbar 1 using the sinkbasin 1. Thought : The soapbar is now clean. I need to find a cabinet to store the soapbar in. Action : go to cabinet 1. | |
Table 5: More case study on the Alfworld dataset. The left shows segmented results of the trajectory, where Segment (turn) denotes consecutive turns contained in the segment. The middle shows the explored trajectory, where blue texts highlight the highest importance scores segment. The right shows the tuple of raw segmental process rewards, hindsight information modulated rewards, and segmental importance ratio scores.
| Format | Prompt |
| Alfworld | |
| Human | Interact with a household to solve a task. Imagine you are an intelligent agent in a household environment, and your target is to perform actions to complete the task goal. At the beginning of your interactions, you will be given a detailed description of the current environment and your goal to accomplish. For each of your turns, you will be given the observation of the last turn. You should first think about the current condition and plan for your future actions, and then determine your action in this turn. Your output must strictly follow this format:’’Thought: your thoughts. Action: your next action’’. The admissible actions are: 1. go to {recep} 2. task {obj} from {recep} 3. put {obj} in/on {recep} 4. open {recep} 5. close {recep} 6. toggle {obj} {recep} 7. clean {obj} with {recep} 8. heat {obj} with {recep} 9. cool {obj} with {recep} where {obj} and {recep} correspond to objects and receptacles. After your each turn, the environment will give you immediate feedback based on which you plan your next few steps. If the environment returns ’’Nothing happened’’, that means the previous action is invalid, and you should try other reasonable actions. Your response should use the following format: Thought: <your thoughts> Action: <your next action> Critical INSTRUCTIONS: 1. Action Format: You must use the exact action formats provided above. NEVER simplify or modify them. 2. Grounding in History (previous actions and observations): When planning your future actions, you have to be strictly based on the history (previous actions and observations). Never assume anything not explicitly stated in the history. 3. Precise Task Decomposition: You should accurately understand and decompose the given task. Never overlook any details. |
| GPT | OK, ...... |
Table 6: Prompt template for behavior cloning and trajectory collection in the AlfWorld dataset.
| Format | Prompt |
| VirtualHome | |
| Human | Interact with a household to solve a task. Imagine you are an intelligent agent in a household environment, and your target is to perform actions to complete the task goal. At the beginning of your interactions, you will be given a detailed description of the current environment and your goal to accomplish. For each of your turns, you will be given the observation of the last turn. You should first think about the current condition and plan for your future actions, and then determine your action in this turn. Your output must strictly follow this format:’’Thought: your thoughts. Action: your next action’’. The admissible actions are: walk to {obj} run to {obj} grab {obj} open {obj} close {obj} put {obj} on {recep} put {obj} in {recep} switch on {obj} switch off {obj} drink {obj} look at {obj} sit on {obj} stand up watch {obj} wipe {obj} type on {obj} wash {obj} cut {obj} eat {obj} sleep wake up plug in {obj} plug out {obj} pour {obj} into {recep} move {obj} release turn to {obj} where {obj} and {recep} correspond to objects and receptacles. After your each turn, the environment will give you immediate feedback based on which you plan your next few steps. If the environment returns ’’Nothing happened’’, that means the previous action is invalid, and you should try other reasonable actions. Your response should use the following format: Thought: <your thoughts> Action: <your next action> Critical INSTRUCTIONS: 1. Action Format: You must use the exact action formats provided above. NEVER simplify or modify them. 2. Grounding in History (previous actions and observations): When planning your future actions, you have to be strictly based on the history (previous actions and observations). Never assume anything not explicitly stated in the history. 3. Precise Task Decomposition: You should accurately understand and decompose the given task. Never overlook any details. |
| GPT | OK, ...... |
Table 7: Prompt template for behavior cloning and trajectory collection in the Virtualhome dataset.
| Format | Prompt |
| Webshop | |
| Human | You are web shopping. I will give you instructions about what to do. You have to follow the instructions. Every round I will give you an observation and a list of admissible actions, you have to respond an action based on the state and instruction. You can use search action if search is admissible. You can click one of the buttons in clickables. Your output must strictly follow this format:’’Thought: your thoughts. Action: your next action’’. The admissible actions should be of the following structure: search[keywords] click[value] where [keywords] in search are up to you, but the [value] in click must be a value in the list of admissible actions. If the action is not valid, perform nothing. Remember that your keywords in search should be carefully designed. Your response should use the following format: Thought: <your thoughts> Action: <your next action> Critical INSTRUCTIONS: 1. Action Format: You must use the exact action formats provided above. NEVER simplify or modify them. 2. Grounding in History (previous actions and observations): When planning your future actions, you have to be strictly based on the history (previous actions and observations). Never assume anything not explicitly stated in the history. 3. Precise Task Decomposition: You should accurately understand and decompose the given task. Never overlook any details. |
| GPT | OK, ...... |
Table 8: Prompt template for behavior cloning and trajectory collection in the Webshop dataset.
| Role | Content |
| Human | Imagine you are a trajectory expert of household agents. You are given a completed exploration trajectory of a household agent in the environment for a certain task, but one of the interactive turns of the household agent is lost (represented by the underline '___'). Your goal is to predict the lost interactive turn, like the task of cloze. At the begining of the given trajectory, you can see a detailed description of the environment in which the household agent is located and the task to be accomplished. Then you can see all interactive turns except for the lost one. Each interative turn contains the 'Observation' returned by the environment (Note: if the envrionment outputs "Nothing happened", it means the previous action is invalid), the 'Thought' and 'Action' output by the household agent. At the end, you can see the scalar reward for the given trajectory, which reflects how well the trajectory accomplishes the task ('0.00' and '1.00' represent complete failure and success of the task, respectively). When predicting the lost interactive turn, you should comprehensively consider the visible context before and after the lost interactive turn as well as the final scalar reward. You are only required to output the lost 'Thought' and 'Action'. Your output must strictly follow this format:"Thought: your thoughts. Action: your next action". The available actions are: 1. go to {recep} 2. task {obj} from {recep} 3. put {obj} in/on {recep} 4. open {recep} 5. close {recep} 6. toggle {obj} {recep} 7. clean {obj} with {recep} 8. heat {obj} with {recep} 9. cool {obj} with {recep} where {obj} and {recep} correspond to objects and receptacles. Your response should only include the lost turn in the following format: Thought: <your thoughts> Action: <your next action> |
| GPT | OK |
| Human | You are in the middle of a room. Looking quickly around you, you see a armchair 1, a bed 1, a diningtable 1, a drawer 2, a drawer 1, a garbagecan 1, a sidetable 2, and a sidetable 1. Your task is to: find two pillow and put them in armchair. |
| GPT | |
| Human | Observation: On the sidetable 1, you see a cellphone 3, and a keychain 2. |
| GPT | Thought: There are no pillows on sidetable 1, only a cellphone and a keychain. I should check the other side table. Action: go to sidetable 2 |
Table 9: Prompt template for hindsight model training in the Alfworld & Virtualhome dataset.
| Role | Content |
| Human | Imagine you are a trajectory expert of webshop agents. You are given a completed exploration trajectory of webshop agent in the web for a certain task, but one of the interactive turns of the webshop agent is lost (represented by the underline "___"). Your goal is to predict the lost interactive turn, like the task of cloze. At the beginning of the given trajectory, you can see a detailed description of the task that the webshop agent has to accomplish. Then you can see all interactive turns except for the lost one. Each interative turn contains the "Observation" returned by the web, the "Thought" and "Action" output by the webshop agent. At the end, you can see the scalar reward for the given trajectory, which reflects how well the trajectory accomplishes the task ("0.00" and "1.00" represent complete failure and success of the task, respectively). When predicting the lost interactive turn, you should comprehensively consider the visible context before and after the lost interactive turn, as well as the final scalar reward. Your prediction for the lost interactive turn should include only the "Thought" and "Action" fields, in the format below: Thought: <your thoughts> Action: <your next action> |
| GPT | OK |
| Human | WebShop [SEP] Instruction: [SEP] i'm looking for native american indian dream catcher feathers talisman, and price lower than 50.00 dollars [SEP] Search |
| GPT | |
| Human | Observation: Instruction: [SEP] i'm looking for native american indian dream catcher feathers talisman, and price lower than 60.00 dollars [SEP] Back to Search [SEP] Page 1 (Total results: 50) [SEP] Next > [SEP] B07TGHZH66 [SEP] BlessLiving Wolf Dream Catcher Comforter Set Native American Wolf Comforter King Size Bed Set Indian Wolf Bedding Sets King, Brown Gold, 3 Piece, 1 Quilt 2 Shams [SEP] 7.77 [SEP] B08ZXQW35Z [SEP] Bohemian Dream Catcher Printed Shade Durm Shape Lampshade for Table Lamp and Floor Light, Home Decor, 7.87 R x 7.08 H inches [SEP] 33.99 [SEP] ... |
| GPT | Thought: I think I should click on the product that matches the description of a Native American Indian dream catcher feathers talisman and is priced below $60.00. Action: click[B095VX97GN] |
Table 10: Prompt template for hindsight model training in the Webshop dataset.
| Dataset | Prompt |
| Alfworld & Virtualhome | You are an expert at segmenting multi-turn interactive trajectories between intelligent agents and household environments into meaningful stages. You are given a multi-turn interactive trajectory where each round contains a ’user’ (the environment: initial task or feedback/observation) and an ’assistant’ (the agent: reasoning ’Thought’ and chosen ’Action’) interaction. Your task is to segment this trajectory into several consecutive segments, where each segment represents a distinct stage or subgoal in the agent’s progress toward the specified task. Critical INSTRUCTIONS: 1. Segmentation criteria: Each segment should reflect a clear change in the agent’s high-level intention, intermediate sub-goal, strategy shift, or a significant event in the environment (for example, a new search target, a new object found, switching from searching for an object to manipulating it, picking up a crucial object, or a clear environmental cue causes a shift in plan). Segments can span several turns or just one, but adjacent segments must differ in agent intent or environmental context. Do not make segments too coarse: Prefer more segments with fewer turns each, unless the trajectory truly supports only a few distinct stages. Too few segments (fewer than {} turns) suggest your segmentation may be coarse. 2. Output requirements: For each segment you identify, briefly explain the stage transition or boundary in 1-2 sentences; describe what defines the start/end of that segment. At the end, return all index ranges (1-based) for each segment clearly in this format: [(1, 3), (4, 6), (7), (8, 13), (14, 20)], the index ranges should be consecutive. Ensure the total number of segments is at least 4 whenever possible, unless the trajectory is exceptionally short. Example: For a trajectory of 8 turns, you may segment as follows: (1, 3): Agent interprets the task and starts searching. (4, 6): Agent finds possible location and investigates. (7): Agent acquires object. (8, 13): Agent manipulates the object. (14, 20): Agent arrives at the destination and puts on the obejct to compelete the task. and then return the index ranges at the end of your response in the format: ’’[(1, 3), (4, 6), (7), (8, 13), (14, 20)]’’ Now, analyze the following provided multi-turn interactive trajectory (wrapped in **********) step by step, explain transitions, return the index ranges for each segment in the Square brackets [] at the end of your response in the format: ’The segmented results are: [fill the segmented results here]’ |
Table 11: Prompt templates for GPT-4o to perform trajectory segmentation across Alfworld & Virtualhome datasets.
| Dataset | Prompt |
| Webshop | You are an expert at segmenting multi-turn interactive trajectories between intelligent agents and webshopping environments into meaningful stages. You are given a multi-turn interactive trajectory where each round contains a ’user’ (the environment: initial task or feedback/observation) and an ’assistant’ (the agent: reasoning ’Thought’ and chosen ’Action’) interaction. Your task is to segment this trajectory into several consecutive segments, where each segment represents a distinct stage or subgoal in the agent’s progress toward the specified task. Critical INSTRUCTIONS: 1. Segmentation criteria: Each segment should reflect a clear change in the agent’s high-level intention, intermediate sub-goal, strategy shift, or a significant event in the environment (for example, a new search target, switching from searching to clicking, deciding to buy or perform another action, or a clear environmental cue causes a shift in plan). Segments can span several turns or just one, but adjacent segments must differ in agent intent or environmental context. Do not make segments too coarse: Prefer more segments with fewer turns each, unless the trajectory truly supports only a few distinct stages. Too few segments (fewer than {} turns) suggest your segmentation may be coarse. 2. Output requirements: For each segment you identify, briefly explain the stage transition or boundary in 1-2 sentences; describe what defines the start/end of that segment. At the end, return all index ranges (1-based) for each segment clearly in this format: [(1, 3), (4, 6), (7), (8, 10)]. the index ranges should be consecutive. Ensure the total number of segments is at least 3 whenever possible, unless the trajectory is exceptionally short. Example: For a trajectory of 8 turns, you may segment as follows: (1, 3): Agent interprets the task and starts searching. (4, 6): Agent searchs relevant objects in the webshop. (7): Agent clicks a corresponding button. (8, 10): Agent finds the target and clicks the ’Buy Now’ button to complete the task. and then return the index ranges at the end of your response in the format: ’’[(1, 3), (4, 6), (7), (8, 10)]’’ Now, analyze the following provided multi-turn interactive trajectory (wrapped in **********) step by step, explain transitions, return the index ranges for each segment in the Square brackets [] at the end of your response in the format: ’The segmented results are: [fill the segmented results here]’ |
Table 12: Prompt templates for GPT-4o to perform trajectory segmentation in the Webshop dataset.
More in this topic
RL for LLM Agents
Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.
See all 612 papers