cs.ROSep 18, 2026
SaveSeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation
Organizations: Carnegie Mellon University
Abstract
Despite rapid progress, generalist robot policies remain brittle on complex, long-horizon tasks that comprise multiple stages or require repeated attempts and deliberation on the same underlying stage before success. Q-value functions can improve these policies by ranking candidate actions or guiding policy improvement, but learning from sparse task-level rewards entails long credit-assignment horizons, difficult Bellman backups, and broad data-coverage requirements. We introduce SeeQ (Subtask-elicited Q-functions), which instead learns Q-values for the currently active subtask. This shortens the value-prediction horizon and enables effective learning with temporal-difference (TD) objectives. During training, subtask-level annotations present in offline robot data provide the decomposition and enable learning from broad, potentially suboptimal robot datasets. To eliminate the need for human annotations or modular subtask prediction systems at test time, our Q-function architecture is trained to autoregressively predict the active subtask in natural language before estimating its value. We instantiate SeeQ using a base vision-language backbone, pretrain it on diverse open-source robot manipulation data, and finetune it on downstream tasks. Across four real-world manipulation tasks on two bimanual robot platforms, the SeeQ value function substantially improves best-of-N policy steering.
Figures & tables
Figure 2 : Overview of the SeeQ value-function architecture. The model first rolls out the active subtask at state in text using a next-token prediction loss, and then predicts a scalar value for the input state-action pair conditioned on the task instruction and the subtask. Ground-truth subtask annotations supervise the rollout during training; at inference the model conditions on its own predicted subtask.
Figure 3 : Our bimanual real-robot evaluation tasks. Top-camera snapshots from dataset demonstrations of the four real-world tasks. Three tasks are set up on a bimanual xArm-7 platform, and one uses a bimanual YAM platform. See Appendix A.1.5 for task definitions and evaluation protocols.
Figure 4 : Qualitative value predictions for our approach compared to basline approaches. Panels (b) and (c) both show the values of selected best-of- action chunks in a SeeQ evaluation. (a) All critics evaluate recorded dataset action chunks. On a held-out RoboCOIN trajectory, task-level TD provides almost no signal for most of the trajectory, consistent with compounding bootstrapping errors, while MC predictions are noisy, potentially reflecting spurious correlations. (b) lid-sealing: The value drops at frame 3 after a failed flap closure, followed by recovery at frame 4. (c) grocery-packing: Value drops at frames 1 and 3 correspond to failed grasps, followed by recovery in both cases; BC often struggles repeatedly with objects such as the Pringles bag. The final subtask switch appears delayed because subtask decoding runs less frequently than action replanning. Solid curves denote SeeQ and TD-BoN; dashed curves denote SARSA and MC. Red vertical dashed lines mark SeeQ -predicted subtask switches and are omitted from the task-level value plot.
| Task | Data | Base | SeeQ |
|---|---|---|---|
| shirt-hang | RaC | 10/24 | 22/24 |
| lid-sealing | RaC | 10/24 | 15/24 |
| grocery-packing | Teleop | 9/24 | 17/24 |
| LEGO-disassembly | Teleop | 5/24 | 10/24 |
Table 1 : Policy steering results. Success rates over 24 trials for the base policy and for the same policy steered by SeeQ with best-of- action selection ( ).
| Value function | Robot pretraining | VLM initialization | Success rate |
|---|---|---|---|
| Base policy | — | — | 10/24 |
| Task-specific (ResNet-50 + MLP, TD-BoN) | ✗ | ✗ | 16/24 |
| SeeQ minus pretraining | ✗ | ✓ | 8/24 |
| SeeQ (Ours) | ✓ | ✓ | 22/24 |
Table 3 : Importance of pretraining and VLM initializations for SeeQ , evaluated on shirt-hang. Columns indicate whether the value function uses robot data pretraining and VLM initialization. Observe that both robot data pretraining and base VLM initialization are important for the success of SeeQ .
| Variant | Success rate |
|---|---|
| Base policy | 10/24 |
| Subtask-level TD No subtask prediction, no subtask conditioning | 8/24 |
| Subtask-level TD Subtask prediction, no subtask conditioning | 15/24 |
| SeeQ (Ours) | 22/24 |
Table 4 : Ablation of subtask elicitation on shirt-hang. Subtask prediction supervision improves success, with further gains from explicitly conditioning value estimation on the predicted subtask.
Figure 5 : Action and image sensitivity on six complete held-out shirt-hang trajectories. (a) Action-gradient norms. (b) Joint image-gradient norms across three cameras. All three critics use the same frames and eight cached policy candidates; gradients are evaluated at each critic’s selected candidate. No pretraining denotes SeeQ finetuned directly from PaliGemma. Bar heights are percentages of frames in shared logarithmically spaced bins; dashed lines denote medians.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6 : Q-values relative to recorded returns during RoboCOIN pretraining. Gray: batch-mean ; red: EMA with a -step half-life; shading: standard deviation of the exponentially weighted prediction-minus-return differences. (a) SeeQ with subtask returns shows a near-zero gap after initial underestimation. (b) Task-level TD-BoN persistently underestimates task returns.
Figure 7 : Subtask conditioning aligns value resets with predicted subtask changes. Both critics evaluate recorded dataset action chunks. Top: predicted values along a held-out shirt-hang trajectory. Bottom: enlarged views of the five shaded regions; colored dashed lines mark each critic’s predicted switch nearest the panel center. Fall span counts frames between the first and of the local peak-to-trough drop, using the maximum value in the frames before that critic’s switch and the minimum from the switch through frames after it. A drop crossing both thresholds in one frame has span . Panel titles report SeeQ versus no conditioning.
Figure 8 : Comparison of the value functions in Section 4.2 . All critics evaluate recorded dataset action chunks. (a) In this held-out episode, the Nesquik container must be reoriented in frame 2 because the soda can obstructs it, causing SeeQ ’s value to drop. In the final frame, MC predictions are noisy despite smooth progress. (b) In this held-out episode, the value jumps in frame 1 as the gripper reorients to grasp the hanger correctly. In frame 4, the left gripper unintentionally releases the collar, causing the value to drop. Frame 5 again shows noisy MC predictions.
Figure 9 : Comparison of the value functions in Section 4.3 on a held-out episode. All critics evaluate recorded dataset action chunks. In frame 3 (red), the right gripper releases the collar, causing the value to drop. The ResNet-50 value peaks about s before this release, so the evaluated action chunk includes the failure; this indicates an incorrect value estimate, unlike the VLM critics in this case. SeeQ generally produces smoother predictions than SeeQ without robot pretraining and the task-specific ResNet-50 critic, indicating better generalization.
| Task | Training steps | LR decay steps | Batch size |
|---|---|---|---|
| shirt-hang | 60k | 200k | 128 |
| lid-sealing | 40k | 60k | 256 |
| grocery-packing | 70k | 70k | 256 |
| LEGO-disassembly | 20k | 20k | 256 |
Table 5 : Downstream base-policy hyperparameters. Training steps refer to the policy checkpoint used in evaluation; LR decay length refers to the full configured cosine schedule.
| Critic | Return scope | NTP weight | Subtask conditioning | |
|---|---|---|---|---|
| SeeQ (TD-BoN) | Subtask | 0.999 | 0.1 | Yes |
| SARSA | Subtask | 0.999 | 0.1 | Yes |
| MC | Task | 0.9995 | 0 | No |
| TD-BoN | Task | 0.9995 | 0 | No |
| NTP, no conditioning | Subtask | 0.999 | 0.1 | No |
| Neither NTP nor conditioning | Subtask | 0.999 | 0 | No |
Table 6 : Critic pretraining variants. All rows use k pretraining steps and the shared schedule above. Conditioning refers to the critic’s value prediction, independently of the base-policy prompt.
| Critic | Downstream task | Finetuning steps | Checkpoint step |
|---|---|---|---|
| SeeQ | shirt-hang , lid-sealing , grocery-packing | 20k | 250k |
| SeeQ | LEGO-disassembly | 10k | 240k |
| Task-level MC | shirt-hang , grocery-packing | 20k | 250k |
| Task-level TD-BoN | shirt-hang , grocery-packing | 20k | 250k |
| Subtask-level SARSA | shirt-hang , grocery-packing | 20k | 250k |
| NTP, no conditioning | shirt-hang | 10k | 240k |
Table 7 : Downstream critic checkpoints. All rows initialize from the corresponding k pretrained critic and use a k LR decay schedule.
Figure 10 : Histogram of RoboCOIN subtask durations in seconds.
| Agilex Cobot Magic (63 tasks) | ||
| box-storage-chopsticks | cap-the-pen-a | catch-the-ball |
| classification-of-fruits-and-vegetables | classification-of-fruits-and-vegetables-a | classification-of-tableware |
| clean-blackboard | clean-up-the-tableware | clear-the-desktop |
| close-book | close-button | cube-reset |
| cut-banana | desktop-organization | drawer-storage-mineral-water |
| fold-clothes | fold-the-towel | fold-towel-a |
Table 8: The RoboCOIN task datasets used for pretraining. Task identifiers are grouped by embodiment, with the embodiment prefix omitted and underscores rendered as hyphens. Released spellings and variant suffixes are retained.