GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
Abstract
Offline reinforcement learning with verifiable rewards (RLVR) trains on a fixed pool of sampled responses and typically supervises every token of every response, although responses and tokens are not equally informative. We study which ones to supervise through the Gibbs-optimal policy of KL-regularized RLVR, which offline training targets, and measure the mismatch between a policy and this target with a bidirectional KL divergence. At the reference policy, this mismatch factorizes exactly and separates the two outcomes: correct responses contribute mainly through the forward term, concentrated on their low-probability tokens, and incorrect responses through the reverse term, concentrated on the responses the policy generates often and on their high-probability tokens. Based on this analysis, we propose GIST (Gibbs-Targeted Hierarchical Data Selection), a two-level selection method that approximates these principles with signals available before training. GIST uses the sampling temperature to select correct responses on-policy and incorrect responses near the policy's mode, and it uses the initial policy's token probabilities to supervise only the tokens where the mismatch lies. Applied to two offline training objectives, GIST raises the average accuracy over GSM8K, MATH-500, and MBPP for all three models under both objectives. In a controlled comparison on Qwen3-0.6B/GSM8K, it outperforms seven selection baselines that choose from the same candidates while supervising 66.6% fewer tokens than training on all candidates of the same prompts.
Figures & tables
| Class | Dominant term | Response level | Token level |
|---|---|---|---|
| Correct | forward (coverage) | under-represented | low-probability tokens |
| Incorrect | reverse (over-commitment) | over-represented | high-probability tokens |
| GSM8K | MATH-500 | MBPP-full | |||||
|---|---|---|---|---|---|---|---|
| Model | Method | Strict | Flex. | Strict | Flex. | pass@1 | Avg. |
| Qwen3-0.6B | Initial | 40.86 | 46.10 | 44.20 | 49.80 | 22.80 | 39.57 |
| SFT | 42.08 (+1.22) | 47.38 (+1.28) | 44.40 (+0.20) | 50.00 (+0.20) | 24.20 (+1.40) | 40.53 (+0.96) | |
| \cellcolor white | + GIST | 43.82 (+2.96) | 48.14 (+2.04) | 44.80 (+0.60) | 50.40 (+0.60) | 28.80 (+6.00) | 42.45 (+2.88) |
| BOLT | 43.29 (+2.43) | 47.46 (+1.36) | 44.60 (+0.40) | 50.40 (+0.60) | 26.60 (+3.80) | 41.49 (+1.92) | |
| \cellcolor white | + GIST | 45.79 (+4.93) | 49.73 (+3.63) | 45.40 (+1.20) | 51.00 (+1.20) | 31.40 (+8.60) | 44.04 (+4.47) |
| Random | PODS | RFT-style | RSI-S | ETS | ProFit | HES | GIST | |
|---|---|---|---|---|---|---|---|---|
| Strict | 43.14 | 43.52 | 44.05 | 43.21 | 43.75 | 42.46 | 44.20 | 45.79 |
| Flex. | 48.60 | 48.22 | 48.14 | 47.84 | 47.54 | 46.47 | 47.31 | 49.73 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | GSM8K | MATH | MBPP |
|---|---|---|---|
| Qwen3-0.6B | 1,593 / 6,973 | 1,619 / 7,000 | 66 / 374 |
| Qwen3.5-2B | 2,574 / 6,973 | 1,100 / 7,000 | 65 / 374 |
| Gemma-4-E2B-it | 1,373 / 6,973 | 1,646 / 7,000 | 58 / 374 |
| Comparison | Metric | Difference (pp) | Bootstrap 95% CI | McNemar |
|---|---|---|---|---|
| GIST BOLT | Strict | 0.027 | ||
| Flex. | 0.044 | |||
| GIST Random | Strict | 0.014 | ||
| Flex. | 0.293 | |||
| GIST All candidates | Strict | 0.072 | ||
| Flex. | 0.005 |
| Quantity | All candidates | Random | GIST |
|---|---|---|---|
| Retained responses | 50,976 | 12,744 | 12,744 |
| Selected epoch | 3 | 5 | 6 |
| Supervised tokens (M) | 13.596 | 6.641 | 4.546 |
| Supervised tokens per epoch (M) | 4.53 | 1.33 | 0.76 |
| Optimizer steps | 4,779 | 1,995 | 2,394 |
| Test Strict / Flex. (%) | 43.82 / 46.70 | 43.14 / 48.60 | 45.79 / 49.73 |
Offline RL
Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.
See all 327 papers