Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

Yunheng Li; Guohong Mu; Hao Li; Shengsheng Qian; Dingwen Zhang; Qibin Hou; Ming-Ming Cheng

cs.CVAug 20, 2026

Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

Authors: Yunheng Li, Guohong Mu, Hao Li, Shengsheng Qian, Dingwen Zhang, Qibin Hou, Ming-Ming Cheng

Organizations: VCIP, School of Computer Science, Nankai University · Brain and Artificial Intelligence Lab, Northwestern Polytechnical University · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · NKIARI, Futian, Shenzhen

Abstract

Multimodal large language models (MLLMs) have become a prevailing paradigm for unified video perception. However, post-training on large multi-task datasets remains challenging, as existing reinforcement learning methods sample on-policy groups with few high-quality rollouts even with costly chain-of-thought (CoT) generation. In this paper, we study the sample efficiency and scalability of RL post-training for video MLLMs and introduce OraRL. We identify an overlooked role for annotations: Beyond scoring rollouts, each can enter its on-policy group as an oracle rollout, a direct positive optimization target. Direct oracle integration, however, is nontrivial: a high-reward oracle raises the group baseline and inverts otherwise positive policy advantages, a failure we term advantage inversion. At the core of OraRL is a decoupled advantage estimator: policy rollouts determine an oracle-free baseline, while the oracle-policy gap modulates both a directional gain and a separate detached oracle advantage. Sign-balanced pruning improves efficiency: by retaining only the oracle and the strongest rollouts of each sign, OraRL requires just 2.2x the step time of SFT, less than half the 4.9x required by GRPO with CoT. OraRL scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts. Without chain-of-thought, Video-ORA-9B decodes in 130 ms instead of 4,780 ms. Compared with the respective prior best models, it raises temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, and the three-benchmark spatial-intelligence macro average from 51.0 to 56.1; on VSI-Bench, it scores 73.1 against 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.

Figures & tables

Panel of Figure 1 — Figure 1: Video-ORA: one model, trained by a single OraRL recipe, for unified video perception. Each column presents a representative input and a family-level score computed over common benchmark coverage. Video-ORA-9B, evaluated without chain-of-thought decoding, outperforms every displayed baseline [ 71 , 118 , 21 , 2 , 96 , 1 ] across seven task families. See Sec. 4 for detailed metrics.

Panel of Figure 2 — Figure 2: Four paradigms for using annotations in model adaptation. Panel (b) shows rewards, whereas panels (c) and (d) show advantages. Unlike mixed-policy normalization, OraRL excludes GT from the on-policy baseline while retaining it as an optimization target, preventing advantage inversion and enabling a sign-balanced update.

Figure 3: Overview of OraRL. The annotation is appended as an oracle rollout to $n$ on-policy rollouts. On-policy rewards define the baseline, while the oracle separately strengthens promising rollouts and provides an adaptive anchor. Sign-balanced pruning retains informative positive and negative rollouts together with the oracle, yielding $K$ rollouts for the policy update.

Model	Charades-TimeLens [ 24 , 108 ]				ActivityNet-TimeLens [ 33 , 108 ]				QVHighlights-TimeLens [ 36 , 108 ]
Model	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU	[email protected]	[email protected]	[email protected]	mIoU
Proprietary Models
GPT-5 [ 67 ]	59.3	42.0	22.0	40.5	57.4	44.9	30.4	42.9	72.4	60.4	46.4	56.8
Gemini-2.5-Pro [ 13 ]	74.1	61.1	34.0	52.8	72.3	64.2	47.1	58.1	84.1	75.9	61.1	70.4
Open-Source Models
Qwen3.5-35B-A3B [ 71 ]	69.5	50.4	27.3	48.2	59.4	58.6	40.1	52.6	81.6	72.6	56.5	66.0

Table 1: Temporal grounding results. We report [email protected], [email protected], [email protected], and mIoU on three TimeLens benchmarks.

Models	RefCOCO [ 101 ]			RefCOCO+ [ 101 ]			RefCOCOg [ 54 ]
Models	testA	testB	val	testA	testB	val	test	val
Perception-R1 [ 100 ]	91.4	84.5	89.1	86.8	74.3	81.7	85.4	85.7
Ground-R1-7B [ 6 ]	93.9	88.0	92.9	90.8	78.8	86.5	90.2	90.1
Game-R1-7B [ 57 ]	92.3	84.7	90.1	89.6	77.3	84.3	86.6	88.1
OneThinker-8B [ 21 ]	93.7	88.9	92.0	91.4	82.7	87.0	88.8	89.2
Qwen3.5-4B [ 71 ]	91.1	87.2	89.6	86.2	77.5	82.9	89.0	87.3

Table 2: Spatial grounding results on the RefCOCO family. All values are [email protected] on RefCOCO, RefCOCO+, and RefCOCOg.

Models	VideoMME	VideoMME-v2	MV-Bench	MMVU(mc)	VideoHolmes	LongVideoBench	MLVU-MC
InternVL3.5-8B [ 76 ]	66.0	26.0	72.1	–	–	62.1	70.2
Keye-VL-1.5-8B [ 92 ]	73.0	23.8	56.9	–	–	66.0	75.0
Eagle2.5-8B [ 7 ]	72.4	24.9	74.8	–	–	66.4	77.6
MiniCPM-V-4.5-8B [ 102 ]	67.9	26.5	60.5	–	–	63.9	70.2
InternVideo3-8B [ 91 ]	73.8	27.6	75.0	–	–	66.8	77.3
LLaVA-OneVision-2-8B [ 1 ]	71.9	19.9	66.2	–	–	66.9	76.6

Table 3: Video question answering results. We evaluate VideoMME [ 22 ] , VideoMME-v2 [ 23 ] , MV-Bench [ 41 ] , MMVU [ 115 ] , VideoHolmes [ 11 ] , LongVideoBench [ 82 ] , and MLVU [ 116 ] . LongVideoBench and MLVU-MC use validation and development M-Avg., respectively.

Models	Frame	GOT-10k [ 30 ]
Models	Frame	AO	[email protected]	[email protected]	[email protected]
Qwen3-VL-8B [ 2 ]	32	33.7	51.1	28.9	10.6
Qwen3.5-4B [ 71 ]	32	45.3	65.5	46.4	25.2
Qwen3.5-9B [ 71 ]	32	46.0	66.9	46.9	25.8
OneThinker-8B [ 21 ]	32	73.0	93.9	84.4	68.8
Video-ORA-4B	32	67.5	87.2	76.8	57.8

Table 4: Tracking results on GOT-10k. We report average overlap (AO) and recall at IoU thresholds 0.3, 0.5, and 0.7.

Models	Frame	STVG [ 38 ]
Models	Frame	[email protected]	tIoU	[email protected]	sIoU
Grounded-VideoLLM [ 73 ]	-	30.0	33.0	-	-
Qwen3-VL-8B [ 2 ]	128	24.4	25.4	11.6	13.6
Qwen3.5-4B [ 71 ]	128	35.8	37.7	6.3	11.2
Qwen3.5-9B [ 71 ]	128	37.0	39.3	11.9	16.5
Video-ORA-4B	128	39.0	40.5	22.2	24.2

Table 5: Spatial-temporal grounding results on STVG. We report [email protected], mean tIoU, [email protected], and mean sIoU.

Models	Numerical Question				Multiple-Choice Question				Avg.
Models	Obj. Count	Abs. Dist	Obj. Size	Room Size	Rel. Dis	Rel. Dir	Route Plan	Appr. Order	Avg.
Proprietary Models
Seed-2.0 [ 63 ]	49.4	25.3	69.5	25.8	61.8	44.9	44.3	71.0	49.0
Grok-4 [ 85 ]	37.1	32.9	60.8	45.4	53.1	39.6	47.4	66.8	47.9
Gemini-2.5-Pro [ 13 ]	46.0	37.3	68.7	54.3	61.9	43.9	47.4	68.7	53.5
Gemini-3-Pro [ 26 ]	49.0	42.8	71.5	41.8	56.6	57.5	61.9	60.0	55.1

Table 6: Complete results on VSI-Bench [ 93 ] . Numerical and multiple-choice tasks use MRA and accuracy; Avg. is their macro average.

Model	Image (cIoU)			Video (J&F)
Model	RefCOCO	+	g	MeViS	ReasonVOS
PixelLM-7B [ 61 ]	73.0	66.3	69.3	–	–
LISA-7B [ 35 ]	74.1	62.4	66.4	37.2	31.1
VISA-13B [ 89 ]	72.4	59.8	65.5	44.5	–
Seg-R1-7B [ 99 ]	74.3	62.6	71.0	–	–
Sa2VA-4B [ 103 ]	78.9	71.7	74.1	46.2	–

Table 7: Segmentation results. RefCOCO/+/g [ 101 , 54 ] use cIoU, whereas MeViS [ 17 ] and ReasonVOS [ 3 ] use J&F.

Model	MMSI	MindCube	Avg.
Proprietary Models
Gemini-2.5-Pro [ 13 ]	38.0	57.6	47.8
Grok-4 [ 85 ]	37.8	63.6	50.7
GPT-5 [ 67 ]	41.8	56.3	49.1
Open-source General Models
InternVL3-8B [ 117 ]	28.0	41.5	34.8

Table 8: Results on MMSI-Bench [ 95 ] and MindCube-Tiny [ 75 ] . Both report accuracy; Avg. is their mean.

Panel of Figure 4 — Figure 4: Model scaling from 0.8B to 9B. Under the same OraRL recipe, Video-ORA improves across all task families with model size ( $x$ -axis, in billions of parameters, log scale) and outperforms its Qwen3.5 backbone at every scale. The final panel reports the macro-average; Spatial Grounding averages RefCOCO-family [email protected], while ST-Grounding averages tIoU and sIoU.

Panel of Figure 5 — Figure 5: Task-group data scaling and reward dynamics under the controlled 9B protocol. (a) The video-perception aggregate is the macro average over temporal grounding, tracking, segmentation, and video question answering. (b) The spatial-intelligence aggregate averages VSI-Bench, MMSI-Bench, and MindCube. (c) Total training reward over one epoch for OraRL and GRPO.

Training data	RL prompts		VSI-Bench [ 93 ]	MMSI [ 95 ]	MindCube [ 75 ]	Avg.
Training data	Video	Spatial	VSI-Bench [ 93 ]	MMSI [ 95 ]	MindCube [ 75 ]	Avg.
Qwen3.5-9B, no training [ 71 ]	–	–	57.9	31.7	41.2	43.6
Spatial intelligence only	–	9k	68.1	33.5	49.2	50.3
$+$ video perception	9k	9k	70.0	34.2	50.3	51.5
Video-ORA-9B , seven tasks	100k		73.1	37.9	57.4	56.1

Table 9: Effect of video perception data on spatial intelligence. Avg. is the mean across the three benchmarks.

Backbone	Stage	Temporal	Tracking	Video Seg.	Avg.
Qwen3-VL-8B	Backbone	51.5	33.7	21.3	35.5
	$+$ SFT	53.2	67.7	55.0	58.6
	$+$ GRPO	58.0	68.5	57.8	61.4
	$+$ OraRL	60.8	71.3	59.9	64.0
Qwen3.5-9B	Backbone	56.2	46.0	26.8	43.0
	$+$ SFT	58.7	68.0	57.2	61.3

Table 10: Backbone generalization. Each model family is evaluated after SFT, GRPO, and OraRL under the same protocol.

Method	Char.	ANet.	QV.	Avg.	Train s/step
CoT + answer
Backbone	50.3	45.0	58.9	51.4	N/A
GRPO	54.8	53.4	67.2	58.5	135.6
Answer only
Backbone	49.4	51.9	64.5	55.3	N/A
GRPO	54.0	54.3	67.7	58.7	93.9

Table 11: Chain-of-thought ablation. Temporal-grounding scores are averaged across Charades, ActivityNet, and QVHighlights; training cost is reported in seconds per step.

Method	Temporal	Tracking	Video Seg.	Avg.
Supervised baselines
Backbone	55.3	45.3	24.2	41.6
SFT init.	57.6	60.2	55.6	57.8
Continued SFT	55.2	63.1	57.7	58.7
On-policy group RL
GRPO [ 65 ]	58.1	63.9	58.9	60.3

Table 12: Training paradigm comparison. All post-training runs use the same Qwen3.5-4B SFT initialization, data, and optimization budget.

Variant	Temporal	Tracking	Video Seg.	Avg.
GRPO	58.1	63.9	58.9	60.3
Naive GT injection	57.2	52.0	57.0	55.4
+ Reward shaping	60.4	64.0	59.2	61.2
OraRL	60.5	67.1	60.6	62.7

Table 13: Annotation injection under the shared three-task protocol. Shaping follows Tempsamp-R1.

Variant	Temporal	Tracking	Video Seg.	Overall
Rollout-level flip rate
Naive oracle mix	18.8	38.7	4.5	22.4
+ reward shaping [ 45 ]	9.7	21.7	1.3	11.9
OraRL, before pruning	1.0	3.5	0.3	1.9
OraRL, after pruning	1.1	0.2	0.0	0.3
Group-level incidence, naive oracle mix

Table 14: Advantage inversion across tasks. Flip rates measure GRPO-positive rollouts assigned negative advantages, so lower is better.

Figure 6: Advantage inversion. Shading marks inverted rollouts; labels give full-data rates among GRPO-positive samples.

Variant	Temp.	Track.	Seg.	Avg.
On-policy scaling
w/o directional gain	59.0	64.4	60.1	61.2
Oracle advantage
w/o detached oracle	58.0	65.5	60.3	61.3
w/o reward-gap weight	58.9	66.1	59.8	61.6
Rollout pruning

Table 15: Component ablation. All variants use the same initialization, data, rollout budget, and optimization schedule.

$\kappa$	$K$	s/step	Mem. (GB)	Avg.	Trade-off (s/pt)
0	8	92.5	62.4	63.1	–
0.25	6	75.0	60.0	62.8	58.3
0.50	4	62.4	50.9	62.7	75.3
0.75	2	45.0	47.5	60.2	16.4

Table 16: Accuracy and efficiency trade-off under sign-balanced pruning.

K

is the number of retained rollouts from each nine-rollout group, memory is peak per-GPU allocation, and trade-off is time saved per point of average-score loss. OraRL uses

\kappa=0.5

Panel of Figure 7 — Figure 7: Training and inference efficiency. (a) Accuracy against training cost, where OraRL gives the strongest trade-off and CoT substantially raises cost. (b) End-to-end single-request latency on one H20 in BF16. All models see the same 2-fps input of about 120K tokens per video. Solid bars give median latency split at the first token, pale tails run to P90, and the numbers report medians.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Composition of the SFT and RL training data. Open and filled markers indicate each task’s mixture proportion during SFT and RL, respectively. Tasks are grouped by output format, and the rightmost columns report exact example counts.

Setting	SFT	RL
Epochs	1	1
Framework	ms-swift	veRL + vLLM
Distributed strategy	ZeRO-2	FSDP full shard
Precision	bfloat16	bfloat16
Trainable module	language model	language model
Global prompt batch	64	64

Appendix

Table 17: Training hyperparameters. The RL batch size counts prompts before rollout generation.

Models	Image Segmentation			Video Segmentation
	RefCOCO [ 101 ]	RefCOCO+ [ 101 ]	RefCOCOg [ 54 ]	MeViS [ 17 ]			ReasonVOS [ 3 ]
	cIoU	cIoU	cIoU	J	F	J&F	J	F	J&F
PixelLM-7B [ 61 ]	73.0	66.3	69.3	–	–	–	–	–	–
LISA-7B [ 35 ]	74.1	62.4	66.4	35.1	39.4	37.2	29.1	33.1	31.1
VISA-13B [ 89 ]	72.4	59.8	65.5	–	–	44.5	–	–	–
Seg-R1-7B [ 99 ]	74.3	62.6	71.0	–	–	–	–	–	–

Appendix

Table 18: Complete segmentation results. RefCOCO/+/g use cIoU on the val set, whereas MeViS and ReasonVOS report J, F, and their mean J&F.

Models	MMSI-Bench												MindCube-Tiny
	Avg.	Positional Relationship						Attribute		Motion		MSR	Avg.	Rot.	Amg.	Ard.
	Avg.	C-C	O-O	R-R	C-O	O-R	C-R	Meas.	Appr.	Cam.	Obj.	MSR	Avg.	Rot.	Amg.	Ard.
Proprietary Models
Gemini-2.5 [ 13 ]	38.0	38.7	34.0	40.7	44.2	38.8	41.0	62.5	30.3	39.2	25.0	33.3	57.6	88.0	44.9	63.2
Grok-4 [ 85 ]	37.8	36.6	35.1	39.5	34.9	45.9	50.6	21.9	22.7	40.5	43.4	38.4	63.6	93.0	54.4	61.6
GPT-5 [ 67 ]	41.8	41.9	33.0	35.8	49.8	42.4	68.7	54.7	37.4	28.3	40.8	36.4	56.3	94.5	38.2	68.4

Appendix

Table 19: Complete results on MMSI-Bench [ 95 ] and MindCube-Tiny [ 75 ] . Scores are accuracies; C/O/R denote the MMSI-Bench camera/object/region categories.

Method	Frames	Numerical Question				Multiple-Choice Question			Avg.	VSI-Bench Avg.
Method	Frames	Obj. Cnt.	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route Plan	Avg.	VSI-Bench Avg.
Chance (frequency)	all	52.2	40.1	17.4	20.9	25.8	31.9	30.2	31.4	34.0
Proprietary models
GPT-5.2 [ 67 ]	64	56.2	41.5	73.9	63.0	48.4	34.9	38.2	50.9	49.2
Gemini-3-Flash [ 26 ]	1 fps	65.7	53.1	77.6	52.8	64.6	47.9	41.8	57.6	55.9
Gemini-3-Pro [ 26 ]	1 fps	60.1	54.7	79.3	51.9	68.1	56.0	56.4	60.9	60.5

Appendix

Table 20: Results on ReVSI [ 111 ] . Numerical and multiple-choice tasks use MRA and accuracy, respectively; Avg. is the reported aggregate over seven tasks, and the last column gives the original VSI-Bench average. All baseline rows, including their VSI-Bench averages, are quoted from [ 111 ] , which evaluates each model at its native frame setting and restricts proprietary models to a 1,093-question subset, so these values differ from our own measurements in Tab. 6 . The backbone and Video-ORA rows are evaluated by us.

Panel of Figure 9 — Figure 9: Temporal-grounding qualitative examples. Frames are decoded at the predicted boundaries and just outside them, and the timeline carries the annotated span above the axis and the predicted span below it. Video-ORA recovers all three intervals exactly, including a two-second event and a segment covering six percent of the timeline.

Panel of Figure 10 — Figure 10: Spatial-grounding qualitative examples. Each card pairs the raw input with a copy carrying both boxes in coordinates normalized to $[0,1000]$ . The two outlines are nearly indistinguishable, with an IoU of 0.98 or above on all three referring expressions.

Panel of Figure 11 — Figure 11: Mask-aware segmentation qualitative examples. The left panel carries the predicted box with its positive (dots) and negative (crosses) point prompts, and the right panel tints the annotation mask decoded from its run-length target. The model never emits a mask itself, so the comparison is between the predicted hints and the tight box of the annotated mask, which agree to within one percent.

Panel of Figure 12 — Figure 12: Visual-tracking qualitative examples. Four of the thirty-two predicted seconds are shown, covering a target that doubles in width, one that barely moves, and one that drifts upward while growing. Mean IoU over the full trajectory stays at 0.96 or above in all three sequences.

Panel of Figure 13 — Figure 13: Spatial-temporal grounding qualitative examples. The timeline compares the annotated and predicted intervals while the boxes compare the referred entity at four integer seconds. Temporal IoU stays above 0.87; the wildlife clip shows a typical failure mode, where the model keeps a single box for the entire span instead of tracking the subject.

Panel of Figure 14 — Figure 14: Video question-answering qualitative examples. One short, one medium, and one long Video-MME item, spanning attribute recognition, a briefly visible detail, and a count that has to be aggregated over a ten-minute broadcast.

Panel of Figure 15 — Figure 15: Spatial-intelligence qualitative examples. Sampled egocentric scans of unseen rooms support first-appearance order, a metric distance regressed in metres, and a relative direction whose anchor and query object are never visible in the same frame.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Reinforcing Structured Chain-of-Thought for Video Understanding
Mar 26, 2026Peiyao Wang, Haotian Xu, Noranart Vesdapunt +6Structured CoT Reasoning RL for VLMs
Stony Brook University · Amazon
EasyVideoR1: Easier RL for Video Understanding
Apr 18, 2026Chuanyu Qin, Chenxu Yang, Qingyi Si +6Reinforcement Learning Video-Language Models
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback
Oct 2, 2025Derek Shi, Ruben Glatt, Christine Klymko +5Video-Language Models RL Fine-Tuning
Stanford University, CA, USA · Lawrence Livermore National Laboratory, CA, USA · Microsoft, WA, USA

Mar 26, 2026J/K move · Enter open · S save

Reinforcing Structured Chain-of-Thought for Video Understanding

Peiyao Wang, Haotian Xu, Noranart Vesdapunt, Rui Hou+5

Stony Brook University · Amazon

Structured CoT Reasoning RL for VLMs Reinforcement Learning Video QA CoT Reasoning Temporal Video Understanding

Open paper AskPDF arXiv

Multi-modal Large Language Models (MLLMs) show promise in video understanding. However, their reasoning often suffers from thinking drift and weak temporal comprehension, even when enhanced by Reinforcement Learning (RL) techniques like Group Relative Policy Optimization (GRPO). Moreover, existing RL methods usually depend on Supervised Fine-Tuning (SFT), which requires costly Chain-of-Thought (CoT) annotation and multi-stage training, and enforces fixed reasoning paths, limiting MLLMs' ability to generalize and potentially inducing bias. To overcome these limitations, we introduce Summary-Driven Reinforcement Learning (SDRL), a novel single-stage RL framework that obviates the need for SFT by utilizing a Structured CoT format: Summarize -> Think -> Answer. SDRL introduces two self-supervised mechanisms integrated into the GRPO objective: 1) Consistency of Vision Knowledge (CVK) enforces factual grounding by reducing KL divergence among generated summaries; and 2) Dynamic Variety of Reasoning (DVR) promotes exploration by dynamically modulating thinking diversity based on group accuracy. This novel integration effectively balances alignment and exploration, supervising both the final answer and the reasoning process. Our method achieves state-of-the-art performance on seven public VideoQA datasets.

Abstract

Reinforcement Learning

Explore similar work

Reinforcing Structured Chain-of-Thought for Video Understanding

EasyVideoR1: Easier RL for Video Understanding

Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback