cs.CVMar 12, 2026
SaveAttend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
Organizations: UC Berkeley · NVIDIA · Clarifai · MIT
Abstract
Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: https://autogaze.github.io/.
Figures & tables
Figure 2: What is AutoGaze paying attention to? For each example, we show the original video, multi-scale gazed patches, and reconstructed video. Note that we only show gazing on two scales to save space while AutoGaze actually uses four. In general, AutoGaze can 1) focus on moving objects while removing redundancy in static regions (a-e), 2) adapt to scene changes by selecting more patches (c, e), and 3) distribute attention with different granularity based on detailedness (c, f). This allows AutoGaze to select a small ratio of patches (gazing ratio) without much information loss, as reflected by the reconstruction quality.
Figure 3: Architecture and training pipeline of AutoGaze. (Left & Middle) Given a video, AutoGaze processes each frame and autoregressively decodes indices of multi-scale patches based on the history of frames and selected patches. Once it believes the previously-gazed patches are sufficient to reconstruct the current frame, it automatically stops gazing and moves to the next frame. (Right) AutoGaze is trained in two stages: next-token-prediction pre-training on collected gazing sequences, and RL post-training with reconstruction reward.
Figure 4: AutoGaze targets patches with higher optical flow. (Left) AutoGaze uses coarser scales to capture higher optical flow. (Right) Across all scales, AutoGaze more frequently selects patches with higher optical flow. Error bars represent SEM.
Figure 5: Gazing scale correlates with patch detail. (Left) At finer scales, AutoGaze selects more detailed patches (measured as Laplacian variance). (Right) With increasing detail, AutoGaze uses finer scales ( ). Sample patches with Laplacian variances are shown below the x-axis. Error bars represent SEM.
Figure 6: Generlizability of AutoGaze to OOD videos. (a) We show model behavior on videos with OOD semantics, including a CCTV clip (left), robot grasping demo (middle), and a video with object swapping (right). In each example, AutoGaze still robustly tracks the changing parts despite the unseen semantics, object categories, and unexpected changes. (b) We show AutoGaze output on the same video with different style transfer. AutoGaze consistently tracks the falling person regardless of visual style, texture and global illumination.
Figure 7: What gazing ratio do we need for different video types? (Left) There is a trade-off between gazing ratio and reconstruction loss: videos with higher FPS or resolution need lower gazing ratio to reach the same reconstruction quality. (Right) Gazing ratios required to reach a loss of 0.7 for videos with different FPS and resolution. 30-FPS, 4K-res videos only need 1% patches.
Figure 8: Efficiency gain on ViTs and MLLMs with AutoGaze. We benchmark the ViT and MLLM latency of encoding one second of video with varying FPS and resolution. AutoGaze can select different numbers of patches to vary latency depending on user needs. When using the gazing ratio required for a reconstruction loss of 0.7, AutoGaze reduces the ViT and MLLM latency by up to 19 and 10 .
Figure 9: Scaling MLLMs to more video tokens. AutoGaze enables scaling to longer videos and higher resolution, while the baseline runs out of memory beyond 256 frames. Performance boosts are especially clear with HLVid, where high-resolution video processing is required.
| general video | long video | high-res & long | ||||||||
| Models | Max #F | Max Res. | VideoMME (w/o sub) | VideoMME (w/ sub) | MVBench (test) | NExT-QA (mc) | L-VidBench (val) | EgoSchema (test) | MLVU (m-avg) | HLVid (test) |
| Gemini 1.5-Pro [ 77 ] | - | - | 75.0 | 81.3 | 60.5 | - | 64.0 | 71.2 | - | - |
| Gemini 2.5 Flash-Lite [ 18 ] | - | - | 65.0 | - | - | - | - | - | 69.3 | 52.2 |
| GPT-4o [ 38 ] | - | - | 71.9 | 77.2 | 64.6 | - | 66.7 | 72.2 | 64.6 | 49.3 |
| LLaVA-OV-8B [ 46 ] | 32 | 384 | 58.2 | 61.5 | 56.7 | 79.4 | 56.5 | 60.1 | 64.7 | 41.1 |
| LongVILA-7B [ 15 ] | 2048 | 384 | 60.1 | 65.1 | 67.1 | 80.7 | 57.1 | - | - | 41.4 |
Table 1: Comparison to state-of-the-art MLLMs. NVILA-8B-Video with AutoGaze is scaled to 1K-frame, 4K-resolution videos, achieving competitive performance on general and long video benchmarks and state-of-the-art result on HLVid.
| Type | Method | ViT lat. | LLM lat. | V-MME (w/o sub) | L-Vid (val) |
| - | No Reduction | 2.20s | 1.42s | 53.4 | 51.1 |
| S-PA | S-Pool | 2.20s | 0.18s | 51.5 | 47.2 |
| ToMe [ 9 ] | 2.23s | 0.11s | 51.5 | 49.3 | |
| VisionZip [ 98 ] | 2.22s | 0.15s | 50.7 | 48.5 | |
| S-PD | FastV [ 14 ] | 2.23s | 0.38s | 53.0 | 46.3 |
| T-PA | T-Pool | 2.20s | 0.13s | 52.2 | 50.0 |
Table 2: Comparing AutoGaze with MLLM token reduction methods. We compare to baselines of spatial/temporal/spatiotemporal (S-/T-/ST-) token reduction approaches that are either prompt-agnostic (PA) or prompt-dependent (PD). All the methods select 6.25% visual tokens on average.
| Pre-Train | Post-Train | Recon Loss | Gazing Ratio |
| ✗ | ✗ | 0.7 | 0.263 |
| ✓ | ✗ | 0.7 | 0.102 |
| ✗ | ✓ | 0.7 | 0.209 |
| ✓ | ✓ | 0.7 | 0.094 |
Table 3: Ablation on AutoGaze training pipeline. Both NTP pre-training and RL post-training helps with the performance.
| Multi-Token Pred. | Multi-Scale Gazing | Recon Loss | Gazing Ratio | Latency |
| 1 | ✓ | 0.7 | 0.074 | 0.949s |
| 5 | ✓ | 0.7 | 0.078 | 0.246s |
| 10 | ✓ | 0.7 | 0.094 | 0.193s |
| 20 | ✓ | 0.7 | 0.109 | 0.156s |
| 10 | ✓ | 0.7 | 0.094 | 0.193s |
| 10 | ✗ | 0.7 | 0.220 | 0.467s |
Table 4: Ablations of AutoGaze model designs.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Recon. Loss | 64 Frames | 128 Frames | 256 Frames |
| No Gaze | 59.1 | 60.1 | 60.5 |
| 0.6 | 59.0 | 60.0 | 60.7 |
| 0.7 | 58.6 | 59.7 | 60.3 |
| 0.8 | 57.8 | 58.4 | 59.0 |
| 1.0 | 56.3 | 56.7 | 57.2 |
Table 5: What reconstruction loss is tolerable for MLLM performance? We find a loss of 0.7 has little performance drop.
Figure 11: Reconstruction quality under varying loss thresholds. Outlined in recon_loss are noticeable visual defects.
Figure 12: Efficiency on streaming videos. We compare the ViT and MLLM efficiency on streaming videos with or without AutoGaze. We measure the maximum FPS of processing different types of videos (i.e., different FPS and resolution). For each plot, the gray area denotes the region where real-time processing of the video is achieved.
Figure 13: Examples of HLVid benchmark. Each sample from HLVid consists of a 5-minute long video at 4K resolution, along with multiple-choice questions that require high-resolution video encoding to answer. The answers to HLVid’s questions can be found at any spatiotemporal location, emphasizing long-context video understanding. Our video content is diverse and includes house tours (top), city driving (bottom), nature videos, etc.
Figure 14: Picture-in-picture whiteboard lecture. After the first frame, AutoGaze focuses on the moving cursor and the lecturer’s face.
Figure 15: Film clip. AutoGaze selects minimal patches to track the drink bottles and glasses as they are moved by hands a rotating plate.
Figure 16: Gymnastics clip. AutoGaze tracks the gymnast across the uneven bars, using finer scales when appropriate.
Figure 17: Claymation cartoon. AutoGaze captures small movements (blinking), scene changes, and enough patches to reconstruct text.
Figure 18: Claymation cartoon. AutoGaze skips the finest scale as it is not needed to achieve the specified reconstruction loss.
Figure 19: Chalkboard lecture. The minimal patches are selected to reconstruct both the lecturer’s movement and the chalkboard writing.
Figure 20: Film clip. In this game of Blackjack, AutoGaze uses finer scales to capture hand movements and cards.
Figure 21: Fisheye lens. AutoGaze can select patches that track moving objects with appropriate scales even with lens distortion.
Figure 22: Fisheye lens. In this overhead video with lens distortion, AutoGaze tracks the walking person and ignores the static warehouse.
Figure 23: Warehouse example. In this video, AutoGaze selects just enough patches to reconstruct the moving person and cart.
Figure 24: Nighttime driving. AutoGaze can be used on nighttime videos such as this one, capturing pedestrians and passing cars.
Figure 25: Robot arm video. AutoGaze uses different scales to gaze at the robot arm and marker.
Figure 26: Multiple perspectives. Given two side-by-side videos, AutoGaze selects patches in both halves to reconstruct the video.
Figure 27: Black-and-white film. In this clip where people are walking, AutoGaze uses finer scales to select people as they get smaller.
Figure 28: Swapping object. In an OOD scenario where the skateboard swaps out halfway through the video, AutoGaze continues to track the moving parts and properly reconstruct each video frame.
Figure 29: Panning over a static image. As the image slides to the left (new pixels highlighted with a red border), AutoGaze does not perfectly track movement; it can select regions that were in different parts of previous frames.
Figure 30: Predicting next frames from physics knowledge. Given the first 6 frames of a ball falling along a parabolic path, VideoMAE is unable to reconstruct the next frames to reflect the continued falling motion.