cs.CVMar 19, 2026
SaveLearning Consistent Temporal Grounding between Related Tasks in Sports Coaching
Organizations: University of Pittsburgh, Pittsburgh PA 15260, USA
Abstract
Video-LLMs often attend to irrelevant frames, which is especially detrimental for sports coaching tasks requiring precise temporal grounding. Yet obtaining frame-level supervision is challenging: expensive to collect from humans and unreliable from other models. We improve temporal grounding without additional annotations by exploiting the observation that related tasks, such as generation and verification, must attend to the same frames. We enforce this via a self-consistency objective over select visual attention maps of tightly-related tasks. Using VidDiffBench, which provides ground-truth keyframe annotations, we first validate that attention misallocation is a significant bottleneck. We then show that training with our objective yields gains of +3.0%, +14.1% accuracy and +0.9 BERTScore over supervised finetuning across three sports coaching tasks: Exact, FitnessQA, and ExpertAF, even surpassing closed-source models.
Figures & tables
Figure 1: Both the generation task (providing feedback on technique) and the verification task (confirming a given feedback statement) require the model to attend to the same keyframes (highlighted in orange) to produce a correct response. We exploit this as a self-supervision signal, enforcing attentional consistency between the two complementary task-views without requiring any frame-level annotations.
Figure 2: Overview of the proposed dual-pathway self-consistency framework. The Generation Pathway and Verification Pathway process the same video through a shared visual encoder and Vid-LLM Decoder ( ) with different task prompts. From a selected visual layer (Sec. 4.1 ), top- attention heads are identified via their Visual Non-Sink Ratio [ 20 ] and paired across pathways via Head Bipartite Matching (exact index matching + Hungarian algorithm). The generation pathway’s attention maps are regularized by for focused attention over specific visual tokens (and frames), while enforces cross-pathway attentional agreement between matched heads. serves as regularization on the generation pathway output.
Figure 3: Visual attention quality score visualization: product of attention sharpness ( ), vision centricity (visual attention sum), and keyframe overlap (AUROC). Both graphs show that layer 8 and head 7 are relatively better at temporal grounding.
| Head Selection | Redistribution | Accuracy |
| Manual Head Selection (layer=8, head=7) | ||
| None | None | 65.8 |
| None | Uniform over all frames | 68.4 |
| None | Proportional over keyframes | 71.8 |
| Adaptive Head Selection (layer=8) | ||
| Top- vis. attention sum | Proportional over keyframes | 69.2 |
Table 1: Proof-of-concept analysis: Effect of manipulating attention distributions using ground-truth keyframes (relevant frames) on the EgoExo4D split of VidDiffBench [ 5 ] . Biasing attention toward annotated keyframes improves accuracy over uniform or no redistribution, demonstrating the benefit of temporal (frame) grounding. Regarding head selection, automatic selection via visual non-sink ratio [ 20 ] performs best.
| Method | Exact | FitnessQA | ExpertAF |
|---|---|---|---|
| Gemini-3-Flash | 79.0 | 54.2 | – |
| Qwen2.5-VL-7B-Instruct | – | 33.3* | 27.3 |
| Qwen3-VL-8B-Instruct | 40.2 | 55.4* | 27.1 |
| PerceptionLM-8B | 27.1 | 39.5 | 26.0 |
| PerceptionLM-8B (Finetuned) | 84.5 | 74.1 | 37.5 |
| +Ours | 87.5 | 88.2 | 38.4 |
Table 2: Performance comparison across sports understanding benchmarks. Our method improves over the finetuned baseline on all evaluated datasets. * denotes discarding answer extraction failures
| Verif. Layer | Head Matching Strategy | Exact | FitnessQA |
|---|---|---|---|
| Fixed ( ) | Ex. Match + Hung. (ours) | 87.5 | 88.2 |
| Fixed ( ) | Ex. Match + Random | 83.0 | 85.4 |
| Fixed ( ) | Ex. Match + Discard | 82.5 | 81.3 |
| Flexible | Hung. only | 83.0 | 87.7 |
| Verif. Summ. | Ex. Match + Hung. | 69.5 | 83.6 |
Table 3: Ablation over head matching strategies and task for . “Fixed” denotes that both pathways share the same selected layer ; “Flexible” allows the verification pathway to select its own layer independently. “Verification Summarization” replaces the verification pathway prompt with a summarization task.
Figure 4: Attention redistribution strategies. Left: The initial attention distribution is diffuse and noisy across the entire visual sequence. Right: Proportional redistribution concentrates the attention weights onto specific keyframe tokens while preserving their original relative importance.
| Example 1 | Example 2 | |
| Question | Is the user’s movement sufficient during shoulder gators? | While performing plank taps on their knees, are the user’s hips elevated? |
| Reference | Yes, the movement is insufficient. | No, the user’s hips are not elevated. |
| A | The movement is excessive. | The user’s hips are significantly elevated. |
| B | The movement is insufficient. ✓ | The user’s hips are slightly elevated. |
| C | The movement is appropriate. | The user’s hips are not elevated. ✓ |
| D | The movement is too slow. | The user’s hips are at an appropriate height. |
Table 6: FitnessQA multiple-choice conversion examples. The correct option (marked ✓) is a paraphrase of the original open-ended reference answer; distractors are plausible but incorrect alternatives for the same exercise.
| SG Direction | Exact | FitnessQA | ExpertAF |
|---|---|---|---|
| Ours (verification fixed) | 87.5 | 88.2 | 38.4 |
| Generation fixed (no on verif) | 0.0 | 2.7 | -4.0 |
| Generation fixed (w/ on verif) | 1.0 | 0.0 | 16.1 |
Table 7: Stop-gradient direction ablation. “Ours (ver. fixed)” applies sg to the verification pathway (Eq. 4 of the main paper). “Gen. fixed” reverses the direction, fixing the generation pathway and training the verification pathway to match it. On Exact, none of the generated answers matched the format.
Figure 5: Selected frames from the ExpertAF clips in Table 8 .
| Sport | Reference | Finetuned baseline | Ours |
|---|---|---|---|
| Soccer | The player’s kicking technique needs improvement, especially the follow-through and body alignment to maximize power, control, and accuracy. | The player’s footwork and body position are affecting the quality of the pass. | The player shows good approach technique to the ball, but needs to improve follow-through and body positioning for maximum power and accuracy. |
| Climbing | The climber is attempting a dynamic move that involves cutting his feet loose and supporting his weight with his arms, which is a challenging and uncertain approach. | The climber’s footwork is not ideal; he should use his left foot to drive upward and bring his right foot back onto the hold. | The climber successfully executed a dynamic move, starting from a good position with effective foot placement to reach the next hold. |
Table 8: Qualitative feedback examples on ExpertAF.
More in this topic
Temporal Video Grounding
Temporal video grounding locates the moments in a video that match a natural-language query, enabling precise search and evidence-based understanding.
See all 166 papers