From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coaching

Yuyang Ji; Yixuan Shen; Shengjie Zhu; Yu Kong; Feng Liu

cs.CVMar 27, 2026

From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coaching

Authors: Yuyang Ji, Yixuan Shen, Shengjie Zhu, Yu Kong, Feng Liu

Organizations: Department of Computer Science, Drexel University · Department of Computer Science and Engineering, Michigan State University

Abstract

We present BioCoach, a biomechanics-grounded vision--language framework for fitness coaching from streaming video. BioCoach fuses visual appearance and 3D skeletal kinematics, through a novel three-stage pipeline: an exercise-specific degree-of-freedom selector that focuses analysis on salient joints; a structured biomechanical context that pairs individualized morphometrics with cycle and constraint analysis; and a vision--biomechanics conditioned feedback module that applies cross-attention to generate precise, actionable text. Using parameter-efficient training that freezes the vision and language backbones, BioCoach yields transparent, personalized reasoning rather than pattern matching. To enable learning and fair evaluation, we augment QEVD-fit-coach with biomechanics-oriented feedback to create QEVD-bio-fit-coach, and we introduce a biomechanics-aware LLM judge metric. BioCoach delivers clear gains on QEVD-bio-fit-coach across lexical and judgment metrics while maintaining temporal triggering; on the original QEVD-fit-coach, it improves text quality and correctness with near-parity timing, demonstrating that explicit kinematics and constraints are key to accurate, phase-aware coaching.

Figures & tables

Figure 2: BioCoach overview. Streaming video is encoded by two backbones: a 3D CNN for visual tokens and a pose extractor for 3D skeletal kinematics. The pipeline has three components: (1) Exercise-Specific DoF Selection uses a lightweight attention head to select the top $K$ biomechanically salient joints; (2) Structured Biomechanical Context builds two representations (individual morphometric context and motion quality context) capturing body measurements, cycles, ranges of motion, and constraint checks; (3) Vision–Biomechanics Conditioned Feedback fuses visual tokens with the morphometric context via cross-attention and prepends the motion-quality context as structured instruction to the LLM. This yields feedback grounded in explicit kinematic evidence rather than pattern matching alone.

Figure 3: Motion-Quality Context module. Given the selected joint set and the 3D skeletal kinematics, the module (a) detects repetition cycles and anchors the feedback moment; (b) time-normalizes each cycle and aligns it to a curated reference trajectory; and (c) evaluates biomechanical constraints: stability for static joints and deviation to reference for dynamic joints. Gray curves denote the reference, blue curves denote the user.

Method	METEOR $\uparrow$	ROUGE-L $\uparrow$	BERTScore $\uparrow$	LLM-Acc. $\uparrow$	LLM-Bio-Acc. $\uparrow$	T-F-Score $\uparrow$
Stream-VLM [ 32 ] (NeurIPS ’24)	0.086	0.108	0.852	1.86	1.72	0.530
BioCoach	0.312 (+262.8%)	0.302 (+179.6%)	0.877 (+2.9%)	3.12 (+67.7%)	3.26 (+89.5%)	0.544 (+2.6%)

Table 1: Evaluation on the QEVD-bio-fit-coach , a newly created benchmark with fine-grained biomechanical ground-truth feedback annotations. Second line in each cell shows % improvement vs. Stream-VLM [ 32 ] . LLM-Bio-Acc. is our LLM-as-judge metric tailored to biomechanics, assessing the biomechanical correctness and specificity of generated feedback.

Method	METEOR $\uparrow$	ROUGE-L $\uparrow$	BERTScore $\uparrow$	LLM-Acc. $\uparrow$	T-F-Score $\uparrow$
Zero-shot Models
InstructBLIP † [ 8 ]	0.047	0.040	0.839	1.56	-
Video-LLaVA † [ 23 ]	0.057	0.025	0.847	2.16	-
Video-ChatGPT † [ 29 ]	0.098	0.078	0.850	1.91	-
Video-LLaMA † [ 49 ]	0.101	0.077	0.859	1.29	-
LLaMA-VID † [ 21 ]	0.100	0.079	0.859	2.20	-

Table 2: Performance on the QEVD-fit-coach with original feedback annotations. Parenthesized values are % change vs. the best baseline (Stream-VLM [ 32 ] ).

\dagger

zero-shot without fine-tuning.

Model Variant	METEOR $\uparrow$	ROUGE-L $\uparrow$	BERTScore $\uparrow$	LLM-Acc. $\uparrow$	LLM-Bio-Acc. $\uparrow$	T-F-Score $\uparrow$
Full Model ( $\tau=3\,\mathrm{s}$ )	0.312	0.302	0.877	3.12	3.26	0.544
w/o Exercise-Specific DoF Selection	0.305	0.296	0.873	3.06	3.14	0.543
w/o Motion Quality Context Module	0.133	0.127	0.864	2.63	2.04	0.544
w/o Individual Morph Context Module	0.284	0.273	0.871	2.91	3.07	0.535
Window size $\tau=2\,\mathrm{s}$	0.311	0.303	0.876	3.10	3.22	0.416

Table 3: Ablation study on QEVD-Bio-Fit-Coach . Each row removes or modifies one component while keeping others fixed.

Figure 4: Qualitative timeline for a squat exercise. BioCoach produces temporally aligned, biomechanics-grounded cues with consistent phase tracking, while Stream-VLM outputs generic or mistimed feedback inconsistent with the ground-truth annotations.

BoT-Feedback: Grounding Multimodal Reasoning in Biomechanical Evidence for Explainable Human Action Feedback
Oct 3, 2026Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye +1Multimodal Reasoning
University of Surrey, Guildford, United Kingdom · University of Wollongong, Wollongong, Australia
Learning Consistent Temporal Grounding between Related Tasks in Sports Coaching
Mar 19, 2026Arushi Rai, Adriana KovashkaTemporal Video Grounding Temporal Attention
University of Pittsburgh, Pittsburgh PA 15260, USA
AIDE: Automated Instruction via Distilled Expertise for Reference-Free Motor Skill Coaching
Aug 4, 2026Yoshiki ItoText Generation
Research & Development Group Hitachi, Ltd. Tokyo, Japan

Oct 3, 2026J/K move · Enter open · S save

BoT-Feedback: Grounding Multimodal Reasoning in Biomechanical Evidence for Explainable Human Action Feedback

Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye, Andrew Gilbert

University of Surrey, Guildford, United Kingdom · University of Wollongong, Wollongong, Australia

Multimodal Reasoning

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in visual understanding and multimodal reasoning, yet they remain fundamentally limited in Human Action Feedback Generation. Existing methods infer coaching feedback directly from visual observations, producing generic advice, limited interpretability, and physically implausible hallucinations. In contrast, expert human coaches diagnose performance through explicit biomechanical reasoning over joint kinematics, posture, and body dynamics. We introduce BoT-Feedback, a framework that grounds MLLM reasoning in structured biomechanical evidence. Our key contribution is Biomechanics of Thought (BoT), a four-stage reasoning framework that progressively identifies the action, localises the critical body regions, analyses quantitative biomechanical differences between expert and student performances, and synthesises interpretable coaching feedback. To support this reasoning process, we develop a plug-and-play Biomechanical Data Parser (BDP) that converts videos into structured biomechanical descriptors and an alignment strategy that temporally matches expert and student motions. We further introduce BiomAF, a benchmark containing paired teacher-student videos, 3D skeletons, biomechanical attributes, and expert-coaching annotations. Experiments across twelve open- and closed-source MLLMs demonstrate that grounding reasoning in biomechanical evidence consistently improves feedback quality, interpretability, and robustness while substantially reducing biomechanical hallucinations. BoT-Feedback improves the average expert evaluation score from 2.07 to 2.95 (+40%), enabling compact open-source MLLMs to approach the performance of substantially larger proprietary systems for explainable action feedback generation.

Abstract

Explore similar work

BoT-Feedback: Grounding Multimodal Reasoning in Biomechanical Evidence for Explainable Human Action Feedback

Learning Consistent Temporal Grounding between Related Tasks in Sports Coaching

AIDE: Automated Instruction via Distilled Expertise for Reference-Free Motor Skill Coaching