BriMA: Bridged Modality Adaptation for Multi-Modal Continual Action Quality Assessment

Kanglei Zhou; Chang Li; Qingyi Pan; Liyuan Wang

cs.CVFeb 22, 2026

BriMA: Bridged Modality Adaptation for Multi-Modal Continual Action Quality Assessment

Authors: Kanglei Zhou, Chang Li, Qingyi Pan, Liyuan Wang

Organizations: Department of Psychological and Cognitive Sciences, Tsinghua University · Department of Statistics and Data Science, Tsinghua University

Abstract

Action Quality Assessment (AQA) aims to score how well an action is performed and is widely used in sports analysis, rehabilitation assessment, and human skill evaluation. Multi-modal AQA has recently achieved strong progress by leveraging complementary visual and kinematic cues, yet real-world deployments often suffer from non-stationary modality imbalance, where certain modalities become missing or intermittently available due to sensor failures or annotation gaps. Existing continual AQA methods overlook this issue and assume that all modalities remain complete and stable throughout training, which restricts their practicality. To address this challenge, we introduce Bridged Modality Adaptation (BriMA), an innovative approach to multi-modal continual AQA under modality-missing conditions. BriMA consists of a memory-guided bridging imputation module that reconstructs missing modalities using both task-agnostic and task-specific representations, and a modality-aware replay mechanism that prioritizes informative samples based on modality distortion and distribution drift. Experiments on three representative multi-modal AQA datasets (RG, Fis-V, and FS1000) show that BriMA consistently improves performance under different modality-missing conditions, achieving 6--8% higher correlation and 12--15% lower error on average. These results demonstrate a step toward robust multi-modal AQA systems under real-world deployment constraints.

Figures & tables

Panel of Figure 1 — Figure 1: Our motivation: non-stationary modality imbalance (a) significantly challenges continual AQA performance (b) .

Figure 2: Overview of BriMA. At each session, incomplete multi-modal inputs are encoded and completed via memory-guided bridging imputation (see Sec. 4.2 ), then scored by the AQA model. The modality-aware replay optimization (see Sec. 4.3 ) selects and updates representative prototypes in the memory bank to maintain consistency in continual adaptation across tasks.

Method	Publisher	SRCC ( $\uparrow$ )					MSE ( $\downarrow$ )					RL2 ( $\downarrow$ )
Method	Publisher	Ball	Clubs	Hoop	Ribbon	Avg.	Ball	Clubs	Hoop	Ribbon	Avg.	Ball	Clubs	Hoop	Ribbon	Avg.
JT-MLAVL [ 46 ]	CVPR’25	$0.826$	$0.829$	$0.871$	$0.866$	$0.849$	$5.57$	$4.20$	$4.11$	$3.99$	$4.47$	N/A	N/A	N/A	N/A	N/A
Modality Missing Rate $\beta=10\%$
ST-MLAVL [ 46 ]	CVPR’25	$0.480$	$0.575$	$0.659$	$0.664$	$0.599$	$14.34$	$9.19$	$7.26$	$8.98$	$9.94$	$3.829$	$4.311$	$2.734$	$3.358$	$3.558$
SI [ 59 ]	ICML’17	$0.481$	$0.597$	$0.605$	$0.685$	$0.597$	$14.61$	$8.70$	$8.63$	$8.74$	$10.17$	$3.903$	$4.083$	$3.249$	$3.271$	$3.626$
EWC [ 18 ]	PNAS’17	$0.524$	$0.594$	$0.597$	$0.692$	$0.605$	$13.35$	$9.41$	$9.16$	$9.11$	$10.26$	$3.565$	$4.415$	$3.448$	$3.409$	$3.709$

Table 1: Performance comparison on the RG dataset. Bold values indicate the best results. Joint Training (JT) and Sequential Training (ST) denote the upper and lower bounds, while rehearsal-free and rehearsal-based methods represent different CL strategies.

\uparrow

: higher is better;

\downarrow

: lower is better. Average SRCC is computed using Fisher-

z

transformation.

Method	Publisher	SRCC ( $\uparrow$ )			MSE ( $\downarrow$ )			RL2 ( $\downarrow$ )
Method	Publisher	PCS	TES	Avg.	PCS	TES	Avg.	PCS	TES	Avg.
JT-MLAVL [ 46 ]	CVPR’25	$0.863$	$0.766$	$0.823$	$7.17$	$19.44$	$13.31$	N/A	N/A	N/A
Modality Missing Rate $\beta=10\%$
ST-MLAVL [ 46 ]	CVPR’25	$0.648$	$0.566$	$0.609$	$17.03$	$33.31$	$25.17$	$2.721$	$2.795$	$2.758$
SI [ 59 ]	ICML’17	$0.630$	$0.556$	$0.594$	$16.81$	$28.77$	$22.79$	$2.685$	$2.414$	$2.550$
EWC [ 18 ]	PNAS’17	$0.584$	$0.551$	$0.568$	$18.36$	$33.43$	$25.89$	$2.933$	$2.805$	$2.869$

Table 2: Performance comparison on the Fis-V dataset.

Method	Publisher	SRCC ( $\uparrow$ )			MSE ( $\downarrow$ )			RL2 ( $\downarrow$ )
Method	Publisher	10%	25%	50%	10%	25%	50%	10%	25%	50%
JT-MLAVL [ 46 ]	CVPR’25	0.90	N/A	N/A	$10.39$	N/A	N/A	N/A	N/A	N/A
ST-MLAVL [ 46 ]	CVPR’25	$0.728$	$0.724$	$0.665$	$32.56$	$35.47$	$54.24$	$1.592$	$1.645$	$2.012$
SI [ 59 ]	ICML’17	$0.719$	$0.725$	$0.656$	$36.77$	$39.83$	$46.74$	$1.650$	$1.631$	$1.932$
EWC [ 18 ]	PNAS’17	$0.716$	$0.710$	$0.635$	$54.97$	$56.58$	$58.06$	$2.150$	$2.041$	$2.354$
LwF [ 23 ]	TPAMI’17	$0.699$	$0.626$	$0.583$	$35.19$	$34.67$	$45.76$	$1.726$	$2.271$	$2.625$

Table 3: Performance comparison on the FS1000 dataset. Subscore results are shown in Appendix Tab. S2 .

Method	SRCC Gain	MSE Drop	RL2 Drop	Params (M)	Training Time (h)	Inference (Sample/s)	Storage (MB)
ST-MLAVL [ 46 ]	0%	0%	0%	8.30	0.416	0.078	0.00
EWC [ 18 ]	+3.1%	+7.5%	+5.8%	8.30	0.597	0.078	0.00
NC-FSCIL [ 54 ]	+5.2%	+13.6%	+9.3%	8.30	1.025	0.078	32.25
MAGR [ 69 ]	+9.5%	+22.8%	+15.2%	8.40	1.154	0.080	32.25
ASAL [ 65 ]	+11.2%	+26.7%	+19.0%	8.30	0.876	0.084	32.25
BriMA (Ours)	+14.6%	+33.4%	+23.1%	8.80	1.417	0.118	32.25

Table 4: Efficiency and relative performance comparison on RG (average across all items). Results are reported relative to [ 46 ] .

ID	Setting	SRCC ( $\uparrow$ )
ID	Setting	Ball	Clubs	Hoop	Ribbon	Avg.
1	Ours	$0.648$	$0.788$	$0.710$	$0.836$	$0.746$
2	Ours w/o MBI	$0.471$	$0.660$	$0.693$	$0.742$	$0.640$
3	Ours w/o Bridge	$0.640$	$0.727$	$0.656$	$0.796$	$0.730$
4	Ours w/o Candidate	$0.639$	$0.755$	$0.628$	$0.773$	$0.699$
5	Ours w/o MRO	$0.640$	$0.727$	$0.656$	$0.775$	$0.702$

Table 5: Ablation study on the RG dataset (

\beta=10\%

Figure 3: Results of different candidates.

Figure 4: Results of different imputation strategies.

Figure 5: Task-wise SRCC performance comparison.

Figure 6: Performance comparisons: (a) loss landscape, (b) error box, (c) accuracy Area Under the Curve (AUC).

Panel of Figure 7 — Figure 7: Case study. See Appendix Fig. S2 for more cases.

Dataset	Subcategory	Score Dimension(s)	Range	Notes
RG [ 56 ]	Ball	Overall	0 – 25	Four rhythmic events
	Clubs	Overall	0 – 25
	Hoop	Overall	0 – 25
	Ribbon	Overall	0 – 25
Fis-V [ 31 ]	TES	Technical Execution Score (TES)	0 – 45	Two judging dimensions
Fis-V [ 31 ]	PCS	Performance Component Score (PCS)	0 – 40	Two judging dimensions

Table S1: Score ranges and evaluation dimensions for all datasets.

Figure S 1 : Non-stationary modality imbalance significantly challenges the model performance: (a) L2 and (b) RL2.

Method	Publisher	SRCC ( $\uparrow$ )								MSE ( $\downarrow$ )								RL2 ( $\downarrow$ )
Method	Publisher	TES	PCS	SS	TR	PE	CO	IN	Avg.	TES	PCS	SS	TR	PE	CO	IN	Avg.	TES	PCS	SS	TR	PE	CO	IN	Avg.
JT-MLAVL [ 46 ]	CVPR’25	$0.92$	$0.89$	$0.90$	$0.90$	$0.88$	$0.89$	$0.88$	$0.90$	$64.89$	$6.39$	$0.23$	$0.24$	$0.50$	$0.25$	$0.26$	$10.39$	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Modality Missing Rate $\beta=10\%$
ST-MLAVL [ 46 ]	CVPR’25	$0.630$	$0.762$	$0.740$	$0.757$	$0.721$	$0.744$	$0.726$	$0.728$	$209.44$	$15.38$	$0.56$	$0.54$	$0.86$	$0.55$	$0.61$	$32.56$	$1.686$	$1.084$	$2.026$	$1.821$	$0.892$	$1.736$	$1.899$	$1.592$
SI [ 59 ]	ICML’17	$0.583$	$0.747$	$0.737$	$0.749$	$0.720$	$0.754$	$0.717$	$0.719$	$238.47$	$15.74$	$0.56$	$0.56$	$0.88$	$0.56$	$0.63$	$36.77$	$1.920$	$1.110$	$2.043$	$1.865$	$0.909$	$1.748$	$1.952$	$1.650$
EWC [ 18 ]	PNAS’17	$0.639$	$0.715$	$0.743$	$0.740$	$0.730$	$0.743$	$0.689$	$0.716$	$358.87$	$22.05$	$0.66$	$0.76$	$0.90$	$0.71$	$0.81$	$54.97$	$2.889$	$1.554$	$2.377$	$2.555$	$0.934$	$2.232$	$2.506$	$2.150$

Table S2 : Performance comparison on the FS1000 dataset (supplementary results of Tab. 3 ). Bold values indicate the best results. Joint Training (JT) and Sequential Training (ST) denote the upper and lower bounds, while rehearsal-free and rehearsal-based methods represent different CL strategies.

\uparrow

: higher is better;

\downarrow

: lower is better. N/A indicates that the metric was not reported in the paper. Average SRCC is computed using Fisher-

z

transformation.

Panel of Figure 16 — Figure S 2 : Case study (supplementary results of Fig. 7 ): (a), (b), and (c) are easy cases, and (d) and (e) are hard cases.

ID	Setting	SRCC ( $\uparrow$ )
ID	Setting	Ball	Clubs	Hoop	Ribbon	Avg.
1	Ours	$0.648$	$0.788$	$0.710$	$0.836$	$0.746$
2	Ours w/o Task-Specific Embedding	$0.579$	$0.702$	$0.656$	$0.834$	$0.693$
3	Ours w/o Replay Prioritization	$0.639$	$0.776$	$0.628$	$0.791$	$0.717$
		MSE ( $\downarrow$ )
1	Ours	$8.65$	$4.88$	$7.12$	$4.39$	$6.26$

Table S3: Ablation study on the RG dataset (

\beta=10\%

JT-EMOE [ 10 ]

(

\beta=0\%

)

SRCC

0.757

MSE

1.108

RL2

3.078

Table S4 : Results on the MOSI dataset.

Action Quality Assessment

Action quality assessment measures how well people perform actions, using video, pose, wearable, and contextual data to score skill or execution.

See all 23 papers

Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment
Jul 8, 2026Kanglei Zhou, Ruizhi Cai, Xinning Wang +4Multimodal Robustness Cross-Modal Alignment
Dept. of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · State Key Lab of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Dept. of Rheumatology and Immunology, Children’s Hospital, Capital Institute of Pediatrics, Beijing, China
MLCR: Multi-Level Cue Refinement for Long-Term Multimodal Action Quality Assessment
May 9, 2026Qiqi Li, Pengfei Wang, Hongyu Chen +1Multimodal Fusion Action Quality Assessment
Qiushi Academy for Advanced Studies (QAAS), Zhejiang University, Hangzhou, Zhejiang, 310007, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, Zhejiang, 310007, China · School of Software Technology, Zhejiang University, Hangzhou, Zhejiang, 310007, China +3
Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization
Oct 8, 2025Kanglei Zhou, Qingyi Pan, Xingxing Zhang +4Continual Learning Action Quality Assessment
Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing 100084, China · Department of Statistics and Data Science, Tsinghua University, Beijing 100084, China · Department of Computer Science and Technology, Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University +3

Jul 8, 2026J/K move · Enter open · S save

Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment

Kanglei Zhou, Ruizhi Cai, Xinning Wang, Yijian Zheng+3

Dept. of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · State Key Lab of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Dept. of Rheumatology and Immunology, Children’s Hospital, Capital Institute of Pediatrics, Beijing, China

Multimodal Robustness Cross-Modal Alignment Multimodal Fusion Action Quality Assessment

Open paper AskPDF arXiv

Action Quality Assessment (AQA) aims to evaluate how well a person performs a movement, which is essential in applications such as sports scoring, skill assessment, and healthcare. However, unimodal approaches often struggle to capture subtle cues of movement quality in real-world settings. Although multi-modal inputs provide complementary information, existing methods still face two major challenges: heterogeneous modalities often lead to cross-modal misalignment and unstable fusion, and reliable multi-modal annotation is costly, resulting in limited dataset diversity. To address these challenges, we propose DualAlign, a two-stage multi-modal fusion framework with adaptive alignment. The framework first constructs a coherent visual representation by maximizing shared structural information across RGB video, optical flow, and skeleton modalities. Textual semantics are then incorporated after visual stabilization, allowing high-level descriptions to complement rather than distort the underlying visual manifold. To evaluate the framework under realistic multi-modal conditions, we introduce MM--JDM, a movement-quality assessment dataset integrating RGB videos, optical flow, skeleton sequences, and structured text. MM--JDM naturally exhibits modality noise, class imbalance, and label scarcity, making it a challenging benchmark for studying multi-modal fusion and alignment. Extensive experiments show that DualAlign improves average correlation on MM--JDM by 21.16% over the state-of-the-art methods and achieves gains of 3.53% and 5.95% on the RG and Fis-V benchmarks, respectively. DualAlign also remains robust under missing-modality and label-scarce conditions.

Abstract

Action Quality Assessment

Explore similar work

Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment

MLCR: Multi-Level Cue Refinement for Long-Term Multimodal Action Quality Assessment

Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization