Do Transformer Temporal Heads and Post-Pooling Motion Gates Help CorrNet-based CSLR? An Empirical Study
Authors: Lisi Wang, Zhidong Xiao, Jianjun Peng
Organizations: University of New South Wales, Sydney, Australia · National Centre for Computer Animation, Faculty of Media, Science and Technology, Bournemouth University, BH12 5BB, United Kingdom · School of Information Science and Engineering, Dalian Polytechnic University, Dalian 116034, China
CorrNet is a strong baseline for continuous sign language recognition (CSLR) because it models inter-frame correlations inside the visual encoding stage. In this paper, we study two natural extensions of a reproduced CorrNet system: replacing the BiLSTM temporal head with a Transformer encoder, and injecting motion cues after temporal pooling. We find that the Transformer head does not outperform the BiLSTM baseline, even with a training strategy adjusted for the Transformer, and the two heads have almost the same computational and runtime cost. For the second extension, we design a lightweight module called MotionGate. In our experiments, MotionGate consistently collapses to an identity-like mapping: the gate loses motion selectivity, and the injected residual becomes a weak, non-selective perturbation of the pooled features. These results suggest that explicit motion injection after CorrNet's correlation-based encoding is largely redundant, and that natural-looking architectural extensions in CSLR should be tested carefully instead of being assumed to help.