Dual-modal gesture recognition using adaptive weight hierarchical soft voting mechanism

Muscle force and morphology information offer complementary perspectives for gesture recognition and its applications. Surface Electromyography (sEMG) provides force and electrophysiological information associated with muscles, while A-mode ultrasound (AUS) reveals muscle morphological information. By leveraging these two modalities, more comprehensive muscle motor unit information relevant to gesture recognition can be obtained. In this article, we introduce the adaptive weight classification (AWC) module and its enhanced version with hierarchical classifiers, adaptive weight hierarchical soft voting (AWHSV), to integrate AUS and sEMG into a fused modality. This approach dynamically adjusts the weights of individual and fused features, compensating for lost details during fusion, leading to a richer information representation and significantly improving algorithm robustness in gesture recognition. The experimental results demonstrate that the proposed method achieves recognition rates that are 0.66%, 2.36%, and 1.30% higher than those of its counterparts using sEMG, AUS, and sEMG-AUS, respectively. Moreover, the method outperforms state-of-the-art approaches, confirming its effectiveness in gesture recognition across both single and multiple modalities. This work demonstrates the advantages of the proposed AWHSV method, providing broader application scenarios for gesture recognition.