Multimodal hand gesture recognition based on the fusion of surface electromyography and vision

In existing hand gesture recognition research, single-modal recognition is commonly used. For example, visual hand gesture recognition uses image information, but it is easily affected by the shooting environment. Another example is using surface electromyography (sEMG) for recognition, but it is susceptible to signal noise. To address the above issues, this paper focuses on the fusion of sEMG and vision of the human hand. We propose a novel approach that fuses the two modalities by using convolutional neural networks (CNN) to improve recognition accuracy. Firstly, using an RGB camera and sEMG armband, we jointly collect sEMG signal and skeleton in real-time, creating our own multimodal dataset for training. Secondly, we design a multimodal recognition network with feature fusion of sEMG and skeleton, to achieve an increase in accuracy. Finally, we built a human-computer interaction system that realizes hand gestures to manipulate a dexterous hand and a robot arm. Experimental results demonstrate that the fusion of the two modalities has complementary effects and effectively improves recognition accuracy.