Skip to content

Hand gesture recognition using multimodal data fusion and multi-scale parallel convolutional neural network for human-robot interaction

Research output: Contribution to journalArticle

Hand gesture recognition plays an important role in human–robot interaction. The accuracy and reliability of hand gesture recognition are the keys to gesture‐based human–robot interaction tasks. To solve this problem, a method based on multimodal data fusion and multiscale parallel convolutional neural network (CNN) is proposed in this paper to improve the accuracy and reliability of hand gesture recognition. First of all, data fusion is conducted on the sEMG signal, the RGB image, and the depth image of hand gestures. Then, the fused images are generated to two different scale images by downsampling, which are respectively input into two subnetworks of the parallel CNN to obtain two hand gesture recognition results. After that, hand gesture recognition results of the parallel CNN are combined to obtain the final hand gesture recognition result. Finally, experiments are carried out on a self‐made database containing 10 common hand gestures, which verify the effectiveness and superiority of the proposed method for hand gesture recognition. In addition, the proposed method is applied to a seven‐degree‐of‐freedom bionic manipulator to achieve robotic manipulation with hand gestures.
Original languageEnglish
Article numbere12490
Number of pages12
JournalExpert Systems
Early online date7 Jan 2020
DOIs
Publication statusEarly online - 7 Jan 2020

Documents

  • Postprint

    Rights statement: This is the peer reviewed version of the following article: Gao, Q, Liu, J, Ju, Z. Hand gesture recognition using multimodal data fusion and multiscale parallel convolutional neural network for human–robot interaction. Expert Systems. 2020; e12490, which has been published in final form at https://doi.org/10.1111/exsy.12490. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions.

    Accepted author manuscript (Post-print), 4.06 MB, PDF document

    Due to publisher’s copyright restrictions, this document is not freely available to download from this website until: 7/01/21

Related information

Relations Get citation (various referencing formats)

ID: 17973135