Skip to content

A two-stream CNN framework for American sign language recognition based on multimodal data fusion

Research output: Chapter in Book/Report/Conference proceedingConference contribution

At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 % on ASL fingerspelling database and is higher than that of baseline methods.
Original languageEnglish
Title of host publicationAdvances in Computational Intelligence Systems
EditorsZhaojie Ju, Longzhi Yang, Chenguang Yang, Alexander Gegov, Dalin Zhou
PublisherSpringer
Pages107-118
Volume1043
ISBN (Electronic)978-3-030-29933-0
ISBN (Print)978-3-030-29932-3
DOIs
Publication statusPublished - Sep 2019
Event19th UK Workshop on Computational Intelligence - Portsmouth, United Kingdom
Duration: 4 Sep 20195 Sep 2019
Conference number: 19
https://www.ukci2019.port.ac.uk/

Publication series

NameAdvances in Computational Intelligence Systems
PublisherSpringer, Cham
Volume1043
ISSN (Print)2194-5357
ISSN (Electronic)2194-5365

Workshop

Workshop19th UK Workshop on Computational Intelligence
Abbreviated titleUKCI 2019
CountryUnited Kingdom
CityPortsmouth
Period4/09/195/09/19
OtherThe UKCI 2019 covers both theory and applications in computational intelligence. The topics of interest include
Fuzzy Systems
Neural Networks
Evolutionary Computation
Evolving Systems
Machine Learning
Data Mining
Cognitive Computing
Intelligent Robotics
Hybrid Methods
Deep Learning
Applications of Computational Intelligence
Internet address

Documents

  • A Two-stream CNN Framework

    Rights statement: The final authenticated version is available online at: http://dx.doi.org/10.1007/978-3-030-29933-0_9.

    Accepted author manuscript (Post-print), 1.66 MB, PDF document

    Due to publisher’s copyright restrictions, this document is not freely available to download from this website until: 30/08/20

Related information

Relations Get citation (various referencing formats)

ID: 16304646