A two-stream CNN framework for American sign language recognition based on multimodal data fusion

Qing Gao; Uchenna Emeoha Ogenyi; Jinguo Liu; Zhaojie Ju; Honghai Liu

doi:10.1007/978-3-030-29933-0_9

A two-stream CNN framework for American sign language recognition based on multimodal data fusion

Qing Gao
, Uchenna Emeoha Ogenyi
, Jinguo Liu
, Zhaojie Ju
, Honghai Liu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

796 Downloads (Pure)

Abstract

At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 % on ASL fingerspelling database and is higher than that of baseline methods.

Original language	English
Title of host publication	Advances in Computational Intelligence Systems
Editors	Zhaojie Ju, Longzhi Yang, Chenguang Yang, Alexander Gegov, Dalin Zhou
Publisher	Springer
Pages	107-118
Volume	1043
ISBN (Electronic)	978-3-030-29933-0
ISBN (Print)	978-3-030-29932-3
DOIs	https://doi.org/10.1007/978-3-030-29933-0_9
Publication status	Published - Sept 2019
Event	19th UK Workshop on Computational Intelligence - Portsmouth, United Kingdom Duration: 4 Sept 2019 → 5 Sept 2019 Conference number: 19 https://www.ukci2019.port.ac.uk/

Publication series

Name	Advances in Computational Intelligence Systems
Publisher	Springer, Cham
Volume	1043
ISSN (Print)	2194-5357
ISSN (Electronic)	2194-5365

Workshop

Workshop	19th UK Workshop on Computational Intelligence
Abbreviated title	UKCI 2019
Country/Territory	United Kingdom
City	Portsmouth
Period	4/09/19 → 5/09/19
Other	The UKCI 2019 covers both theory and applications in computational intelligence. The topics of interest include Fuzzy Systems Neural Networks Evolutionary Computation Evolving Systems Machine Learning Data Mining Cognitive Computing Intelligent Robotics Hybrid Methods Deep Learning Applications of Computational Intelligence
Internet address	https://www.ukci2019.port.ac.uk/

Access to Document

10.1007/978-3-030-29933-0_9

A Two-stream CNN Framework
The final authenticated version is available online at: http://dx.doi.org/10.1007/978-3-030-29933-0_9.
Accepted author manuscript (Post-print), 1.66 MB

Cite this

Gao, Q., Ogenyi, U. E., Liu, J., Ju, Z., & Liu, H. (2019). A two-stream CNN framework for American sign language recognition based on multimodal data fusion. In Z. Ju, L. Yang, C. Yang, A. Gegov, & D. Zhou (Eds.), Advances in Computational Intelligence Systems (Vol. 1043, pp. 107-118). (Advances in Computational Intelligence Systems; Vol. 1043). Springer. https://doi.org/10.1007/978-3-030-29933-0_9

Gao, Qing ; Ogenyi, Uchenna Emeoha ; Liu, Jinguo et al. / A two-stream CNN framework for American sign language recognition based on multimodal data fusion. Advances in Computational Intelligence Systems. editor / Zhaojie Ju ; Longzhi Yang ; Chenguang Yang ; Alexander Gegov ; Dalin Zhou. Vol. 1043 Springer, 2019. pp. 107-118 (Advances in Computational Intelligence Systems).

@inproceedings{a3027f26e33b4713944efd5b546c5f44,

title = "A two-stream CNN framework for American sign language recognition based on multimodal data fusion",

abstract = "At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 \% on ASL fingerspelling database and is higher than that of baseline methods.",

author = "Qing Gao and Ogenyi, \{Uchenna Emeoha\} and Jinguo Liu and Zhaojie Ju and Honghai Liu",

year = "2019",

month = sep,

doi = "10.1007/978-3-030-29933-0\_9",

language = "English",

isbn = "978-3-030-29932-3",

volume = "1043",

series = "Advances in Computational Intelligence Systems",

publisher = "Springer",

pages = "107--118",

editor = "Zhaojie Ju and Longzhi Yang and Chenguang Yang and Alexander Gegov and Dalin Zhou",

booktitle = "Advances in Computational Intelligence Systems",

address = "Netherlands",

note = "19th UK Workshop on Computational Intelligence, UKCI 2019 ; Conference date: 04-09-2019 Through 05-09-2019",

url = "https://www.ukci2019.port.ac.uk/",

}

Gao, Q, Ogenyi, UE, Liu, J, Ju, Z & Liu, H 2019, A two-stream CNN framework for American sign language recognition based on multimodal data fusion. in Z Ju, L Yang, C Yang, A Gegov & D Zhou (eds), Advances in Computational Intelligence Systems. vol. 1043, Advances in Computational Intelligence Systems, vol. 1043, Springer, pp. 107-118, 19th UK Workshop on Computational Intelligence, Portsmouth, United Kingdom, 4/09/19. https://doi.org/10.1007/978-3-030-29933-0_9

A two-stream CNN framework for American sign language recognition based on multimodal data fusion. / Gao, Qing; Ogenyi, Uchenna Emeoha; Liu, Jinguo et al.
Advances in Computational Intelligence Systems. ed. / Zhaojie Ju; Longzhi Yang; Chenguang Yang; Alexander Gegov; Dalin Zhou. Vol. 1043 Springer, 2019. p. 107-118 (Advances in Computational Intelligence Systems; Vol. 1043).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A two-stream CNN framework for American sign language recognition based on multimodal data fusion

AU - Gao, Qing

AU - Ogenyi, Uchenna Emeoha

AU - Liu, Jinguo

AU - Ju, Zhaojie

AU - Liu, Honghai

N1 - Conference code: 19

PY - 2019/9

Y1 - 2019/9

N2 - At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 % on ASL fingerspelling database and is higher than that of baseline methods.

AB - At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 % on ASL fingerspelling database and is higher than that of baseline methods.

U2 - 10.1007/978-3-030-29933-0_9

DO - 10.1007/978-3-030-29933-0_9

M3 - Conference contribution

SN - 978-3-030-29932-3

VL - 1043

T3 - Advances in Computational Intelligence Systems

SP - 107

EP - 118

BT - Advances in Computational Intelligence Systems

A2 - Ju, Zhaojie

A2 - Yang, Longzhi

A2 - Yang, Chenguang

A2 - Gegov, Alexander

A2 - Zhou, Dalin

PB - Springer

T2 - 19th UK Workshop on Computational Intelligence

Y2 - 4 September 2019 through 5 September 2019

ER -

Gao Q, Ogenyi UE, Liu J, Ju Z , Liu H. A two-stream CNN framework for American sign language recognition based on multimodal data fusion. In Ju Z, Yang L, Yang C, Gegov A, Zhou D, editors, Advances in Computational Intelligence Systems. Vol. 1043. Springer. 2019. p. 107-118. (Advances in Computational Intelligence Systems). Epub 2019 Aug 30. doi: 10.1007/978-3-030-29933-0_9

A two-stream CNN framework for American sign language recognition based on multimodal data fusion

Abstract

Publication series

Workshop

Access to Document

Fingerprint

Cite this