Skip to content
Back to outputs

A two-stream CNN framework for American sign language recognition based on multimodal data fusion

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Standard

A two-stream CNN framework for American sign language recognition based on multimodal data fusion. / Gao, Qing; Ogenyi, Uchenna Emeoha; Liu, Jinguo; Ju, Zhaojie; Liu, Honghai.

Advances in Computational Intelligence Systems. ed. / Zhaojie Ju; Longzhi Yang; Chenguang Yang; Alexander Gegov; Dalin Zhou. Vol. 1043 Springer, 2019. p. 107-118 (Advances in Computational Intelligence Systems; Vol. 1043).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Harvard

Gao, Q, Ogenyi, UE, Liu, J, Ju, Z & Liu, H 2019, A two-stream CNN framework for American sign language recognition based on multimodal data fusion. in Z Ju, L Yang, C Yang, A Gegov & D Zhou (eds), Advances in Computational Intelligence Systems. vol. 1043, Advances in Computational Intelligence Systems, vol. 1043, Springer, pp. 107-118, 19th UK Workshop on Computational Intelligence, Portsmouth, United Kingdom, 4/09/19. https://doi.org/10.1007/978-3-030-29933-0_9

APA

Gao, Q., Ogenyi, U. E., Liu, J., Ju, Z., & Liu, H. (2019). A two-stream CNN framework for American sign language recognition based on multimodal data fusion. In Z. Ju, L. Yang, C. Yang, A. Gegov, & D. Zhou (Eds.), Advances in Computational Intelligence Systems (Vol. 1043, pp. 107-118). (Advances in Computational Intelligence Systems; Vol. 1043). Springer. https://doi.org/10.1007/978-3-030-29933-0_9

Vancouver

Gao Q, Ogenyi UE, Liu J, Ju Z, Liu H. A two-stream CNN framework for American sign language recognition based on multimodal data fusion. In Ju Z, Yang L, Yang C, Gegov A, Zhou D, editors, Advances in Computational Intelligence Systems. Vol. 1043. Springer. 2019. p. 107-118. (Advances in Computational Intelligence Systems). https://doi.org/10.1007/978-3-030-29933-0_9

Author

Gao, Qing ; Ogenyi, Uchenna Emeoha ; Liu, Jinguo ; Ju, Zhaojie ; Liu, Honghai. / A two-stream CNN framework for American sign language recognition based on multimodal data fusion. Advances in Computational Intelligence Systems. editor / Zhaojie Ju ; Longzhi Yang ; Chenguang Yang ; Alexander Gegov ; Dalin Zhou. Vol. 1043 Springer, 2019. pp. 107-118 (Advances in Computational Intelligence Systems).

Bibtex

@inproceedings{a3027f26e33b4713944efd5b546c5f44,
title = "A two-stream CNN framework for American sign language recognition based on multimodal data fusion",
abstract = "At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 % on ASL fingerspelling database and is higher than that of baseline methods.",
author = "Qing Gao and Ogenyi, {Uchenna Emeoha} and Jinguo Liu and Zhaojie Ju and Honghai Liu",
year = "2019",
month = sep,
doi = "10.1007/978-3-030-29933-0_9",
language = "English",
isbn = "978-3-030-29932-3",
volume = "1043",
series = "Advances in Computational Intelligence Systems",
publisher = "Springer",
pages = "107--118",
editor = "Zhaojie Ju and Longzhi Yang and Chenguang Yang and Alexander Gegov and Dalin Zhou",
booktitle = "Advances in Computational Intelligence Systems",
note = "19th UK Workshop on Computational Intelligence, UKCI 2019 ; Conference date: 04-09-2019 Through 05-09-2019",
url = "https://www.ukci2019.port.ac.uk/",

}

RIS

TY - GEN

T1 - A two-stream CNN framework for American sign language recognition based on multimodal data fusion

AU - Gao, Qing

AU - Ogenyi, Uchenna Emeoha

AU - Liu, Jinguo

AU - Ju, Zhaojie

AU - Liu, Honghai

N1 - Conference code: 19

PY - 2019/9

Y1 - 2019/9

N2 - At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 % on ASL fingerspelling database and is higher than that of baseline methods.

AB - At present, vision-based hand gesture recognition is very important in human-robot interaction (HRI). This non-contact method enables natural and friendly interaction between people and robots. Aiming at this technology, a two-stream CNN framework (2S-CNN) is proposed to recognize the American sign language (ASL) hand gestures based on multimodal (RGB and depth) data fusion. Firstly, the hand gesture data is enhanced to remove the influence of background and noise. Secondly, hand gesture RGB and depth features are extracted for hand gesture recognition using CNNs on two streams, respectively. Finally, a fusion layer is designed for fusing the recognition results of the two streams. This method utilizes multimodal data to increase the recognition accuracy of the ASL hand gestures. The experiments prove that the recognition accuracy of 2S-CNN can reach 92.08 % on ASL fingerspelling database and is higher than that of baseline methods.

U2 - 10.1007/978-3-030-29933-0_9

DO - 10.1007/978-3-030-29933-0_9

M3 - Conference contribution

SN - 978-3-030-29932-3

VL - 1043

T3 - Advances in Computational Intelligence Systems

SP - 107

EP - 118

BT - Advances in Computational Intelligence Systems

A2 - Ju, Zhaojie

A2 - Yang, Longzhi

A2 - Yang, Chenguang

A2 - Gegov, Alexander

A2 - Zhou, Dalin

PB - Springer

T2 - 19th UK Workshop on Computational Intelligence

Y2 - 4 September 2019 through 5 September 2019

ER -

ID: 16304646