A feature fusion network for skeleton-based gesture recognition

Xiaowen You; Qing Gao; Hongwei Gao; Zhaojie Ju

doi:10.1007/978-981-99-6486-4_6

A feature fusion network for skeleton-based gesture recognition

Xiaowen You
, Qing Gao
, Hongwei Gao
, Zhaojie Ju

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

72 Downloads (Pure)

Abstract

With the development of the times, the requirements for human-computer interaction methods have gradually increased, and naturalness and comfort are constantly pursued on the basis of traditional precision. Gesture is one of the innate ways of human communication, which is highly intuitive and can be employed as an effective means of natural human-computer interaction. In this paper, dynamic gestures are investigated based on the 3D skeletal information of gestures, and different cropping boxes are placed at generate global and local datasets respectively according to whether they depend on the motion trajectory of the gesture. By analyzing the geometric features of skeletal sequences, a dual-stream 3D CNN (Double_C3D) framework is proposed for fusion at the feature level, which relies on 3D heat map video streams and uses the video streams as the input to the network. Finally, the Double_C3D framework was evaluated on the SHREC dynamic gesture recognition dataset and the JHMBD dynamic behavior recognition dataset with an accuracy of 91.72% and 70.54%, respectively.

Original language	English
Title of host publication	Intelligent Robotics and Applications 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part II
Editors	Huayong Yang, Honghai Liu, Jun Zou, Zhouping Yin, Lianqing Liu, Geng Yang, Xiaoping Ouyang, Zhiyong Wang
Publisher	Springer
Pages	67-78
Number of pages	12
ISBN (Electronic)	9789819964864
ISBN (Print)	9789819964857
DOIs	https://doi.org/10.1007/978-981-99-6486-4_6
Publication status	Published - 10 Oct 2023
Event	International Conference on Intelligent Robotics and Applications - Hangzhou, China Duration: 5 Jul 2023 → 7 Jul 2023

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	14268
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Name	Lecture Notes in Artificial Intelligence
Publisher	Springer
ISSN (Print)	2945-9133
ISSN (Electronic)	2945-9141

Conference

Conference	International Conference on Intelligent Robotics and Applications
Country/Territory	China
City	Hangzhou
Period	5/07/23 → 7/07/23

Keywords

3D skeleton
pseudo heat map
3D CNN

Access to Document

10.1007/978-981-99-6486-4_6

Final_paper_78_Xiaowen_You
This version of the article has been accepted for publication, after peer review (when applicable) and is subject to Springer Nature’s AM terms of use, but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: http://dx.doi.org/10.1007/978-981-99-6486-4_6.
Accepted author manuscript (Post-print), 1.26 MB

Cite this

You, X., Gao, Q., Gao, H., & Ju, Z. (2023). A feature fusion network for skeleton-based gesture recognition. In H. Yang, H. Liu, J. Zou, Z. Yin, L. Liu, G. Yang, X. Ouyang, & Z. Wang (Eds.), Intelligent Robotics and Applications 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part II (pp. 67-78). (Lecture Notes in Computer Science; Vol. 14268), (Lecture Notes in Artificial Intelligence). Springer. https://doi.org/10.1007/978-981-99-6486-4_6

You, Xiaowen ; Gao, Qing ; Gao, Hongwei et al. / A feature fusion network for skeleton-based gesture recognition. Intelligent Robotics and Applications 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part II. editor / Huayong Yang ; Honghai Liu ; Jun Zou ; Zhouping Yin ; Lianqing Liu ; Geng Yang ; Xiaoping Ouyang ; Zhiyong Wang. Springer, 2023. pp. 67-78 (Lecture Notes in Computer Science). (Lecture Notes in Artificial Intelligence).

@inproceedings{97be33c14d7b4c2688d3da74183f28f7,

title = "A feature fusion network for skeleton-based gesture recognition",

abstract = "With the development of the times, the requirements for human-computer interaction methods have gradually increased, and naturalness and comfort are constantly pursued on the basis of traditional precision. Gesture is one of the innate ways of human communication, which is highly intuitive and can be employed as an effective means of natural human-computer interaction. In this paper, dynamic gestures are investigated based on the 3D skeletal information of gestures, and different cropping boxes are placed at generate global and local datasets respectively according to whether they depend on the motion trajectory of the gesture. By analyzing the geometric features of skeletal sequences, a dual-stream 3D CNN (Double\_C3D) framework is proposed for fusion at the feature level, which relies on 3D heat map video streams and uses the video streams as the input to the network. Finally, the Double\_C3D framework was evaluated on the SHREC dynamic gesture recognition dataset and the JHMBD dynamic behavior recognition dataset with an accuracy of 91.72\% and 70.54\%, respectively.",

keywords = "3D skeleton, pseudo heat map, 3D CNN",

author = "Xiaowen You and Qing Gao and Hongwei Gao and Zhaojie Ju",

year = "2023",

month = oct,

day = "10",

doi = "10.1007/978-981-99-6486-4\_6",

language = "English",

isbn = "9789819964857",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "67--78",

editor = "Huayong Yang and Honghai Liu and Jun Zou and Zhouping Yin and Lianqing Liu and Geng Yang and Xiaoping Ouyang and Zhiyong Wang",

booktitle = "Intelligent Robotics and Applications 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part II",

address = "Netherlands",

note = "International Conference on Intelligent Robotics and Applications ; Conference date: 05-07-2023 Through 07-07-2023",

}

You, X, Gao, Q, Gao, H & Ju, Z 2023, A feature fusion network for skeleton-based gesture recognition. in H Yang, H Liu, J Zou, Z Yin, L Liu, G Yang, X Ouyang & Z Wang (eds), Intelligent Robotics and Applications 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part II. Lecture Notes in Computer Science, vol. 14268, Lecture Notes in Artificial Intelligence, Springer, pp. 67-78, International Conference on Intelligent Robotics and Applications, Hangzhou, China, 5/07/23. https://doi.org/10.1007/978-981-99-6486-4_6

A feature fusion network for skeleton-based gesture recognition. / You, Xiaowen; Gao, Qing; Gao, Hongwei et al.
Intelligent Robotics and Applications 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part II. ed. / Huayong Yang; Honghai Liu; Jun Zou; Zhouping Yin; Lianqing Liu; Geng Yang; Xiaoping Ouyang; Zhiyong Wang. Springer, 2023. p. 67-78 (Lecture Notes in Computer Science; Vol. 14268), (Lecture Notes in Artificial Intelligence).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A feature fusion network for skeleton-based gesture recognition

AU - You, Xiaowen

AU - Gao, Qing

AU - Gao, Hongwei

AU - Ju, Zhaojie

PY - 2023/10/10

Y1 - 2023/10/10

N2 - With the development of the times, the requirements for human-computer interaction methods have gradually increased, and naturalness and comfort are constantly pursued on the basis of traditional precision. Gesture is one of the innate ways of human communication, which is highly intuitive and can be employed as an effective means of natural human-computer interaction. In this paper, dynamic gestures are investigated based on the 3D skeletal information of gestures, and different cropping boxes are placed at generate global and local datasets respectively according to whether they depend on the motion trajectory of the gesture. By analyzing the geometric features of skeletal sequences, a dual-stream 3D CNN (Double_C3D) framework is proposed for fusion at the feature level, which relies on 3D heat map video streams and uses the video streams as the input to the network. Finally, the Double_C3D framework was evaluated on the SHREC dynamic gesture recognition dataset and the JHMBD dynamic behavior recognition dataset with an accuracy of 91.72% and 70.54%, respectively.

AB - With the development of the times, the requirements for human-computer interaction methods have gradually increased, and naturalness and comfort are constantly pursued on the basis of traditional precision. Gesture is one of the innate ways of human communication, which is highly intuitive and can be employed as an effective means of natural human-computer interaction. In this paper, dynamic gestures are investigated based on the 3D skeletal information of gestures, and different cropping boxes are placed at generate global and local datasets respectively according to whether they depend on the motion trajectory of the gesture. By analyzing the geometric features of skeletal sequences, a dual-stream 3D CNN (Double_C3D) framework is proposed for fusion at the feature level, which relies on 3D heat map video streams and uses the video streams as the input to the network. Finally, the Double_C3D framework was evaluated on the SHREC dynamic gesture recognition dataset and the JHMBD dynamic behavior recognition dataset with an accuracy of 91.72% and 70.54%, respectively.

KW - 3D skeleton

KW - pseudo heat map

KW - 3D CNN

U2 - 10.1007/978-981-99-6486-4_6

DO - 10.1007/978-981-99-6486-4_6

M3 - Conference contribution

SN - 9789819964857

T3 - Lecture Notes in Computer Science

SP - 67

EP - 78

BT - Intelligent Robotics and Applications 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part II

A2 - Yang, Huayong

A2 - Liu, Honghai

A2 - Zou, Jun

A2 - Yin, Zhouping

A2 - Liu, Lianqing

A2 - Yang, Geng

A2 - Ouyang, Xiaoping

A2 - Wang, Zhiyong

PB - Springer

T2 - International Conference on Intelligent Robotics and Applications

Y2 - 5 July 2023 through 7 July 2023

ER -

You X, Gao Q, Gao H, Ju Z. A feature fusion network for skeleton-based gesture recognition. In Yang H, Liu H, Zou J, Yin Z, Liu L, Yang G, Ouyang X, Wang Z, editors, Intelligent Robotics and Applications 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part II. Springer. 2023. p. 67-78. (Lecture Notes in Computer Science). (Lecture Notes in Artificial Intelligence). doi: 10.1007/978-981-99-6486-4_6

A feature fusion network for skeleton-based gesture recognition

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this