3D human pose estimation in video for human-computer/robot interaction

Rongtian Huo; Qing Gao; Jing Qi; Zhaojie Ju

doi:10.1007/978-981-99-6498-7_16

3D human pose estimation in video for human-computer/robot interaction

Rongtian Huo, Qing Gao, Jing Qi, Zhaojie Ju

School of Computing

Sun Yat-Sen University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

3D human pose estimation is widely used in motion capture, human-computer interaction, virtual character driving and other fields. The current 3D human pose estimation has been suffering from depth blurring and self-obscuring problems to be solved. This paper proposes a human pose estimation network in video based on a 2D lifting to 3D approach using transformer and graph convolutional network(GCN), which are widely used in natural language processing. We use transformer to obtain sequence features and use graph convolution to extract features between local joints to get more accurate 3D pose coordinates. In addition, we use the proposed 3D pose estimation network for animated character motion generation and robot motion following and design two systems of human-computer/robot interaction (HCI/HRI) applications. The proposed 3D human pose estimation network is tested on the Human3.6M dataset and outperforms the state-of-the-art models. Both HCI/HRI systems are designed to work quickly and accurately by the proposed 3D human pose estimation method.

Original language	English
Title of host publication	Intelligent Robotics and Applications
Subtitle of host publication	16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part VII
Editors	Huayong Yang, Honghai Liu, Jun Zou, Zhouping Yin, Lianqing Liu, Geng Yang, Xiaoping Ouyang, Zhiyong Wang
Publisher	Springer
Pages	176–187
ISBN (Electronic)	9789819964987
ISBN (Print)	9789819964970
DOIs	https://doi.org/10.1007/978-981-99-6498-7_16
Publication status	Published - 13 Oct 2023
Event	International Conference on Intelligent Robotics and Applications - Hangzhou, China Duration: 5 Jul 2023 → 7 Jul 2023

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer Nature
Volume	14273
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	International Conference on Intelligent Robotics and Applications
Country/Territory	China
City	Hangzhou
Period	5/07/23 → 7/07/23

Keywords

Human Pose Estimation
Human-Computer Interaction
Human-Robot Interaction
Deep Learning

Access to Document

10.1007/978-981-99-6498-7_16

Embargoed Document

2023.522-ICIRA-V9_1_
Accepted author manuscript (Post-print), 29.1 MB
Due to publisher’s copyright restrictions, this document is not freely available to download from this website until: 13/10/24

Cite this

Huo, R., Gao, Q., Qi, J., & Ju, Z. (2023). 3D human pose estimation in video for human-computer/robot interaction. In H. Yang, H. Liu, J. Zou, Z. Yin, L. Liu, G. Yang, X. Ouyang, & Z. Wang (Eds.), Intelligent Robotics and Applications: 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part VII (pp. 176–187). (Lecture Notes in Computer Science; Vol. 14273). Springer. https://doi.org/10.1007/978-981-99-6498-7_16

Huo, Rongtian ; Gao, Qing ; Qi, Jing et al. / 3D human pose estimation in video for human-computer/robot interaction. Intelligent Robotics and Applications: 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part VII. editor / Huayong Yang ; Honghai Liu ; Jun Zou ; Zhouping Yin ; Lianqing Liu ; Geng Yang ; Xiaoping Ouyang ; Zhiyong Wang. Springer, 2023. pp. 176–187 (Lecture Notes in Computer Science).

@inproceedings{11c0e2739b5840c99b9109c5c66e5c7a,

title = "3D human pose estimation in video for human-computer/robot interaction",

abstract = "3D human pose estimation is widely used in motion capture, human-computer interaction, virtual character driving and other fields. The current 3D human pose estimation has been suffering from depth blurring and self-obscuring problems to be solved. This paper proposes a human pose estimation network in video based on a 2D lifting to 3D approach using transformer and graph convolutional network(GCN), which are widely used in natural language processing. We use transformer to obtain sequence features and use graph convolution to extract features between local joints to get more accurate 3D pose coordinates. In addition, we use the proposed 3D pose estimation network for animated character motion generation and robot motion following and design two systems of human-computer/robot interaction (HCI/HRI) applications. The proposed 3D human pose estimation network is tested on the Human3.6M dataset and outperforms the state-of-the-art models. Both HCI/HRI systems are designed to work quickly and accurately by the proposed 3D human pose estimation method.",

keywords = "Human Pose Estimation, Human-Computer Interaction, Human-Robot Interaction, Deep Learning",

author = "Rongtian Huo and Qing Gao and Jing Qi and Zhaojie Ju",

year = "2023",

month = oct,

day = "13",

doi = "10.1007/978-981-99-6498-7_16",

language = "English",

isbn = "9789819964970",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "176–187",

editor = "Yang, {Huayong } and Liu, {Honghai } and Zou, {Jun } and Yin, {Zhouping } and Liu, {Lianqing } and Yang, {Geng } and Ouyang, {Xiaoping } and Wang, {Zhiyong }",

booktitle = "Intelligent Robotics and Applications",

note = "International Conference on Intelligent Robotics and Applications ; Conference date: 05-07-2023 Through 07-07-2023",

}

Huo, R, Gao, Q, Qi, J & Ju, Z 2023, 3D human pose estimation in video for human-computer/robot interaction. in H Yang, H Liu, J Zou, Z Yin, L Liu, G Yang, X Ouyang & Z Wang (eds), Intelligent Robotics and Applications: 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part VII. Lecture Notes in Computer Science, vol. 14273, Springer, pp. 176–187, International Conference on Intelligent Robotics and Applications, Hangzhou, China, 5/07/23. https://doi.org/10.1007/978-981-99-6498-7_16

3D human pose estimation in video for human-computer/robot interaction. / Huo, Rongtian; Gao, Qing; Qi, Jing et al.
Intelligent Robotics and Applications: 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part VII. ed. / Huayong Yang; Honghai Liu; Jun Zou; Zhouping Yin; Lianqing Liu; Geng Yang; Xiaoping Ouyang; Zhiyong Wang. Springer, 2023. p. 176–187 (Lecture Notes in Computer Science; Vol. 14273).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - 3D human pose estimation in video for human-computer/robot interaction

AU - Huo, Rongtian

AU - Gao, Qing

AU - Qi, Jing

AU - Ju, Zhaojie

PY - 2023/10/13

Y1 - 2023/10/13

N2 - 3D human pose estimation is widely used in motion capture, human-computer interaction, virtual character driving and other fields. The current 3D human pose estimation has been suffering from depth blurring and self-obscuring problems to be solved. This paper proposes a human pose estimation network in video based on a 2D lifting to 3D approach using transformer and graph convolutional network(GCN), which are widely used in natural language processing. We use transformer to obtain sequence features and use graph convolution to extract features between local joints to get more accurate 3D pose coordinates. In addition, we use the proposed 3D pose estimation network for animated character motion generation and robot motion following and design two systems of human-computer/robot interaction (HCI/HRI) applications. The proposed 3D human pose estimation network is tested on the Human3.6M dataset and outperforms the state-of-the-art models. Both HCI/HRI systems are designed to work quickly and accurately by the proposed 3D human pose estimation method.

AB - 3D human pose estimation is widely used in motion capture, human-computer interaction, virtual character driving and other fields. The current 3D human pose estimation has been suffering from depth blurring and self-obscuring problems to be solved. This paper proposes a human pose estimation network in video based on a 2D lifting to 3D approach using transformer and graph convolutional network(GCN), which are widely used in natural language processing. We use transformer to obtain sequence features and use graph convolution to extract features between local joints to get more accurate 3D pose coordinates. In addition, we use the proposed 3D pose estimation network for animated character motion generation and robot motion following and design two systems of human-computer/robot interaction (HCI/HRI) applications. The proposed 3D human pose estimation network is tested on the Human3.6M dataset and outperforms the state-of-the-art models. Both HCI/HRI systems are designed to work quickly and accurately by the proposed 3D human pose estimation method.

KW - Human Pose Estimation

KW - Human-Computer Interaction

KW - Human-Robot Interaction

KW - Deep Learning

UR - https://icira2023.org/

U2 - 10.1007/978-981-99-6498-7_16

DO - 10.1007/978-981-99-6498-7_16

M3 - Conference contribution

SN - 9789819964970

T3 - Lecture Notes in Computer Science

SP - 176

EP - 187

BT - Intelligent Robotics and Applications

A2 - Yang, Huayong

A2 - Liu, Honghai

A2 - Zou, Jun

A2 - Yin, Zhouping

A2 - Liu, Lianqing

A2 - Yang, Geng

A2 - Ouyang, Xiaoping

A2 - Wang, Zhiyong

PB - Springer

T2 - International Conference on Intelligent Robotics and Applications

Y2 - 5 July 2023 through 7 July 2023

ER -

Huo R, Gao Q, Qi J, Ju Z. 3D human pose estimation in video for human-computer/robot interaction. In Yang H, Liu H, Zou J, Yin Z, Liu L, Yang G, Ouyang X, Wang Z, editors, Intelligent Robotics and Applications: 16th International Conference, ICIRA 2023, Hangzhou, China, July 5–7, 2023, Proceedings, Part VII. Springer. 2023. p. 176–187. (Lecture Notes in Computer Science). doi: 10.1007/978-981-99-6498-7_16