An efficient RGB-D hand gesture detection framework for dexterous robot hand-arm teleoperation system

Qing Gao, Zhaojie Ju, Yongquan Chen, Qiwen Wang, Chuliang Chi

Research output: Contribution to journalArticlepeer-review

Abstract

Aiming at the problems of accurate and fast hand gesture detection and teleoperation mapping in the hand-based visual teleoperation of dexterous robots, an efficient hand gesture detection framework based on deep learning is proposed in this article. It can achieve an accurate and fast hand gesture detection and teleoperation of dexterous robots based on an anchor-free network architecture by using an RGB-D camera. First, an RGB-D early-fusion method based on the HSV space is proposed, effectively reducing background interference and enhancing hand information. Second, a hand gesture classification network (HandClasNet) is proposed to realize hand detection and localization by detecting the center and corner points of hands, and a HandClasNet is proposed to realize gesture recognition by using a parallel EfficientNet structure. Then, a dexterous robot hand-arm teleoperation system based on the hand gesture detection framework is designed to realize the hand-based teleoperation of a dexterous robot. Our method achieves high accuracy with fast speed on public and custom hand datasets and outperforms some state-of-the-art methods. In addition, the application of the proposed method in the hand-based teleoperation system can control the grasping of various objects by a dexterous hand-arm system in real time and accurately, which verifies the efficiency of our method.

Original languageEnglish
Number of pages11
Journal IEEE Transactions on Human-Machine Systems
Early online date26 Sep 2022
DOIs
Publication statusEarly online - 26 Sep 2022

Keywords

  • data integration
  • dexterous robot
  • hand gesture detection
  • interference
  • location awareness
  • RGB-D
  • robot kinematics
  • robot vision systems
  • robots
  • teleoperation
  • visualization

Fingerprint

Dive into the research topics of 'An efficient RGB-D hand gesture detection framework for dexterous robot hand-arm teleoperation system'. Together they form a unique fingerprint.

Cite this