Parallel dual-hand detection by using hand and body features for robot teleoperation

Visual hand-based robot teleoperation provides a powerful guarantee for robots to complete complex tasks. However, detection and distinction of dual hands on images are difficult because of the small differences between left and right hands. To solve this problem, a parallel dual-hand detection and distinction method that combines the features of hands with the relationship features between the dual hands and body pose is proposed to achieve robust and accurate dual-hand detection. This parallel dual-hand detection method includes a hand detection module, a body pose estimation module, and a fusion module. In the hand detection module, a hand detector that realizes fast and accurate hand detection by detecting the center and corner points of hands is designed. In the body pose estimation module, a body pose estimator with dual-hand positions is proposed. The fusion module is designed to fuse hand detection and dual-hand estimation results to achieve distinction between left and right hands. Finally, the parallel dual-hand detection method is applied to a bimanual robot teleoperation system by using a designed dual-hand teleoperation framework. The proposed parallel dual-hand detection method can achieve 98.54% mAP of hand detection with 18 frames per second on a custom dual-hand detection dataset, and the bimanual robot teleoperation method can achieve 95.4% average accuracy for teleoperation tasks. Experimental results show the high accuracy and speed of our proposed parallel dual-hand detection method and its practicability in bimanual robot teleoperation.