Dual-hand motion capture by using biological inspiration for bionic bimanual robot teleoperation

Bionic bimanual robot teleoperation can transfer the grasping and manipulation skills of human dual hands to the bionic bimanual robots to realize natural and flexible manipulation. The motion capture of dual hands plays an important role in the teleoperation. The motion information of dual hands can be captured through the hand detection, localization, and pose estimation and mapped to the bionic bimanual robots to realize the teleoperation. However, although the motion capture technology has achieved great achievements in recent years, visual dual-hand motion capture is still a great challenge. So, this work proposed a dual-hand detection method and a 3-dimensional (3D) hand pose estimation method based on body and hand biological inspiration to achieve convenient and accurate monocular dual-hand motion capture and bionic bimanual robot teleoperation. First, a dual-hand detection method based on body structure constraints is proposed, which uses a parallel structure to combine hand and body relationship features. Second, a 3D hand pose estimation method with bone-constraint loss from single RGB images is proposed. Then, a bionic bimanual robot teleoperation method is designed by using the proposed hand detection and pose estimation methods. Experiment results on public hand datasets show that the performances of the proposed hand detection and 3D hand pose estimation outperform state-of-the-art methods. Experiment results on a bionic bimanual robot teleoperation platform shows the effectiveness of the proposed teleoperation method.