Motion retargeting using graph neural network for vision-guided dexterous robot teleoperation

Dexterous robot teleoperation enables remote access to specialized therapies, addressing geographic barriers in healthcare. However, traditional methods suffer from computational complexities, local minima issues, and reliance on expensive motion capture devices, yielding inaccuracies for unseen motions. We propose a novel vision-guided motion retargeting approach using only an RGB camera, offering a simple, low-cost solution. Our method introduces a graph encoder network for reasonable initial encodings and optimizes in the latent space, reducing complexity and avoiding local minima entrapment. It transforms retargeting into latent code optimization, enabling true end-to-end automated retargeting without manual rule design. Even with limited data or unseen motions, our approach achieves high-precision retargeting through iterative latent space optimization, mitigating reliance on massive datasets. Through simulated experiments retargeting human demonstrations onto a bi-manual robot, we validate our method's effectiveness in successfully reproducing motions, demonstrating feasibility and accuracy for human-robot motion retargeting.