SignRobot: Sign Language Recognition for Robot Interaction Based on Dual-Stream Multi-Fusion with Frame Enhancement Network

Deaf and hard-of-hearing individuals often rely on signs for communication, but limited translation resources restrict their daily needs. Robots equipped with sign language recognition and interaction capabilities can assist in bridging this gap. To enhance sign language translation resources, we design a robotic system called SignRobot, which accurately recognizes and responds to signs. To improve recognition performance, we develop a sign language recognition network based on dual-stream multi-fusion and frame enhancement (MFE-Net), using RGB and heatmaps as inputs. Specifically, an frame enhancement module with parallel spatial and motion guidance is introduced to emphasize key spatial regions and movement changes. By employing a multi-fusion strategy, we achieve feature-level interaction and adaptive late fusion between modalities, improving accuracy and robustness. Experimental results show that MFE-Net surpasses state-of-the-art methods on the PHOENIX14 and PHOENIX14-T datasets. Additionally, our SignRobot successfully demonstrates sign recognition and robotic responses in sign language, representing a promising advancement in robot-assisted communication for deaf people.