Recurrent attention unit: a new gated recurrent unit for long-term memory of important parts in sequential data

Zhaoyang Niu, Guoqiang Zhong, Guohua Yue, Li-Na Wang, Hui Yu, Xiao Ling, Junyu Dong

Research output: Contribution to journalArticlepeer-review

87 Downloads (Pure)

Abstract

Gated recurrent unit (GRU) is a variant of the recurrent neural network (RNN). It has been widely used in many applications, such as handwriting recognition and natural language processing. However, GRU can only memorize the sequential information, but lacks the capability of adaptively paying attention to important parts in the sequences. In this paper, we propose a novel RNN model, called recurrent attention unit (RAU), which can seamlessly integrate the attention mechanism into the interior of the GRU cell by adding an attention gate. The attention gate enhances the ability of RAU to remember long-term information and pay attention to important parts in the sequential data. Extensive experiments on adding problem, image classification, sentiment classification and language modeling show that RAU consistently outperforms GRU and other related models.
Original languageEnglish
Pages (from-to)1-9
Number of pages9
JournalNeurocomputing
Volume517
Early online date2 Nov 2022
DOIs
Publication statusPublished - 14 Jan 2023

Keywords

  • attention mechanism
  • gated recurrent unit (GRU)
  • memory
  • recurrent attention unit (RAU)
  • recurrent neural networks (RNNs)

Cite this