Research Area: Machine Learning
This paper summarizes the evolution and application of attention mechanism in natural language processing.We searched attention with the title or topic fields of WoS, ACM Digital Library, arXiv and CNKI from January 2015 to October 2019. Then, we manually screened the topic literature in the field of natural language processing, and obtained 68 related papers.We first summarized the general attention mechanism, and sorted out its derivations. Second, we thoroughly reviewed their applications in natural language processing tasks.The application of attention mechanism in natural language processing focused on sequence labeling, text classification, reasoning and generative tasks. There were adaptation rules between tasks and the various attention mechanisms.Some adaptations between the mechanisms and the tasks were obtained from the overall performance of the model. More research is needed to examine the performance of different mechanisms.The study of attention mechanism has effectively promoted the development of natural language processing. However, the mechanism of action is not yet clear. Future research should focus on making attention mechanism closer to those of the human beings.
Keywords:
Author(s) Name: Shi Lei,Wang Yi,Cheng Ying,Wei Ruibin
Journal name: Data Analysis and Knowledge Discovery
Conferrence name:
Publisher name: magtech
DOI: 10.11925/infotech.2096-3467.2019.1317
Volume Information: 2020, Vol. 4 Issue (5): 1-14
