Amazing technological breakthrough possible @S-Logix

Office Address

  • #5, First Floor, 4th Street Dr. Subbarayan Nagar Kodambakkam, Chennai-600 024 Landmark : Samiyar Madam
  • +91- 81240 01111

Social List

Review of Attention Mechanism in Natural Language Processing - 2020

Review Of Attention Mechanism In Natural Language Processing

Research Area:  Machine Learning


This paper summarizes the evolution and application of attention mechanism in natural language processing.We searched attention with the title or topic fields of WoS, ACM Digital Library, arXiv and CNKI from January 2015 to October 2019. Then, we manually screened the topic literature in the field of natural language processing, and obtained 68 related papers.We first summarized the general attention mechanism, and sorted out its derivations. Second, we thoroughly reviewed their applications in natural language processing tasks.The application of attention mechanism in natural language processing focused on sequence labeling, text classification, reasoning and generative tasks. There were adaptation rules between tasks and the various attention mechanisms.Some adaptations between the mechanisms and the tasks were obtained from the overall performance of the model. More research is needed to examine the performance of different mechanisms.The study of attention mechanism has effectively promoted the development of natural language processing. However, the mechanism of action is not yet clear. Future research should focus on making attention mechanism closer to those of the human beings.


Author(s) Name:   Shi Lei,Wang Yi,Cheng Ying,Wei Ruibin

Journal name:  Data Analysis and Knowledge Discovery

Conferrence name:  

Publisher name:  magtech

DOI:  10.11925/infotech.2096-3467.2019.1317

Volume Information:  2020, Vol. 4 Issue (5): 1-14