Amazing technological breakthrough possible @S-Logix

Office Address

  • #5, First Floor, 4th Street Dr. Subbarayan Nagar Kodambakkam, Chennai-600 024 Landmark : Samiyar Madam
  • +91- 81240 01111

Social List

Revisiting Simple Neural Probabilistic Language Models - 2021

Research Area:  Machine Learning


Recent progress in language modeling has been driven not only by advances in neural architectures, but also through hardware and optimization improvements. In this paper, we revisit the neural probabilistic language model (NPLM) of~citet{Bengio2003ANP}, which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. When scaled up to modern hardware, this model (despite its many limitations) performs much better than expected on word-level language model benchmarks. Our analysis reveals that the NPLM achieves lower perplexity than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. Inspired by this result, we modify the Transformer by replacing its first self-attention layer with the NPLM-s local concatenation layer, which results in small but consistent perplexity decreases across three word-level language modeling datasets.

Author(s) Name:  Simeng Sun, Mohit Iyyer

Journal name:  Computer Science

Conferrence name:  

Publisher name:  arXiv:2104.03474

DOI:  10.48550/arXiv.2104.03474

Volume Information: