Amazing technological breakthrough possible @S-Logix

Office Address

  • #5, First Floor, 4th Street Dr. Subbarayan Nagar Kodambakkam, Chennai-600 024 Landmark : Samiyar Madam
  • +91- 81240 01111

Social List

Domain-specific language model pretraining for biomedical natural language processing - 2021

Domain-Specific Language Model Pretraining For Biomedical Natural Language Processing

Research Area:  Machine Learning


Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition.


Author(s) Name:  Yu Gu , Robert Tinn,Hao Cheng ,Michael Lucas ,Naoto Usuyama,Xiaodong Liu,Tristan Naumann,Jianfeng Gao, Hoifung Poon

Journal name:  ACM Transactions on Computing for Healthcare

Conferrence name:  

Publisher name:  ACM

DOI:  10.1145/3458754

Volume Information:  Volume 3,Issue 1,January 2022 ,Article No.: 2,pp 1–23