Research Area:  Machine Learning

Abstract:

Recently pre-trained models have achieved state-of-the-art results in various language understanding tasks. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring information, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entities, semantic closeness and discourse relations. In order to extract the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which incrementally builds pre-training tasks and then learn pre-trained models on these constructed tasks via continual multi-task learning. Based on this framework, we construct several tasks and train the ERNIE 2.0 model to capture lexical, syntactic and semantic aspects of information in the training data. Experimental results demonstrate that ERNIE 2.0 model outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several similar tasks in Chinese.

Author(s) Name:  Yu Sun ,Shuohuan Wang , Yukun Li,Shikun Feng ,Hao Tian, Hua Wu, Haifeng Wang

Publisher name:  Association for the Advancement of Artificial Intelligence

DOI:  10.1609/aaai.v34i05.6428

Volume Information:  

Paper Link: https://ojs.aaai.org/index.php/AAAI/article/view/6428