Generating text from structured and unstructured data with the knowledge of the human language is referred to as text generation or natural language generation. Text generation is a challenging task and applied to various tasks in Natural Language Processing (NLP). One of the emerging techniques for text generation is deep learning models. The deep learning model utilizes deep neural networks to learn the vector representations and generates informative text.
The availability of large-scale datasets and the need for a huge number of parameters to learn are considered a drawback of the deep learning model. The pretrained language model is developed to convert a large amount of linguistic knowledge from the corpus and produce universal representations of language. The significance of pretrained models is better model initialization, learning universal language representation, avoiding training a new model from scratch, and regularization to avoid over-fitting. Text generation is guided under pretrained deep learning models to achieve better performance.
• Text generation strengthens knowledge in computational linguistics and artificial intelligence for automatic natural language text generation to satisfy certain communicative requirements.
• With the help of a pre-trained deep learning model, more accurate text generation is effortlessly performed.
• Pre-training of denoising auto-encoders learns more suitable representations for text generation. Lately, it has been equipped with transformer and pointer-generator networks to speed up convergence.
• The popular language representation model, Bidirectional Encoder Representations from Transformers (BERT) developed to pre-trained deep bidirectional representations from the unlabeled text.
• A novel improvement on a large unlabeled corpus that collaboratively pre-trains an autoencoding and autoregressive language model, specially designed for generating new text based on context.
• As the prevalent trend of pre-trained models, the mainstay of such models moved from deep learning models to transformers or transformer-based pre-trained models.
