Text generation is a widely used sub-field of Natural Language Processing (NLP) and generates informative text similar to human written text. It automatically generates text by utilizing the knowledge of computational linguistics and artificial intelligence. Word embeddings are the vector representation that is dominantly used in text generation for words with similar meanings that possess similar representation. Contextual word embedding for text generation allocates each word a representation based on its context by utilizing words across various contexts and encoding knowledge that transfers over languages. Contextual word embedding outperforms traditional word representation by capturing many syntactic and semantic properties under different linguistic contexts. Deep learning approaches are efficient in NLP tasks and especially achieve great progress in text generation. Deep neural networks own the ability to generate smooth, topic-consistent, and even personalized text. Deep neural network-based contextual word embedding for text generation learns the distributed representation efficiently and generates meaningful text.

  • Learning effectual representations of text words is a long-standing research focus in Natural Language Processing (NLP).

  • Word embeddings which is a representational basis for downstream NLP tasks, characterized by deep learning, have captivated extensive attention and also made many NLP problems a lot trouble-free.

  • In recent years, text generation based on deep learning has made great progress, owing to the fast, accurate locate and use of effective information from huge text data has turned into an urgent research problem in the field of NLP.

  • Generative automatic summarization is a significant part of text generation, and its research hotspots have moderately shifted from extractive to generative.

  • Fine-tuning pre-trained language models that came to be trained on large-scale corpora have constantly evolved state of the art for many NLP tasks.

  • Owing to the lag in the evaluation metrics of pre-trained word embedding models, a novel pre-training scheme (BLEURT) with learned robust metrics that utilizes millions of synthetic examples to help the model generalize for text generation.

  • Non-autoregressive generation is another emerging concept in text generation with pre-trained language models due to its quick inference speed.

  • The immense increase in available data and computational power, also the novel improvements in deep learning, imparts more new potentialities for researchers and practitioners to investigate unaccustomed developments for text generation.