Generally, word embedding is a method of learned representation for text, where words have the same meaning and a similar representation. Contextualized Word Representations are significantly improving in Natural Language Processing (NLP) tasks and learning the highly transferable and task agnostic properties of language. Contextualized word representation builds the representation for each word based on the context that uses words across different contexts and encodes the knowledge transfer across languages.

Discovering a single representation for each word is an important problem in static word embeddings. Contextualized word representation overcomes the issues mentioned above by creating context-sensitive word representation. Some deep neural language models used in downstream NLP tasks, Embeddings from Language Model (ELMo), Contextual Word Vectors (CoVe), Bidirectional Encoder Representations from Transformers(BERT), and Generative Pre-trained Transformer 2(GPT-2). Such models possess successful contextualized word representation.

NLP application tasks for contextualized word representation are question answering, textual entailment, named entity extraction, semantic role labeling, sentiment analysis, sentiment classification, and conference resolution. Future research directions of contextualized word representation are the generation of static word embedding from contextualized word representation, multi-task learning approaches, noise combination models, and robustness models to protect against the vulnerability.