Research Area:  Machine Learning

Abstract:

Attention mechanisms are now a mainstay architecture in neural networks and improve the performance of biomedical text classification tasks. In particular, models that perform automated medical encoding of clinical documents make extensive use of the label-wise attention mechanism. A label-wise attention mechanism increases a models discriminatory ability by using label specific reference information. This information can either be implicitly learned during training or explicitly provided through embedded textual code descriptions or information on the code hierarchy; however, contemporary studies arbitrarily select the type of label specific reference information. To address this shortcoming, we evaluated label wise attention initialized with either implicit or explicit label-specific reference information against two common baseline method target attention and text encoder architecture specific methods to generate document embeddings across four text encoder architectures a convolutional neural network, two recurrent neural networks, and a transformer. We also present an extension of label wise attention that can embed the information on the code hierarchy. We performed our experiments on the MIMIC III dataset, which is a standard dataset in the clinical text classification domain. Our experiments showed that using pretrained reference information and the hierarchical design helped improve classification performance. These performance improvements had less impact on larger datasets and label spaces across all text encoder architectures. In our analysis, we used an attention mechanisms energy scores to explain the perceived differences in performance and interpretability between the text encoder architectures and types of label attention.

Keywords:  
Codes
Transformers
Task Analysis
Recurrent neural Networks
Text Categorization

Author(s) Name:  Christoph,Shang Gao,Drahomira,Herrmonnova,Heidi

Publisher name:  IEEE

DOI:  10.1109/JBHI.2024.3355951

Volume Information:  volume 28,pages 2247-2258(2024)

Paper Link:   https://ieeexplore.ieee.org/document/10409522