Amazing technological breakthrough possible @S-Logix

Office Address

  • 2nd Floor, #7a, High School Road, Secretariat Colony Ambattur, Chennai-600053 (Landmark: SRM School) Tamil Nadu, India
  • +91- 81240 01111

Social List

Q-bert:Hessian based ultra low precision quantization of bert - 2020

Q-Bert:Hessian Based Ultra Low Precision Quantization Of Bert

Research Area:  Machine Learning


Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use Hessian-based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most 2.3% performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to 13× compression of the model parameters, and up to 4× compression of the embedding table as well as activations. Among all tasks, we observed the highest performance loss for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as well as visualization, we show that this is related to the fact that current training/fine-tuning strategy of BERT does not converge for SQuAD.


Author(s) Name:  Sheng Shen, Zhen Dong , Jiayu Ye, Linjian Ma, Zhewei Yao , Amir Gholami , Michael W. Mahoney,Kurt Keutzer

Journal name:  

Conferrence name:  Proceedings of the AAAI Conference on Artificial Intelligence

Publisher name:  Association for the Advancement of Artificial Intelligence

DOI:  10.1609/aaai.v34i05.6409

Volume Information: