Research Area:  Machine Learning
Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added domain, typically as many as the original network. We propose a method called Deep Adaptation Modules (DAM) that constrains newly learned filters to be linear combinations of existing ones. DAMs precisely preserve performance on the original domain, require a fraction (typically 13 percent, dependent on network architecture) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance. When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3 percent of the original with negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.
Keywords:  
Author(s) Name:   Amir Rosenfeld; John K. Tsotsos
Journal name:   IEEE Transactions on Pattern Analysis and Machine Intelligence
Conferrence name:  
Publisher name:  IEEE
DOI:  10.1109/TPAMI.2018.2884462
Volume Information:  ( Volume: 42, Issue: 3, March 1 2020)
Paper Link:   https://ieeexplore.ieee.org/abstract/document/8554156