A dynamic neural network is a hot research topic in deep learning that can scale up neural networks with sub-linear increases in computation and time by dynamically adjusting their computational path based on the input. Dynamic neural networks become a promising solution due to adapting their structures or parameters to different inputs, capable of making data-dependent decisions during inference to transform their architectures or parameters, allowing both model pre-training with trillions of parameters, and some notable advantages in terms of accuracy, computational efficiency, adaptiveness, compatibility, generality, and interpretability.

   Instance-wise dynamic networks, spatial-wise dynamic networks, and temporal-wise dynamic networks are the types of dynamic neural networks. Based on the input data modality, the typical applications of dynamic DNNs are For image recognition - dynamic CNNs are used. For text data - temporal-wise dynamic RNNs are used for video-related tasks, and the three types of dynamic inference (instance-wise, spatial-wise, and temporal-wise) can be implemented simultaneously.

   Even though many advances have been made in the research of dynamic deep neural networks, there are some significant research challenges in dynamic networks, such as architecture design, decision-making scheme, data parallelism, robustness against adversarial attack, and optimization technique.