How deep should be the depth of convolutional neural networks: a backyard dog case study
Abstract
The work concerns the problem of reducing a pretrained deep neuronal network to a smaller network, with just few layers, whilst retaining the network's functionality on a given task The proposed approach is motivated by the observation that the aim to deliver the highest accuracy possible in the broadest range of operational conditions, which many deep neural networks models strive to achieve, may not necessarily be always needed, desired, or even achievable due to the lack of data or technical constraints. In relation to the face recognition problem, we formulated an example of such a usecase, the `backyard dog' problem. The `backyard dog', implemented by a lean network, should correctly identify members from a limited group of individuals, a `family', and should distinguish between them. At the same time, the network must produce an alarm to an image of an individual who is not in a member of the family. To produce such a network, we propose a shallowing algorithm. The algorithm takes an existing deep learning model on its input and outputs a shallowed version of it. The algorithm is noniterative and is based on the Advanced Supervised Principal Component Analysis. Performance of the algorithm is assessed in exhaustive numerical experiments. In the above usecase, the `backyard dog' problem, the method is capable of drastically reducing the depth of deep learning neural networks, albeit at the cost of mild performance deterioration. We developed a simple noniterative method for shallowing down pretrained deep networks. The method is generic in the sense that it applies to a broad class of feedforward networks, and is based on the Advanced Supervise Principal Component Analysis. The method enables generation of families of smallersize shallower specialized networks tuned for specific operational conditions and tasks from a single larger and more universal legacy network.
 Publication:

arXiv eprints
 Pub Date:
 May 2018
 arXiv:
 arXiv:1805.01516
 Bibcode:
 2018arXiv180501516G
 Keywords:

 Computer Science  Neural and Evolutionary Computing;
 Computer Science  Machine Learning;
 Statistics  Machine Learning
 EPrint:
 Edited and extended version with more detailed description of numerical experiments