FreezeNet: Full Performance by Reduced Storage Costs

Paul Wimmer, Jens Mehnert, Alexandru Paul Condurache

Abstract

Pruning generates sparse networks by setting parameters to zero. In this work we improve one-shot pruning methods, applied before training, without adding any additional storage costs while preserving the sparse gradient computations. The main difference to pruning is that we do not sparsify the network’s weights but learn just a few key parame- ters and keep the other ones fixed at their random initialized value. This mechanism is called freezing the parameters. Those frozen weights can be stored efficiently with a single 32bit random seed number. The pa- rameters to be frozen are determined one-shot by a single for- and back- ward pass applied before training starts. We call the introduced method FreezeNet. In our experiments we show that FreezeNets achieve good re- sults, especially for extreme freezing rates. Freezing weights preserves the gradient flow throughout the network and consequently, FreezeNets train better and have an increased capacity compared to their pruned counter- parts. On the classification tasks MNIST and CIFAR-10/100 we outper- form SNIP, in this setting the best reported one-shot pruning method, applied before training. On MNIST, FreezeNet achieves 99.2% perfor- mance of the baseline LeNet-5-Caffe architecture, while compressing the number of trained and stored parameters by a factor of ×157.
OriginalspracheEnglisch
Seiten1-17
Seitenumfang17
PublikationsstatusVeröffentlicht - 01.12.2020
Veranstaltung15th Asian Conference on Computer Vision - Virtual , Kyōto, Japan
Dauer: 30.11.202004.12.2020
https://accv2020.github.io

Tagung, Konferenz, Kongress

Tagung, Konferenz, Kongress15th Asian Conference on Computer Vision
KurztitelACCV 2020
Land/GebietJapan
OrtKyōto
Zeitraum30.11.2004.12.20
Internetadresse

Fingerprint

Untersuchen Sie die Forschungsthemen von „FreezeNet: Full Performance by Reduced Storage Costs“. Zusammen bilden sie einen einzigartigen Fingerprint.

Zitieren