Recurrent dropout without memory loss

Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth

Abstract

This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to forward connections of feed-forward architectures or RNNs, we propose to drop neurons directly in recurrent connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on three NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.

Original languageEnglish
Title of host publicationProceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Number of pages10
PublisherAssociation for Computational Linguistics (ACL)
Publication date12.2016
Pages1757-1766
ISBN (Print)978-487974702-0
Publication statusPublished - 12.2016
Event26th International Conference on Computational Linguistics - Osaka, Japan
Duration: 11.12.201616.12.2016
Conference number: 136517

Fingerprint

Dive into the research topics of 'Recurrent dropout without memory loss'. Together they form a unique fingerprint.

Cite this