Abstract
This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to forward connections of feed-forward architectures or RNNs, we propose to drop neurons directly in recurrent connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on three NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.
Original language | English |
---|---|
Title of host publication | Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers |
Number of pages | 10 |
Publisher | Association for Computational Linguistics (ACL) |
Publication date | 12.2016 |
Pages | 1757-1766 |
ISBN (Print) | 978-487974702-0 |
Publication status | Published - 12.2016 |
Event | 26th International Conference on Computational Linguistics - Osaka, Japan Duration: 11.12.2016 → 16.12.2016 Conference number: 136517 |