On the importance of sluggish state memory for learning long term dependency

Tepper, JA ORCID: https://orcid.org/0000-0001-7339-0132, Shertil, MS and Powell, HM, 2016. On the importance of sluggish state memory for learning long term dependency. Knowledge-Based Systems, 96, pp. 104-114. ISSN 0950-7051

Preview

Text PubSub4260_Tepper.pdf - Post-print Download (288kB) | Preview

Abstract

The vanishing gradients problem inherent in Simple Recurrent Networks (SRN) trained with back-propagation, has led to a significant shift towards the use of Long Short-term Memory (LSTM) and Echo State Networks (ESN), which overcome this problem through either second order error-carousel schemes or different learning algorithms respectively. This paper re-opens the case for SRN-based approaches, by considering a variant, the Multi-recurrent Network (MRN). We show that memory units embedded within its architecture can ameliorate against the vanishing gradient problem, by providing variable sensitivity to recent and more historic information through layer- and self-recurrent links with varied weights, to form a so-called sluggish state-based memory. We demonstrate that an MRN, optimised with noise injection, is able to learn the long term dependency within a complex grammar induction task, significantly outperforming the SRN, NARX and ESN. Analysis of the internal representations of the networks, reveals that sluggish state-based representations of the MRN are best able to latch on to critical temporal dependencies spanning variable time delays, to maintain distinct and stable representations of all underlying grammar states. Surprisingly, the ESN was unable to fully learn the dependency problem, suggesting the major shift towards this class of models may be premature.

Item Type:	Journal article
Publication Title:	Knowledge-Based Systems
Creators:	Tepper, J.A., Shertil, M.S. and Powell, H.M.
Publisher:	Elsevier
Date:	15 March 2016
Volume:	96
ISSN:	0950-7051
Identifiers:	Number Type 10.1016/j.knosys.2015.12.024 DOI
Divisions:	Schools > School of Science and Technology
Record created by:	Linda Sullivan
Date Added:	04 Feb 2016 11:36
Last Modified:	04 Feb 2022 12:21
URI:	https://irep.ntu.ac.uk/id/eprint/26882

Actions (login required)

Edit View

Statistics

Views

Downloads