INTERNATIONAL SCIENTIFIC CONFERENCE

17-18 November 2017, GABROVO

LONG SHORT TERM MEMORY IN MLP PAIR

Todor Balabanov
Institute of Information and Communication Technologies - Bulgarian Academy of Sciences
akad. Georgi Bonchev Str., block. 2, office 514, 1113 Sofia, Bulgaria
+359898237103, todorb@iinf.bas.bg

Abstract
This study focuses on the capabilities to introduce long short-term memory in classical multilayer perceptron.
In MLP there are not recurrent links and because of that it has no memory capabilities. The idea of the solution
proposed is to improve MLP time series forecasting possibilities by introduction of modified long short-term memory
model as pair of artificial neural networks.

Keywords: artificial neural networks, time series forecasting, backpropagation, machine learning.

INTRODUCTION In the classical MLP links between the

Artificial neural networks (ANNs) as
math models with the goal to describe natural
neural systems was introduced in the middle of
20th century. In their organization ANNs are
oriented weighted graph. ANNs are basically
used in two modes - training and usage. The
training mode is determined by the search for
such weights values [7,9,10] that the ANN
learns the information from the input to output
information as well as possible. In the most
used case of ANNs the MLP the signals are
transmitted from the input to the output. The
input information in each neuron is coming
from the outputs of the previous neurons. The
external impact obtained by this way is then
processed in a summation function. The most
used summation function is the linear function
(neurons' output signals are multiplied by the
weights of the connections). This is the general
case but functions different than the linear
function also can be applied. The output of the
summation function is then transferred to a
function for normalization which determines
the activation level of the neuron. In the
literature a set of activation functions are
proposed. Some of them are well established
when others not so well [11].
neurons are only from the input to the output.
It is not possible for the signals to travel from
the output to the input. Backward only the
neurons' error is propagated in the backward
pass. In ANN with such topology it is not
possible some past information to remain
circulating in the network. Because of this
MLPs are not so suitable for forecasting tasks
and generally in the field of the financial time
series forecasting. The lack of memory in MLP
is overcome by the introduction of recurrent
links as it was done in Jordan and Elman
networks.
MLP PAIR
If we put two MLPs in pair we can
achieve network memory (Fig. 1). As input of
MLP1 past values of the time series are
provided with combination of MPL2 output.
At the output of MLP1 the forecast is
expected. MLP1 output is also used as input
for MLP2. After that signals are propagated in
MLP2. By this way MLP2 is used as network
memory. In this implementation both MLPs
are trained with backpropagation of the error.
The values of the time series are normalized
before to be supplied in MLP1. The time series

International Scientific Conference UNITECH 2017 Gabrovo

itself is conditionally divided on past frame disadvantaged in its derivative is that it has
(lag) and future frame (lead). two break points.

(2)

Fig. 1. MLP Pair.

ACTIVATION FUNCTION
Fig. 3. Fading sin derivative.
ALTERNATIVES
With less periodic influence is the
It is possible to use other function second proposal of exponent regulated sin in
instead of the most used activation functions. Fig. 4 with formula Eq. 3
In this research to different activation
functions are proposed (Fig. 2-5).
(3)

(1)
The derivative of exponent regulated
sine is presented in Fig. 4.
The fading sin function is presented on
Fig. 2 and its formula with Eq. 1.

Fig. 4. Exponent regulated sin activation function.

Fig. 2. Fading sin activation function.

(4)
Fading sin derivative is presented with
Eq. 2 and it is visualized in Fig. 3. A small

International Scientific Conference UNITECH 2017 Gabrovo

 It can be provided as formula by usage
of the function itself as it is shown in Eq. 5.
(5)
Disadvantage in this derivative is that it
has breakpoint when x is zero.
Fig. 5. Exponent regulated sin derivative.
In both derivatives there is periodic
component Fig. 3 and Fig. 5. Periodic
functions are problematic in gradient descent
methods such as backpropagation of the error.
Fig. 6. Alternative activation function derivative.
An efficient way to escape from these
complications is artificially to replace the
derivatives with much smoother and much
more effective function as proposed in Fig. 6.
Such artificial replacement of derivatives is
International Scientific Conference UNITECH 2017 Gabrovo
possible because of the way in which Encog
ANN programming library is designed.
The usage of fading function gives to
the neurons a behavior of input signals
saturation. If the input signals are too strong
the neurons will stop working. By such barrier
the responsibilities of the neurons in the
hidden layers are much better divided.
EXPERIMENTS AND RESULTS
All the experiments performed and the
results achieved are done with the Encog
program library.
Fig. 7. Single MLP Convergence.
All experiments runs are conducted on
a desktop computer with the MacOS Sierra
10.12.2 operating system, the 2.3 GHz Intel
Core i5 processor and 8 GB of RAM, and the
Encog Core v3.3.0 Java Version, Java 8
Update 112 (64bit).
Fig. 8. MLP Pair Convergence.
The same experiment was performed
with two different networks [6], the first
having no recurrent links (topology 15-11-5),
and the second network was a combination of
two multilayer perceptron (MLP1 with
topology 20-11-5 and MLP2 with topology 5-
6-5). As a time series, publicly available data
on the number of earthquakes per year with
magnitude of 7.0 or greater for the period from
1900 to 1998 year. The paired multilayer
perceptron has slightly better predictive
properties than the three-layer MLP as it is
shown in Fig. 7 and Fig. 8. In terms of training
time, the pair performs slightly worse than the
single MLP, which is shown in Fig. 9 and Fig.
10.
Fig. 9. Single MLP Training Epochs.
The proposed model of MLP pair leads to
results with improved training time and
forecasting accuracy.
Fig. 10. MLP Pair Training Epochs.
CONCLUSION
MLP pair shows promising results when
it is compared with single MLP in the problem of
financial time series forecasting. Training of the
MLP pair is more time consuming, but it is
International Scientific Conference UNITECH 2017 Gabrovo
logical because the network consists of more
neurons and much more connections than the
single MLP. The proposed alternatives for
activation functions are leading to better
distribution of the information in the hidden
layers.
As further development it will be
interesting this kind of training to be implemented
as parallel or distributed computing as it is
proposed in [3,5,8]. Another direction of further
development is the application in big data
problems as described in [12]. In this study
backpropagation of error was used, but
evolutionary or population based training
heuristics can be applied as proposed in [1,2].
ACKNOWLEDGMENTS
This work was supported by a private
funding of Velbazhd Software LLC.
REFERENCE
[1] Balabanov, T., Zankinski, I., Barova, M.:
Strategy for Individuals Distribution by
Incident Nodes Participation in Star Topology
of Distributed Evolutionary Algorithms.
Cybernetics and Information Technologies,
Sofia, Bulgaria, vol. 16, no. 1, 80--88 (2016).
[2] Balabanov, T., Zankinski, I., Dobrinkova, D.:
Time Series Prediction by Artificial Neural
Networks and Differential Evolution in
Distributed Environment. International
Conference on Large-Scale Scientific
Computing, Sozopol, Bulgaria, Lecture Notes
in Computer Science, vol. 7116, no. 1, 198--205
(2011).
[3] Balabanov, T.: Heuristic Forecasting
Approaches in Distributed Environment (in
Bulgarian). Proceedings of Anniversary
Scientific Conference 40 Years Department of
Industrial Automation, UCTM, Sofia, Bulgaria,
163--166 (2011).
[4] Balabanov, T.: Distributed evolutional model
for music composition by human-computer
interaction. Proceedings of International
Scientific Conference UniTech15, University
publishing house V. Aprilov, Gabrovo, Bulgaria,
vol. 2, 389--392 (2015).
[5] Balabanov, T.: Avoiding Local Optimums in
Distributed Population based Heuristic
Algorithms (in Bulgarian). Proceedings of
 XXIII International Symposium Management
of energy, industrial and environmental
systems, John Atanasoff Union of Automation
and Informatics, Sofia, Bulgaria, 83--86 (2015).
[6] Balabanov, T.: Long Short Term Memory in
MLP Pair with Encog,
https://github.com/VelbazhdSoftwareLLC/Long
-Short-Term-Memory-in-MLP-Pair-with-Encog
Sofia, Bulgaria (2017).
[7] Keremedchiev, D., Barova, M., Tomov, P.:
Mobile application as distributed computing
system for artificial neural networks training
used in perfect information games, Proceedings
of 16th International scientific conference
UNITECH16, Gabrovo, Bulgaria, vol. 2, 389--
393, (2016).
[8] Tashev T., Monov V., Tasheva R.: Load
optimization in a grid structure for parallel
simulations of the throughput of a packet switch
node. Journal Information Technology and
Control, Volume 12, Issue 2, ISSN (Online)
1312-2622, DOI: 10.1515/itc-2015-0013, 23--
30, (2014).
[9] Tomov, P., Monov, V., Artificial Neural
Networks and Differential Evolution Used for
International Scientific Conference UNITECH 2017 Gabrovo
Time Series Forecasting in Distributed
Environment, Proceedings of International
conference Automatics and Informatics, ISSN
1313-1850, 129--132, Sofia, Bulgaria, (2016).
[10] Zankinski, I., Stoilov, T.: Effect of the Neuron
Permutation Problem on Training Artificial
Neural Networks with Genetic Algorithms in
Distributed Computing, Proceedings of XXIV
International Symposium Management of
Energy, Industrial and Environmental Systems,
ISSN 1313-2237, Bankya, Bulgaria, 53--55,
(2016).
[11] Zankinski, I., Tomov, P., Balabanov, T.:
Alternative Activation Function Derivative in
Artificial Neural Networks, Proceedings of
XXV International Symposium Management of
Energy, Industrial and Environmental Systems,
ISSN 1313-2237, Bankya, Bulgaria, 79--81,
(2017).
[12] , ., , ., , .:
.
, 193--198, ISSN:1314-1937, (2016).