approach did not show improvements for GRU and showed small improvement, at around +2% for LSTM.

Figure 6: RNN accuracy within 100 epochs

Figure 7: LSTM loss within 100 epochs

Figure 8: LSTM accuracy within 100 epochs

Figure 9: GRU loss within 100 epochs

Figure 10: GRU accuracy within 100 epochs

Figure 11: Prediction accuracy for different ranges:

from 1 to 15 days

Architecture

Log loss

Accuracy

RNN

0.725 0.625

LSTM

0.629 0.665

GRU

0.629 0.67

LSTM + Dropout 0.681

GRU + Dropout Figure 12: Losses and accuracy after training different architectures

INTERNATIONAL JOURNAL OF MATHEMATICS AND COMPUTERS IN SIMULATION

Volume 11, 2017

ISSN: 1998-0159 11

Architecture

Time (sec)

RNN

49

LSTM

189

GRU

216

Figure 13: Time for training 100 epochs are supposed to be totally random, while, in reality, next days trend could be influenced by some earlier fluctuations. In figure 11, results for 1-15 days of prediction have been reported. The accuracy for next day is still not bad, about but it is much better for 5 days horizon, about 72%, with a small jump to 71% of accuracy on days horizon prediction.

Nevertheless it is worth to mention that the latter is a test which is dataset dependent result, and it should vary for different assets.

Generally, this plot is showing, that the better horizon of prediction is 1 to 5 days, which totally makes sense.

B- Hidden Dynamics Analysis

To discover hidden behavior of RRNs we have provided a visualization of activations after first recurrent layer. This idea is inspired by the LSTMVis tool, see [18], that can be used to understand hidden state dynamics in LSTM. We have the hypothesis that RNNs can early detect trend of time series movement because

Figure 14: Activation examples on random time windows of given task to solve. On the figure 14, the black line corresponds to some input time window, and the blue dashed line shows the activations. As we can see, RNNs can discover some useful patterns. In particular, if activation in some moment goes to −0.5, this could be a signal that price will go up in next couple of days, and vice-versa. Namely, if activation goes to 0.5 it could mean that price is going to fall in the closest future. The same holds for activations of second recurrent layer. Such an approach can be used as a powerful indicator also in more complex

financial applications or, for example, as an algorithmical trading signals. We intend to deeply go through this latter topic , particularly from the machine learning point of view.

VII.

C

ONCLUSIONS

In the present paper we have applied some of the most promising RRNs architectures, namely basic RNNs, LSTMs and GRUs, to stock market price movement forecasting. We have compared results trained on a daily basis for GOOGL

stock prices with respect to the last five years, showing that the

LSTMs approach is able to

provide a high enough accuracy,

up to 72% for 5 days prediction horizon. This means that it can be successfully applied in practice. We also show that to avoid overfitting to the dataset, RNNs have to be trained for large number of epochs, choosing final weights carefully with early stopping. Furthermore we have also performed the analysis of RNNs hidden dynamics. The latter allows us to prove that NNs aren’t not black box learning models with non interpretable inner structure. In fact, visualizations of activations clearly show,

that NNs can learn useful patterns. In particular, they can detect short term ups and downs in time series. These activations can be used as indicators for further time series analysis. INTERNATIONAL JOURNAL OF MATHEMATICS AND COMPUTERS IN SIMULATION

Volume 11, 2017

ISSN: 1998-0159 12

In future research we plan to apply more bleeding-edge deep learning approaches to financial time series. We will mainly focus on the explanation of how neural attention mechanism,

bidirectional RNNs and more complex structures that were successfully applied in NLP problems, can help in learning important parts of time series of interest. We also plan to perform more in-depth research of hidden behaviour of RNNs to use inner activations as technical indicators or feature selectors.

R

EFERENCES

[1]

Y. Bengio, D. Bahdanau, H. Schwenk et al, Learning Phrase Representations using RNN Encoder–Decoder for Statistical

Machine Translation, C. Benazzoli and L. Di Persio Default contagion in financial networks, International Journal of Mathematics and Computers in Simulation Volume 10, 2016, Pages L. Di Persio and M. Frigo, Maximum likelihood approach to markov switching models, WSEAS Transactions on Business and Economics Volume 12, Pages 239-242, L. Di Persio and M. Frigo, Gibbs sampling approach to regime switching analysis of financial time series, Journal of Computational and Applied Mathematics Volume 300, Pages 43-55, L. Di Persio and O. Honchar, Artificial neural networks architectures for stock price prediction Comparisons and applications, International Journal of Circuits, Systems and Signal Processing, Volume 10, Pages 403-413, X. Ding, Y. Zhang et al, Deep Learning for Event-Driven Stock Prediction,

*Proceedings of the Twenty-Fourth **International Joint Conference on Artificial Intelligence **(IJCAI)*, 2015

[7]

N.Srivastava, G. Hinton et al, Dropout A Simple Way to Prevent Neural Networks from Overfitting

*Journal of **Machine Learning Research 15*, A. Saxe, J.L. McClelland, S. Ganguli, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2014

[9]

Diederik Kingma, Jimmy Ba Adam A Method for Stochastic Optimization, arXiv:1412.6980, 2015

[10]

Jie Wang, Jun Wang Forecasting energy market indices with recurrent neural networks Case study of crude oil price fluctuations, 2016

[11]

S.Hochreiter, J.Schmidhuber Long-short term memory,

*Neural Computation*, K. Greff, R. K. Srivastava et al, LSTM: A Search Space Odyssey, 2015

[13]

Girosi, Federico, M. Jones, Regularization Theory and Neural Networks Architectures,

*Neural Computation **p.219–269* , F. Butaru, Q. Chen et al, Risk and Risk Management in the credit Card Industry , E. Hurwitz, T. Marwala, State of the Art Review for Applying Computational Intelligence and Machine Learning Techniques to Portfolio Optimisation

*preprint*, M. Naeini, H. Taremian, Stock Market Value Prediction

Using Neural Networks,

*International Conference on **Computer Information Systems and Industrial **Management Applications (CISIM)*, Xavier Glorot and Yoshua Bengio, Understanding the difficulty of training deep feedforward neural networks.

*International conference on artificial intelligence and **statistics*, H. Strobelt, S. Gehrmann, B. Huber et al, Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks,

2016

INTERNATIONAL JOURNAL OF MATHEMATICS AND COMPUTERS IN SIMULATION

Volume 11, 2017

ISSN: 1998-0159 13