See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/228571349

The use of NARX neural networks to predict chaotic time series

Article in WSEAS TRANSACTIONS ON COMPUTER RESEARCH · March 2008

CITATIONS READS
387 15,065

1 author:

Eugen Diaconescu
University of Pitesti
46 PUBLICATIONS 447 CITATIONS

SEE PROFILE

All content following this page was uploaded by Eugen Diaconescu on 15 May 2014.

The user has requested enhancement of the downloaded file.

WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

The use of NARX Neural Networks to predict Chaotic Time Series

EUGEN DIACONESCU, PhD
Electronics, Communications and Computer Science Faculty
University of Pitesti
Targu din Vale, Nr. 1
ROMANIA
eugend@upit.ro

Abstract:
The prediction of chaotic time series with neural networks is a traditional practical problem of dynamic systems. This
paper is not intended for proposing a new model or a new methodology, but to study carefully and thoroughly several
aspects of a model on which there are no enough communicated experimental data, as well as to derive conclusions
that would be of interest. The recurrent neural networks (RNN) models are not only important for the forecasting of
time series but also generally for the control of the dynamical system. A RNN with a sufficiently large number of
neurons is a nonlinear autoregressive and moving average (NARMA) model, with “moving average” referring to the
inputs. The prediction can be assimilated to identification of dynamic process. An architectural approach of RNN with
embedded memory, “Nonlinear Autoregressive model process with eXogenous input” (NARX), showing promising
qualities for dynamic system applications, is analyzed in this paper. The performances of the NARX model are verified
for several types of chaotic or fractal time series applied as input for neural network, in relation with the number of
neurons, the training algorithms and the dimensions of his embedded memory. In addition, this work has attempted to
identify a way to use the classic statistical methodologies (R/S Rescaled Range analysis and Hurst exponent) to obtain
new methods of improving the process efficiency of the prediction chaotic time series with NARX.

Key-Words: - Chaotic Time Series, Hurst Exponent, Prediction, Recurrent Neural Networks, NARX Model

1 Introduction Viewed this way, prediction becomes a problem of

Many processes in domains as physics, technique, function approximation, where the purpose of the
biology, and economics are described by time series. method is to approximate the continuous function Γ
In formal terms, a time series is a sequence of as closely as possible. Therefore, in the case of
vectors, depending on time t: function approximation or regression problems,
many methods from that domain can be applied
y(t), t = 0, 1, 2, … (1) here.
The applications of type prediction or forecasting of Usually, the evaluation of prediction performance is
time series is largely presented in the literature about done by computing an error measure E over a
time series [10][11][12][13]. The prediction of number of time series elements, such as a validation
futures values of vector y is helpful or strict required or test set:
to decide upon a strategy of control or to optimize N
the activity, production, selection, etc. Formally, the
problem of prediction can be formulated as finding a
E= ∑ ( yˆ (t − k ), y(t − k ))
k =0
(3)

function Γ so as to obtain an estimate yˆ (t + D) of the

E is a function measuring the error between the
vector y at time t+D (D = 1, 2, …), given the values estimated (predicted) and actual sequence element.
of y up to time t, plus a number of additional time- Typically, a distance measure (Euclidean or other) is
independent variables (exogenous features) ui: used, but depending on the problem, any function
ŷ (t+D) = can be used (e.g. a function computing the cost
(2) resulting from an incorrect prediction of y(t+D))[2].
=Γ(y(t), … y(t-dy), u(t), …, u(t-du)) The problem of chaotic time series prediction is
where u(t) and y(t) represent the input and output of studied in various disciplines now including
the model at time t, du and dy are the lags of the engineering, medical and econometric applications.
input and output of the system and Γ a nonlinear Chaotic time series are the output of a deterministic
function. Typically, D = 1, meaning one-step ahead, system with positive Liapunov exponent. Therefore,
but can take any value larger than 1 (multi-step unless the initial condition are specified with infinite
ahead) [2]. precision, the time series behavior becomes

ISSN: 1991-8755 182 Issue 3, Volume 3, March 2008

WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

unpredictable, and the prediction of chaotic time functional. Although NN had been shown to be
series is a difficult task. universal approximators, it has found that NN had
From a historical point of view, before the 1980s, difficulty modeling seasonal patterns in time series.
prediction of time series used linear parametric When a time series contains significant seasonality,
autoregressive (AR), moving-average (MA) or the data need to be deseasonalized.
autoregressive moving-average (ARMA) models -The number of samples in time series. Researchers
introduced by Box and Jenkins [11][13]. These have found that increasing observation frequency
models are linear and are not able to cope with does not always help to improve the accuracy of
certain non stationary signals, and signals whose prediction.
mathematical model is not linear. An obvious -Stationarity – the classical techniques for time
drawback is that these algorithms are linear, and are series prediction, require a stationary time series,
not able to cope with certain nonstationary signals while most real time series are not stationary
and signals whose model is chaotic nonlinear. On (stationarity refers to a stochastic process whose
the other hand, neural networks (NN) are powerful mean value, variances and covariances – first and
when applied to problems whose solutions require second order moments do not change in time). After
knowledge which is difficult to specify, but for NNs have been introduced one can use original time
which there is an abundance of examples. series as forecasting targets.
The prediction of chaotic processes implies finding -The problem of long time dependencies - is related
the interdependences between time series to the problem of vanishing gradient or forgetting
components. The dependences are minimal in behavior.
random time series and maximal in a complete Time series prediction is the same as system
deterministic process. But, random and deterministic identification; this paper shows that the dynamics of
are only margin of the large set of chaotic time nonlinear system that produce complex time series
series signals with weak dependences between can be captured in a model system. The model
components on short or long term. A special case is system is an artificial RNN. The main idea of RNN
represented by the fractal time series characterized is providing a weighted feedback connection
by auto similarity or non-periodic cycles. between layers of neurons and adding time
significance to entire artificial NN. Therefore, RNNs
are the most suitable for Time Series Analysis
1.1 Prediction with neural networks
After 1980, there has been resurgence in the field of 1.2 Chaotic time series
time series prediction, when it becomes clear that
this type of prediction is a suitable application for a 1.2.1 A short characterization of some example of
neuronal network predictor. chaotic time series
The NN approach to time series prediction is non- To evaluate the prediction capability of proposed
parametric, in the sense that it is not necessary to algorithms, the best data are the chaotic time series
know any information regarding the process that [14], generated by some linear dynamical systems.
generates the signal. It is shown that the recurrent The degree of irregularity is different, from on type
NN (RNN) with a sufficiently large number of of series to another, depending on the sort of iterated
neurons is a realization of the nonlinear ARMA difference equation, chaotic map or flows. In this
(NARMA) process. [1][12][5]. paper, four time series were been tested: logistic,
Neural Networks (NN) based prediction has been Mackey-Glass, fractal Weirstrass and BET index.
explored from their beginning and development, The logistic time series (4) generate a chaotic map
because of NNs approximation and generalization with extremely short memory length. It is a difficult
property. Many research papers are published in test for prediction algorithm. They do not exhibit
scientific literature and some commercial companies cycles as we see sometimes in system practice.
claim or market the so-called advanced statistical
y ′(t ) = a ⋅ y (t ) ⋅ (1 − y (t )) (4)
programs, using neural networks, for modeling and
prediction. The Mackey-Gloss equation (5) is classified as a
However, some difficulties and limitations remain simpler system generating chaotic flows. This type
nevertheless of current actuality and cause of chaotic time series is a relatively easy mission for
researches for new NN models and learning prediction algorithms. Non-periodical cycles appear
techniques to be conducted[4][5][7][8]. due the delay included in the equation. Chaotic
-Outliers make it difficult for NNs (and other
prediction models) to model the true underlying

ISSN: 1991-8755 183 Issue 3, Volume 3, March 2008

WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

flows are frequently encountered models in real life N

yt (8)
practice. y=∑
a ⋅ y (t − τ ) t =1 N
(5)
y′ = − b ⋅ y (t )
1 + y c (t − τ ) This aspects do not are an objective for this work.
The time series can be used immediately for
The fractal function of Weirstrass (6) is an infinite processing in only few cases. In most cases, it is
sum of a series of sine or cosine waves in which necessary to preprocess the time series to ensure an
amplitude decreases, while the frequency increases optimal outcome of the processing. The most
according to different factors. The Weirstrass common operations are removing known
function used has a limited sum of four terms. systematicies as linear or nonlinear trends
n =3 (6) (constantly rising or descending of the average
∑ (( a
1
y (t ) = n
) cos(b n ⋅ ω ⋅ t )) value), seasonality (periodic patterns due to a
n =0 periodic influencing factor)[2][9][12]. In this work,
the data were only normalized to be included in [-1,
On the financial market, multivariate time series are 1] range.
encountered, for example the market indices BET
(figure 1), BET-C, BET-FI from the Romanian stock
market. These indexes also have, although they are 1.2.3 Stochasticity of time series. Long memory
presented only by their value, other important processes
components, such as the daily transaction volume or Long memory process is a process with a random
number of transactions. component, where a past event has a
decreasing/decaying effect on future events. The
process has some memory of past events, which is
“forgotten” as time moves forward.
The mathematical definition of long memory
process is given in terms of autocorrelation: when a
data set exhibit autocorrelation, a value yt at time ti
is correlated with a value yt+d at time ti+d , where d is
some time increment in the future. In a long memory
process autocorrelation decrease over time and the
Fig. 1 BET index for 2005-03.2008 years decreasing follows a power low [10].
In a long memory process the decrease of the
1.2.2 Preprocessing time series autocorrelation function (ACF) for a time series is a
Usually, the time series y(t) and the input sequence power low:
ui(t) are analyzed in a process named “data ACFTS(k)=Ck -α, (9)
selection” for the prediction optimization, or
computing time minimization. The rule of input where C is a constant, and ACFTS(k) is the
variable selection is that the input variables should autocorrelation function with log k. The Hurst
be as predictive as possible [21]. Selection consists exponent is related to the exponent α in the equation
in partitioning the data series in prediction intervals by:
about contents of measured information with
H = 1 – α/2, (10)
autocorrelation coefficient rk. For time series, the
autocorrelation coefficients rk are given by: The value of the Hurst exponent ranges between 0
N
and 1. A value of 0.5 indicates a random evolution
∑ ( y(t ) − y )( y(t + k ) − y )
t =1
in time (a Brownian time series). In a random
process there is no correlation between any element
Rk = N (7) and a future element.
∑ ( y(t ) − y ))
t =1
2
A Hurst exponent value 0.5 < H < 1 indicates
“persistent behavior” (e.g. a positive
where y(t) is a data value at time step t, k is the lag, autocorrelation). Persistence means that if the curve
and the overall mean is given by: has been increasing for a period, it is expected to
continue for another period. If there is an increase
from step ti-1 to ti, there will probably be an increase

ISSN: 1991-8755 184 Issue 3, Volume 3, March 2008

WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

from ti to ti+1, or a decrease will be followed by a those problems for which the output of a system at
decrease. time instant k depends on network inputs presented
A Hurst exponent of 0 < H < 0.5 shows at times r << k. The researchers have analyzed
antipersistent behavior. After a period of decreases, learning algorithms for systems with long time
a period of increases tends to show up. The dependencies and showed that for gradient-based
antipersistent behavior has a rather high fractal training algorithms, the information about the
dimension, corresponding to a very “noisy” profile- gradient contribution m steps in the past vanishes for
like curve (which highly fills up the plane). This large m. This effect is referred to as the problem of
behavior is sometimes called “mean reversion”[10]. vanishing gradients, which partially explains why
In principle, fractal dimension D and Hurst gradient descent algorithms are not very suitable to
coefficient H are independent of each other because estimate systems and signals with long time
the D is considered a local propriety, and long- dependencies. For instance, common recurrent
memory dependence is a global characteristic [22]. neural networks encounter problems when learning
For self-affine processes (e.g. fractals), the local information with long time dependencies, a problem
properties are reflected in the global ones, and it is in the prediction of nonlinear and no stationary
possible the relationship D + H = n + 1, where the n signals. The vanishing gradients problem makes the
is dimension of self-affine space [22]. The long- learning of long-term dependencies in gradient-
memory dependence (persistence) is linked with the based training algorithms difficult if not virtually
case 0.5 < H < 1 and a feature of the surfaces with impossible in certain cases [1].
low fractal dimensions. The antipersistent processes A state space representation of recurrent NARX
are linked with the surfaces with higher fractal neural networks can be expressed as[12]:
dimensions (rougher) with 0 < H < 0.5.
Φ(u (k ), zi (k )), i = 1, 
z k (k + 1) =   (11)
 zi (k ), i = 2,3,...N ,
2 NARX networks where the output y(k) = zi(k) and zi, i=1,2, … N, are
In this paper, the architectural approach proposed to state variables of recurrent neural network. The
deal with chaotic time series is one based upon recurrent network exhibits forgetting behavior, if:
“Nonlinear Autoregressive models with eXogenous
input (NARX model)”, which are therefore called ∂z i ( k )
lim =0 ∀k , m ∈ K , i ∈ Ο , j ∈ Ι , (12)
NARX recurrent neural networks [1][4][5]. This is a m→∞ ∂z j ( k − m)
powerful class of models which has been
where z is state variable, “I” denotes the set of input
demonstrated that they are well suited for modeling
neurons. “O” denotes the set of output neurons and
nonlinear systems and specially time series. One
K denotes the time index set.
principal application of NARX dynamic neural
Several approaches have been suggested to get
networks is in control systems. Also, is a class
around the problem of vanishing gradient in training
computationally equivalent to Turing Machines [1].
RNNs. Most of them rest on including embedding
Some important qualities about NARX networks
memory in neural networks, whereas several others
with gradient-descending learning gradient
propose improved learning algorithms, such as the
algorithm have been reported: (1) learning is more
extended Kalman filter algorithm, Newton type
effective in NARX networks than in other neural
algorithm, annealing algorithm, etc.
network (the gradient descent is better in NARX)
Embedded memory is particularly significant in
and (2) these networks converge much faster and
recurrent NARX and NARMAX neural networks.
generalize better than other networks [4][5].
This embedded memory can help to speed up
The simulated results show that NARX networks are
propagation of gradient information, and hence help
often much better at discovering long time –
to reduce the effect of vanishing gradient. There are
dependences than conventional recurrent neural
various methods of introducing memory and
networks. An explanation why output delays can
temporal information into neural networks. These
help long-term dependences can be found by
include creating a spatial representation of temporal
considering how gradients are calculated using the
pattern, putting time delays into the neurons or their
back-propagation-through-time (BPTT) algorithm.
connections, employing recurrent connections, using
Recently, several empirical studies have shown that
neurons with activations that sum input over time,
when using gradient-descent learning algorithms, it
etc.
might be difficult to learn simple temporal behavior
with long time dependencies [7][9], in other words

ISSN: 1991-8755 185 Issue 3, Volume 3, March 2008

WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

3 Architecture and learning For the NN models used in this work, with two
levels (level 1 surnamed input layer and level 2 or
3.1 The NARX models output layer), the general prediction equations for
The NARX model for approximation of a function Γ computing the next value of time series y(k+1)
can be implemented in many ways, but the simpler (output) using model in figure 2, the past
seems to be by using a feedforward neural network observation u(k), u(k-1), …, u(k-du) and the past
with the embedded memory (a first tapped delay outputs y(k), y(k-1), …, y(k-dy) as inputs, may be
line), as is shown in figure 2, plus a delayed written in the form:
connexion from the output of the second layer to  N du

input (a second tapped delay line). Making the y(k + 1) = Φ o wb0 +


∑ h=1
who ⋅ Φ h (who + ∑ w u(k − i)) +
i =0
ih

network dependent on du previous sequence

elements is identical to using du input units being fed (13)
with du adjacent sequence elements. This input is
dy 
usually referred to as a time window since it
provides a limited viewed on part of the series. It
+ ∑w
j =0
jh ⋅ y ( k − j )) 

can also be viewed as a simple way of transforming
the temporal dimension into another spatial For the model in figure 3, the prediction equations
dimension. for computing the output value y(k+1) using the past
observations u1(k), u1(k-1), …, u1(k-du) for the first
time series, the past observations u2(k), u2(k-1), …,
u2(k-du) for the second time series and the past
outputs y(k), y(k-1), …, y(k-dy) as inputs, may be
written in the form:
 N d u1
y(k + 1) = Φo wb0 +
 h =1
∑
who ⋅ Φh (who +
i1= 0
∑
wi1hu1(k − i1) +

(14)
du 2 dy 
+ ∑w
i 2= 0
i 2 h u 2 (k − i 2) + ∑w j =0
jh ⋅ y ( k − j )) 


Fig. 2 NARX model with tapped delay line at input

In practice it was observed that forecasting of a time

series will be enhanced by simultaneously analyzing
related time series. For example, electrical power
consumption for the next day will be better
predicted if taken together, last pc day consumptions
and last pt environment temperatures are
simultaneously applied as inputs to the neural
networks. The architectural model in figure 3 is
make to test this hypothesis A generalized
implementation of this model allows the input and
output to be multidimensional, and thus applying to
the “multivariate” type of time series.
For the architectural model in figure 2 the notation
used is NN(du, dy; N) to denote the NN with du input
delays, dy output delays and N neurons in layer 1.
Similarly, for the architectural model in figure 2 the
notation used is NN(du1, du2, dy; N).
Fig. 3 NARX with two tapped delay lines for two
time series applied at NARX input

ISSN: 1991-8755 186 Issue 3, Volume 3, March 2008

WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

3.2 Learning algorithms the correct combination so as to produce a network

For learning purposes, a dynamic back-propagation which generalizes well. The process is called
algorithm is required to compute the gradients, Bayesian regularization.
which is more computationally intensive than static In general, in function approximation problems, for
back-propagation and takes more time. In addition, networks that contain up to a few hundred weights,
the error surfaces for dynamic networks can be more the Levenberg-Marquardt algorithm will have the
complex than those for static networks. Training is fastest convergence. This advantage is especially
more likely to be trapped in local minima [6][12]. noticeable if very accurate training is required.
The selected training method in this work uses the However, as the number of weights in the network
advantage of availability at the training time of the increases, the advantage of this algorithm decreases.
true real output set. It is possible to use the true Other training algorithms [15] were tested, but with
output instead of the estimated output to train the a less good result:
network witch has the feedback connections - The network training function that updates weight
decoupled (cut). The decoupled network has a and bias values according to gradient descent with
common feedforward architecture which can be momentum.
trained with classical static back-propagation -The network training function that updates weight
algorithm. In addition, during training, the inputs to and bias values according to the gradient descent
the feedforward network are just the real/true ones – with adaptive learning rate.
not estimated ones, and the training process will be -The network training function that updates weight
more accurate. and bias values according to the gradient descent
The training process has some difficulties. One is momentum and an adaptive learning rate.
related to the number of parameters, which refers to The neural network training can be made more
how many connections or weights are contained in efficient if certain preprocessing steps on the
network. Usually, this number is large and there is a network inputs and targets are performed. The
real danger of “overtraining” the data and producing normalization of the input and target values mean to
a false fit which does not lead to better forecasts. For mapping them into the interval [-1, 1]. This
NARX neural network model the number is given simplifies the problem of the outliers for the
by p = (du + dy + 2)N. A solution is penalizing the network. The normalized inputs and targets that are
parameter increase [15]. This fact motivates the use returned will all fall in the interval [-1, 1].
of an algorithm including the regularization
technique, which involves modifying the
performance function for reducing parameters value. 4 Experimental results
Practically, the typical performance function used in
In all the experiments performed, a one-step-ahead
training, MSE, is replaced by a new one, MSEreg, as
prediction is considered; that is, the actual observed
follows:
values of all lagged samples are used as inputs. (If
N N multistep-ahead predictions are required then, it is
∑ ∑ (t
1 1
MSE = ( ei ) 2 = i − yi ) 2 (15) possible to proceed by adding the first one-step-
N i =1
N i =1 ahead prediction to time series, and then the new
time series is used to predict the second step-ahead,
n
and so on).
∑w
1 2
MSW = j (16)
n Various NN models having different numbers of
j =1
lagged input steps (or time windows), different
number of lagged output connected steps as inputs,
MSEreg = ξMSE + (1 − ξ ) MSW (17) and different number of neurons in layer 1 have
where ti is the target and ξ is the performance ratio. been compared. All the models had layer 1 with N
The new performance function causes the network neurons and a single neuron in layer 2 (output
to have smaller weights and biases, and in this way layer). The input has been rescaled in most cases to
forces the network response to be smoother and less be included in [-1, 1] range.
likely to overfit. An interesting comparison about length of input lags
The network training function that updates the in feedforward neural network and the NARX model
weight and bias values according to Levenberg- presented in this paper can be made with [3]. It is
Marquardt optimization was modified to include the well known from practice of NN that the input
regularization technique. It minimizes a combination variables of NN should not to be much correlated,
of squared errors and weights and, then determines because the correlated input variables may degrade

ISSN: 1991-8755 187 Issue 3, Volume 3, March 2008

WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

the forecasting by interacting with each other as well Table 1

as other elements and producing a biased effect [23]. a b c τ H sse ssw N du dy R
In [3] it is shown that a feedforward model as NN(1- 0.210.1 10 30 0,25 0.22 13.6 3 12 1 0.99545
0.2 0.12 10 30 0.36 0.62 27.2 3 12 1 0.99966
13;4) lead to poor prediction. The notation (1-13) 0.2 0.1 5 75 0.37 0.62e-3142.8 3 12 1 0.94592
means 13 inputs, and (1, 13) means 2 inputs. In 0.2 0.1 10 50 0.37 0,102 29.7 3 12 1 0.9931
general, the success model in [3] have only 3 or 4 0.2 0.1 10 30 0.41 0.15 14 3 12 1 0.99185
0.2 0.1 10 80 0.51 2.93 45.39 3 12 1 0.74984
inputs, as in corresponding notation NN(1,12,13;1) 0.2 0.1 10 80 0.51 2.41 75 5 12 1 0.06015
or NN(1,2,12,13;2). By his architecture, the NARX 0.2 0.1 10 80 0.51 0.02 59 5 24 1 0.99968
model applies simultaneously all the values stored in 0.2 0.1 10 80 0.51 21,3 10 2 5 1 0.29047
0.2 0.1 8 70 0.50 0.014 43.9 3 12 1 0.99981
the tapped delay line as inputs to the RNN. It is easy 0.2 0.1 5 70 0.72 2.4e-5 2413 3 12 1 0.9244
to observe from the table 1-5 that, in general, the
optimum number of lags is in 12-30 range, the best
value being found from one case to another case. A
bigger number of lags at input do not improve the
prediction, it is even worse. That is, a conflicting
condition results in selecting the number of lags of
the delay line at input: a bigger number should
assure input sequence as predictive as possible, but
many variables are too correlated to each other to
produce unconfused outputs.

Fig. 4 The traces for original and predicted

4.1 Criteria to estimate prediction chaotic Mackey-Glass time series are very
Usually, the performance of a trained network can close; H=0.51, a=0.2, b=0.1, c=10, τ=80,
be measured by the errors on the training, validation, NN(24, 5; 5) model. R=0.99573
and test sets. In this work, the correlation coefficient
R between the outputs and targets of neural network
is used. R is a measure of the variation between the 4.3 Example 2: Fractal Weirstrass time series
generated outputs and targets. If this number is equal The data contained in table 2 give a motivation
to 1, then there is perfect correlation between the about the conclusion that in that case of Weirstrass
targets and outputs [15]. In the following examples, fractal time series the values of exponent Hurst do
the fact that the number is very close to 1 (R>0.98 or not represent an indication about the prediction
better R>99), indicates a good fit. success. The column labeled H shows a strong
variation of about more 300% of the values of Hurst
coefficient which do not can be put in relation with
4.2 Example 1: Chaotic Mackey-Glass time the rate of success represented by the value from the
series column labeled R.
Table 1 depicts the experimental results with chaotic
Mackey-Glass time series as input, for a total of Table 2
1000 observations, test set being different from a b ω H sse ssw N du dy R
3.2 6.1 25 0.252e-11 5.2 18 15 2 0.99998
training set. Column labeled R (correlation 2.7 2.1 11.9 0.2611e-8 4,9 10 30 2 0.99988
coefficient between input and output of neural 3.2 5.1 32 0.3 1e-11 8.32 10 30 1 0.99739
network) shows the results in different runs 3.2 5.1 32 0.3 1.6e-9 5.61 10 30 2 0.99744
respectively. The rows 3 and 4 from table 1 show 2.3 1.5 15 0.363.8e-8 5.28 10 30 2 0.99988
3.2 4.1 50 0.461.1e-8 4.07 10 30 2 0.99821
that for two different time-series but with identical 3.2 4.1 50 0.467e-10 10.66 10 30 2 0.99906
H, the same NN model gives different predictions. 2.1 2.07 15 0.531.8e-9 21 10 35 1 0.99964
The rows 6, 7, 8 and 9 indicate the influence of 3.2 5.1 38 0.571.2e-7 24.77 5 15 2 0.9980
3.2 5.1 38 0.577e-10 10.66 10 30 2 0.99906
architecture on prediction quality: for the same time- 3.2 6.1 25 0.662.3e-7 6.58 2 10 1 0.99971
series, only the NN(24,1;5) model gives the best 3.2 6.1 25 0.660.6 30 2 4 1 0.9655
prediction. 1.2 1.4 10 0.822e-7 61.52 2 10 1 0.99929
2.7 2.1 11 0.841.2e-8 5,38 10 15 8 0.98409
2.7 2.1 11 0.841.8e-8 5,39 10 30 2 0.99996

ISSN: 1991-8755 188 Issue 3, Volume 3, March 2008

WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

4.4 Example 3: BET time series Table 3 Table 4

The BET-index time-series is an average of daily N du dy R N du dy R
closed prices for nine representative, most liquid 6 30 2 0.95185 6 30 3 0.25758
listed companies at Bucharest Stock Market. 7 30 2 0.96206 7 15 2 0.53296
8 30 2 0.9535 9 24 2 0.88662
Although the Hurst coefficient of BET-index time 10 30 2 0.72639 10 24 1 0.83436
series was close to 1, it was proved that prediction in 10 35 2 0.72639 10 24 2 0.2868
the case of BET index time series, (figure 1) is a 11 30 2 0.54259 10 30 2 0.87045
much more very difficult task. For 800 daily 11 45 2 0.92045 11 40 1 0.64629
observations, the result of simulations was obtained 11 45 3 0.92947 11 40 2 0.57197
including the test set in training set (Table 3), or 12 30 2 0.54337 15 50 3 0.07878
12 50 3 0.9592
excluding the test set from training set (Table 4). 12 50 4 0.82927
15 50 3 0.96116
18 50 3 0.86318

Table 5
N du dy R
6 30 2 0.8207
7 30 2 0.9299
8 30 2 0.8281
11 45 2 0.8933
11 45 3 0.8690
12 30 2 0.4266
12 50 3 0.4266
Fig. 5 Weirstrass fractal function, easy to be 15 50 3 0.7904
predicted: H=0.82, a=1.2, b=1.4, ω=10,
R=0.99971. Original and predicted time series are
approximately the same, NN(8, 2;1).

Fig. 7 BET prediction, NN(50, 3;15)

Fig. 6 A difficult Weirstrass case: H=0.53, a=2.1,
b=2.07, ω=15, R = 0,99964, NN(35,1;10) The RNN model presented in figure 3, for two
inputs presented simultaneously to the network, was
From the first set of simulations, one of the best tested for two related time series: BET - index value
prediction is graphic represented in figure 7 with the and BET - index volume (figure 9 and 10). TDL
correlation R=0.96116. The training algorithm with means time delay line. The volume (figure 8) should
the best results was the back-propagation with be enhanced the prediction for the time series
Levenberg - Marquardt optimization and Bayesian values. The results are in table 6. Another test was
regularization. To be compared with the results in made by using as second input the delayed time
Table 3, some results obtained after training with an series (with 15 days) of the same BET-values time
algorithm that updates weight and bias values series (figure 10). The results are in the table 7.
according to the gradient descent momentum and an
adaptive learning rate, were presented in Table 5.

ISSN: 1991-8755 189 Issue 3, Volume 3, March 2008

WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

5 Conclusions
In this paper, the performance of the prediction for
different time series was tested using a NARX
dynamic recurrent neural network. Comparative
experiments with real and artificial chaotic time
series from diverse domains have been made.
The first conclusion of this paper is that NARX
recurrent neural networks have the potential to
capture the dynamics of nonlinear dynamic system
such as in the examples shown, for the Mackey-
Fig. 8 BET index volume 2005-03.2008 Glass system with different delays. This affirmation
is based on the fact that correlation coefficient R
Table 6 Table 7 estimated for the original and generated (1000
N du1 du2 dy R N du1du2 dy R points) time series is close to 1 in many cases, and
10 25 25 3 0.8147 10 25 25 3 0.2432 the prediction can be considered of real interest or
12 6 6 2 0.5607 12 6 6 2 0.6143
12 25 25 3 0.8093 12 25 25 3 0.5320
significance if R>0.98.
14 14 14 2 0.5916 14 14 14 2 0.0301 The paper has attempted to use traditional statistical
15 25 25 3 0.7145 15 25 25 3 0.7308 methodologies (R/S Rescaled Range, from where
Hurst coefficient) to obtain indications to make
efficient the process of prediction chaotic time series
with RNN. To certain extent, the Hurst coefficient
may give a clue, otherwise vague, about existence of
long time memory in the analyzed time series. The
prediction may fails however, even the values of
Hurst coefficient are encouraging, in conformity
with R/S theory.
The second conclusion is that the nonlinear NARX
models are not without problems, they have
limitation in learning long time dependences due to
the “vanishing gradient”, and like any dynamical
system are affected by instability, and have lack of a
procedure of optimizing embedded memory.
The last conclusion, and the most important, is that
the architecture of the tested RNN model affects the
Fig. 9 NARX RNN with simultaneous inputs: BET- performance of prediction. The most favorable
index value and BET- index volume behavior of NARX model is dependent upon the
dimension of embedded memory of input and output
and the number of neurons in the input layer. The
determination of these architectural elements, in an
optimal way, is a critical and difficult task for the
NARX model, and remains an objective for future
works.
The followings lines contain several directions to
explore:
- The avoiding of saturation and over-fitting because
of too many neurons in network. Too many hidden
neurons lead to poor prediction. A solution is the
including in NARX models the penalizing terms as
Bayessian Information Criteria (BIC), Akaike
Information Criteria (AIC) – a process named
regularization.
- The data input analysis may show how the length
Fig. 10 NARX RNN with simultaneous inputs: BET- of sequences and their correlation influence the
index value and BET- index value 15 days lagged

ISSN: 1991-8755 190 Issue 3, Volume 3, March 2008

 WSEAS TRANSACTIONS on COMPUTER RESEARCH Eugen Diaconescu

interval and value of predictability (selection the [13] M. Tertisco, P. Stoica, T. Petrescu, Modeling
length of lags). and forecasting of time series, Publ. House of
- The finding other training algorithms than back- Romanian Academy, 1985
propagation, for the NARX models. Consecutive [14] Garnet P. Williams, Chaos Theory Tamed,
restarts of the program with back-propagation Joseph Henry Press, 1999
training function, give different results, indicating [15] H.B. Demuth, M. Beale, Users’ Guide for the
the end of iteration process in local optima. Neural Network Toolbox for Matlab, The
Mathworks, Natica, MA, 1998
References: [16] L.F. Mingo, J. Castellans, G. Lopez, F. Arroyo,
[1] Simon Haykin, Neural Networks, Second Time Series Analysis with Neural Network,
Edition, Pearson Education, 1999 WSEAS Transactions on Business and
[2] Georg Dorffner, Neural Networks for Time Economics, Issue 4, Volume 1, October, ISSN
Series Processing, Neural Network World, Vol. 1109-9526, pp. 303-310, 2004
6, No. 4, 447-468, 1996 [17] A.C. Tsakoumis, P. Fessas, V. M. Mladenov,
[3] J.Farawey, C.Chatfield, Time Series Forecasting N. E. Mastorakis, Application of Neural
with Neural Networks: A Comparative Study Networks for Short Term Electric Load
Using the Airline Data, Applied Statistics, Prediction, WSEAS Transaction on systems,
Volume 47, No.2, 1998, pp. 231-250 Issue 3, Volume 2, July 2003, ISSN 1109-2777,
[4] Tsungnan Lin, Bill G. Horne, Peter Tino, C. Lee pp. 513-516
Giles, Learning long-term dependencies in [18] A.C. Tsakoumis, P. Fessas, V. M. Mladenov,
NARX recurrent neural networks, IEEE N. E. Mastorakis, Application of Chaotic Time
Transactions on Neural Networks, Vol. 7, No. 6, Series for Short-Time Load Prediction, WSEAS
1996, pp. 1329-1351 TRANSACTION on SYSTEMS, Issue 3,
[5] Yang Gao, Meng Joo Er, NARMAX time series Volume 2, July 2003, ISSN 1109-2777, pp. 517-
model prediction: feedforward and recurrent 523
fuzzy neural network approaches, Fuzzy Sets and [19] Theodor D. Popescu, New method for time
Systems, Vol. 150, No. 2, 2005, pp.331-350 series forecasting, WSEAS TRANSACTIONS
[6] M. T. Hagan, O. D. Jesus, R. Schultz, Training on CIRCUITS and SYSTEMS, Issue 3, Volume
Recurrent Networks for Filtering and Control, in 2, July 2003, ISSN 1109-2734, pp. 582-587
(editors) L.R. Medsker, L.C. Jain, Recurrent [20] O. Valenzuela, I. Rojas, I. Marquez, M.
Neural Networks – Design and Applications, Pasados, WSEAS TRANSACTION on
CRC Press, 2001 CIRCUITS and SYSTEMS, A novel Aproach to
[7] Tsungnan Lin, C. Lee Giles, Bill G. Horne, S.Y. ARMA Time Series Model Identification by
Kung, A Delay Damage Model Selection Neural Network, Issue 2, Volume 3, April 2004,
Algorithm for NARX Neural Networks, IEEE ISSN 1109 – 2737, pp. 342-347
Transactions on Signal Processing, “Special [21] Wei Huang, Shouyang Wang, Lean Yu, Yukun
Issue on Neural Networks”, Vol. 45, No. 11, Bao, L. Wang, A New Computational Method of
1997, pp. 2719-2730 Input Selection for Stock Market Forrecasting
[8] H. T. Siegelmann, B. G. Horne and C. Lee with Neural Networks, International Conference
Giles, Computational capabilities of recurrent on Computational Science, 2006, pp. 308-315
NARX neural networks, IEEE Transactions on [22] Tilmann Gneiting, Martin Schlather, Stochastic
Systems, Man and Cybernetics, Part B, Vol. 27, Models That Separate Fractal Dimension and
No.2, 1997, 208-215 Hurst Effect, SIAM review, 46, 2004, pp. 269-
[9] Jingtao Yao, Chew Lim Tan, A case study on 282
using neural networks to perform technical [23] G. P. Zhang, Neural networks in Business
forecasting of forex, Neurocomputing, 34, 2000, Forecasting, Ideea Group Inc., 2003
pp. 79-98
[10] Edgar E. Peters, Fractal Market Analysis, John
Wiley & Sons, 2001
[11] D.S.G. Pollok, A Handbook of Time – Series,
Signal Processing and Dynamics, Academic
Press, 1999
[12] D.P. Mandic, J.A. Chambers, Recurrent Neural
Networks for Prediction, JohnWiley&Sons, 2001

ISSN: 1991-8755 191 Issue 3, Volume 3, March 2008

View publication stats