RNN Example with Keras SimpleRNN in Python


   Recurrent Neural Network models can be easily built in a Keras API.  In this tutorial, we'll learn how to build an RNN model with a keras SimpleRNN() layer. For more information about it, please refer to this link.
   The post covers:
  • Generating sample dataset
  • Preparing data (reshaping)
  • Building a model with SimpleRNN
  • Predicting and plotting results
We'll start by loading the required libraries.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, SimpleRNN

Generating sample dataset

For test purpose, we'll generate simple sequence data.

N = 1000    
Tp = 800    

t=np.arange(0,N)
x=np.sin(0.02*t)+2*np.random.rand(N)
df = pd.DataFrame(x)
df.head()
        0
0  0.224671
1  0.490856
2  0.567755
3  0.974433
4  0.805705 
 
plt.plot(df)
plt.show() 
 


Next, we'll split 'df' dataset into a training and test parts.

values=df.values
train,test = values[0:Tp,:], values[Tp:N,:]


Preparing data (reshaping)

RNN model requires a step value that contains n number of elements as an input sequence. Here, we define it as a 'step'. This is an important part of RNN so let's see an example:
x has the following sequence data.
x = [1,2,3,4,5,6,7,8,9,10]
for step=1, x input  and its y prediction becomes:
x  y
1  2
2  3
3  4
4  5
..
9  10
for step=3, x and y contain:
x         y
1,2,3   4
2,3,4   5
3,4,5   6
4,5,6   7
...
7,8,9   10

As you have noticed the sizes of x input and y output become different. We'll fix it by adding step size into the training and test data.

step = 4
# add step elements into train and test
test = np.append(test,np.repeat(test[-1,],step))
train = np.append(train,np.repeat(train[-1,],step))

Next, we'll convert test and train data into the matrix with step value as it has shown above example.

# convert into dataset matrix
def convertToMatrix(data, step):
 X, Y =[], []
 for i in range(len(data)-step):
  d=i+step  
  X.append(data[i:d,])
  Y.append(data[d,])
 return np.array(X), np.array(Y)

trainX,trainY =convertToMatrix(train,step)
testX,testY =convertToMatrix(test,step)

Finally, we'll reshape trainX and testX to fit with Keras model. RNN model requires three-dimensional input data. You can see the shape of testX below.

trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
trainX.shape
(800, 1, 4) 
 

Building a model with SimpleRNN

Next, we create the keras Sequential model.

# SimpleRNN model
model = Sequential()
model.add(SimpleRNN(units=32, input_shape=(1,step), activation="relu"))
model.add(Dense(8, activation="relu")) 
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='rmsprop')
model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
simple_rnn_26 (SimpleRNN)    (None, 32)                1184      
_________________________________________________________________
dense_50 (Dense)             (None, 8)                 264       
_________________________________________________________________
dense_51 (Dense)             (None, 1)                 9         
=================================================================
Total params: 1,457
Trainable params: 1,457
Non-trainable params: 0
_________________________________________________________________

Predicting and plotting result

We'll fit model with trainX data and predict testX data.

model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)
trainPredict = model.predict(trainX)
testPredict= model.predict(testX)
predicted=np.concatenate((trainPredict,testPredict),axis=0)

Next, we'll check the loss

trainScore = model.evaluate(trainX, trainY, verbose=0)
print(trainScore)
0.35334232926368714 


Finally, we check the result in a plot. A vertical line in a plot identifies a splitting point between training and test part.

index = df.index.values
plt.plot(index,df)
plt.plot(index,predicted)
plt.axvline(df.index[Tp], c="r")
plt.show() 



    In this post, we've learned how to fit and predict sequence data with keras SimpleRNN model. The full source code is listed below.
   Thank you for reading!

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, SimpleRNN

# convert into dataset matrix
def convertToMatrix(data, step):
 X, Y =[], []
 for i in range(len(data)-step):
  d=i+step  
  X.append(data[i:d,])
  Y.append(data[d,])
 return np.array(X), np.array(Y)

step = 4
N = 1000    
Tp = 800    

t=np.arange(0,N)
x=np.sin(0.02*t)+2*np.random.rand(N)
df = pd.DataFrame(x)
df.head()

plt.plot(df)
plt.show()

values=df.values
train,test = values[0:Tp,:], values[Tp:N,:]

# add step elements into train and test
test = np.append(test,np.repeat(test[-1,],step))
train = np.append(train,np.repeat(train[-1,],step))
 
trainX,trainY =convertToMatrix(train,step)
testX,testY =convertToMatrix(test,step)
trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))

model = Sequential()
model.add(SimpleRNN(units=32, input_shape=(1,step), activation="relu"))
model.add(Dense(8, activation="relu")) 
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='rmsprop')
model.summary()

model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)
trainPredict = model.predict(trainX)
testPredict= model.predict(testX)
predicted=np.concatenate((trainPredict,testPredict),axis=0)

trainScore = model.evaluate(trainX, trainY, verbose=0)
print(trainScore)

index = df.index.values
plt.plot(index,df)
plt.plot(index,predicted)
plt.axvline(df.index[Tp], c="r")
plt.show()

4 comments:
  1. Hey,

    Nice example, it was helpful. How would it be if the input data consisted of many features (let's say 40) and not just one ?

    ReplyDelete
    Replies
    1. You are welcome! You need to create combined X array data (contains all features x1, x2, ..) for your training and prediction. It goes like this;
      x1, x2, y
      2, 3, 3
      3, 4, 4
      2, 4, 4
      3, 5, 5
      4, 6, 6

      Here, each window contains 3 elements of both x1 and x2 series.
      2, 3,
      3, 4,
      2, 4, =>4

      3, 4,
      2, 4,
      3, 5, => 5

      2, 4,
      3, 5,
      4, 6, => 6

      Delete
  2. Hello!

    Very good example, it showed step by step how to implement a RNN. I am struggling to reuse your knowledge and build a Jordan network.

    I am attempting to translate your Sequential to Functional API but summary shows different network. :(

    This is what I am doing:

    visible = Input(shape=(None, step))
    rnn = SimpleRNN(units=32, input_shape=(1,step))(visible)
    hidden = Dense(8, activation='relu')(rnn)
    output = Dense(1)(hidden)
    _model = Model(inputs=visible, outputs=output)
    _model.compile(loss='mean_squared_error', optimizer='rmsprop')
    _model.summary()

    By using same data input, I can have some result, but then, when predicting, I am not sure how Tensorflow does its recurrence. I would like to use only one output as input, then, what should I change?

    Could you help me out, please?

    ReplyDelete
  3. Hello again!

    I am trying very hard to understand how I build a RNN with the following features
    1. x1, x2 and x3 are input signals that are measurements.
    2. o1, o2 are outputs from the last prediction of the NN and o is the actual output

    x1, x2, x3, o1, o2 --> o
    2, 3, 3, 10, 9, 11,
    3, 4, 4, 11, 10, 12,
    2, 4, 4, 12, 11, 13,
    3, 5, 5, 13, 12, 14,
    4, 6, 6, 14, 13, 15,

    3. how do I train and predict?

    ReplyDelete