Practical 4.3 – RNN training Video Lecture - Torch A Practical Hands

in this segment we will see how to train
a recurrent neural network without easy
example we start by introducing our
study case which means the problem we
will try to solve with a recurrent
neural network training then we will
review how to use Optim for training a
neural network and in this case a
recurrent neural network then we will
run the code and see the output
we will try also to change some hyper
parameters and see their corresponding
result finally we will delve deep into
the code we will see how everything
works together so let's introduce our
learning problem we are going to have a
sequence to sequence case so we are
going to start from a sequence here
which is sent to the hidden layer we are
going to just use one hidden layer and
then we expect to output a prediction
based on the current input and from the
previous input so what is our task so
our X is going to be a sequence of
characters for example a be a P P a a
and let's say we would like to be able
to recognize the sequence of letter that
make the word up like the famous music
band so more specifically we would like
that when we see this last character the
last day of the world ABBA we would like
our model to output 2 instead for all
the other cases we just output a 1
so let's say this one is our sequence of
label if we get it right not here I
expect at 2:00 and I actually get my
natural predictor - then I'm gonna
highlight these four letters in one way
if instead I predict the last one to be
the correct sample I will have to say
that these letters here are a false
positive instead we can have that if I
have to predict the number two here and
I don't predict so I predict a one I
will highlight these four letters as a
false negative so in our visualization
we had to take care of true positives
false positives true negative which are
actually the only one that we won't be
highlighting and then we have false
negatives
and we will see how we can do so in the
UNIX environment so actually our X has
to be one hot encoding so this sequence
that we have just written here on the
left hand side so let's write it here
it was a b a b b:a a we had to
correspond to a 1 0 0 1 1 0 0 1 0 1 1 0
1 0 for example and so on our labels are
going to be so for the first one we have
1 for the second one we have one third
one can be one fourth one is going to be
1/5 1 then we had a sixth we have a 2
and then again a 1 and so on
so our target labels are going to be one
for all the sequences that don't
correspond to the
sequence and then it's going to be
turning to when the sequences being
found
so we call it this find V Appa sequence
problem so how do we plan to train such
a system we have seen for convolutional
neural network that we can use very
efficiently the optimum package we will
do here the same and therefore we will
follow a very similar workflow let's
summarize it just for convenience so at
the beginning in the training part we
are going to have get the data set use
the Train then we are going to craft our
model our recurrent neural network
together with its time replicas
we had to reset our state initial state
because we start from a from a clean
network so we haven't seen any sequence
before then we should get the parameter
in a pictorial form and not anymore as a
object collection of objects and guiding
parameters so the partial derivative of
the error with respect to the global
vectorial parameter then we will define
our function eval to which we send theta
is argument and which returns error e
and the partial derivative of the error
with respect to the vectorial form of
the parameters finally we can run our
optimizer for testing purposes we just
need to get the data set in this case is
the validation and then reset
the state since the state would have
been polluted by the training at this
time of the code we are going to feed
each example of X one at a time to the
RNN and we don't need any more the timer
replica format as we have said in the
previous video we can find the code at
the ELab website under the torch 7 demo
repository and in the folder RNN - train
- sample let's start by having a look at
how this program runs so let's call th
main we have the training progress and
here we have the output notation we can
observe in green the true positives in
white simply the true negative in white
with red background the false positives
which means the network tells that the
sequence is positive but it is not and
then underlined it's the false negative
basically it's when the network doesn't
recognize the sequence so in green again
is when the network recognized the
sequence in white is when the network do
not identify wrongly any sequence read
the ground is when a sequence has been
identified as positive but is not in the
underlined it's when a sequence has not
been recognized as you can see here from
this the sequence ba
a P P B and then we have twice up ba
that is has been recognized so it is in
green then there is one more Abba which
has not been recognized therefore is
underlined then we have a B and then
three B's followed by a a which the
network thinks is ABBA
it's not and therefore it is highlight
in red background then it correctly
identify true negatives so we have all a
white sequence then correctly identified
the up a sequence and we go on like this
up to the end of the sequence let's see
how this training is implemented this is
the main dot Lua file we require n n
graph and up
so we can use them both in this script
we set approach manual seed of six in
order to have repeatability of this
script if you run this script you will
observe the same result that I'm showing
you right now
then we set the torch default tensor
type to float dancer this is a common
practice in machine learning we do not
require very high precision during
computation moreover there are a set of
hyper parameters and is the
dimensionality of the input that is 2d
is the dimensionality of the hidden
representation then we have number of
hidden layers we had just one hidden
layer and then K is the output
dimensionality that is again 2 n is 2
because it can be a or b so the notation
since it's one hot encoding will be 1 0
or 0 while in the same way we will have
4k dimensionality of 2 because we will
have a probability distribution across
the classes T is a maximum length of the
sequence which is equal 4 since Alba has
4 characters the training size is set to
10,000 and a test in size is set to the
150 let's see what happens if we change
the hidden dimensionality going from two
to three we run again the script and we
have all correct predictions so we have
seen now that one singular hidden layer
with three neurons it's able to solve
this simple problem let's go back to the
previous screen
we set a learning rate of 0.02 in a
smoothing factor of 0.9 t5 which are
then sent to the optimum state we define
some coloring shortcuts that are showing
you the different colors of the
background and the underline features
then we do data equal required data and
then we call data dot get data with the
training size and the maximum length of
the sequence and we expect to receive x
and y so let's have a look to data and
here we have data so the function get
data defines a torch tensor of
dimensionality size of the training
which is filled with ones and twos
randomly we define a Y which is a tensor
of ones and then if we do find from the
fourth position to the last element of
the training size the sequence a be BA
or 1/2 to 1 then we set the
corresponding label of Y to 2 then we
define our X X is gonna have as many
roles as the training size and as many
columns as the features since it's a one
hot encoding with two symbols we will
have just two columns so for the first
two the last feature if the symbol is a
then we are going to set a 1 so
basically it's going to be 1 0 otherwise
if the picture is a B so it's a 2 we are
going to set 0 1 basically and then
we'll turn x and y and then we have
ironin equal required Arnon and then we
have that we call the CAD model from the
ironin package we have rolled with the
dimensionality of the input the
dimensionality of the hidden layer the
number of hidden layers the
dimensionality of the output K and in
max length of the sequence T and then
the output is gonna be a model which is
the prototype clone over the capital T
time steps and the prototype which is
one of these loan models then we define
a criterion which is a negative
glyco last criteria so it's our
classical cross entropy if you don't
know what is a cross entropy criterion
just check CNN loss video lecture then
we get our one dimensional vectors
containing the weights of the model and
the grad parameters which are the
derivatives of the loss function with
respect to the parameters and then we
print on screen the number of elements
that are in W so the number of
parameters we then define H 0 and H
tables and we fill them with tensors of
zeroes so these are the initial state
and the current state of the network but
since we haven't started using the
network we are going to fill them with
zeros then we forward our first four
samples in the network that is grown
over time and in the prototype so that
we can plot the two graphs with full
annotations let's have a look at these
two graphs this is the prototype we can
see that after the first reverse maps
and splits we have the X of T in orange
on the left hand side and then on the
right hand side we have in red the first
and actually only hidden layer at the
time step t minus 1 these two inputs are
fed inside a join table which is then
sent to a linear which is then sent into
a non-linearity which computes the first
hidden layer output at time T if we
would have had more layers than the
would have stacked one after each other
the output of the first and only hidden
layer then it sent one under linear and
then to the final log softmax which
allows us to use the cross entropy
criterion here we have the full model
flown over time in blue we are going to
see the are NN prototypes so we have our
n n 1r n n time step to our n n time
step three and our nine time step four -
the first are n n time step one we are
going to feed the previous value of the
hidden state and then we send also the
first element of our sequence X of 1
then the output of the RN n the red
circle which contains the hidden layer
at the time step one it's fed together
with the second element of our extensors
into the second clone or an end
the output of RNN 2 which is the age at
time 2 it's fed into the RN n 3 together
with the X at time 3 finally in the last
H of timely together with the X at time
4 it's fed into the r9 4 which produce
the last output age moreover each of
these are named blue blocks produce the
Y prime softmax output so we are going
to have here Y hat 1 Y hat - Y hat 3 and
Y hat for which we are going to be using
in order to to train and to write the
prediction at the testing time
here we define two helper functions one
allows me to go from tables to tensors
and the other one allows me to go from
tensor to table it also adds some zeros
that are gonna be sent into the state
when we perform a grappa gation we will
see this in a few line of code here we
initialize our training error to zero
then we go from the first element to the
last element of our training set with a
time step of capital T we are gonna
narrow the whole data set X along the
first dimension ality we start from the
item iteration and we take just capital
T elements we do the same for dy answer
we define then our F evaluation we set
our model to training mode then we have
our current state which is gonna be the
collection of the output wise followed
by the ages States is going to be equal
our model to which we forward our X
sequence and we unpack the table-saw
age in this case we just have one hidden
layer so we just send there 1 tensor but
in the case you would have chosen
multiple hidden layers the unpack would
have sent separately each and every
intenser then we have the prediction we
convert the state's table which contains
the Y prediction and the age hidden
state into just prediction so we
basically extract just the prediction
from these states then we compute the
error via the criterion to which we
forward the prediction and the y
sequence perform the backward pass we
compute the gradient of the error with
respect to the output of the system by
forwarding again the prediction and the
labels to the criterion using the
backward method then we convert the
output tensor from the criterion into a
table which we are gonna use in order to
perform back propagation to the model
but before sending it into the model we
have of zero degrade parameters in order
to perform stochastically in the same
otherwise people have been accumulating
degrade parameters over the previous
values finally we take care of storing
all the hidden layers state into the H
table in this way when we go back above
and we are
eh we are gonna unpack the current state
of the network and we do not have a zero
State every time we switch to the next
sequence and that's it we simply call up
team with the rmsprop optimization
function which will provide the function
evaluation the parameters W in the
optimum state which was containing the
learning rate and the learning rate DK
now we go on to testing we start our
testing by setting the model and the
prototype to evaluation so we turn off
any kind of training related features we
again initialize our hidden state to the
h0 state which is a table of tensors set
to zero moreover we require a new data
set of size s size which was 150 so in
order to be able to test our system on
data that is not the training data then
we define some pointers for the Q we are
going to use later in order to visualize
nicely the result and here we have the
test function get the first element of
the X validation set and it sends to the
prototype which is one of the RNN
replicas is sent the first symbol of the
sequence together with the state of the
network which we start from a zero state
and then we obtain the output which
contains the next state and will cache
it and then we extract the prediction by
checking the value after the States then
we check whether the first symbol was a
1 0 or 0 1 we converted to the character
a or B then if we are just processing
the first 3 element of the whole
validation set we insert them into our
sequential buffer and we do not do
anything otherwise if we are at the
fourth symbol we can start evaluating
whether the sequence is correct sequence
or is a bad sequence index is the max
index
of the prediction so prediction is gonna
be a log likelihood and we take the max
so if the index it's equal D label then
we are gonna draw it in green otherwise
if we got it wrong there are two cases
one case is that the label is true so
it's basically we have a sequence but we
haven't identified the sequence so we
have a false negative or otherwise if
it's not at all we have identified a
sequence then we have a false positive
and then we update the pointers we get
the next value from the cue and we write
it on the screen at the end we simply
print the legend or the visualization
and we iterate the test function over
the test size dimension and let's run it
once more and since we have restored the
dimensionality of the hidden state to
two we are gonna see again different
mistakes so we train here number of
parameters are 16 it's very small
network and then we can see again we
have a true negative true positive then
we have a false negative we didn't see
the a becoming then we have again
through negative and then we got a false
positive the output of the network was
saying that it was a positive but it was
not there and that's it thank you for
listening