Practical 4.1 – RNN forward and backward Video Lecture | Torch: A Practical Hands On Tutorial - AI & ML

in this section we go through on the
forward and backward pass for a
recurrent neural networks we we start
with reviewing the equations for Avenida
neural network with a new notation which
allows us to introduce later the
equation for the forward propagation of
recurrent neural networks finally we
will see how big the propagation can be
applied on an unrolled version of the
network this technique gets the name of
big propagation through time recurrent
neural network can be thought is fancy
in neural network where the current
output Y hat depends on the given input
X and the state of the system the start
viewing what are the basic fundamental
equations of a vanilla neural network in
this case we have a three layer neural
network I could draw here one more
circle representing the first input so
we have three layers representing
vectorial quantities not as before where
each circle was representing a scalar
and then we go from X to H through the G
non-linearity and the WH matrix and then
we go from H to y hat through the genome
linearity with W Y matrix so let's draw
the equations with this new notation and
let's see how they connect to what we
have seen previously in the previous
lessons we can write here that the
vector H it's equal our nonlinear
function that we just simply called G of
the affine transformation of X so we
have W a which multiplies my vector X
and then we have the bias vector for the
Aged activation then we have that our
prediction Y hat is going to be again a
nonlinear function of the product of W Y
with our H+ again the bias term vector
for my Y term
is a reminder in the previous lessons we
saw that this one was simply the Sigma
and this part over here was my theta
coming from the first layer which was
multiplying my x2 which I have added the
one for the bias but basically we are
doing the same where I simply took out
the bias from DX and moreover we have
that D again representation here on the
left hand side represent vectors
not anymore scalars and moreover we had
that this first guy here it was called a
two vector on the same way this guy was
called Sigma this content here was theta
for the second layer multiplied a hat to
and this guy was called our final
hypothesis on the input so here we have
slightly changed the notation which is
going to make our life easier when we
switch to the current neural networks so
we have that our X it's still a
n-dimensional vector our Y is a I know
this symbol represents something else
but let's say I can write this symbol in
the opposite direction is our RK and
let's say our dimension here of our
hidden state is simply let's call it D
dimensional so we are gonna have that
our WH D times n because then it's
multiplied by n dimensional vector and
then we have of course our bias H it's
going to be d dimensional and on the
same way we had the wy
is going to be shooting towards sky by
starting from T and then the bias term
for the Y it's going of course to be a K
dimensional vector G in this case can be
sigmoid can be at an age or any kind of
other non-linearity it's not that
relevant so far all quantities are being
matrices or vectors so we can think that
our neural network transfer function y
hat it's actually a function of our
input X so we have that Y hat goes from
our n which is our input to our K which
means shoot vector X into vector Y so it
tries to shoot our vector X into a
mapping to our wire labels basically or
at least we are trying to so we have a
hat here but it's doing this going
through the hidden States
so actually we could write that Y goes
yes from our end
y it goes through our D before going to
our okay where we have that D the hidden
state user is quite larger than both the
input and the output dimensionality this
is done in order to generate a hidden
space or internal representation which
disentangle the combination of input
features and therefore allows to discern
meaningful characteristics of the input
data the matrices WH in Wi contain our
linear kernel as row vectors each of
these row vectors is scalar Li
multiplied by the corresponding input so
if we are in the age
lair we are going to be using the
projection of these horizontal corners
with these row corners with the input
and then if we are in the last layer we
perform the projection of the hidden
state with this output kernels and then
we applied the non-linearity in order to
boost the prediction and also to use
separation and break linearity of the
system so we can think W to be something
like w1 w2 and so on and every time we
perform a multiplication and every time
we perform a multiplication we perform
simply the projection of the input here
with respect to the each and every
kernel and then we apply a translation
with a bias term which is then followed
by the application of a nonlinear
function in order to be able to stack
more than one layer since the again the
linear combination of multiple linear
system is still a linear system and this
would translate as in total collapse of
any hierarchy of layers and finally we
come to the recurrent neural network
so all it changes here is that we have
here a loop with a whh matrix and then
we have introduced the square bracket
with a time index double kht so in the
input we don't have anymore just the IMP
X vector but we can have like as well a
sequence of input and then we have a
hidden state which is function of the
time index T and then at the output we
can have the sequence Y hat of T or
simply consider the last element of the
sequence Y hat capital T which can be
thought as the final output of the
network given that we provided all the
input and then the processing has been
completed let's write the equations for
this network and let's see how they
compared to the question we just wrote
for the classical standard neural
network
so we have now that our H it's a
sequence so we have time index T we have
square brackets because the time is
discrete we would have have parenthesis
instead if we would have been working
with time continuous domains so here we
are in discrete domain this is going to
be equal our nonlinear function G which
is fed with W H X which multiplies the
input at the time index T plus the
weight matrix which is waiting the
previous state H vector time index t
minus one plus the Associated bias term
for the H and then below we have the D
prediction Y hat vector of time index T
it's actually exactly as it was for the
previous slide so we have our nonlinear
function which is fed with the matrix W
Y which is multiplying our hidden state
of the same time index T plus the offset
B of Y again here if each and every of
these elements belongs to the RK
dimensionality the state and previous
state belongs to the D dimensionality
and our input is a sequence of n
dimensional vectors moreover we had to
specify the the initial state so the
hidden state at the time interval 0 is
by definition set the zero vector so
that we start with a reset system so
regarding the dimensionality of the
matrices we have that the wo h x
the same as for the wh of before which
was shooting towards D and when coming
from N and then we had now the new wo h
h instead goes to D coming from D and
the last one which is the same we headed
WY instead shoots towards
D K output dimensionality coming from
the D dimension of the hidden state the
term B H is of course ad dimensional
vector and B Y again of course is going
to be a K dimensional vector what is the
main takeaway of this equation here the
recurrent nature of this equation tells
us that the current output vector Y of
index T not only depends on the current
input x time index t but also to the
internal state of the system which is h
of index t minus 1 this means that apply
multiple times the same very input X may
lead to different output Y we see here
the current state age of index T depends
on the previous state h of t minus 1 via
the matrix wh h which is modeling the
relationship with which the current
state depends from the previous state
the hidden state equation can be
rewritten
in a different and more compact way
which is going to be even more similar
with the previous equation we just saw
in the previous slide so we have now
that H vector
on time T index T can be also written
and preferably written as our Jean
nonlinear function where I just use one
matrix wh
likewise I was doing for our vanilla
neural network which is multiplying
something a bit particular
is the concatenation of my input X a
time index T in the system state Y of
time index t minus one and both of them
are state vectors and then we simply sum
our biased vector for the H there where
wh it's the horizontal concatenation of
w h x and wo h h which therefore belongs
to D coming from n plus D this should
not be surprising at all because instead
of doing the scalar product of the first
row and the column vector plus the first
row the second Matrix and the column
vector of the state we simply perform
the whole first row which is a
concatenation of the two first rows the
concatenation of the two columns so the
computations are exactly the same but
this notation allows us to express in a
more compact way and actually we perform
just one matrix vector multiplication so
it's even more efficient in terms of
computations when performed on a
computer furthermore we can see that a
recurrent neural network can be easily
converted back to a normal neural
network when the matrix whh it's set to
the null matrix so basically if the
input is a sequence which does not
contain any kind of information across
the time index the network can converge
to a system where the memory matrix it's
set to zero and therefore the system
will be simply a combinatorial system so
how do we train the system
the cool part is that we simply apply
forward propagation back propagation and
then we apply gradient descent so
nothing changes from the way we have
been training our networks so far so
let's start and see how we can perform
the first for propagation so we are
going to insert our first input then we
suppose we do not have any previous
States so we actually input here a zero
vector and therefore we simply kill the
recursion from the first element if this
is the first element of our sequence
then we can therefore compute our
internal representation given simply the
first input and then via the matrix wy
we perform and compute the first
prediction finally we are going to have
here our loss block which is which is
fed with the network output and accounts
for the first error in this case it's
called t minus 1 here then we insert our
second input and then we can compute the
internal hidden state given the previous
hidden state and the whh matrix once the
new state is computed we can compute the
output given dwy matrix then the new
prediction it's sent forward and we
compute again the error with whatever is
expected finally we input let's say the
last element of our sequence so there
are not more element we can produce the
new state given the previous state with
the whh matrix and the wh x matrix and
then we produce the output given the wy
matrix and we feed the last prediction
into our los block
what we had to notice is that each and
every circle here share the same
parameters and also the gradient with
respect to the parameters so although
this one may seem a simple neural
network actually is a lighter version
because the parameters and decode
parameters are shared across each and
every instance what is not sure is the
internal temporal buffers meaning if I
send my first input then my first input
temporal buffers inside the network will
be different from whatever happens for
the second instance and the third
distance otherwise the network would be
over writing its own its own activations
in the same way also the three nodes
here share the same parameters and grad
parameters so we started with a sequence
of input length which take we can call X
1 X 2 and X 3 then we have set of
parameters which is again similarly to
what was happening before and although
there are again multiple branches we
just have one set because they are
shared so parameters are shared and then
at the end we end up having a sequence
of factorial output
and corresponding errors then we can
simply compute the corresponding
gradient and back propagate them down
the network so here we have our last
gradient a second us gradient first
gradient we are converted into the Delta
tool in deltas then of course here we
have to say we don't want any gradient
coming from the future so the gradient
that comes from here is going to be zero
gradient because again we don't have the
gift of precognition opera science so
again we have to input a zero from the
future and then we are gonna have that
this Delta coming down we split it in
two so one goes down here and of course
we don't know we don't do anything to
the input so we don't care but we could
have more layers stacked so we can
represent here the Delta coming down so
these are relative to the layer two and
then we have the gradient here coming
down this way the same way we go one
down here and the other one coming down
another one down here and stop because
we don't go in the past before the
initial time then we can update all the
grad parameters the e in D theta given
gradient input coming from higher levels
in the hierarchy and from the layers in
the future there are some assumptions we
have done so far so we have four
training
we fix capital T which is the number of
repetitions of the network it limits
also the length of influence which we
can catch from the data so if if data
have relationships that are spread at a
time interval time period greater than
capital T then we won't be able to
capture those kind of relationships if
we train with a capital T that is lower
than such period for training purpose we
fix T training interval so that was
number one then we have number two zero
is the vector the comes from the past so
we already set zero initial state so we
start with a reset state unless we would
like to keep learning specific sequence
so we actually can input let's say in
this case three elements then for the
new element we can send to the first
state the last state of the previous
training shot in this way we have the
state keeps updating based on the
several inputs and then we can have a
system that remembers what has happened
in longer time periods so zero initial
state for the first element of the
sequence and then the other assumption
is that zero future knowledge which
means we don't have the ability of
recognition means that at the end here
we will have a 0 gradient input 0 Delta
coming from future symbols we haven't
input into the system while training
these recurrent neural networks are
powerful tools it can extract
information from sequences can condense
temporal information into vectors can
expand vectorial information into
sequences can compress and we expand
back and even they can perform vector to
vector mapping which is going to be
simply killing
the whh matrix so they are very powerful
and versatile models which can be
trained with the same tools we have seen
so far but requires a little bit perhaps
of tricky implementation for applying
the back propagation through time which
is simply India again the classical back
propagation algorithm which computes the
partial derivatives of the error
function with respect to our parameters
on an unrolled version of the network
which shares the parameters and also the
gradient of the error with respect to
the parameters in the next episode we
will see how we can implement this kind
of tricky combination of connections of
forward and backward signals in a
network indoors