Practical 4.2 – RNN nngraph package Video Lecture - Torch A Practical Hands

in this segment we will see how to use
the NN graft package to build a
recurrent neural network in torch we
will start by comparing the technique to
create a multi-layer perceptron with the
NN package and the NN graph package so
we will introduce the parentheses and
the - notation alternatives moreover we
will show how to use the graph dot
visualization with the graph reads
application we will write a gate node
function additional - what comes with
the G module in order to display the
content of a specific node then we will
see more fun architectures that can be
easily done with the NN graft package
and we will implement our very own
vanilla recurrent neural network in just
3 or 4 lines of code finally we will use
a script we have developed here at the
ELab at Purdue which can create a
recurrent neural network model and its
time index replica with some annotation
that is the understanding of its own
architecture let's see where we can find
the N n graph package and how to install
the graph feeds program we are going to
have some fun with the n n graph package
of torch which is a graph package to
craft neural networks you may need to
install the grub feed utility on Mac or
Ubuntu
if you would like to visualize also the
graphs we are going to make let's start
our interpreter and require n n graph we
are also going to require pretty n n
moreover we are gonna set the torch
manual seed to 0 so that the results
that I make can be reproducible by you
let's create a simple neural network so
we are going to have
sequential then i'm going to add a
linear rate cause for example from when
twenty to ten then we can add a
non-linearity then we can add one more
linear in this case let's say from 10 to
10 and then we can add one more
non-linearity finally we can add our
last layer so in order to write a
network which has three layers
I had to write six lines of code we can
print here our network let's see how we
can do the same by using the NN graph
module instead our first module is going
to be and handle on the first module of
our network that we had generated here
and we add an extra parenthesis here so
what is h1 h1 now it's a NN graph note
if you would like to look inside h1 we
can use the curly bracket operator and
then we can see the content of our h1 we
can recognize that inside data and
module we will see the first module of
our previous network let's actually
check if it's the same module so we can
do for example from the Prius net we can
get the first module dot by us let's
view it in in one line so it's easier to
view and let's do the same for the other
one so h1 the we go inside data module
then we go to the bias and we view it
the same way and we can see the
evaluates match so let's finish to write
this network so we had that each one was
simply the first module of our network
we had generated before and then it's
called h2 the rest of the module so we
are going to apply net module 2 on h1
then we are going to apply net module
three on this combination then we are
going to apply net modules for to this
combination and then finally we are
going to apply net modules 5 on this and
here we have h2 it's the second node of
the graph which has been built from h1
to which we have applied the module to
function the module 3 function the
module 4 function and the module 5
function so in two lines of code I have
my network now let's create the graph
net which is going to be a G module
which I pass my h1 as the input and the
h2 as the output so if you'd like to
visualize this we can do graph dot dot G
net then we specify for Warcraft and we
can call this MLP
let's open it open MLP dot SVG and here
we have it so we have our first input
which is mapped to a linear then we have
at an age another linear at an age at
last linear and enrich the output of our
model okay so let's try to forward now
in an input to both networks so we can
write x equal torch dot ran and 20
because our first linear was from 20 to
10 and then we can see net which is our
sequential network we can do net forward
x and we get a point eighteen fifty six
and then we can try g net forward x and
we have the same exact result let's say
we would like to check the content of
the node number four so I'd like to
usually I write this handy function so I
usually write a function perhaps you can
do it another way but I'm not aware at
the moment so let's add one more
function get node specifically my ID so
I have for whatever and node in my peers
self for word nodes
please do if n dot IT its equal my ID
then return and data module and and four
of the four and then we do return needle
if you didn't find anything and then we
can end so now if I get my Jeanette and
I ask get note number for that is the
one we were looking for I get my dinner
and if you would like to check inside of
as usually as usual we can use the curly
bracket so can we make it more easy yes
we can
so let's see how to do so let's forget
actually we had this network here let's
make a new network from scratch but with
the same architecture but without
sharing specifically the same modules so
I'm gonna write my first hander I'm
gonna call it G one and I can write it
this way it's a linear that goes from is
the first pinger so twenty to ten and
then I write my second module my second
handle which is going to be G one
attached to n n attached to an end of
linear the boss from 10 to 10 and then
attached to alternates again and then
finally our last linear which goes from
10 to 1 and then I can put those two
guys together and we have our MLP equal
n n dot G module of my first G 1 which
is the first hinder and then the G 2
which is the second handle let's
visualize this guy for fun so let's go
on graph dot we have MLP forward graph
and we call it a millipede - and I made
an M&P - and then we can do we can open
we can open it
MLP to dot SP and here it is so our
second module is basically the same as
the first one and it's the way to write
this down it's much more compact respect
to the classical sequential is this the
only advantage no actually the fun just
starts now we can make some interesting
architecture by using the NN graph
package for example let's play with this
let's call it input it's gonna be my
first hand identity so then we have our
first block I'm gonna call it input 2
which attached to which I attached a
linear 10 to 20 and then a non-linearity
my l2 instead is going to be the input
here above attached to the output of l1
together with joint and then we say make
a linear so since my input was 10 and
the output of the first one is 20 we
will be going from 30 and let's say 2 we
go to 16
and then we apply a non-linear function
then we have our third clock it's let's
say the concatenation of the first block
and the second block
so we do again join table on the first
dimension here we can make a linear
solid cl1
was 20 and l2 is 60 so 20 plus 60 is
going to be from 18 here and let's go on
to scalar and then we apply a nonlinear
function and let's make our G motor so
the first one is going to be my input
and then the output just the l3 let's
see how this looks like so we can go
graph dot G dot forward graph let's call
it fancy fancy
and open the fancy dot SVG and here we
have it so let's see we start here with
the initial node then we have an
identity which is sent to two part
because we see here that the input was
used in vs. by l1 and also by l2 so on
the left hand side we have l1 and on the
the node 10 we have L 2 then L 2 is
processing and then we have node 4 which
is going to be including our l3 which
gets the input from the output of l1 and
l2 and fact here we get the last 10 H
now node 5 from l1 and the output of the
l2 which is again another tonnage which
is going to be sent to the node 4 which
is joining the tables and performing the
last linear now that we have sharpened
our weapons we can start tackling the
RNN so let's say we have the
dimensionality of the input it's equal
to 3 then we have the output
dimensionality it's equal 1
the dimensionality of the hidden layers
it's five then the number of hidden
layers let's say we have two hidden
layers in this case and then we have the
length of the longest sequence if you
would like to tackle let's say it's four
we are going to make now our model and
we have to define some specific inputs
and some specific outputs so to this
model I will have three inputs the first
one is going to be called X X I'm going
to explain some of what it is the second
one it's going to be called
age age number one and then the third
one is going to be age age number 2 X X
is a terminal for my input sequence so
this one is going to be going here then
the H h1 it's simply the age of the
first layer at time t minus 1 and this
one is fed here and the last one is
going to be the previous age for the
layer number two at the time t minus 1
and this one is going to be fed here on
the right hand side we are going to be
producing the first hidden
representation which is going to be
called simply h1 we output also the
second representation in order to be
able to use it for the following steps
in the sequence and then we provide the
YY output and this will correspond to my
age first layer of time T
the second one instead it's gonna be a
gate for my age second layer again time
interval T and then the last one I will
have my prediction sequence so let's
make now this central block with all
these terminals we are going to have
three input and three output X X equal
and an identity H h1 equal
- and n dot identity and then h h2 equal
- and an identity then we have the h1 is
going to be X X and H h1 the previous
state which I divide join table on the
first dimensionality then I have a
dinner layer going from n plus T to D
and then we have a tonnage nonlinearity
then here we have our h1 concatenated to
h h2 then we have the same join table 1
and n dot linear which goes from two
times T D no RIT and then the output is
going to be H 2 connected to
linear d ok and then we have an end
damage
so our RN n is going to be equal to n n
dot G module of X X H h1 h h2 and is
output with sin h1 h2 + YY sweet let's
try to forward a note here so we're
gonna have my X is gonna be torch round
of n and then we are gonna have that H 0
the initial state we said it's gonna be
zeros of the dimensional so if I do RN n
forward and then I send my X 0 state and
a 0 state because it's for example the
first term first element of my sequence
we are going to have the output which is
going to be current h1 h2 and then the Y
output let's visualize this network so
we can do dot RN n dot for Warcraft and
then we have our name and our name
and we can obtain our n n dot SVG and we
have our n n written in four lines of
code so let's see what's happening here
so at the beginning we have our input
that is going to be the three sensors
the three-dimensional input X and the to
implead States then surf items 4 5 the
node 10 we have the identity connected
to DX not eleven we have the identity
connected to the previous state for the
h1 and not 12 we are going to have the
identity which is connected to the
previous state of the second hidden
layer then we compute a concatenation of
the input and the previous state in the
node 8 and then we perform the linear
mapping to which we apply then the
teenage non-linearity then in the output
its first send down for the next
iteration and also it's sent to the next
block which is going to be having an
input which is tensor fight and survive
because both of them are the
dimensionality of D we join the table so
we are going to have 10 which shoots to
5 and then below we have at an age and
then age 2 is sent again out for the
next state for the next iteration
finally we have the last linear and
nonlinear editing and also that is going
to be sent out in fact as we saw here
when we send forward our ex and to null
States we are going to have the output
which is going to be the first hidden
layer the second hidden layer and then
the final output perhaps was a bit
confusing so let's make it easier by
using a script we have developed in our
lab RN n equal require RN n you can get
this file if you go on the ELab github
account find the torch seven demos and
then on the RN n training sample you can
find the RN n dot lua script i'm using
right now so we can say time net and net
equal RN n dot get model and we send the
dimensionality of the input and the
dimensionality of the hidden states D
the number of hidden layers which is
going to be true in this case deem K
dimensionality of the output and the
axial length of the sequence done finish
so one line and we have
everything we written before in the
previous screen plus also on the cloning
and with parameter sharing across time
steps
in order to perform a training which we
will go across in the next lesson so
let's see what we have generated here so
we go graph dot dot net dot forward
graph and we have net net we can open
net dot SVG so we have the input then
node number ten and it's sent together
with a node 11 previous state for the
first hidden value it sent a number node
number eight where we have them joined
together then they go through a linear
mapping and then we apply the
non-linearity tonnage we can see the
linear goes from 8 to 5 because the
lower case n was set to 3 whereas T was
set to 5 so 5 plus 3 equal 8 so we go
from 8 to 5 then below we are gonna have
the depleted state for the second hidden
layer it's sent down to the joined table
which is joined together with the state
of the first hidden layer we have a
concatenation linear which goes from 10
to 5 because we have 2 times T then we
applied the nonlinear function and then
we have last linear mapping followed by
the last non-linearity let's go back
here and let's see instead what is time
net this time so let's go and type graph
dot dot time net forward graph and we
call it time net and time net that we
can open it
and here we have the monster let's try
to understand what's going on
I screwed up okay we we see that in blue
we have our NN module one iron in module
2 iron in module 3 in iron in module 4
we have replicated our same very same
model four times in order to catch
events that are at most four apart then
the first learning module it's sent with
the input number one the second one with
the input number two the third one with
the input number three and the fourth
with the last fourth input more over the
previous states of layer 1 note 13 and
14 are sent into the iron and model
number number one then the state from
the model number one it's sent forward
to the model number two which state is
in forward to number three and which
state it's sent forward to model number
four at the end we have the last two
states which are sent out to the network
moreover to the output of the network we
had the prediction from the iron and
model 1 the prediction from the model
RNN model to the prediction from the RNN
model number 3 and the final prediction
from the model RNN model for in this way
we can easily train our system by
providing the initial state and the
sequence of input and then we are going
to get in output a sequence of
prediction and the final state of the
system then we will simply compute the
loss function with respect to the
prediction and a nd labels and embed
propagate the the gradient of the loss
function with respect to each and every
prediction back into the networks and
this is handled automatically by the NN
graph package in the next video we will
go through the complete training script
to have a working example on how such a
system can be
trained