Practical 3.3 – CNN training Video Lecture | Torch: A Practical Hands On Tutorial - AI & ML

in this chapter we will see how to use
the optimum package for training any
kind of neural network we will start by
mentioning the non convexity
optimization problem that neural
networks training are affected by then
we will see what are the advanced
optimizers that the optimal packing
offers us in order to be able to train
complex network we will see then how to
reformulate the training of a network is
an objective function optimization
problem we will see also some basic
optimization equations and at the end we
will see what is the optimum training
workflow and we will write a script
which is going to be training a network
on a CPU and a GPU using any kind of op
team advanced optimizer
the objective function used to train
neural networks is highly non convex
this means that finding the global
minimum is not a trivial task as you can
see from this animation by adequate fold
we can see how different optimizers
managed to escape a saddle point
training this network is not trivial
because the loss function is populated
by local minima and also there is who
claims that the major obstacle for the
training are instead saddle points you
can find more details in the description
below the optimum package from torch
provides a series of different
optimizers for example we can see
stochastic gradient descent averaged
stochastic gradient descent l-bfgs
conjugate gradient a delta at the grad
adam at a max fist an ester of
accelerated gradient method rmsprop our
prop and semis a description of these
algorithms is provided below
or also it can be reached from the
comment line so far we have seen that
this set theta is the collection of all
the parameters at different layers like
layer 1 layer 2 and so on until the last
layer and in the case of fully connected
layers or classical neural network we
have seen that these elements here are
matrices instead for a convolutional
layer we have that this element over
here it's up for the kernel so at the
end capital theta will be a collection
of very diverse objects in order to
apply more powerful optimization
techniques we have to convert this
collection of different sized object
into a one dimensional vector that we
are going to call theta lower case which
is a point in our D dimensional space
where D is the number of trainable
parameters so our D is also referred as
parameter space
therefore now when we compute the
propagation we aim to compute the
gradient of our loss function function
of the parameter theta with respect to
its parameter theta some of the most
common optimization techniques are the
following we will start with revising
gradient descent in this case we're
gonna have that our new theta is going
to be our previous theta to which we
subtract the gradient with respect to
the parameters of our loss function
computed on the whole dataset so I can
also write this one as theta minus ETA
gradient with respect to the parameters
of cost function of theta given D whole
input X design matrix X and the label
capital y then we can see the stochastic
in the same in this case we can write
that theta takes theta minus ETA
gradient with respect to the parameters
of our cost function function of
parameters computed only for the example
I with the label y mini-batch
gradient descent instead can be written
this way we have theta takes theta minus
e turn gradient with respect to the
parameters of the COS function and here
we are going to have a sub matrix X I
and Y I so if we think about X or Y
being the design matrix with n examples
and n features we may think that X I
it's as a subset of this dataset for
example this could be of height batch
batch size and and still here we have
the n features one more interesting
method which I think it's pretty easy
and I like to mention here is the
momentum which update is as follow we
have that velocity we have gamma
previous velocity plus ETA gradient with
respect to the parameters of J here I
don't specify if it's a mini batch if
it's stochastic if it's complete classic
batch gradient descent I just put a
generic J of theta and then we have that
theta becomes theta minus the velocity a
link to a very nice blog post by
Sebastian rudder it's included in the
description below I highly recommend to
go over it to have an overview of
several optimization techniques that are
used currently in deep learning
so how are we going to train with opting
first thing we are going to build our
dataset we are going to have our data
contained into the design matrix X and
then here we are going to have for
example our labels Y so this is my
training data set then I'm going to
build my network layer after layer which
is fed with our examples at the end here
I have to define my loss function to
which I feed my output of the network
and of course I will be feeding also my
labels finally from the gradient I'm
going to compute all the derivative of
the loss function with respect to each
and every parameter in the network is
finite step I will get finally from
every layer I'm going to get the
parameters which is my set of query
object and from this one actually I'm
going to make my parameter vector theta
and moreover I got to get these guys
here and I'm gonna have my gradient with
respect to the parameter theta of my
cost function so both those two elements
here are going to be a mono dimensional
vector which I will feed to my optimizer
together with the evaluation of my cost
function so that the optimizer can find
the best solution to minimize my loss
function given the parameters and the
derivatives of the loss function with
respect to the parameters themselves so
we are gonna right now our training
script which is going to be a general
training script and it can be used for
training any kind of network in this
case that we just simply train a classic
and neural network so let's open our
editor and we start with require and end
then I can actually set manual seat to
one two three four so that we have
consistent results across multiple runs
then I can find my model model click on
an end of this equation I will also
define the number of input features and
in this case I'm gonna plan to help a X
or kind of problem so I'm gonna have
just two input features and therefore
for the output I'm gonna just have one
output a scalar up there for my size of
the network is going to be and let's put
ten neurons in the hidden layers and
then our K so it's a three layer neural
network let's and therefore our first
linear going from the first
dimensionality to the second
dimensionality
then we are going to have model a
nonlinear function for example in this
case I would like to use that on each
linearity then let's add another linear
function another linear projection and
that's it for for my network we can
decide to visualize this the total by
typing print into torch and there we go
our neural network
so let's keep going and let's define the
close function we are going to define
now our loss function so we're gonna
have local dos equal to n n dot MSE
criterion and now we can build our data
set so let's say that our data set has
128 instances therefore I can build my
design matrix X equal torch dot double
double tensor of M rows and n columns I
would have written put a tensor if I
would have liked to run this on a GPU
instead then I can build my labels so
it's going to be porch dot double tensor
of just M elements and again here I
would have written good the tensor if I
would have liked to create a data set
this stays in the device memory and so
let's fill fill up this
intersect with some values so we can run
or I index I'd it goes from one with the
number of elements of examples let's
create our local X which is lowercase X
means the singular example is going to
be in porch don't run and often this is
this is a normal distribution to
elements of our normal distribution then
have let's have our Y which is going to
be an X or output so let's say if x1
times x2 it's greater than zero so it
means or both are positive or both are
negative then why it's equal minus 1
because it's X or otherwise if it's
lower than 0 means they have opposite
signs we are going to have a plus 1 now
we can copy this single samples into our
main design matrix so I'm gonna have my
I throw copy over the example I and this
works perfectly fine Buddha and the same
we are gonna have a y of I example it's
going to be equal in this case and Y
which is a scalar so I don't use the
copy and that's it and here we have
built our dataset now before starting
the training I'd like actually to
extract all the capital theta relative
to each layer in which in this case they
are matrices but it doesn't matter they
are just object with different sizes I
just like now to have a one-dimensional
tensor which contains all the parameters
of my network
so let's have actually a global variable
net which is going to be equal our local
model so that I can access outside this
script so let's call porch interactive
train so if I print net okay we're gonna
have our model if I do net one for
screen if I do net parameters then this
one returns me theta and the partial
derivative of the cost function with
respect to each and every of this object
with the Haitian notation let's see also
the dimensionality we said that the
first dimension we were going from two
elements to ten elements so here we see
that the first matrix theta one it's 10
by two of course should be 10 by 3 but
we have here as well our 10 elements
bias which actually in our notation it's
in front of the other part and the
second layer we're going to have 1 times
10 plus 1 so actually it's going to be 1
times 11 in our notation in the part
below we can see in the gradient of T
loss function with respect to DT these
objects have the same dimensionality
because again the Hessian notation has
been used we would like now to have
instead just two vectors the parameters
vector and the grad parameters vector or
theta and grad theta we can do this by
typing net get parameters so if we have
cat parameters instead we can see now
that it has output two vectors the first
one it's one dimensional vector of 41
elements which is our current weights
that are randomly initialized
and then we have the grad parameters
that are initially all zero because we
haven't run all right we haven't run the
backpropagation already okay so we can
go back to our script so we can have
their for local lowercase theta and grab
it down this should be lowercase but
doesn't really work in the coding so I
will write model get and this gives me
the one dimensional theta and the
gradient with the lot of the loss
function with respect to the theta then
we are also defining a optimum state
which is going to be all the parameters
that we are going to send to our
optimizer in this example I will use the
stochastic gradient descent so we can
check what are the options available by
typing required of the team we can do
question mark
opting dot sgt we press enter we can see
here what we can specify example ready
learning rate learning rate DK weight
decay weight decades momentum dampening
Nesterov so we can specify multiple
options in our case we are going to
specify that the learning rate is going
to be 0.15 for example and let's start
the training
so we require opting and then we are
going to write our epoch so let's say I
don't know from one
1,000 bucks so we write f you value in
theta basically f evaluated in theta
should return the scalar value of the
objective function computed in theta and
the derivative of this cost function
with respect to these parameters let's
provide this element so at the beginning
we are gonna zero on the ground
parameters of course so that we don't
overwrite a precursor site this is the
same of actually calling module 0 grab
doors
ok since we already picked them out I
just simply 0 the degrading this way
would be the same then I can define my
hypothesis of the network in the output
of the network which is equal in model
forward of my matrix X then I will
define a local J which is my loss
to which I forward my output of the
network and the labels let's even print
this value of changes so we can see how
how it performs are the training
performs so this is just for just for
debugging purpose then we have our local
DJ in th X is going to be our loss which
I to back part of our output X H of X
and why then we have molted and backward
to which I send my input X and the DJ
the age finally our function as I said
before has to return the scalar J and
the grad theta after with the finest
function I will simply called octave sgt
to which I send my F evaluated my
parameters theta and then the optimum
state I the found before and that's
pretty much it let's actually have again
a cobalt net so I can play with it
afterwards inquire how our model which
were below let's a torch interactive and
train so let's try our network forward
with George answer minus 1 minus 1 and
this gives us negative value minus 1 and
20 then let's try to have minus 1 1 and
this one gives us a positive result then
let's try plus 1 minus 1 and this also
gives us a positive results and then
let's try both positive and it's perfect
it uses a negative result something I'd
like to show as well it's like in this
case we train the network and we got his
final value design so let's save this
value
here for example I can do print reviews
previous J listen and let's see how the
cost function changes if instead of
having a three layer your network I have
a four layer neural network with the
same number of neurons so let's make
these one fine euros also here fine
yours and then I simply increase those
two got two guys so same number of
neurons simply one more layer
let's go below and let's see what
happens here and there we go from Oh
point eighteen we went down to 0.06 so
we can see now that stacking multiple
layers improve the capabilities of
abstraction of a network which can
converge to a better minima in the same
number with the same number of epochs
furthermore if I would have liked to use
other kind of network I would have
simply define a different model here the
field art now they require operations to
have the script running on a GPU so so
first of all you're gonna have just go
here and we are gonna have GPU so first
of all we have to say require
see you CUDA neural network then we have
mobile center CUDA we have lost same to
pull down
then we are gonna have X equal x it's
time to cool down
y equal Y same to cool down and that's
it now if you run this script instead of
running on my CPU it will be running on
the GPU off my machine
so let's write if it works on this Mac I
don't have a GPO so I will go on my
machine and let's so and then here we
have my file it's the same one so let's
try to run torch train bomb and then we
can also try to train without these
instructions and we saw you can see it
took some a bit more time