PyTorch Lecture 03: Gradient Descent Video Lecture | Pytorch: A Complete Guide - AI & ML

in this lecture we are going to use
gradient descent algorithm in our last
lectures we introduced the loss function
for a given W we can compute errors
using MSE manners also we show that loss
graph something like this when you talk
about the training what we do is that we
want to identify W value that minimize
the loss in the graph also we can think
of in this way so let's say this is our
true graph so true W value which we
don't know and we can start any W random
value and then minimizing this loss
means that by changing this W we want to
make these graphs close to our true so
that our prediction is as good as its
true real values so in summary we can
say it'll earning what is the learning
or training is finding W that minimize
the loss so we can express this one
using just one mathematical notation is
called hard max so it means that we want
to identify W the argument that minimize
this so of course in this case is it so
simple so intuitive just look at this
graph we can identify what is the W take
give us minimum loss for imagine that we
are using thousands even millions of
weights in these cases is impossible to
compare to this W manually so we have to
develop some systematic way that
automatically identified this w this
algorithm is called the gradient descent
algorithm so idea is quite simple
suppose we have this kind of loss graph
since if we don't know what is the write
W we can just start any random points
suppose our starting point tested here
then we have to decide that this is the
right place or we have to move in or
move out we have to move in we have them
about how we're gonna
decide the best way to decide here is
that we can compute the gradient of this
point where where the W is and the based
on this gradient we can decide either
moving in or moving out
for example this gradient is closed then
we're going to move in so my W becomes
negative if we are standing here our
gradient becomes minus then W will go to
plus so this is Paraclete a simple idea
and then we can express this one using
mathematic notations as well so here is
how we're going to update this is our
gradient and then what we're going to do
is that we do W minus this gradient
because the gradient is plus we got a
move in W is getting smaller is gradient
as - is getting bigger so that's why we
use - in here and then how much you're
gonna move in it will be defined by this
alpha where your causal usually running
late so it's usually small value 0 0.1
or something so based on this gradient
we move in by this alpha and then and
then we can compute the gradient again
and then compute the gradient again and
again and again so once we get here
usually the gradient at this point is
almost zero so we don't really move too
much if you start here it's basically
the same thing so this is a systematic
way to identify collabo minimum and in
mathematical form we can just express in
using two equations so our loss is
defined something like that once we
define the loss then we can define how
we're going to update our weights
it's basically w nu W is previous W
minus alpha which is related with the
gradient which can be computed with this
derivative then the question is how can
you compute
in our case the equation of laws is
rather simple so that we can compute
this one but if you don't remember how
to compute the domore if there are many
tools many webpage they can help you for
example you can go to this derivative
calculator that Ned and just provide
your loss function here and then you
want to ask what is the derivative and
then it will automatically compute and
give us the answer which is 2x X W minus
y will be the derivative of this area
and then we can just use you can plug in
this one and we can really implement it
so let me just summarize it so W now
will be updated using this equation
alpha plus 2x X W minus y this is our
gradient so let's get into
implementation so first of all like
people we're gonna define our X&Y data
and our weight can start from any random
value maybe you can start from three
point zero five zero minus three it
depends on your guess and the like
previously we're going to define our for
network which is a linear x and y head
and then we cannot define the loss based
on our definition and then importantly
we can also define our gradient 41 cuban
X&Y
we can compute using this equation
that's exact implementation of this
equation and the ones we have this the
forward and the loss an ingredient what
we can do is that we can update our
weight the key idea of our update is
this
so this is our learning rate and this is
our gradient computation using this
function and that this is how we can
update that's it
so in this epoch decide how many loops
we can ago and then we're gonna except x
and y value
each time we can compete this gradients
and they be update his W and then we
move on and then at each time you're
gonna print out W and loss then we're
gonna ask we're gonna test our system be
poor training and at the training this
is the result the beginning we ask what
will be the points if I study for hours
it's sort of a completely rendom along
value but as time goes on as the
progress moving then you see the lost
value is getting smaller and smaller and
then the W value is getting to towards
to two we know that two is the right W
value an entertaining risk to those
machine learning system that oh I start
the powers or we might score and then it
will predict almost eight points which
is correct so in this lecture we
introduced a very simple algorithm
called the gradient descent but it's
very simple but very efficient and very
powerful to automatically identify W
which minimized our loss means that we
trained our models using this algorithm
before we move on to next lectures I
provide a few exercise first let's say
we have a slightly modified model in
these cases we have a two which W 1 and
W 2 so we have to compare the two
different derivatives and update them
separately can you complete a pair
gradient first and there once we compute
the gradient
can you please implement is to exactly
same thing like machine learning
automatically in the next lecture we're
going to talk about how I'm gonna
compute this gradient automatically
using graph it's called back propagation
algorithm