Practical 2.1 – NN backward Video Lecture - Torch A Practical Hands

in this video we will go through the
derivation of the back propagation
algorithm we will start with a review on
the forward propagation then we will
introduce the training data and training
labels for a supervised training
framework the training will help as
objective the minimization of a loss
function we will introduce therefore a
local error through the network we will
start by defining an output error and
then we generalized the error to a
nonspecific
air layer we will also review briefly
how to compute the Jacobian of the
composite of functions finally we see
how to compute the parameters gradient
and therefore we review the equations
for the back propagation algorithm which
can be used for stochastic and
mini-batch gradient descent for
minimizing the loss function that we
have introduced we start with an
overview of the equations we have
already seen so far X is going to be the
vector of components X 1 X 2 up to X n
which belongs to our size of layer 1
then we have dead a of layer L plus 1
it's equal to our nonlinear function in
this case the sigmoid of a quantity they
recall weighted input and we write Z of
layer l plus 1 which is simply sigmoid
off of our mappings for the layer L
which is mapping the activations of
layer L to the L plus 1 where and here
it's equal to 1 to up to n minus 1 and a
hat is equal to the vector a with the
bias on top a L plus 1 will be of size s
L plus 1 finally we have the H the
hypothesis based on the parameter theta
capital theta applied to our input X its
equal to the activation of the last
layer capital L which therefore belongs
to our s capital N or we can also say R
of K capital K our theta J is the
mapping that is mapping the layer J to
the J plus 1 and therefore has
dimensionality size of the layer J plus
1
and gets as many input as the current
layer so SJ plus one is which is the
barriers and our capital theta is the
set of all the mappings for all the
layers which is mapping the layer the
layer on 2 to the 2 to 3 up to the last
one which is mapping the layer capital L
minus 1 to define a layer capital L to
train our system we provide some
examples here we have a capital X which
is providing the actual data which is
organized in this way so X is the matrix
of row vectors x1 which is the first
example then we are gonna have x2 which
is the second example we have again x3
up to the last one which is going to be
X
and why is the matrix of labels which
has m rows like X and then we have the
first label by one second label white
top third label y3 up to the last one
why M the number of columns of M as we
saw it's K because we saw that H of
theta of X it's belong to RK and we have
that of course we have M rows in a
similar way of course we have M rows for
the X matrix and the number of columns
is going to be equal and let's write
this in a more extended way so X can be
written as a table basically so here we
have detector 1 X 1 here we have the
vector X 2 up to the last one vector xn
and therefore we have here X 1 of first
sample then we have X 2 of first sample
up to X and offer sample and saying
below so we have first feature sample
umberto second feature sample number 2
up to the last feature still for the
sample number two and then we can go
down to the last one which is gonna be
the M example feature number one an
example feature number two and up to the
last one feature number and of the M
example we are going to have something
similar for y as well so Y can be
written as y 1 y 2 up to y K and then
you're gonna have first sample label
number 1 first sample table number 2 and
broke a and so on
the last one so we have pi am first
label why am ii label
ym k label if k it's equal 1 then the
matrix y is going to be exactly just the
vector Y we can now define a loss
function which is evaluating how far is
our prediction from the actual label our
loss function egg which is a function of
the parameters only can be expressed as
an average over all the all the errors
of the given example so I equal 1 to n
we provide to our neural network all the
M examples and we can compute the
average and this is gonna be our loss
function here we are going to drop the
index I because otherwise it becomes a
little bit messy but this one basically
expressed the output error of the
network the error which is a function of
the output of the network will tell us
how far we are from the true labels in
an example a standard example of error
function could be the mean square error
which is equal to 1/2 the one half term
is going to be helpful
in later steps of the difference between
Y which is for the I example but we said
we are gonna exclude this for
moment - the output of our network of
which we have fed the example I and this
one it's simply one-half summation over
K of Y K - so we have that this one can
be also written as our a of layer L so
YK minus a component K of layer and
square it this is the considered for the
example I but we are going to take away
this notation otherwise it becomes
really unbearable we want to keep in
mind that e it's a function on the
output of our network where x and y are
fixed and theta can be wired
our goal would be to find the rate of
variation of our error when changing a
given parameter theta IJ of layer ad in
this way we can move from the evaluation
of the error done in theta towards a
point which is going down the slope of
the error based on the parameter space
in this way if we have our cost function
and we are here we can try to go towards
the minima of our cost function
let's introduce now the local error if
we have our weighted input Z I of layer
L and we replace it with Z I so itself
plus a small variation so Delta Z I
layer and then we expect the error in to
change from E to e plus what the
variation rate of the error with respect
to this particular way to the input
multiply of course the variation of the
perturbation here we can see two
different cases the first case we
consider the absolute value of the
variation of the error
- did specific way to the input to be
greater than zero and therefore we know
that some variations in the way to the
input that I can bring some improvement
instead when we have when the absolute
value of the variation of the arrow with
respect to the specific way to the input
is nearly zero we have that this neuron
is near optimal and there are four
variations in the specific weighted
input won't bring any relevant influence
on the final error e we see that the
partial derivative of the error with
respect to the specific weighted input
can be considered as a term of error so
if this error is greater than zero then
we can still improve our cost function
if the error is 0 then we cannot we can
no longer change much so we define Delta
I of layer L as the partial derivative
of the error with respect to the
specific weighted input of layer l
component I and this is the error at
I've neuron and layer and let's compute
therefore the output error Delta I of
layer capital n so we can simply write
the partial derivative of the e with
respect to the a of last layer component
I and then we have the I the a component
I off last layer with respect to the Z I
component I of last layer which is equal
to what
so the first one is the partial
derivative over our error function with
respect to the output of the network
multiplied by simply the derivative of
our sigmoid for the component I of the
weighted input of the layer capital n
last layer and let's see what these two
elements are because we already know
basically everything so let's say we
consider our error e equal 1/2 the
summation over K of Y K - a K of last
layer squared well therefore we have 1/2
x 2 and the summation disappears because
the all the components are going to be 0
but the is component which is going to
be there for a I and minus y I
so this one here if we use the sum of
square errors it's simply a I L - weii
and here we are gonna have the
derivative of the sigmoid I'm gonna make
let's make ads on if we have sigmoid of
Z which is equal to 1 plus X of minus
that minus 1 and we simply have that the
derivative of the sigmoid is going to be
so the minus 1 comes down here at 1 plus
X - Ted - 2 multiplied by X minus side
by x minus 1 now we can multiply and we
can write this 1 1 plus X minus side
squared times 1 plus X - Dead and minus
1 and therefore we have these two
simplifies and they become
one plus X minus said minus 1 and then
minus 1 plus X minus read and I stop and
we can easily recognize this one is to
be the sigmoid of step and this other
one Sigma square of Z so we can write
sigmoid outside come on which is
multiplying wine- sigmoid of that and
also we have that Sigma top set it's
simply a so this is a which is
multiplied which multiplies one - hey so
in this case we can simply write this
one it's simply a I of layer ad which
multiplies one - a I lay your head for
sake of generality we can keep our more
compact form where we still have the
partial derivative of the error and the
derivative of T nonlinear function just
in case you would like later on to
change the error function or the
non-linearity finally we can write the
vectorial form of the Delta the error at
layer at the last layer which is going
to be equal to the gradient with respect
of the input of the
error function which is component-wise x
the derivative of the nonlinear function
utilized again in our specific case this
component would simply be the last
activation minus the label and this
component would be last element
component-wise multiplied 1 minus a of L
we would like now to compute the error
at any L layer of the network to do so
we need to express our error at layer L
as function of the error at the
following layer we have written the
error at layer L as the composition of
the error at a layer n and the error at
layer l plus 1 and therefore we had to
compute the partial derivative let's
refresh how to compute the Jacobian of a
composite function let's say we have a
function G which goes from R to R and
differentiable in U 0
and then we have a function f which goes
from our end and which is scalar field
so I goes to R and which is also
differentiable in X 0 which is equal to
G of U 0 we have that H which is a
composition of F and G which goes from R
to R is the driveable in U 0 and the
derivative of H nu 0 it's equal the
Jacobian of DF in X 0 multiplied by the
Jacobian of G in u 0 and what is this
this is the classical matrix
multiplication but more specifically
since F is a scalar field we had the
Jacobian it's simply the row vector the
F the x1 in x0
they have the X n in X 0 multiplied by
the derivative combined twice of G so we
have G 1 prime u 0 down to the last one
G n Prime user which is equal to a
scalar product of the gradient of the
first function computed in X 0 and the
derivative of the second one so we have
here the current error at layer L it's
been expressed as a function of the
error at the following layer we have
that the Delta I of layer L
it's simply de respect to the red
component I of layer l and delta j of
their life l plus 1 is going to be equal
to de and here we have the set J of
layer l plus 1 we can also write this
one is discolored product of the two
jacobians so it's decimation of all the
components some on J of D in the set
J of Halloween layer multiplied by D
said J how the following layer with the
current layer we actually know what is
this first part which is actually Delta
J of layer n plus 1 we just have to find
out what this second part is which is
not that complicated so this one is the
partial derivative of the weighted input
at layer l plus 1 component J with
respect to the weighted input component
I of the layer and so as we said partial
derivative of the weighted input
component J of layer plus 1 with respect
to the weighted input component high of
layer L to perform this partial
derivative we can simply write Z J so
the the J component of the weighted
input layer l plus 1 is function of the
current layer so this one is going to be
equal to the summation over K of the
parameter theta J ek of layer add
multiply by a cut layer and ok
this one can be simply written as
summation over K and theta J ok now your
end of the sigmoid of z ok
by your end and then if we perform the
partial derivative for respect with Z I
we simply have the decimation goes away
and the output is going to be the
argument argument of the summation where
K is said to Y so simply this one is
going to be theta J I of layer add the
derivative of the non-linearity of the I
lie here L finally we saw from the
previous slide we had the error Delta
component I at layer L it can be
expressed as the summation over J of the
component J's of the error at the layer
plus n plus 1 multiplied with the
partial derivative of the weighted input
component J with respect to the
component I of the weighted input at
layer add and we can simply substitute
substitute and we have that this is
equal to summation over J of theta J I
layer egg which is multiplying the Delta
J of L plus 1 and the final variable T
for the sigmoid said I layer L
finally we can write the vectorial form
so given that the component I of the
error at layer L it's equal the
summation of the parameter J I not I J
of layer L multiplied by the error
component J and layer l plus 1
multiplied by the derivative of the
nonlinear function of the same component
can be written in a vectorial form as
the matrix theta at layer head
transposed which multiplies the error at
the following layer and then we have
component multiplied by the derivative
of the nonlinear function under your
head for the weighted input at layer n
our last effort will be done to compute
the partial derivative of the error with
respect to the specific parameter now
that we have all the tools and amenities
over this slide we can see that the
final equations are pretty easy to write
so we have the partial derivative is
going to be equal to the partial
derivative with respect to the following
layer weighted summation Delta set I of
following layer multiplied by derivative
of Z I of the following layer
we respect to the parameter IJ of layer
n for every I and J and we know that
this one it's simply our error component
I of layer l plus 1 and this guy over
here it's simply a J component of the
activation at layer n we computed this
one from the forward pass and we have
computed this one on the left hand side
from the backward pass
finally we can write the last formula
which is writing the total partial
derivative of the error with respect to
the parameter at layer L this can be
written as a at layer L which multiplies
the error at layer l plus 1 transposed
so we have the first one it's a column
vector and the second one is going to be
a row vector so we can easily tell that
the product is going to be the matrix
which has the same dimensionality of
capital theta of layer L and finally we
have the list of five equations we are
required to compute in order to perform
back propagation so at the beginning as
first that we have the first activation
it's equal to the input to which we
additionally add the extra plus one
there's a bias term then a second point
we compute every other weighted input at
layer L plus 1 which is equal to the
mapping of layer L applied to the
activation of layer L then we have the
deactivation it's simply the
non-linearity applied to the weighted
input and then again we have the a hat
is going to be simply the a with +1 bias
additional term is a third point we
compute the error at the last layer so
we have the gradient with respect to the
input to the error or basically to the
with respect to the output of our
network which is then multiplied
component-wise with the derivative of
the nonlinear function utilized
then we can back propagate the error
through the network by utilizing this
formula where we see the transposition
of the weight matrix multiplying the
error at the following layer and then
component wise multiplied by the
derivative of the activation function we
can see here that we haven't specified
any particular error function or any
activation function so based on your
choice of error or activation function
you can simply modify this formula to
adapt to your specific training finally
we can compute the variation of the
error with respect with any of the
parameters used in the network is simply
the product of the activation of a
specific layer L multiplied the
transpose of the error at the following
layer finally if you would like to
perform gradient descent we can choose
between the stochastic gradient descent
so stochastic gradient descent which is
simply saying that the current parameter
should be updated with the following
rule so equal to the current parameters
- learning rate beta which is
multiplying the activation at layer L
and the error at layer L plus 1 as we
just saw or if we instead would like to
perform a batch gradient descent or mini
batch
we simply have that the update rule is
going to be that theta n it's equal to
theta and minus the learning rate and
the average of our errors so our updates
as well
for the example I