Practical 3.4 – CNN loss Video Lecture - Torch A Practical Hands On Tutorial

in this chapter we will see in greater
detail how the loss function will take
its own place in our learning system we
will start by drawing a basic diagram
where we can identify our model our
input data and the loss function which
can be expressed as function of the
output of the model and the label or as
a function of the parameters of the
model we will go forward and we will
revisit the mean square error loss
function then we will discuss how we can
use this mean square error in order to
perform recreation and how to perform
classification and we will see which are
the drawbacks of using MSE for
classification then in order to cope
with these limitations we will introduce
the sigmoid plus cross-entropy loss
function in order to be able to train on
classification tasks and eventually we
will see the softmax plus negative log
likelihood loss which is something
similar to the sigmoid plus
cross-entropy loss but with a additional
assumption which is that the output of
my network is going to be a conditional
class probability in define across the
classes at the end we will see how to
craft a totally new loss function for a
completely different task which is going
to be the task of recognizing different
identity given different positives of
faces so we have to introduce the
triplet embedding loss we will give the
motivations why we have to specify a new
loss function how we actually come up
with our loss function in terms of the
mathematical expression and finally how
we can get a working implementation of
this loss function so let's draw our
basic diagram where we can see where the
loss function is actually taking its own
space so we have a input X here and then
we have a label Y which is our in
put data then the X is going to be fed
into a block where I called which I call
network now we have here we fit the X
and then we have a we have a hypothesis
age based on the parameters of the
network theta on the specific input X we
failed then here we define our loss
function which is fed with our
prediction and our label in this case
that you can write here our loss
function is a function of the prediction
H of theta on the example X and our
current level y decent by definition
it's a scalar sometimes instead of using
the notation L lost based on the
prediction and the label since both this
input X and the y label are fixed we
usually write this as J function of the
network parameters and what we do with
gradient descent we basically compute
the derivative of J with respect to the
parameters and use this derivative in
order to minimize the loss function J by
using the update the theta becomes theta
minus learning rate ETA DJ in the theta
Q I should mention I should use a
transposition in the case I use the
jacobian mutation
but is not relevant the only loss
function we have seen so far is the MSC
or mean square error which I can write
in this way
so we have our J function of the
parameters of the model theta capital
theta was equal to 1 above 2m the
summation of 4i that goes from 1 to m
samples of specific errors square errors
so that's why we also call it error
function because in this case I specify
a error e and my J the loss function is
a average across all the errors which
are computed per sample and this one can
also be written as 1 above 2 M summation
4 pi that goes from 1 to M of the norm
of H theta of X example I - my label for
the example I square and if you'd like
to write this one in a vectorial
notation you simply would write 1 above
2 m summation from for I because from 1
to M of H theta X minus y transpose H
theta X minus o pi our last way to
summation said last layer vector it's
also called logits of my network and
each component would be allotted
if you would like to perform regression
we would have that our hypothesis on the
X would simply be our logit we don't
apply at the last sigmoid so basically
we don't apply the last nonlinear
function for example we could have
bounding boxes prediction and for
example age of theta could be four
numbers giving me the coordinates of the
bounding box or for example if I'd like
to learn the steering angle of a
self-driving car steering angle then it
would be a scalar so I could have my age
of X simply a scalar and so on otherwise
for classification we have that our age
of X its equal to a L which is the
sigmoid applied to the largest
and we saw that if you would like to use
it for multiple classes we can use a
wine which is simply a OneNote hand
coding so for example could be like 0 0
1 0 and it would be my label for a
specific example and my output here are
all belonging to the interval in this
case 0 1 the power of number of classes
in this case here number of classes is
it's equal 4 and in this case we have
that every a K represents the
probability of the class ok and all
these classes are independent in this
case so there is no whatsoever
Motorola's cluesive ET we lisa later how
we can estimate a different kind of
probability across all classes for the
world we can add a another term which is
going to be something like lambda above
2 m summations for J different from 0 of
all my weight I J player L so this
summation is done for all eyes all J's
but the bias the the term relative to
the bias term and for all layers so this
is die cap 3 times summation and of all
my weight squared so in this case we
introduced the regularization term
in order to fight over fifty or high
variance models using MSE mean square
error in order to perform some
classification create some problems
those problems are related to the speed
of convergence of the algorithm this is
because if we have our sample X and our
label Y which let's say it's equal to
one so actually it's a scalar and we
have that my network predicts a 1 so we
are going to have the error in this case
is going to be 1 over 2 times 0 which is
0 let's instead have H of X equals 0 so
it's completely wrong in this case we
have that the error will be 1/2 1 square
which is 1/2 unfortunately if we try to
learn this system although we are far
away from the correct solution the MSE
will contribute just for 1/2 and
moreover we have the dissolution here
this means that in our chart since we
have applied a sigmoid we are way down
this way
what does it mean it means that when we
compute the derivative
the derivative is basically zero and the
system will not do anything in order to
improve this solution so on the contrary
if the solution if our hypothesis would
have been let's say 1/2 then I would
have actually been able to correct the
solution to the correct value to 1 but
if we are like anything very very small
then my solution won't be able to move
any more because again the derivative
outside this little region with within
the y axis it's basically drops to a
very very small value can we fight this
problem yes we can we can fix this by
using a different loss function which
actually fights this problem by
increasing drastically the power with
which the update is going to be
performed and more specifically if we
work out the algebra it computes a
function that is 1 above this one so
they simplify away and basically we have
a factor of 1 instead of having a factor
that is going from 0 to 1/4 and 0 again
this loss function it's called
cross-entropy
let's see how we can come up with this
loss function in an intuitive way which
also will end up having a very nice
mathematical consequences and as I said
we'll be cancelling out this derivative
factor due to the sigmoid so summarizing
using a messing with sigmoid is terribly
slow and so we should never use mean
square error for performing
classification tasks therefore we're
going to introduce the cross entropy
loss function for speeding up
convergence so let's start with my
example where I have my label it's equal
1 and my hypothesis of the network after
applying the last sigmoid on the logit
it's outputting
a 1 so this is just okay right we are
fine we shouldn't be penalizing so my
loss function should be 0 this case my
loss
should be 0 instead let's say we said
label is 1 and I predict 0 this is bad
KO and therefore my loss function should
be let's say
tending to plus infinity I'm saying we
can say 10 to plus infinity because
actually my hypothesis will be never
able to be 0 it can tend to 0 because we
are applying a sigmoid which are defined
on an open range from 0 to 1 so let's
see how we can make this happen
so we have our
we have our output here H and here I
have my loss we can go from zero to one
the loss should be only positive and we
said we would like to be zero when we
are outputting one so we should be go
here and when we go we go to zero we
should be going up this way here so do
we know some function that works this
way well I think we do and it sounded
like this one which is of course is the
natural logarithm so which one is our
function is going to be simply the
opposite
and of course we don't care anymore we
don't care what's happening here because
we never are gonna be reaching that area
so we can have any - logarithm of H for
when the label is one so if my label is
one then I'm going to put apply this on
with the logarithm of my H of X and here
we have our first term what if y instead
it's zero so again we have two cases we
have the first case in which we output
actually zero and it's fine
or the case in which we output a totally
opposite solution which is so bad and in
this case the loss should be equal to
zero and in this case the loss should
tend to plus infinity let's draw this
one as well
here we have our one this is our age
here this is zero this is the loss and
we would like something that is going to
infinity
near the one and zero for zero well we
just use the one above and we flip it
and isn't it simply so we said it's the
one above so minus log we flip it minus
8 and then with some one so we actually
when we subtract 1 to H so we shift to
the right hand side by one if I would
have simply flipped with age I would
have had this function here and then by
subtracting 1 to H so becomes minus H
minus minus 1 so plus 1 I would I just
simply bring this chart one unit to the
right hand side so I can finish this
equation here below so we can have that
I can call this one actually entropy we
sum this one to what the case in which
we have 1 minus y so if Y is 0 this
becomes 1 and this becomes 0 so this one
basically goes away if Y is equal to 0
and this term is going to be actually a
nonzero term for y equals 0 so we
multiply this on with the logarithm of 1
minus H theta of X and actually we are
going to have here minus in front of
everything so we make this amount
positive and there we go and we have
made the assumption that age of
theta of X its equal the sigmoid of the
logit which belongs to the open interval
0 to 1 to the power of K so we don't
have to worry about the region infinity
because it actually never reached the
zero or t1 let's write it down come
fully so we have our j of the parameters
theta is going to be minus 1 over m
examples sum over my examples going from
1 to n some also over all the classes k
equal 1 to capital k and then we are
going to have y ok i example which
multiplies the hypothesis on the edge on
the x on the ice example kate component
plus the opposite so we are going to
have t 1 minus y k eyes example which
multiplies the logarithm of 1 minus H of
theta X is example as we said before we
are going to be using the assumption
that H of theta H theta of X its equal
the sigmoid of the logit
and which tell us that this belongs to
the open 0 1 ^ k so this expression here
J
it's called cross-entropy and basically
it's a measure of surprise basically if
the network is very wrong our surprise
would be very huge so if we expect to
have a label that is 1 and instead I get
a prediction that is very very towards 0
the logarithm of zero is going to be a
very very negative number of course
there is a minus here so we are gonna
have the the surprise for the specific
example is going to be a very huge
surprise instead if my prediction is
quite close to 1 the logarithm of one
it's going to be a very tiny number
negative number I have a minus here so
it's going to be a small positive number
and therefore my surprise it's gonna be
small in the same way on the right hand
side you're gonna have in case I expect
a zero the first part goes away in this
case we have if I predict a zero my
logarithm is gonna be a logarithm of
basically 1 which is going to be a small
number a negative number and in case
instead I predict a 1 in here I will
have a number which is very close to
zero and therefore the logarithm of a
number that is quite close to zero is
going to be a very very negative number
and again with a minus in front here it
turns out to be a very positive number
so it is connected to a very high
surprise in this way we fight the fact
that if we are way down the 1 of the
size of the sigmoid this surprise which
turns out to grow very quickly if we are
very wrong helps us going away from
plateaus with a derivative of the
nonlinear function are very tiny in
addition we can add a one more term like
in the case before
we have a lambda divided by 2m and then
we have a summation for all terms but
the one relative to the bias of the
parameters of the network for all layers
squared and this is called also weight
decay regularization finally I will
introduce the softmax plus negative log
likelihood criterion which actually is
going to be almost the same of our cross
entropy criterion so when someone speaks
about cross entropy it's better to be
sure in which kind of cross entropy he
or she might be speaking the sigmoid
plus cross entropy or the softmax plus
negative log likelihood so this one
actually is sigmoid plus percent repiy
but basically they perform the same
operation and if we perform the the
algebra you actually get to a similar
simplification of D and derivative of
the final nonlinear function which could
have helped slow down the training let's
introduce this softmax layer which is
very kind of popular in literature so we
have that softmax of my Z my logit so
the last weighted summation and this is
a vector it's going to be a vector which
component lowercase K is going to be
equal the exponential of the K component
of the logit
/ the summation for K that goes from the
first to the capital K component of all
the other all the actually components of
the logit so you can easily see that if
i have the exponential of one term
divided by the exponential the summation
of the exponential of all the other
terms this cannot be greater than one so
this one will always always be lower
than one moreover been an exponential
simply a positive function we also have
that this is true
so the softmax layer turns out to give
results which are always between the
range 0 and 1 so we can write simply
dissin belongs to the open 0 1 capital k
moreover we can say something else we
can say that if I perform the summation
of the old K so for K that goes from 1
to capital K of softmax of my logit
Keith's component this is of course
who's gonna be the first one plus the
second one plus the sake of third one so
this is simply going to be equal to one
so this is by definition
let's get by construction it's gonna be
equal to one so what can we see here
every terms of the softmax layer
corresponding to the logit gives us a
value between 0 & 1 and the summation of
all these values is 1 well this can be
thought as probability over all the
classes of this specific problem so if
you would like to predict what is the
likelihood of having one moodily
exclusive class within a specific sample
we can use this kind of layer because
again each and every output will
represent the likelihood of each and
every class to be the correct class for
the specific example for example this
has been used for the image net
competition where we have 1,000 classes
and we have to estimate which is the
correct label 1 out of 1,000 for a
specific given image therefore we can
use this soft max to have the
probability of each and every class
being the correct class for the specific
sample associated to this soft max layer
we have to use a negative log likelihood
loss function which is as follows so J
of theta is going to be minus 1 over m
summation across all my samples of
simply the natural logarithm of my
prediction
example I demand why this one says my
loss function for a given sample X I I'm
going to compute the natural logarithm
of the probability corresponding to the
correct class so if my correct class
it's let's say the fourth out of ten and
my age of theta is going to be
representing ten numbers from 0 to 1
which sums 2 to 1 then the this term
here corresponds to the fourth
probability to which a computing natural
logarithm and then I have a minus in
front and then I average out all this
negative log likelihoods furthermore
this can be rewritten
also as 1 over m summation the eyes go
from 1 to M we have logarithm of
summation for K they goes from 1 to
capital K of our Exponential's which
were the denominator of this softmax
layer and this is for the I example -
simply simply the logit
to which I select the white glass and
again this is for the eyes the example
and it's missing a parenthesis here I
simply computed the logarithm of the
exponential which gives us the same plea
the argument of the explanation
sometimes instead of using the softmax
another layer is use it which is called
log softmax is used for numerical
robustness this is because the
probabilities can easily turn down to
very small numbers let's say again if we
have the imagenet competition
we may have one thousand classes so we
start usually with a probability random
probability and if it's uniform of ten
to the power of minus three and can
easily go to smaller values and we have
problems with the Machine Epsilon and
instead if we use a log softmax all
these tiny tiny small numbers turn out
to be actually very very negative
numbers which are less prone to give
problems of annealing finally we can
always add our lambda over two n
summation for all terms but the bias of
our parameters by J the N squared
are there other loss functions yes there
are and they are made for specific cases
forgot for example let's consider this
problem we would like to be able to
identify different identities so we
would like perhaps to UM create a
network which is able to recognize faces
as belonging to the same
person why should this be difficult
let's see why here we can see the faces
of two different people share more
pixels across similar views then for the
same person but different views
this creates some troubles for the
network to learn actually what an
identity means because the input samples
are really really different although we
human we can easily recognize that these
faces for example in the first row
belong to the same person but we can do
so because we have a specific Center in
our brain which is dedicated for this
purpose neural network will struggle in
order to be able to actually make this
Association let's see how to make the
neural network life a bit easier we can
come up with a new framework for
training against identities we can
provide as we see here two examples of
the same identity for example we provide
the face of Liza through the Machine and
then we provide another face belonging
to a different subject like Bart then
the network will encode these input
samples into an embedding space which
for convergence practicality lives on
the unitary sphere we had the three
embeddings corresponding to the three
input images and we would like to have
embedding belonging to the same identity
coming closer together whereas the
embedding belonging to a different
identity we would like to push it away
in this way if we train on several
samples belonging to several identities
we will end up having multiple clusters
across our embedding space which are
defining different identities in this
way the network will be able to
recognize identity across different
poles of the face of the same subject so
let's try to write down these equations
so let's call this one as my input uncle
this is my input positive sample and
this is my input
negative sample which after going
through network will become an embedding
a here a positive here and a negative
here and basically what I'm writing here
is simply that a is defined as my
embedding of my uncle sample and so on
for the P and n so I have my uncle here
I have my positive example here and I
had this distance then I'd like to have
a extra guard here which I call which is
gonna be size of square root of alpha
and here I can have my negative so I'd
like the distance of my negative being
further away a square root of alpha with
respect to the distance of the positive
nd anchor so I can write this one as my
loss function J is going to be one over
m summation across my whole data set the
distance between my anchor and my
positive which I square
plus my alpha and then minus distance
between my anchor and my negative square
so in this case here I'd like to have
this element greater than this element I
have here three positive parts I have
one positive element a second positive
element and a third positive element and
if I'd like to minimize this function I
can simply make this term grow in order
to make it bigger than this positive
term to minimize this function would be
to minimize this distance here anyway
even though this first distance may
shrink down to zero we still have this
alpha here which imposed us to have a
distance between the anchor and the
negative that is at least as big as
alpha if we leave this equation is like
this cause function works pretty well
but it can become negative because this
term can grow in dimension a lot and can
overcome this first term and then we
start having a negative loss which is
something that we don't really want so
in order to do not make the loss become
negative we simply take the positive
part it's like a plane array go in this
way whenever these terms become bigger
than the summation of these two our loss
function becomes zero and then we don't
have anymore any contribution to the
learning I wrote an implementation of
such triplet embedding you can find it
on my github account here we can see the
criterion equal n n dot triplet
embedding criterion which parameter
alpha can be set as an argument and then
the loss function is defined over the
embedding a P and n equal one over the
number of sample summation or the
positive part of the summation of the
three parts the squared distance between
the anchor and the positive plus the
alpha minus the distance the squared
distance between the anchor and the
negative sample