Convolutional Neural Networks [Part 5] -Translation aftermath Video Lecture

hi explorers and welcome back to another
convolution neural network series
video and last time we talked about the
softmax layer and the layer before the
softmax layer is this at this
fully connected layer and it sees it in
the pool no it's not in pool it's not
normalizer i guess might not have that one
open of course it should be in is it in
the drop yeah so this is the fully
connected layer and that could be the
reason why the number of classes isn't
used in the softmax layer because it's
fully connected layer build number of
outputs that softmax layer needs to
operate so that could be the
reason so in this function we have here
we set output the depth to the number of
neurons and then we have a number of
inputs and we have a flat structure we
don't have any volume it's just an a row
of values and we have some bias here
this value is taken and set to 0.1 and
that comes from the logic in the network
from the creation here so you have
depending on which layer comes first and
in what order you create this you should
either have a bias of 0.1 or 0 and in my
case we have this relu activation then we
will get a 0.1 value as a bias so that's
why I add that but in this case let's
see here on the convolution and if we
have an activation but I think on the
fully connected connected layer we don't
have any activation so in this case it
should actually be 0.0 so we
found a bug they're good so let's
continue here we were looking at this
fully connected pretty much translated
everything these are optional so I removed
those they aren't used in this layer at
all so I why have them then we have some
computed values and this layer type is
used either so remove that i set the
bias create the filters and create the
bias and set the output we have seen
this before the forward layer very easy
as well we just have this for loop here
that goes through the depths and the
number of inputs and we calculate a
filter here we get to filter weights and
add the filter weight to the input weights
on the different positions
that's pretty much it we have seen this
function before in other
implementations so this is just a simple
translation of this JavaScript and here
we have the backpropagation where we
create a gradient for a new layer for
the input layer and also for this tfi
where we set the gradient and I guess
that yeah that's the filters gradient
that we change so I don't understand
this code yet but hopefully when we have
translated and then look into how these
different things work we and we changed
names on it could become clear for us
I'm learning by writing code getting it
to work and then go backward and
actually translate the different things
so I get a better picture of how it
works and this translation of the
backpropagation we have seen before so
that's not interesting to look more at
we can look at the pooling layer as well
if there's anything interesting here we
have some optional values that is not
some are used some not so i removed
stride which is not used in this pooling
layer we have two stores of switches so
these are int positions of values that
we want to switch out I guess
and yeah we store the output size as
well we on our forward loop we will
calculate these winx and winy and
those are the switches that we put here
and on the after this implementation and
setting this new weight on the <
back propagation we will actually use
these arrays with switch x and switch
y to figure out where to put the
gradient on the back propagation so
that's how the pooling layer works the
little things that I have found out of
the pooling layer so far we will
probably get more about that later on
see here normalization was this local
normalization layer that weird name that
i couldn't remember and I get it on the
right side there we go close this up
again see if we have any specific things
to talk about yeah these are interesting
I actually when I implemented this found
out that i needed a K, N, alpha and a
beta and those were never mentioned in
the JavaScript code I looked a lot for
them and I couldn't find any values
anywhere that they should set these
values anywhere as we see here they have
some standard values but no those are
not k is set to 0 here
okay and beta 1 and beta 2 I set but we
actually have the input of K, N,
alpha and beta and later on in the
implementation here we can see that we
we use beta 1 beta 2 see if i can
find alpha ok don't see alpha
that's weird I could have sworn that I had
alpha used in the code here weird anyway
I actually look these up in order to use
them no I look into trainer oh that's
why here we see that we had required
values k & alpha beta now we are back in
business again and these required fields
are never set I search through all the
github code for the name alpha and alpha
is never mentioned so I don't know where
they set these values in their code but
i found this research paper here for the
image net and they talked about these
values and what they are they are
hyperparameters that you could set two
different values in order to get this
normalization layer to work and the
suggested values where 2 5 10 up to 10
up
I don't really know how to say that in
English but yeah 0.0001 and then you
have beta2 0.75 so these are
just constant values that you can fiddle
with back and forth those will never
mention in the JavaScript code but I
set them from this paper and hope
that that's good enough and then we have
this in out and scache that we use we
define the computed values in this case
we should also set the output values
right I always do that so why not do it
here let's add that that's a bug I find
I find a few bugs here while when we go
through this code that's good one of the
reason to do this to look closely at
that so now we have set that we have
some output definition then again we get
the output values from the definition so
that's not needed and the forward loop
here is a bit strange in in the
javascript code we do the first set up here
that's pretty much the same as we are
here we go through the full volume
nothing strange about that we get the
first weight and then we will calculate
this den value so that's what we have
here and then we will time that will
alpha divided by N and add k and set
that to our cache
and we will do a power of beta and set
that value den value divided by ai
which is our input vector and I can't
see really that we use this ai value
anywhere else so this is some some value
to get this normalized very strange math
here but it's pretty much the same as we
have over here and I get this s_cache
makes the value no we will save the
value so we can use it in the back
propagation I guess yeah so we always
see that we get is that value back so
here will it create the input vector we
have an output act and then we just do
work on the input the output is never
used that's a bit weird that's because i
have done got nope so do we use a
doesn't look like it you can have any a
dot nope so a is never used here so can
remove that in the backward propagation
good to know good to know and we run
this through and see it run it through
the full volume again and we will
calculate the normalization gradient
using this s_cache
and this math is also something that
I would look have to look into more
later on it just translated back or to
java so let's see here do we
have anything in the network that we
want to mention yeah so this create we
have this function here make layers I'm
not using that at all at the moment
so this make layers is disregarded I will
set up my full network by hand then we
have some helper functions here to do
the forward propagation pretty easy cost
loss never used so far backward
propagation just run through the
different layers but we do something
different with the first layer because
or the last layer there we actually set
the correct value so that handle a bit
differently so in this case we require
that the last layer is a loss layer and
then we have this back propagation
result it just accumulates all the
difference back propagation results and
we have some prediction that we don't
use at the moment I guess that could be
interesting to know later on if we want
to see what value is actually correct
one so let's see here we have the
mystery that we have already talked
about the test class we saw earlier the
difference here is that I actually add
this mnist reader and I have a for loop
where I do the actual training
and I will print the current training
results I will see how much better the
actual
network will get over time that's the idea
at least let's see if we can come to
that later on at the moment it doesn't
work so I have some debugging to do the
training interface very simple as well
trained something get a training results
so not much to look at and this ADA
grade trainer is one of the trainers
specified here in order to do training
we need a lot of data values we need a
list of the different summations we
need a learning rate we need some decay
a batch sizes and so on and everything is set
in different values here in this trainer
here you set a few more values but that's
because other training methods need
require more or less data so depending
on what kind of trainer we get we needed
a different set of data so that's why
and this training function is also
translated straightforward we have this
is the forward loop with the time
calculation so we see how much time it
takes this is the backward loop also
with the time so both of these are
pretty much the same just translated
straight through and we see here that we
get a cost loss that we will have later
so we can see how much better or worse
we're doing and then we have this very
strange regression and guess that one of
the one of the loss or the
one of the loss function is regression
and in that case you could use a full
data block as your training data or your
result my case I'm not doing that at
the moment I've removed that logic
because it's confusing and then we will
go through and handle batch sizes I'm
not sure if this is needed that if you
want to run forward and backward
propagation over a number of different
images before you actually change your
gradients and improve so you will get
more data before you do changes I guess
or is this just to get some output could
be as well so here we will get the
backward propagation result we do a lot
of summation here then we will get all
the weight and get all the
learning decay parameters and go through
here and sum up even more data and then
we will return all a lot of different
information in this training result and
the training result is your setting this
information and then printing some of
this so it's not that fun either so I
think that I've talked extensively about
pretty much everything that I
implemented so far some of it is very
clear and some of it is still a mystery
for me
and I think when I go in and debug it
and when I find out that what works and
what doesn't I might get a clear picture
and when I get a clearer picture and
understand things more I'll try also to
do some changes in in the naming of
stuff so it will be easier to read and I
might change a few of the names that I
used so far but I'm very curious to see
what your thoughts are about this
project what names do you think are
appropriate to use in the different
blocks and so on and have you played
around with this before um yeah give me
your thoughts you're like your comments
everything I want you to do this with
you I want to learn about convolutional
neural network and every comment and
every suggestion and discussion that I
can have with you will make my process
better and I will learn a lot about
learn a lot from it as well so please go
in and converse with me I will answer
all the comments and I read all of them
so yeah I hope to see you in the comment
thread I hope to see you in the next
video I will go and debug this and try
to get it working and also do some
tweaks here and there and then I will do
another of these updates videos and talk
about how the new network is so I really
hope to see you in the next video