Convolutional Neural Networks [Part 4] -Translation aftermath Video Lecture

hello explorers and welcome back to
another convolution neural net
episode and I thought we should do a
little bit of a catch up and talk a
little bit about where I'm at at the
moment I have done a lot of translation
and have pretty much most of the
structured done so I think the next part
of my project is to actually debug and
get it working and then try to rebuild
some stuff that might be a bit confusing
when you look at it most of these
components that we are use and layers
that we use are quite easy they don't
have that much logic in them so you
should be able to write them in a very
concise and describable way and maybe
it's because of JavaScript that you
like to keep it concise so it's easier
to minimize or something but in my case
I think they have been a bit course with
their naming of stuff so you can't
really follow along and I don't like
that I like more of a broader namespace
that you really describe every element
and so on so just start with what I've
added so far we we have this project
structure over here it's a little bit
small so I don't know if you can read it
and at the moment I have a few data
objects I have a back-propagation result
a data block and an output definition
and also training results output
definition might not be the right word
for it might change that to definition
later on the data block is
the actual work unit that we send
through network the output definition is
used when you set up the network to give
the next layer its input data for the
size of the object of the current
working data and back propagation
results it's for the back propagation as
it sounds you get a result for each
backward loop and that is summed up and
put in the back propagation result and
sent back and so then you have a
training data which is the actual result
of the training data run so we can look
at those individually later on if we at
the time but those is those objects are
objects that I have discovered during
the implementation of the different
layers so those are more of my design of
course when I had data to go with I used
the same variables so I could easily see
where they came from and then we have
all the layers whether that we already
talked about I had added a mnist reader
to read the mnist data this numeric
data set we have an ADA grade trainer
that in the old JavaScript library you
have a trainer library and a lot of
training methods in each of those I
wanted more to have one trainer per class
and then have an interface that
implement every trainer implements and
then we have our network and our test
case so the class I have open here the
mnist reader it just reads the mnist data and the interesting thing is
read next label and also read next
image which gives you a number of bytes
which is the 28 x 28 number of bytes in
the raw data from the file so that's the
way we get our input data to our network
so let's look at some things that I have
translated and if we go for this we can
start at the data block in the old
implementation this was called volume so
we have the new implementation over here
and the old implementation here and in this
volume we had a few things to
setup the object which were dependent
on the different input data and then we
have get set add get grad set grad at
grad we have some cloning information
have some add from and set constant we
change the whole weights metric to one
value and then we have some JSON if a
some functions to go from data to JSON
and back these function i have skipped
over so far so if we look at my
implementation i have discovered that i
need the size of this data block and the
depth and then i need arrays that i
work with one with weight and i kept
that naming i might change that later on
and one for the gradient weights so and i
kept that name as well and then we have
this data block initialization where we
initialized the two weights
either with the value that we give or
with a random value for the weights and
the gradients always gets filled with
zeros that's pretty much what we see up
over here as well but in JavaScript
coding instead and then i have this
function to add my image data I thought
that was easier to put in a byte array
of image data into this and so that's
how I actually propagate my image data
that put it into the first block and
then i let it run through the network and
then I have some helper functions to get
the size of this data block because
those are needed in the different
functions then the getters I change
those so you actually can get a weight on a
specific index if you calculate the
index in advanced and you can get a weight
to a specific position in the matrix
dependent on X Y and depth and then you
can set the same information either
by the index or the position the
getters and setters are equal and then
we have an add weight that we don't use at
the moment so i think these gray things
in IntelliJ is quite nice that you get a
visual representation that this is never
used you have you you are added a
function that is not needed at the
moment and because I haven't implemented
all the different layers and all the
different functions in this JavaScript
library I don't know if I need it in the
future so I will keep it at the moment
and then we have the gradient getters
and setters and those are quite similar
and I change the name so i have get
gradient and sets gradients and so on
and then we have clone and zero and
that's pretty much
just create a new data block with the
current sizing information and just set
every value to zero and then we have a
full clone and then that is the same but
it also over transfers the weight values
to the new data block so the gradients
isn't transferred but the data
in the weight are and that's pretty much
what's done in this function and then we
have clear gradients that was a function
that I thought were nice to have because
in many situation you have functions in
in the different layers that actually
clears the gradients at different
intervals and they actually do it by
pointing directly to the array so I
created a function for that and then we
have the functions to get the full weights
and the full gradients because some functions
needed that to work well and then we
have the three functions add from
scale and consts never used at the
moment but I implement them anyway in
case that I needed in the later
stages so that's that so let's look at a
simple layer then and start thought
that I open the first layer I didn't so
let me just open the input layer
as well so here we have the input layer
and that should be in the layers
input class and it's very simple in this
case you don't know if you have any
definition with outputs
so it just sets the different values
into this class and all the out
information in each of these layers are
transferred to in values on the
definition in the next layer so to do
this I actually create this out
definition def and I put those values
into to this definition so I can pull
them out in the next class over and
then we have forward which is pretty
much just setting the value and
returning the current data block and
backward propagation does nothing and
also this parents and grads that I
change to get back propagation result
because that I thought that was a
better name for that function that's the
input layer so now we can close that out
and concentrate on something else so
let's go in and look at the next layer
up which should be a convolution didn't
I open that either I was sloppy when I
opened stuff here let's see that we
actually have everything opened and
convolution like so
this is pretty much a little bit more
involved so we open it up here it's in
the dot product so the convolution layer
here we have a lot of different values
that are needed we have some decay
multipliers we have the data blocks for in and out we have a bias we
have some sizing information both in and
out we have some other sizing
information and we have a stride
and padding I think these sizing
information is for the filter and
striding padding so as well and then we
have the list of data blocks
that actually has all the filters and we
have the data block with the biases and
then we go through and initialize
everything here we calculate new output
from the different values of the
filters because each of these filters
will create a new layer with a new
size dependent on how large a filter you
choose how much padding you have in
stride the output array will differ a
bit so this is that calculation and they
look here i have just transferred that
calculation over with the same names and
everything this layer type for me is not
interesting because every class has
specific names I can just look if it's
an instance of a specific class then I
know it's that kind of a layer so I
didn't bother with actually setting a
type and then we have the
bias if we have any input bias I just use
this bias def value and I add up here
and biases is a new array as well which
I create down here and it doesn't
have any bulk it's just the output
depth so you have one value per
depth so per filter that is and then we
set the current output information so we
can use it in the next layer up and the
forward is also translated directly from
this function you can pretty much
read it the first point here that I said
in_act from this volume a jump use that
value either in the back propagation or
in this function so I never said it and
I believe so at least see here because I
have it up here right now I set it up
there okay I didn't see it I use it yep
oh my bad I didn't see it and but in
this this function we create new block
with the sizing information of out and
we set the the different values of the
filter and then we go through and
calculate the new value so everything
here should just be the same information
the only difference is that i have added
types and changed the different name
some
what padding was pad before so you see here
pad and so on but I don't like that you
just create shorter names for things
when when pad well it explains something
but I think padding is better name for
that and I will do more of that work
later on when i get this working because
i haven't run a full test of this yet so
yeah this calculates the weight by
taking the filter weight and calculated
against the position of the current data
block and then we get a value that we
add and we use some biases and add those
as well and then we set new weight for the
output so that's pretty easy to
understand in broad strokes backward
propagation did the same with that one
pretty much just kept the name added
types and as you see up here we actually
do this zero out of all the gradients
but I wanted to use a function to make
that more clear I think that's a clear
way to describe it at least and here you
see I use these function instead of
looking directly into the object and
making those public and yeah that's
pretty much it here we see how I use the
add gradient function instead of just
this plus operator on the raw value I
also think that's bit nicer to keep it
separated and the same with biases down
here and the result function here I
create a result array
and I go through all the filters and for
each filter I get the weights and the
gradients for the filters and just add
the multipliers they have calculated in
this layer to that result and
lastly I add the biases weights and
gradients as well with no decay and that
pretty much what this function over here
does so that should be the convolution
layer and that's the most intricate
layer of them all I should say the other
layers are much smaller and easier to
understand so let's go for this is a
another thing that I saw that all the
layers that are last in the operation we
do some different calculations with
those and we also do a different step in
the back propagation of those so to
separate those out a bit and give those
a little bit more of a definition i
actually created a layer that i called
loss layer where I have this backward
function that you could give a y-value
and this Y value is the actual
prediction or the actual correct value
so when you train the network you will
give this layer the answer let's this Y
value and then I needed a way to get the
the output act as well from the loss
layer in the back propagation so this is
that interface or doing that
and i think the softmax is the only loss
layer that's the last layer in my
network it has some values here that I
implement from the beginning we have the
the output from the softmax and number
of classes so that's how many number of
inputs why do I have a number of classes
when I don't use those that's
interesting let's look at this
information here we have number of
inputs that's calculated and then we set
a depth for the output to number of
inputs that's interesting so the
previous layer gives us how many
classifiers we have that's a bit weird
isn't it I thought that this layer
should actually have a depth of the
number of classes that we'd want but we
can look at that later this might be
something i want to change because i
think that's a bit strange and why
will we it you actually could give the
definition number of classes that's
because i have added that here because
of that so why it's not used that's a
bit curious and the softmax layer in if
we look at the forward function again we
create these the new data block up here
we get all the weight we get the first
weight and set that as our amax value
and then we go through
all the values in the weights and try to
find the largest amax here and then we
so that's the largest weight that we
find in our data block going into this
function and see here compute
Exponential's carefully do not blow up
what a very insightful comments they
have put here so I create this layer
with outputs I fill it with zeros I have
this sum value and I go through and
find the sum value for okay so I look
for the differences or distances from
this max value I guess don't really
understand this function math and we'll
look into that more later on and make
that more understandable we
compute Exponential's okay and then
we normalized output to all sum to one
that's so we actually get the value
let's say that we have one value that is
eighty percent right and then two other
values to ten percent right so we want
all the values that we have in our array
to sum up to 1 so we can say how
confident we are that one of the values
are correct and then we save the
information for the back propagation
and here we say this specific
function for back propagation for this
layer the loss layer this back
propagation is a bit different it takes
an actual correct value we set the data
block clear the gradients go through and
add new gradients with this simple
function where we go through all of the
values in the depths if the current
value is right then the indicator
will become one otherwise it becomes
zero so that's pretty easy and then we
look how far this indicator is from the
current value so this is the current
guessed value and we will then say how
far from it it is and then we will add
that as the gradient so we can use that
in our back propagation and then we
calculate all the losses with this
function on here loss is class negative
log likelihood also very strange
comments and we don't need any
back propagation results because
the only thing that we are interested in
is the actual loss and we want to
minimize that in our function so that was
softmax layer see here close these out
fully connected should be over here oh
it doesn't want to you I think I need to
take a little break here because my
throat is a bit sore at the i'll see you
in the next video