Multi Layer Perceptron Part 1 (Java by example) Video Lecture | Machine Learning with Java - AI & ML

hi and welcome back to another java by
example and today i'm gonna do some of
my personal musings i have been really
fascinated by different machine learning
techniques lately and when i used a tool
that actually and from new zealand i
think called Weka
it's bird and it's also machine learning
software tooling and you can take
courses on their site and I've taken one
of them and where you can learn more
about how to do machine learning and
they have this tooling that you put in
training data that the machine will
learn from and then you can put in some
examples that it should test on test
data and it will actually see how
accurate the different machine learning
techniques are so you can get a sense of
how to use machine learning to solve
real problems and I've also tried to use
Weka for some real problems and have the
had somewhat good results but
one of the implementation in Weka that
have been a little bit strange to me is
the multi-layer perceptron
and that
was not what I think very efficiently
implemented in Weka perhaps it's just me
I don't really know much about how to
create really great efficiency in this
method because I have not looked at it
closely yet but in my case I added a lot
of data so I have a training set of
about 400,000 examples and it actually
crashed because the tree or the training
or not tree but the set of the
training data got so huge that it
couldn't hold it in the memory at one
time and what I see from this example
that I they have it's actually just an
amount of floats or integers probably
float no floats float weights in a mesh
network so it shouldn't be that huge
even in Java so I don't know why takes
so much memory I gave it about 12 GB
and that was not enough so a bit weird
so I found this tutorial about the
multi-layer perceptron and it explains
how it works in mathematical terms and
it also provides source code you can
look into and I will look at it briefly
and read the tutorial and I can say that
the math around it is not familiar to me
but the code is really familiar so I
guess I'm more of a code person I like
to read code and understand how
sequences work more than actually have
formulas to explain math for me and but
to understand that we have to look at
a simpler network called a perceptron
neural network and the difference
between these are that this is as the
perceptron that actually takes multiple
inputs and arrives at one decision
because the multi-layer can actually
have multiple answers to one question
but this
the preceptor network is just choosing
between either that or that and it works
quite simply by taking a number of
inputs in this case four inputs and
weight these inputs into a function and
in the other end you get an Y which is
the answer if it's either that or that
and in the case of this example they
have have they have talked a bit about
neurons and how they work in the brain
and the example they have here is colors
simply is this a bluish color or a
reddish color or it is there somewhere
in between and in this method you can
say that it's blue if it's a high number
or red if it's a low number in the
result and if it's somewhere in the
middle you can say range where you don't
really know what the answer is and
that's undetermined I don't know if
this method will actually yield that
and but to look at this code a little
bit it's quite small you have defined
either blue is 1 and 0 is red so if the
number is closer to 1 it's blue value
and if it's closer to red it's a 1
that's just definitions and then they
have a few examples here where they put
in the RGB values for the different
colors and the resulting class
is the answer in machine learning and
then they create some normalizing
vectors and some more static
definitions
how many input patterns there are they
it's 13
and the least mean square error is where
you want to stop when you have an error
rate of about 0.001 then you should stop
and the training step is a value that
determines how fast the machine learning
should learn so smaller value that would
take a longer time and yeah then they
have this a n n which is the
actual learning method called a
perceptron so they define it as an a n n
and the this gets to values the number
of the the input values and the method
to use in this example they use a
sigmoid which is defined on the page
with a mathematical formula let's go
back to that and see no it was oh no it
wasn't this one so you have three
different here you have the threshold
function the sigmoid function and the
hyperbolic tangent function so if you
are interested you can read it on this
page I will link them in the video and
these are the mathematical formulas the
threshold function is quite simple it
just says is it 1 and if it's larger than
0 or equal to 0
then then its value if not it's not a
value so it will strip away the negative
values
so and all these functions are to
normalize the values so they are closer
to the values we are looking for we're
looking for a value between one and zero
because blue is one and red zero so to
be able to find a value that is closer
to either of those two we need to
normalize it with a function so we get
in the ballpark so to say and here we have a
while loop so it runs until either the
least mean square error is divided by
the current mean square error
is larger than this value 0.0001
so when we are closing into
this breakpoint we will stop and in here
we just define the error and we have our
counter go through the input patterns
and this is actually just training the
model we put the inputs from our
values here we normalize them so they
are between 1 and 0 nope between
-0.5 and 0.5 so and then we
calculate a network we calculate the
error from the result so how close are
the input to the result value and then
we adjust the weights depending on how
we actually did and then we increment
our counter and try again and so this
will run through all the patterns and
when we reach this line here we could
take the patterns that we have and
divide the error with the patterns which
will calculate the mean squared error
for us which is the value that we will
test a in our while loop and then we
just increment epochs so we know
how many times we have actually trained
the model and the function down
here is after we train the model we will
give the program a value for the red
green and blue values and it will try to
recall what's the correct solution for
for these values that you have given it
so this function is quite simple or
the flow is quite simple there could be
things in it that is a bit
strange but I think you can follow
through it and understand pretty much
what it does and here we have
the perceptron and it takes this
input number and a function so it just
sets the activation function so we know
what it will use for function
in the loops and it will actually push a
number of random values into our input
numbers and it will populate
these with values from -0.5
and 0.5 so we
have a lot of floating values between
these edge points and then we have a
function to set the input values in the
calculate networks we just take the input
vector which is the value we want to
Train and we take that and multiply it
with the weights in the same way in the
weighted column and then we get an
action value and that we will increment
with all the different weights
and this action value we then take and
run one of these methods with so we will
have the we will let give it to one of
these methods and then we will return to
actions so we know what this calculate
network will return the actual value of
the this calculation which is just
multiplication of each input value with
the weights and the input values are
normalized and so let's see here where
did we get that value back
it was the output here and output from
this calculate Network we gave two fabs
to see how far from how far our inputs
was from the output to get an error
so I guess that our input see here just
to see what's the definition of
that so we can follow the yeah so it's
from here we check the input counter hmm
yeah yeah okay so incrementing a input
counter for each of these loops which
means that here the input counter will
always be on the class so we will
calculate the value that we get from the
calculate network and see how far from
the blue and the red value it is and then
we go back to this one and in the case
that we know what the difference was we
will push it into adjust weights
they are the error and now the output
will be push into the adjust weights and
we will give it training step and the
actual value that we want to have and
adjust weights goes through the input
vector size and then we will change the
weights for each and using the previous
weights and add the training step times
the target minus output so the target
value that we want to reach minus the
actual output that we got times the input
vector so that will slightly modify the
weights in the network
and then we just have this recall which
is the same function that we have seen
before but it's just simplification to
do all do one of these tests in one go
so it will set each of the input vectors
and calculate a network and get a value
back that that value will either be
close to zero or close to one so I think
this this method is quite simple to
understand it's a lot of steps but I
think it's reasonably small
problem with good namings you can
actually understand it so that's the
simple function but if we go to the
multi-layer that's a bit harder to
understand because it takes a number of
inputs and push this through a set of
hidden layers and then it will arrive
out at it output layer and the input
layer and the output layer could have
how many nodes as you want so you can
have 400,000 inputs inputs if you like
and 300 outputs that's no problem and
just the the accuracy of this network is
in how your hidden layers are structured
and I think that's a trial and error or
there could be some some math behind it
so you can actually figure out how a
really good network should look in order
to find your solution and this page has
a lot of math
and different methods but the activation
functions are quite similar and so
on so I'm not going to go into this
paper you can read it on your own if you
like just to understand the problem and
this sample code will train on pictures
so you will get monochrome pictures with
just black and white values and so it's
either 1 or 0 in a grid and then it will
try to figure out what number each of
these images represents so here we have
an example of the different values it
could be and then you push that through
the your multi-layer perceptron and on
the other end it will end up at one of
these nodes at nodes either 0 1 2 3 4 5
6 7 8 or 9 and so let's look a bit
closer at that code I have set up
repositories here with my own
implementation I want to do this in Java
and this code is written in C so or C++
so and I want to translate it into Java
and by doing that understand it further
and let's look at the main program
program here it start off by setting up
a number of input values so it will
fetch something but not values from
the user
it will check how many hidden neurons
there should be and how many hidden
layers so that's neurons is the number
of nodes in each layer and then you have
a number
layers so if you go back to this picture
up here each node here is an neuron
and each row is a layer so in this
program you can define how many neurons
and layers you want and then you will
give it a number of bit maps a training
step which we have discussed at least
mean square error which we also
discussed this momentum that's the same
as the value we had before called
training step so it will actually or it
will work in the same way we will take
the momentum value and multiply it with
a previous step in order to push the
function closer to the value that we
actually want to end up in or end up at
and then we will set up our network and
we will train the network and then we
when the training is complete we will
ask for a number and we will recall that
number from the neural net so I think
this program isn't that hard to
understand either there is very few
steps here it's much it's very mean it's
many it outputs and descriptions and so
on but not much code so the code is
actually more into the neural
code here so here we have this
multi-layer perceptron constructor and
that was the first we gave here hidden
layers and hidden neurons so we get
that and we set it here we say how many
outputs we have in our case we have all
the possible numbers so that's 10 and
for some reason he creates the reader
twice here I don't know why and then
he reads the dimension from the pictures
and reserves memory for that in the
weights so here he needs to reserve the
value of inputs of course times the
number of hidden layers so he needs
values to represent the step before from
the inputs to the hidden layers and then
he wants all the hidden layers values in
between so you want the value of one
neuron and the weight it has against
another neuron so it's actually the
relationship between each neuron so in
order to create the amount of data you
need you take the neurons that in your
network times the neuron in your network
times the layers minus one so you get
the number of neurons that you have in
the middle and in all the layers and
then you want to have the same on the
other side so you want to have the
relationships between the neurons
and the outputs so this will allocate a
large amount of the memory and then we
resize I guess it's resized some vectors
to the actual sizes if they you need to
be so you have the input neurons you
have the hidden neurons and output
neurons so these are just vectors of all
the different neurons and then we
randomized all the weights and give them
values from -0.5 to 0.5 and then we
go down here and do this that's for the
weights between inputs and the first
layer of neurons and then
we will go through all the hidden layers
and set all the weights
for those and then we will go through
the last layer weights from the last
layer to the output layer and then we
have a destructor that will clear out
all this memory we have something that
calculates the network and that will go
through the hidden neurons and then set
the hidden value for the first layer and
that is calculated from the input
neurons and then we will time that with
weight between the input and the hidden
layer so it's pretty much the same as we
did before so there we will calculate
the first weight
and of the first layer or the first
result you can say in the in the first
row so there we set all the neurons
numbers and then we will use sigmoid
again to normalize these values we
will get values closer to yeah to 0 to 1
and 0 and then we go through each layer
in turn the hidden layers and calculate
each weight for them so that's pretty much
the same as the first layer we will go
through the layer 2 until the end on
hidden layers and calculate those and
then we do the same to calculate the
output neurons that's from the last
layer to the output neurons so we will
get all those numbers as well so here we
have actually calculated the whole
network with all the hidden layers and
then we will we have a function here to
populate input from a file number so
that's pretty explanatory it will read a
amount of data from disk it will figure
out if this neuron should either have a
1 because the value in the image data is
black or if it should have a zero if the
input is white or you can turn it the
other way around depending on the input
but the interesting thing is that
one color is one or under the color is
zero and so this will just populate the
input then we come to the training stage
and here we do the same we will keep
track of our mean error we will keep
track of the epochs and we have a
counter here as well don't really know
what that is and we will have the error
that we will calculate for each step and
here we allocate some deltas
I guess that will be used when you do
the reverse calculation to actually
update our weights and we have some
templates that is to remember the
previous results a buffer of weights and
used to keep previous weights before
modification for momentum yeah well that
was what I said before that we will use
the momentum to push the network in the
direction that we want and then we read
the image goals to know if we have
actually ended up where we want to end
up and this is just to check that we
have goals and here we have some
differences between how Windows 32
solves something and Linux that's the
calculation of the the mean square error
and the least mean square error in win
32 you need to use a function as well
I don't know why
could be important
guess it's because the libraries are a
bit different and then we will calculate
our mean square error we will run
through all the training files that we
know of and we will populate our input
with the counter so that's the actual
file that we want to read now so when we
have done that we calculate the network
and we take the target okay so that's
the target that we want to hit so we
will just fetch that value from pointer
that we have we will calculate the
output Delta so we will look at the last
layer if we are close to how close we
are to the actual result and if the
value is not equal to the target see
here if I is not equal to the target
okay so if the value that we want to hit
is not the correct value then we think
that is this layer should be closer to 0
and if we are closer to the target we
want to give it a higher number so it
should be closer to 1 so that's just the
difference here we want to skew the
result in the neural network so it will
actually give values closer to
the results that we look at so we put
some inputs in that this end we want
when we go through the network we want
to end up at the node with the value
closest to one in the result so this is
pretty much it and we use their sigmoid
to normalize or stabilize
the values here and stabilize? and
see here um then we will create a look
at the layers the hidden layers so here
we will go through the last row and see
how close those were to the actual
output so this is actually
calculating backwards just looking at
the last layer and then we go one step
further and we go through each layer in
turn from the next last layer all the
way through and we would calculate
deltas for those and see how close we
are and then we will store our weights
for the last value and so we need to
have these templates so we can set those
as the previous result later on and then
we will calculate the last layer weights
from the hidden layer to the input
values using this function here where we
multiply the momentum
this value that we give the formula how
much we should push it oh we have a
training step that's well interesting
and so the momentum is an other value to
push the or to try to shake out the
values we get the the right value
quicker and we will take the previous
value that we had in the input
to hidden layer and we will remove that
amount from the actual hidden input to
hidden value that we know now and then
we will multiply that with momentum to
give the they give the machine learning
a push so it probably will resolve
quicker because the the closer we are
the more correct the values will be and
this value will be smaller and the
further we are from the truth the value
will be larger so the momentum will give
a larger kick so this will actually end
up to push your way closer to the actual
value that we want to end up at and
then we take the training step and
multiply that with the Delta that we
have calculated and the inputs that we
know about so we get a new value
the last input layer and so we know that
weight the difference in that weight so
we will actually go through all the
layers back and update the first weights
with the differences that we calculate
all the way back so up here we start to
look at the outputs and then we will
calculate Delta all the way back to the
first layer and then apply this Delta to
the first layer and after that we will
also apply the Delta for each of
the next occurring layers as well so we
will go all the way through the network
to train it then calculate the deltas
back and then apply them on the other
way back through the network and then we
will store the previous weights we will
assign the mean square error and then we
will set the error to 0 again so yeah we
got the error up here when we looked at
how close to the truth we were and then
we will add our counter and start off
again and when we come to 1,000 epochs
or to thousands and so on when we reach
a modulus of epochs we type out how many
epochs we have the ran and we ask the user
if
they won't stop execution now because
they have ran through a lot of steps
back and forth and then we increment the
epochs later on so that's pretty much the
training which is the interesting part
there the recall network just takes a
file number again it will populate the
inputs calculate the network and then
just check the output neurons to look
for the winner which value is closest to
zero know closest to to one so which
is the largest number for this
input file number so that's pretty much
it now we have gone through the actual
implementation and we know a little bit
more about how a neural network works
and how to how an implementation of a
multi-layer
perceptron Network looks and also how
much simpler perceptron neural network
could look and I hope you thought this
was as interesting as I think it is and
I hope you will leave an interesting
comment a nice comment if you know more
about these functions and if you want to
correct anything or if you want to
give me suggestions about how to
continue this exploration into machine
learning and if you liked this video
give it a like share it with your
colleagues friends and subscribe if you
want to see more with videos like this
one and I really hope to see you in the
next one