Convolutional Neural Networks [Part 6] - Wrap up and networks Video Lecture | Machine Learning with Java - AI & ML

welcome explorers and hello world I'm
going to do a wrap up today talk about
the Java CNN where I have this video
series for a couple of weeks now and
I've been running a lot of different
tests to test this framework and see how
well it does with different networks so
today I'm going to talk a little bit
about what I've done in code and what
I've done in tests way where which
different tests I've run so first the
code changes and I'm some additions i
added a lot of documentation so you can
go to the github page and read up on the
different layers and so
on and you can also look into the code
and try to decipher it and if you have
any additions please let me know and
give me a pull request and we will
integrate your changes as well so we
learn by changing the code together and
create good java CNN repository and then
I added it abstract trainer so I
actually broke out all the common code
that different trainers use which just
handles the different gradients so you
and give it to the specific training
method in the normal code that was
handled by a large switch case or a lot
of if statements but I've done it by
having classes that extend the common
core functionality then I'm added the
adaptive gradients Adam nesterov SGD and
window gradient trainers to the mix so
we have all those and those were
implemented in the convnet
JavaScript all that is regression and is
supportive vector machine loss layers
also in the convnet implementation
and a lot of things from the
ConvNet was the dropout max out rectified
linear units sigmoid and Tang layers
so we have all of those at one so now I
try to run this test that they got from
the ConvNetJS so they have a model
already set up that they run on their
page and get a quite good result I'm not
sure if they do the testing right I've
heard that you need to first test on or
train on your training set and then you
do the testing on the test set and those
should never intertwine or a mix so you
have a separate phase for training and
a separate phase for testing and
those should it be intermingled in order
to ensure that you don't actually
trained on the test data so i ran
different trainers on the same network
and adaptive gradient the did 91% Adam
did 93 and the interesting thing of
these is that they are in different
corners of their implementation where
adaptive gradient goes for the result
quite fast and then pendel back and
forth and actually have quite a bit of a
difference in what it ends up at in
values so it's not that stable it can go
quite far from the actual right loss so
it's it's not getting to the point it's
getting to the point faster but it might
not get the correct or the best-trained
network the Adam network is much slower
but it's much more stable and we'll
actually find the specific points where
the trained set is at its max which is
good and all the other trainers are
somewhere in between and the best result
I got with the trainers with the
adaptive Delta trainer as 94 29 and then
I did a small test where changed all the
local normalization in the network to
rectified linear units instead and I got
it to 94.52 so it was an
improvement there as well by changing
some part of the network which made me
think what what else can I change what
can I try so I've tried
some different networks I prefer to try
to fully connected their network just to
see what that would give me and this is
the best fully connected network i have
found so far which is quite terrible
it's not even guessing you should you
should be at around fifty percent if
it's just flipping a coin and it's not
even doing that so it's worse than
guessing so I think you should be able
to create a fully connected network that
could work for this data set but I can't
really say that I've found a good kind
of this good setup for this yet so if
you have any suggestion how fully
connected network should look in order
to get the good results then please give
me a comment in the below and another
point where this fully connected network
is that it's very slow to run so it's
another
bad thing about the network then we have
this large small network where I tried
to have a large window and went down to
the smaller window with 10 different
filter layers 20 different filter layers
and 30 different filter layers so I went
from a windows seven then down to five
and lastly to 3 and that gave me a result
of 94.12 which is quite good and
the next try try a did was to add a big
pooling layer after this run and I
have done a lot of these together so the
large big pooling I did for a network
with three five and seven so that goes
from small to large and adding a fully
connected layer at the end with 500
nodes and it actually gave a lot worse
result at 84.2 which is interesting that
a large network at the end which should
be better and I actually got this
network from a movie at the math some
math guys actually is suggested this in
the movie so I wanted to try it out and
it actually gave me a really bad result
could have the same vibe to it as the
earlier fully connected that I don't
have a good fully connected layer set up
so can be something fishy there perhaps
and then I have this small to large
which is even better so I have a small
window first with a few filters 10
filters then I go up and try a little
larger window with 20 filters and then I
go up even further to 30 filters at
seven which gives me a really good
result 95.49 so that's quite nice
so i know that if i look at small detail
and then zoom out i will get a really
good result from from that work
instead of having a large window and
zooming in that will get give me a
worse result so then i try to actually
give more filters to the mix so i have
30 30 30 filters and as you see i get a
marginal improvement there i go to a 95.83 and that's because i have more data i
guess so more data you get more better
results so then I try with just more
layers just add more layers so at the
small window I have 20 filters then I go
to 30 filters and 40 filters and that I
can see a point one point actually
increase up to 96.26 as I had
earlier 95.49 so that that's a
really good push up so just by adding a
few more filters in the mix and they
this took a lot longer to run but it it
gave me a better result so that's a good
improvement and then I wanted to try out
the tensorflow set up so i looked at the
simple setup the 10th floor who uses for
their example so they have on their page
tensorflow Network that you should try
in order to get a good result with
tensorflow and this has just two
convolutional networks and then fully
connected layer at the end and that gave
me 95.98 and so almost 96
percent which is really good but when
you look at this network
performing in the Java CNN and then look
at the same result in the tensorflow
model when you run it in tensorflow
in tensorflow you get a result of
99.18 which is much better than
running in this model which is a little
little strange and this also takes a
lot more time to run and the
tensorflow model took a couple of minutes I
think three minutes to run on my machine
and this here it was days running this
so yeah I'm not that impressed by my
optimization skills in the Java code
but then again we have at least been
able to create a pretty good result with
all my networks so I'm quite happy with
this and I hope that you found this in a
video interesting and I hope that you go
in and look at the code and comment the
code perhaps give me a pull request with
some changes and perhaps you learned
something about from this where you
found a new insight into how to build a
network perhaps I'm very
fascinated about how to think when you
build your network because all guides
that I have seen is you have this data
and then you set up this network run it
and you get this result okay but I on
the other hand have a set of data I know
what kind of results I would like to get
but I'm no clue which way to go about
actually creating my network in order to
improve my result so I'm really curious
about how to figure out which kind of
network should be run on which data so
if you have any suggestions on reading
material or if you have any clues that
you can tell me then please hit me up in
the comments I read all comments and
mostly answer all of them as well so I'm
really looking forward to hear from you
and yeah I hope you learned something
today and I'll see you in the next one