Convolutional Neural Networks with TensorFlow - Deep Learning with Neural

the convolutional neural network with
tensorflow is what we are going to be
doing in this tutorial so to start we
are going to grab that multi-layer
perceptron code that basic deep neural
network code that we worked on in the
beginning you can go to Python program
at net search for how the network will
run it'll be your first result scrolling
down to the bottom we're just going to
take that code and we're going to copy
it move this aside go into TF tuts oops
open it twice close one and I'm going to
new documents and we'll call it CNN
example DUP hi actually don't want it G
edit open in sublime paste and here's
our code so we're going to make the
changes that we need to make you should
already be able to guess maybe some
different variables or changes up here
and then mostly just change this model
function and we should be good to go
so if that was you you're right so what
we're going to start off by doing is
getting rid of these variables we're not
actually using them anymore
number of classes is still ten that
hasn't changed we're going to change
this to 128 for the same reasons that we
did it in the recurrent neural network
tutorial and then now we're going to
mess with this neural network model
function and I'm going to just go ahead
and delete everything except for to the
dictionaries because we're going to use
dictionaries also just to note I am
going to try to use the same ish varial
variable names as you could find in the
deep m-miss for experts tutorial it's
not the exact same code word there are
only third just scripting this code
basically there is no function as far as
I know and they're kind of they're
actually running the iterations a little
different but so our code is actually a
little different but it might actually
even be helpful to you to go through our
code and then go through their code and
kind of see what the differences are
because you can do you can do tensorflow
in a bunch of different ways obviously
it's just like any other thing you could
program
so let it be useful but also if you're
confused about some of the the
parameters I suppose that we set you
could come here and learn a little bit
more and kind of keep diving into their
documentation if you want but hopefully
I'll explain them as best as possible
also as always if you do have questions
comments concerns whatever you can
always leave them below munna long let's
go ahead and first of all we'll call
this convolutional neural network and
again it is going to take data but will
replace that with X again same reason we
were doing it before it's just easier I
think if we just keep everything as X
now we are going to instead of having
like layer dictionaries we're just going
to have basically our weights dictionary
and a BIOSes dictionary I think this is
a better way to go about doing it and
we'll start with the weights part and
this will be with W underscore comm one
so this is following extra so for Casey
if we're going to do it just right this
is following the same kind of names for
variables as this tutorial so if you did
weight underscore comp one you'll find
that oh yes look at that so you probably
should be able to see what's about to
come it's going to be something like
this
but anyway you get the following code
here so now what we're going to go ahead
and do is is do this only we're not
going to define them as straight-up
variables these are going in our
dictionary and again there's going to be
a lot of not a lot but some subtle
differences here but it would be
probably a good idea to go through both
again if you're confused about how to
build your own neural network models
it's good to see as many examples of
people doing it as possible so anyways
moving along we've got the weights for
the comp the first convolutional layer
basically the first convolution that's
as usual going to be a TF variable it's
a good TF random normal it is not
anymore going to be 784 by the number of
nodes and hl1 will delete that and
instead it'll be 5 5 132 because those
are magical blackbox parameters written
now this will be 5x5 convolution it's
going to take one input and it's going
to produce 32 features or 32 outputs now
I am going to just loops really I can't
get away with that huh well delete that
and I'm just going to copy come down
here paste oh my goodness really alright
and let's see if I can get away with
this yes
great okay so we're gonna have four
things in here obviously not the same
things so you've got your convolution
one convolution two then you're going to
have your fully connected layer and then
you're going to have your output so
we'll just whoops we'll just call this
out so the fully connected I think the
tutorial refers that as the densely
connected layer I think fully connected
is the proper term densely just means
it's not sparse its dense but does dense
necessarily mean all of them are
connected I don't think so so fully
connected I think is the proper term
next you have a five by five again
convolution instead of one input it's
going to be 32 inputs in the output
we're going to say is 64 you can tinker
with these if you'd like but that's what
we're going to go with the fully
connected layer is fully connected it's
not a convolution anymore so we're going
to say the image at this point we're
going to say is a seven by seven image
remember we started with 28 by 28 images
when you do a convolution it's going to
you're going to significantly compress
those images down to be feature Maps
they're not it's not even an image
anymore and if you saw a picture of what
ends up happening I wonder if I can pull
one up really quick I'll see if I can
should had it prepared I don't see any
and a quick Google search but they're
just like here we go this is close they
look like this I pull this over they
look like this
oh are you really gonna do me like that
okay a few image all right so they look
just like this right they're just
colored squares and then you might
actually look for horizontal lines or
edges or whatever and so on and as you
go deeper and deeper you might look for
more complex types of features like eyes
here or an entire face and so on but at
the most basic level they're just these
little things and in this case is a four
by four okay but they look like that
they're nothing special so uh so this
will be a seven by seven and then by 64
which is from the 64 here and then we'll
have 1024 nodes in the lair so if you
recall our typical our before our fully
connected layers were what 784 right
that was the the input was 784 initially
but now it's a seven by seven times 64
features and then we're going to just
say okay we want a 1,024 nodes there
then the output is going to be a
thousand twenty-four and then number of
glasses so it can be any of Minah zero
through nine now we get to the biases
and really the biases are I think I'm
just going to take let's do this I think
I'm just going to copy this pace so I
don't like reformat it biases and then
we're going to say rather than wait conv
one it'll be be Khan one be calm be
fully connected and out can stay the
same that's no problem
now for here the biases would just be 32
here 64 here what do you think 1024 huh
and then here again number of classes so
the biases are just for the number of
you know the output then now that we've
done the weights and the biases
dictionary most of the work has already
been done we're going to first reshape
equal x equals tre
shape we're going to reshape X from the
input and we're going to say the new
shape is going to be a nice flat 28 by
28 by 1 okay someone in the last
tutorial was asking about the like
reshape and I think I mean we covered
the what if we do transpose there was
reshape transpose and something else I
forget what the other thing was but you
can always open just go to numpy and
take the example like in the video we
use that nice example and do it in numpy
and just print out at each step of the
way to kind of get an idea because these
are kind of like abstract manipulations
this one's fairly simple this one we are
simply reshaping a 784 pixels image to a
flat one or 28 by 28 image so that one's
pretty simple I don't actually but I
don't think in this one we're going to
be doing anything besides reshaping but
in the other one we did like transpose
transpose is oh it like I can't explain
transpose very well but if you if you
just write out the code and see it
happen there's also like pretty images
where they show like transposing
happening but that's not actually how it
works every time so I don't really like
that but anyway moving along if you are
confused I think I always think the best
thing to do is run these out you could
do it in the interactive tensorflow
session as well I just I think it's
quicker and easier just to open up a
quick terminal use numpy and do that
sometimes the syntax is slightly
different in the actual function that
you're going to run but for the most
part it's the same like here rather than
I think in reshape for numpy I don't
know that they use brackets they might
use like parentheses or you know
something like that anyway moving along
reshapes fairly simple next conf one for
that first convolution layer that's
going to be a function we have not yet
written we are going to use the exact
same function from tensorflow I think
it's kind of silly to be right to keep
making a function out of this it's
literally a return one line of code but
they're making a function out of it and
in theory you could
make a more complex function in time I
just don't know that it's absolutely
necessary but we'll go ahead and do what
they did in the tutorial so we're gonna
say define calm 2d and we're going to
pass the data and the weights here and
all this function is going to do is it's
going to return TF n n calm 2d and this
is basically just going to do that two
dimensional convolution so take note
this doesn't do not that I mean this is
your first intro to convolutional neural
networks doesn't really matter but this
is two dimensions you're not going to
get the depth in here so keep that in
mind
X capital W strides oh let me write
these out and then we'll explain them
but strides 1 1 1 1 there we go
your clothes and then padding equals
same and then let's do let's write the
other function and don't worry I'll
explain what strides are because we're
about to see them again and then also
the case eyes now define max pool 2d
again we're going to pass just X here
and then we're going to return to f TF n
n dot max underscore pool again from X K
strides is going to be a 1/2 to 1 and
then strides strikes strides 1 this week
to 2 1 again and then padding again will
be so convolution recall that when we
did convolution I'm gonna so it when we
did convolution we were going to we took
that image and then we kind of like slid
over a moving window over the pixels and
we just didn't want to skip any pixels
right at each time and then with pooling
we were doing basically the same thing
only with convolution you're trying to
extract features with pooling you're
just trying to simplify things so in
this case like with strides 1 1 101 this
is just simply taking it's going to take
your
your size and each time it moves it's
just going to move it's just going to
take basically one pixel at a time right
to go through in that convolution now in
the pooling we're going to take a 2x2
pooling here and we're going to move it
two by two at a time and so in this case
you're just that's just moving your pool
window twice over so you're not actually
if I'm understanding that correctly
you're not actually overlapping anything
there for padding on its 28 by 28 using
such small strides I don't see why we
would ever actually need padding
hopefully but that would be the case
like if you've got a five by five window
and you've got it let's say 28 by 28
image there's going to come a time when
you reach like the edge of the image and
to get those last pixels you might have
to have part of your window hanging off
in theory the image so how do you want
to pad those edges in this case we're
just saying the same so it's just going
to pad those same exact pixels to the
edge so those are your two functions
again it's not absolutely necessary to
make them a function but it's
conceivable you could have your
convolution be doing a lot more complex
things and then it's not just one line
of code so we'll leave it that way I
suppose so um oh did I call it okay
sighs okay this is the size of the
window size of window that was going to
give us an error and then this is the
actual movement let's say of the window
okay so this is the size it's a two by
two we're moving that window two pixels
each time so there is no overlap here as
opposed to here where your stride is
simply one pixel over at a time now
let's we those are the last functions we
need so we're coming back down
underneath that reshape and what we're
going to do now is we're going to say
conv
one is going to be cons duty which we
just wrote we pass X and then the
weights will be actually we can just
pass the weights like this weights I'm
pretty sure the parameter is weights
equals but we're going to do this
because it's the second parameter W
underscore con one so that's our con one
then we do pooling so we're going to say
comp line equals max pool to D again we
just pass the data but the data this
time is not going to be X we're actually
working with conv one remember that that
con that full layer is actually the
result of the convolution and the
pooling and together they are a layer so
actually we're going to apply this pool
to that convolution and then we're going
to take con two and that's going to be
in fact let's just take copy paste and
this will be con tu con to this the
input to that secondary layer that's not
going to be X anymore that'll be com1
the weights on - max pool will be
against cond - now we're going to do the
fully connected layer so FC that's going
to be F dot re shape we're going to
reshape that con - and the shape will be
that a negative one and then seven by
seven by ya 6-4 that's what we did up
here so so we're reshaping that next
what we're going to say is fully
connected equals t f dot n n this should
start looking relatively familiar
rectified linear of the TF dot and what
are we pretty much always going to do a
matrix
occasion of what's the input FC what are
going to be the weights it's going to be
weights and this will be the the fully
connected layer mat and then plus from
at time I put the plus and we did the TF
add a horrible biases and then for the B
F C so my news okay now we're going to
do finally the output layer have so much
space so output will be again because
keep in mind remember the fully
connected layers are super simple they
basically just do this every time as
opposed to the convolution now are the
convolution layers now the output again
is just going to be F dot mat mole and
this time the input is FC that fully
connected layer the weights will be the
weights put the out layer mantle and
then we're going to just do plus the
biases for the out layer and that's its
so then we take convolutional neural
network copy/paste and we should be
finished
let's see what happens will this be the
first neural network that we've built
that doesn't have an error on running if
I was putting money on it I would say no
Python 3 come really
oh we you seen anything
could it be oh my goodness let's see if
we actually train guys can you believe
this oh my gosh I hope this recording is
working I'm looking at like the output
it's not looking so hot anyway yeah
we'll see we will see so we are running
through I think at how we did pride 10
epics and we didn't move that that
variable which hopefully we would have
moved but that's okay oh it's going so
slow
I think it's cuz like I'm recording and
it just it just really messes with my
recording okay accuracy 97.5%
actually I believe that's worse than the
the recurrent net that we did but it's
because the size of the data is not
ideal so the recurrent net actually can
seemingly get by with smaller data sets
if at least from what I've found the
content will be superior with a much
larger data set the other thing that I
do want to just show is especially with
a confident dropout this this idea of
dropout is pretty important the idea of
dropout is like it from our brain for
example we not every neuron actually
works in our brain so neurons sometimes
are dead neurons okay so with with
dropout we're trying to we're kind of
like mimicking just a missing neuron
from time to time so you can add dropout
to your neural networks seems to be the
most popular with the comp net so I'll
go ahead and show it here I can't I
can't remember if it's in this tutorial
let's look for joy yup it is ok so again
even with dropout you can look at the
example here we're going to add ours
just to the end anyways but yeah so
after our you know X&Y variable
definitions let's go ahead and add a
keep right the keep right will do 0.8
and then keep underscore prob will say
is T F dot placeholder and it's going to
be T f dot float
32 okay now what I'm gonna change is
down here someone made fun of me for
using notes in my tutorials just for the
record this is the first time I'm
teaching it confident so you should be
thankful I'm using notes now let's go
down to it should be just at the end of
this fully connected layer is the yeah
we'll just add it there I think that's
all we're going to do is just add it
right at before the output so you could
in theory apply drop out like right here
you could say fully connected equals TF
n n dot dot drop out what are we going
to apply that to that fully connected
layer what's the rate of keeping is keep
right let's look that up really quick
actually not that I am looking at that
it doesn't sound right that it would be
dot drop out and then it would be keep
rate but let's let's find out
TN n drop tensorflow Doc's linked me to
the right thing drop out that is keep
Rob okay cool and you can even throw in
these I had no idea they even had these
other parameters anyway all it does is
computer drop out and we're going to
just throw in that keep rate
so basically 80% of our neurons will be
kept
woops all right now that should be all
we have to do can we go for two models
in a row of changes and no errors I
really think that's all the two major
changes to add drop out would need so
let's run it see what happens so let's
wait see if we're actually training and
then I'll talk okay
wow that's a lot of error it's like
significantly more air than the other
one started with but anyway dropout is
probably gonna add a little bit more
error but the idea is to it's almost
it's AIT's a it's it's not really so
much a form of noise I don't even know
how to describe what dropouts
necessarily like it's kind of its own
unique snowflake
but in this small of a dataset dropouts
probably not actually going to help us
at all but again in a much larger
dataset dropout is going to help because
they're going to and in fact we even did
worse this one more time
dropout is kind of like like not
everything is exactly the same and
sometimes like maybe one neuron is given
way too much weight or something like
that so these things can happen so
dropout and kind of help fight those
kind of local maximum kinds of things
but anyway once we're done with this
we'll be done with this tutorial but
again the in general for image data even
though in this case the recurrent net
actually did better spoiler alert I'm
more of a fan of recurrent Nets anyways
but in all reality on image data
generally the the convolutional neural
network it still did worse under 97 the
convolutional neural network is the
champion you're just going to need a
bigger data set in this case I think
we're using 50,000 images and 10,000
tests if I'm recall if I recall right
something like that it's not a huge
number and in reality you would want a
very large number to work with so
anyways that's convolutional neural
network in tensorflow if you have
questions comments concerns whatever
feel free to leave them below otherwise
as always thanks for watching thanks for
all the support and subscriptions until
next time