Neural Network Model - Deep Learning with Neural Networks and TensorFlow

what is going on everybody and welcome
to part three of our deep learning with
neural networks tensorflow and python
tutorial in this tutorial we're going to
be going down the rabbit hole and
beginning to build our first deep neural
network so to start I'm just going to go
ahead and copy this will rename it to
deep net and we will edit with spline
minimize this make this big and we'll
just delete all this okay so before we
get started we're going to be using the
M NIST data set for a few reasons the
main reason we're going to do this is
because gathering data and just kind of
preparing it to be used in machine
learning is a pretty long and tedious
process so we will eventually do
something like that but if you are
familiar with maybe the investing with
the support vector machine series I did
probably 80% of that tutorial who was
spent just getting the data in the right
format right so the eminence data set
it's really nice because the data is
already in the right format and stuff
like that and so we can very easily work
with this data set and first focus on
the neural network itself modeling the
neural network training the neural
network working with tensorflow we can
focus on that first and then we can
worry about okay from here how do we do
our own project and stuff like that so
so that's all we're going to do we're
going to be working with the emne set to
begin now the MS dataset is a set of
60,000 training examples of handwritten
digits so anything from 0 to and 9 and
they are all 28 by 28 pixels so 784
total pixels and then you have 10,000
testing examples so we train against
those 60,000 and then we test against
those 10,000 samples and those 10,000
samples are unique you know right
they're not the same as the stuff that
we trained on so the objective is to
take these these m nests
by 28 images of handwritten characters
you know they're just black-and-white
thresholded images take those and
somehow pass that through a neural
network to the point where the neural
network can somehow model what's going
on here and output when it comes time to
predict output the correct number of
whatever handwritten digit we pass
through that's in a 28 by 28 at least so
that's the objective
now with this basic neural network the
way that we're doing that are basically
each of our features is just a pixel
it's just the pixel value right is this
pixel on it basically well it's not a
pixel value but is it's basically it's a
0 or 1 right is there something there in
that pixel or is it just white space
right is it part of the number or is it
white space and that's it those are our
features and from that somehow we're
hoping our neural network is going to
somehow model this relationship and be
able to do things like figure out
whether it's a 1 or a 7 or a 4 or a 9
and so on okay
so first before we start writing code
I'd like to kind of condense a little
bit what we're going to be doing so I'll
just kind of write it on here just
really quickly just as a quick recap so
first of all we have our input data and
we send the input data to the let's say
hidden layer one okay
on the way we send the input data we
weight it and then it goes a hidden
layer one and that's with unique weights
and then from hidden layer one we're
going to run that through some sort of
activation function okay and then we
take that and send it on along with
weights to hidden layer - I'll just say
hidden l2 for now already huh nice okay
whatever let's do this okay and we
repeat this process up until again
hidden layer two has an activation
function
which then will send off with weights to
let's say the output layer okay
something like that now I guess I'll
leave it that simple for now so so
that's what happens and in a typical
neural network we pass data straight
through okay that passing of data that
goes straight through to the very end
that's feed-forward so this is a
feed-forward neural network so we pass
that data straight through and then at
the end of it we compare the output to
intended output okay how close is it and
the way that we compare that is with the
cost or loss function and I'm just gonna
say or here just just in case you saw
that and it's like cost divided by laws
now anyway compare the intended output
that's a cost last wash function I'm
just gonna say cost function because I
don't want to say it were either anyway
um so you compare the the output to the
intended output with the cost function
and then finally and like an example of
a cost function I don't know
cross-entropy okay this is how close to
or how wrong are we or how close to what
we are intended to target are we or how
not close then that's your your cost
function and then we're going to use an
optimization function or an optimizer
call it optimizer and that is going to
attempt to minimize that cost okay and
the example here that we're going to use
is we're going to use the atom optimizer
but there's also stochastic gradient
descent
adda grad or ADA grad and like even in
tensor flow there's like eight or
something that you could use so there's
a lot of options there and what that
does is it goes backwards and
manipulates the weights that backwards
motion and the manipulation of weights
that's your back prop bug
chin okay and then feed-forward plus
back prop B equals an epoch okay
that's one like cycle of feed forward
and back prop that's an epoch and you're
going to do that maybe 10 times 15 times
20 times and after so long hopefully
each time you cycle through you're
lowering the cost function and
eventually you'll get to a point like
initially you'll start like the cost
will be really high and let's see if I
can do this right if it was over here so
the cost is like really high and then as
time goes on it kind of drops down and
then it kind of like levels out and
sometimes it'll kind of wiggle a little
bit and you might go backwards your cost
might actually go up or something and
then at that point you've hit your
diminishing returns it's no good and
you're just not really making any more
progress so we'll probably do like 10 or
something epics but anyway um that is
basically what we're doing hopefully
that was a quick but useful summary just
in case some of you were a little
confused about what goes on now I'm
actually going to get rid of this
because without my talking along with
writing this out this is probably really
confusing looking so I'm gonna get rid
of it so now that we have that let's
prepare for liftoff
so we import tensorflow stf there's
really nothing fancy going on there and
then from um oops I'm not even in my
window here from tensorflow
dot examples dot tutorials dot M NIST
we're going to import input data and
then we're going to say I don't want
that on that one we're going to say M
missed equals input data read data sets
and then we'll put a bit in temp in data
and then we're going to say one hot
equals true great
so the only thing to note here is the
one hot parameter okay this term comes
from electronics and circuit board stuff
we're like one component will be hot and
that like dictates something so and then
the rest are off like one hot literally
means one is on and the rest are off
okay and it's like literally like if
it's physically has electricity running
through it it's probably hot so an
example here is well first of all why
are we using this one is it can be
useful for multi-class classification so
the output from our our neural network
is going to have say like 10 10 output
nodes or something so for example we
have this is multi-class we've got 10
classes right so we have 10 classes and
they are either a zero to a nine through
handwritten digits 0 to 9 so normally
you might say a 0 we want the network
and in fact let me just make a comment
out of this just in case I leave it you
might say you want 0 you want the output
to be a 0 a 1 you want the output to be
a 1 to a 2 and so on
well what one hots going to do instead
is it's going to say 0 actually outputs
to be a this 1 2 3 4 5 6 7 8 9 10 ok so
a 0 is actually a 1 0 0 0 0
ok so I'll copy this just for a couple
examples so 1 2 and a 3 and it'll
actually be like a 1 would be a 0 1 0 0
0 and then a 2 would be a 0 0 1 0 0 0 0
and a 3 would be the following and so on
so under a one hot situation it's just
literally one pixel or one element is on
or hot and the rest are cold or off
alright so we have our data and now what
we're going to do is actually start kind
of defining our our model so we're going
to start by saying n nodes for the
hidden layer 1 and we're going to say is
500 and we're just going to say this so
we're going to have three hidden layers
so this is a deep neural network so
and three okay so five hundred for each
now these don't have to be identical you
could have this one be 500 this one be
1500 and this one be 15 okay 500 500 500
will be good enough to start with but
feel free to tweak that depending on
what you're doing how many layers you
have and what kind of model your what
kind of thing you're trying to model
this can be totally different but this
this will be fine hopefully and then
we're going to have number of classes
equals ten
I would imagine through em missed you
can probably derive this value but we'll
just do this and then like I was saying
before M missed is a small enough data
set that I would wager most of you could
probably load all of em nasteria by 28
but most you can probably load the MS
data set end up into your memory or so I
think but that doesn't really prepare
you for the future and the reality is
that you're going to be you're going to
have like a data set that might have
millions and millions of samples and
might like for example with common crawl
that's got petabytes of data well you
probably don't have petabytes of RAM so
unless you're watching this tutorial and
like I don't know 50 years or something
maybe even less who knows but anyways
you probably don't have petabytes of RAM
I'm still trying to think about how many
years will probably be so we have
petabytes of RAM right maybe probably
not that long anyway regardless we're
gonna say the batch size back on topic
is 100 okay
and so what this is going to do is it's
going to go through batches of 100 of
features and feed them through our
network at a time and manipulate the
weights and then do another batch and
manipulate the weights a little bit so
it's just by batches of a hundred images
we could do a thousand images or
something else but hundred should be
fine enough so now what we're going to
do is define a couple placeholder in
variables so X is going to be TF
placeholder and then it's going to be a
float and then we're going to define Y
is the same thing one thing I will note
is it's not necessary but
you can define these so for example X
this is your input data okay so the
input data will have a very specific
shape it's just going to be 784 pixels
wide okay and a matrix is height by
width so the height let's say of the X
data set what would the height be well
you know I have any height really so
that would be none and then the width is
28 by 28 because what we're going to do
is we're just gonna flat their squash I
think is the proper turn or flatten I'm
not sure which one's the proper ones so
I can comment below and tell me which
one's the proper term in this case maybe
they're both synonymous I don't know
anyway uh and then also seems to be a
quote different or a comma that's a
different parameter anyway the second
parameter here can be a shape so none
and then it's a 28 by 28 which we're
going to just flatten out or squash out
to be 784 um pixels or actually just 784
values so even though it's coming in as
a picture that's a unique you know 28 by
28 so in theory you might be thinking oh
the matrix should be 28 by 28 it's
really not necessary as long as they
maintain the exact same position that
can just be a string of values it
doesn't actually have to maintain the
initial shape of the array so anyway you
can do this and the you don't have to
have these values here but what happens
is if you attempt to feed through
something that is not of this shape into
this x value tensorflow tensorflow
will throw an error if you don't have it
there and something not of that shape
gets attempted to be placed into X
tensorflow won't throw an error so
sometimes it can be useful to like leave
something in there because a lot of this
stuff is happening behind the scenes and
you're like absolutely not privy to
what's going on so when things don't go
the way you hoped this can be kind of
helpful but anyway just keep that in
mind that sometimes it's useful
sometimes not so so we've got our
starting values right the X is our data
the Y is the label of that data and
basically what will happen is this is
just a placeholder and at no point are
you actually going to have any code well
you kind of will write the code that
manipulates that handles x and y but it
because x and y is just a placeholder at
any given time for some data to be
shoved through the network but I guess
we'll wait till we get to the weights
but that's where I'm trying I just want
to forget because the weights is a
perfect example where like we have no
code that suggests that oh let's modify
weights here tensorflow just does it so
anyway we'll get there so now we're
going to go ahead and do is we're kind
of done defining our constants and our
variables so now we're going to define
our neural network network model and the
neural network model just needs some
data to be passed through it right this
is your raw input data I'd use the word
input there but that would violate some
Python so anyways here we're going to
start just with some some dictionaries
so we're going to say hidden one layer
and hidden one layer is going to have
two values in the dictionary so you're
gonna have weights and you're gonna have
weights equal to and this is a be kind
of long but anyway TF dot variable and
then hopefully this won't run us off the
screen TF dot random normal and then of
a shape so it's a it's a tensorflow
variable the weights right the weights
here are a tensorflow variable and then
that variable is a tensorflow random
normal and what we're going to do is
we're going to specify the shape of that
normal okay so the weights what you know
what would this be like so for the
weights for example we're going to have
it'll be 784 by the number of your nodes
in the hidden layer 1 HL 1 that's your
weight so those are all of your weights
so this is just going to create a tensor
or an array of your data using a bunch
of random numbers and that's just that
those are your weights now so now
have got your like weights in one giant
tensor basically and that tensor is a
tensor flow variable of course and it
was generated with TF dot random normal
but obviously soon enough we're going to
start modifying that but we have to
start with something so we just start
with something random next we are going
oh I really wish it didn't do this idle
doesn't do this this is why idle is the
best hopefully this won't throws for an
error later on ah like normally if you
hit it unless I did close it off right
yeah hopefully hopefully this will be
kosher anyway
weights and then we're going to have
biases and the bias is similar will be
TF variable the TF variable will be a TF
dot random whoops random normal normal
and then this one the bias is a recall
let me write it out first a number known
as a hidden layer one okay so so in this
case do you need it to be any sort of
shaped tensor not really the bias is
something that's added at the very end
and actually just realizing everybody's
like probably like what the f our bias
is what the hell
okay so biases are something that's
added in after the weights okay so as
the weights come through so you've got
your input data and then you've got
weights and so the weight is like the
input data is multiplied by the weight
and then we were saying before that the
weight comes through and is just also
all those inputs are some together after
their weights are multiplied by the
input data well what a bias does is adds
to that so you've got basically the the
formula it's going to happen is you're
going to have input data we'll call it
input data times your weights plus bias
and it'll be like this okay
so why do we even have a bias right
because we haven't talked about a bias
at all up to this point now we're going
to be using this activation function
that's rectified linear and so the
biggest benefit for a bias in this case
is if all of the input data is a zero
that would be 0 times weights which
would be zero of course
so no neuron would ever fire okay and
that's sometimes that's not really ideal
in every scenario and so you have a bias
which is yet another parameter that we
will have to like deal with but a bias
comes in and just adds a value to that
so at least some neurons could still
fire even if all inputs were a zero so
we had biases in there and that's that
it also can make your your network
overall a little bit more more dynamic
but mostly it's for if all input data
was zero this guy's just thrown in
biases see why is it acting like this
why can I not like did I not close this
off correctly like I feel like I did a
good job closing that off hmm I feel
like I'm gonna get a syntax error or
something let me yeah I'm just curious
if I'm gonna get a syntax error I don't
want to keep using this method here
let's open up a terminal really quick CD
/ desktop
so I am getting a thing did I not close
out Olesya in this case I didn't close
off the variable thing so that's why it
was probably giving me such a hard time
then I closed it off on this one I did
it's just not this one okay now what
happens beautiful so now what if I tab
this over sure it's still pretty
unfortunate
come on sublime okay but it still looks
like it's going to accept my code but
pulling out data everybody chill
all right so cool so we've got our
hidden one layer okay so now we're going
to take this and we are just gonna paste
oh gosh just killing me let's see if we
can do this actually so one and then we
want really we're going to do this so
hidden layer one two three and then we
have the output layer that's a lot of
copy and paste
I encourage somebody to come up with
some sort of dynamic for loop or
something to iterate through a variable
number of hidden layers that would be
interesting of course you'd still have
to have you'd have to then you'd
probably have to set like the max number
of nodes but it is possible probably
because imagine if you want to have like
100 layers that wouldn't be very good
probably at least for this problem but
anyway I digress so this would be 2 this
will be 3 and then this is our output
layer okay so what is going to change so
one thing that can change and one thing
to note is that these can change right
you can one layer could have five
hundred nodes that is one 1,500 nodes so
those things can change that's why we
have unique values here because you
could change them in this case we've got
them all as 500 but they can be variable
so generally what's going to happen is
your weights like in this case the
weights here would not be it wouldn't be
necessarily 784
that's not necessarily the input the
input will be the number of nodes in the
hidden layer one right multiplied by the
number of nodes in the hidden layer two
okay and then we'll take this paste this
in this would be a hidden layer two
hidden layer three and in fact this will
and then down here write the biases this
would be for the number of nodes in
hidden layer one this would be the
number of nodes in layer two hidden
layer three and then in this case and
the output layer we just explained what
one hut read of it but we explained what
one hot was right what is the output
about out output layer to write how many
biases are we going to need in this case
good answer it's going to be what is
happening here this is why I don't like
fancy IDs anyways the right answer is
going to be the number of classes right
however many classes we have because
that's the one hot right zero one zero
zero zero and so on so that is going to
be how many biases we need in the output
layer now kind of zoom through this
let's see down here this would be we
should have finished this this so this
would be three by the number of classes
cool all right so we've got that all
defined okay so hopefully everyone's
clear on what weights and biases are I
know I just kind of threw the biases at
you I didn't want to throw that at you
initially because just the model I think
makes more sense just with the weights
part but I meant to talk about biases
before we get to the code of actually
writing the bias so I'm sorry but
hopefully that's easy enough you have
questions feel free to leave them leave
them below so now all we have done at
this point is of course just create
variables basically for our layers we
haven't actually modeled anything so
remember the model let's see if I got
that I do well at least the
yeah so this is the actual model let's
cut I'm going to cut that and I'll just
paste it down here that's the model that
we have for each layer okay so to start
let's say layer 1 is equal to TF ad and
then that would be TF mat mole for
matrix multiplication and we're going to
multiply what in layer 1
well the input data right so data and
data remember data is coming from this
variable here data so we multiply the
data the data by the weights right and
that would be hidden this is kind of
nice although I don't know how to get
down probably the arrow keys hit in one
layer and then weights okay right
weights and then outside of those
parentheses now don't make that mistake
plus a hidden one layer biases biases
okay so one mistake that I made before
and could easily make as we go through
is instead of having the closing
parentheses there I had it here so
instead it was TF add together the
matrix multiplication of data and hidden
layer one weights plus biases so it was
like the weights plus the bias times the
data right and basically adding nothing
so but it still gives you that value
like if you do TF dot add like five the
output would be five I think anyway
because everything ran there was no
error it's just my accuracy was in the
in the dirt and I had to like figure out
what the hell is going on
so mmm so make sure you don't make that
mistake so let's fix that so again input
data times weights plus the biases right
and and as if you recall a little pretty
picture I drew right you're the input
data times the weights and it went into
that sum box right so this is your some
box so then this information eventually
after we pass it through like this is
just layer one right we haven't gone
very far so then it goes through though
an activation function
right and that's whether or not the
neuron fire well
the threshold function would have been
whether or not it fired the activation
function is this nice beautiful sigmoid
function so it's not as on or off or all
or none as a step function would have
been but anyways continuing layer one
this is just TF n n dot R Lu which is
rectified linear that's your activation
function which is not synonymous but it
does the same thing as SOTA castigating
descent and I'm blanking no sorry does
not I knew that sounded wrong okay so
the Cassegrain isn't as your optimizer
sorry rectified linear is like your
threshold function and we just apply
this to layer one okay
so it takes whatever this value is and
then passes it through that activation
function whenever you've done that
you're going to bebop on to layer two
layer two is TF dot ad and in fact I'm
going to we're going to take this copy
paste paste again this is why for loop
would be great but I cannot think of a
really great way to like somehow just
dynamically set the layers someone
really wants to try though
and I also it had it would have to like
work with tensorflow
there's got to be a good way but even in
like the tensorflow docks and all the
examples I've ever read nobody seems to
have that one figured out but I know
there's some some Python you stay out
there it's going to go and get that
fixed for people anyway so for layer two
what's coming through to layer two what
is the input data is it data now it's
the whatever that activation function
returns for layer one okay so we take
layer one that goes through are using
layer ones weights now we're using
layers layer twos weights and then same
thing on the biases who's already using
two and then while we're at it let's
just make that three and then this will
be three and what's the information
coming through to layer three well for
layer three you have layer 2 data layer
2 data
and then um make sure this was all good
for we okay yeah I'm gonna do this okay
so this needs to be layered to you
there's so many places to go wrong again
that's why for loop would just be superb
wouldn't have such messy code okay so
then what happens after these three
hidden layers what happens I say you got
your output layer pasted okay output
layer basically does the same thing the
only difference is the output layer
doesn't go through a well mm well the
output layer is a little different
sorry it's not basically the same thing
because first of all you're not adding
you're not going through summation and
you're not going to run through the
activation function but you are going to
matrix multiply so this would be layer 3
would it be hidden 3 layer no it's going
to be the output layer weights and the
output layer bias and again you could
screw up with these parentheses but
since we've been copying pasting we did
fine hopefully and then at the end of
the day we return the output boom whoo
okay so at this point we now have
modeled a neural network and at this
point we are basically done with with
the computation graph okay and setting
up all the basics of the computation
graph and we are almost ready to launch
into the session I actually trained this
thing see how we did
there's like a couple variables that
we're going to define right before we
actually start the session but they make
more sense to define outside of this
function which is just our neural
network model but at this point we've
actually coded the model like the model
is done now all we have to do is explain
to tensorflow what to do with this model
and what to do in the session so anyways
if you have questions comments concerns
or you come up with a for loop that's
going to accurately handle for this copy
and paste here this
nonsense here sort of I still don't know
how you're going to handle for those if
you can make them dynamic but I'd still
like to see the for loop even if every
layer has 500 nodes but this copy and
paste BS and then all this
at least this BS or I can understand if
the output layer is unique but even then
the output layer would just be like an
addition at the end of the for loop or
something anyway gotta come up with that
anyways questions comments concerns
leave them below otherwise till next
time