Running our Network - Deep Learning with Neural Networks and TensorFlow

what is going on everybody and welcome
to part four of our deep learning with
neural networks tensorflow and python
tutorial in the last tutorial what we
did was we basically built up the
computation graph for our tensor flow
model and also basically we built up the
neural network model so those two are
very synonymous as far as what they are
so your model if your machine learning
model is very similar to setting up the
the computation graph and then what we
have left to do now is specify how we
want to run data through that model in
the session ok and what we want to kind
of do with that model in the session so
to do that let's go ahead and we're
going to create a new function that's
going to be trained neural network this
is going to take X which is just your
input data and we're going to say the
prediction is equal to the neural
network model X so what's happening here
you're taking input data you're passing
it through your neural network model
which shoves it all through there's and
returns an output recall that output is
that one hot array right that thing
where it's like 0 0 1 0 0 0 and so on
that's what it returns so your
prediction is that and then what we're
going to do is we're gonna say we're
going to say the cost right this is our
cost function is equal to TF dot reduce
mean and we're going to do that on the
TF n n soft max underscore cross
underscore and tropi with logits against
the prediction and why so in this case
warden is we're using cross-entropy
with logits as our cost function and
that's going to calculate the basically
the difference let's say of the
prediction that we got to the known
label that we have in both of these are
those that one hot format that's why we
basically you know this one hot equals
true that's why we have that and that's
why our output layer
is of that shape and everything so but
of course if you're not using one Hut
that's totally fine
you can have your output be whatever
shape your output basically will just
always be the shape of your testing sets
labels right so just keep that in mind
training in testing says hopefully you
don't have those different because
you're going to be screwed anyway so now
we have the cost function we know what
the cost is what do we do with that cost
well we want to minimize that cost we
want to minimize the difference of
prediction in Y right we want that to be
as small as possible ideally not no
difference in zero button but that
rarely happens so anyway we're gonna
have an optimizer and the optimizer is
going to be TF trained Adam optimizer
and this is synonymous with stochastic
gradient descent
eita grad and so on and this would be so
using the Adam optimizer what do we want
to do well we want to minimize what do
we want to minimize cost done optionally
the Adam optimizer does have a parameter
that is learning rate but learning rate
default equals 0.001 which is a fine
enough learning rate so I see no reason
to modify that okay so we've got those
values and we have our cost function
defined but again what is anything
actually happening in this cost function
I mean just look at it like nothing is
happening in the cost function how about
in the optimizers anything actually
happening here we're minimizing cost but
are we actually running anything or we
running it through layers or telling the
optimizer how to minimize cost I don't
see how but okay now we're going to
start with how many epics do we want and
this can be any value we'll start with
10 just so it goes a little fast if you
are following along on a really really
slow computer uh
probably do like a really low number
there or don't do this at all I don't
know what you're doing with neural
networks on a really slow computer but
it'll run it just might take a while so
yeah anyway
how many epics 10 like if you're on a
netbook I wonder how slow this would be
any way with TF dot session as cess what
do we want to do well first we need to
initialize our variables so we'll do
cesta run TF dot initialize all
variables I feel like they need to just
have like a TF dot in it or something
rather than like knit in it VARs or
something that's a long thing to type
out and I'm just saying okay so that
initializes our variables in that that
actually begins now the session has
started running right thus Justin has
begun we're done defining our
computation graph at this point so we're
going to run through these epics and
again the epics is just cycles right of
feed forward plus your back prop right
so you're for data and then back prop
fixing all the weights
hey those are epics so once you've done
that what we're going to say is we're
going to say for um let's see for epic
in how many epics do we want we're going
to start by saying the epic loss is
currently at 0 and we're going to kind
of just like calculate the loss as we go
it could say epic cost or something too
but we're going to calculate that as we
go now we're going to say for underscore
in range underscore just is like
shorthand for variable we don't care
about by the way in range of the int
value of M NIST trained num examples
divided by the batch size okay
so this is just basically we have the
total number of samples we're dividing
by our bat size and that tells us how
many times we need to cycle based on
that dynamic batch size that we set way
up here so we do that and we're just
gonna iterate through as we iterate
through we're going to say x and y so
data labels equals m missed dot train
net
first batch batch size so what this does
for you magically is basically kind of
chunks through the data set for you now
when you do this in the reality in the
reality in the real world
in reality you would have to like build
your own little function that does this
for you right it just so happens that
they've got this pre-built to do batch
sizes for us but this kind of makes it
look a lot easier than it really is but
I suppose it's not that hard to make
something that does this but there are a
lot of like little helper functions like
this in in tensorflow is it's actually
quite nice I was a pretty surprised
given what I had heard about tensorflow
versus my actual reality of using it
anyway variable we don't care about C
equals cess dot run C is cost by the way
what are we going to run in this session
we're going to run the optimizer and the
cost right recall optimizer this cost
this right they are just simply
definitions in our graph we are going to
run the optimizer with cost and then
using their feed dict the data is a
whoops feed dick sorry my friends in the
way of my keyboard feed dict equals and
we're going to say x XY Y done so we run
through the data and we're running that
we're optimizing the cost with X's and
Y's and we're passing the X's and Y's
right we see X's and Y's how are we
optimizing that cost you ask right we
are doing that by modifying the weights
and yes the weights are or by modifying
these weights and yes those weights are
being passed through these layers but
somehow magically tensorflow knows that
an biases that it gets to modify those
so incredible it's it's it's a little
high level I mean that was a kind of
like one of the most confusing things
for me as I was coming into tensorflow
but anyway epic loss plus equals
whatever see is okay and then for each
epoch we're obviously resetting the
epoch loss but we kind of want to track
that each time and then after that for
loop what we can do is we can just print
we can do some beautiful formatting here
epic epic completed out of what was it
how many epics HMF X and then we say
loss loss no epic loss perfect
done done okay now that will give us the
answers or the loss function and tell us
where we are obviously if you don't
print something out you'll have no idea
like how much longer do I have to wait
so it's kind of useful do that you could
also do something in the batches like if
you had like a huge data set and your
batch sizes were huge you could you
could do that there too just so you kind
of and then you could kind of generate
like a percentage complete kind of thing
but for this data set that won't be
necessary
anyway now hats side of the for loop for
the epoch and how many epics I do like
the low lines that are there I will say
idle is still the best correct equals TF
dot equal so we're just kind of
asserting TF dot Arg max prediction 1 TF
dot Arg max Arg max y 1 so all this is
doing TF to Arg max is going to return
the index of the maximum value in this
these arrays and we're hoping that those
index values are the same right they're
both 1 hots and all that so we're
expecting that this is going to tell us
whether or not these are basically
identical so that's that now we can
compute accuracy as being equal to TF
dot reduce mean
TF cast which just changes the variable
to a type so we're casting correct to a
float before it's just I don't even know
what the default type would be I'm in
tensor of some kind but I don't anyway I
don't know
anyway this float now and then we can
just say print II accuracy coma accuracy
dot eval and we are going to evaluate X
: m-miss dot test images and then y : M
missed
testable x' okay so after we've trained
because like basically this is just
training the network or at least this so
at this point we're just training we're
dealing with training data right Emnes
train we're doing all with training and
train here we're dealing with the
training data and then once we have
optimized those weights we come down
here and run them through our model and
we're saying that to know what's correct
or not we compare the prediction to the
actual label accuracy is just whatever
the float is of that correctness and
then we're going to evaluate all of
those accuracies of basically the test
images to their test labels again we're
not telling it like feed this data like
through the model or anything like it
just does it great I mean I don't know
some of this stuff is maybe too high
level I think for my review of
tensorflow but at the same time it's
pretty cool like like this is a I have
some some white space so may I say it's
decent but this is 66 lines of code
that's incredible
and by the way that yeah we're done so
now what we do
everyone's like run it we're just gonna
train network X right we pass X that's
it done
so now let's just this bad boy so we
come over here and that's just from
before we'll just up yep Python three
deep net dot pie please don't get me
near
Oh God
knew you're gonna do this okay so layer
one we're missing one required
positional argument add mat mole mean I
missing here uh-huh
all right so I positive I was trying to
figure this out and I've figured out so
so I should paid a little bit more
attention but anyway add missing one
required positional argument of Y so
when you add you have an x and a y right
you add and then XY and instead what I
had done which I probably actually could
have gotten away with after we showed
the whole multiplication and all that
but I'm doing TF add this Plus this
right which you know kind of made sense
if it wasn't within a TF dot add so
actually rather than this plus you have
a comma okay
and then another comma here here and
this one is a mat mole without the add
so let you tell you right there that you
actually can get away without the add so
you can do TF table add or just a
straight-up plus you actually don't need
the add interesting okay
so we'll save that let's try again see
what happens this time local variable
why what are you talking about so this
times telling us why was referenced
before the assignment
and that would be the Y in the softmax
function up here
I don't know about y'all but I honestly
already see why being defined but I
wonder maybe of what's happening is here
that it's getting angry so why don't we
say epic X epic y I wonder if like this
is you know screwing that up because
again it doesn't work like Python works
so you can't necessarily use bad code
that you write in Python so epic X and
then let's try this to be epic why sorry
for the errors but I'll probably end up
leaving them in because I think they're
fairly useful because it's easy to make
stupid mistakes like this the first one
was a stupid mistake I'm guessing this
will fix this one but we'll see I should
stop surmising and just find out okay
okay so here we have an invalid argument
and but it's not telling me like where
that would be really useful so clearly
we've got an empty something was empty
that it didn't want to be empty ah
that's it okay so as painful as that is
or that was recall these are all
basically needing to be arrays here so
here all these biases here here here
here and this one I actually did
correctly interestingly enough anyway
that was a little hard harder to get get
to but anyway yeah you get that first
trace back and then another exception
occurs and that's where I got to read
here I was like reading that sounds like
what's wrong with this line but that's
it this needed to be you know square
brackets okay so let's taste that again
oops what just happened okay save that
again and let's try again ah yes we left
this commented out at least we know who
made it to that point I shall try again
this is getting brutal
clearly I'm just having a bad day for
epoch in range of how many epics all
right
this is going to be the death of me
please don't get me there I want to jump
out my window
luckily I'm on the first story here we
go that's a big loss but that's a
probably decent massive improvement
that's what we need to see we need to
see improvement here guys hmm see I wish
I knew how to make this bigger someone
knows how to make this text and this
bigger would be great hopefully you guys
are running though and I imagine the
batch thing or the the square brackets
you guys for sure made that mistake I'm
not sure if everybody made the range
mistake though that is pretty dumb
anyway so you can see this is pretty
slow now the good news is once you've
trained a model right you don't have to
keep doing it so if you've been
following along in this series you've
seen that like what scikit-learn you can
pickle the model and you don't have to
retrain it again anyway so we're done
the accuracy is ninety four point eight
five percent so perfect accuracy would
be a 1 point so we got almost 95% and on
the Emnes data set that's actually
considered like laughable but that's
only considered laughable because this
isn't quite the best model to be using
on the m-miss data set
mostly because like consider again that
all we did was feed through raw pixel
values just one by one we didn't help it
generalize we didn't help it kind of
have a different perspective maybe a
little bit like for example the
convolutional neural network which is
going to kind of take chunks of pixels
at a time well it might go down to a raw
pixel but it's like a raw pix well then
maybe chunks of pixels and stuff like
that and those are the layers or even
you might just
even have layers but he's just like a
chunk and then slides down a chunk and
slides on trunk anyway that means
nothing to you that's fine but that has
a little bit more of a logical approach
to it whereas here we're just like
throwing data at the neural network and
we're just like yeah I figured out and
the neural network figured it out and
while 95% isn't like industry standard
for that exact specific task the point
is we threw a pretty challenging task at
the machine just gave it the most basic
form of data possible and it still came
up with a pretty darn good model like I
think that's just amazing
right the the accuracy could be better
with a better model in a convolutional
neural network maybe and some more work
but honestly that's a really good score
for what we like the effort that we put
into it and I think that all that does
is while not being the greatest score it
does kind of show you the incredible
value of a neural network or a deep
neural network anyway at just coming up
with models like that's pretty cool
anyway I think that's cool so not
without errors but I'll probably leave
them in there just simply because it
seems like people don't really mind when
I leave the errors in and I think it
actually helps because you know working
in this code is like confusing as it is
and I've probably done too much of this
today but working in the code is just
confusing you're going to make mistakes
and you're going to get a bunch of big
errors as you play around with this like
that's going to be really common so so
I'm going to leave them in there you
probably get used to these errors and
those Big C errors like like Python plus
C errors those are kind of annoying to
work through but at the end of the day
the biggest thing that you need to look
for is the stuff that like line like
I've seen line 728 I'm like I didn't
write some 128 lines so I'm just like
scrolling through looking for my code
right and I was still able to find my
code and figure out what the problem was
because like some of this stuff is
talking about vector shapes and all this
and clearly it's getting an empty shape
somewhere and that probably would have
been a useful ish
but that didn't tell me enough I just
knew it was empty but I didn't know okay
what vector is empty and stuff so
I'm the wearing your line of code and
your code it messed up so anyway maybe
more errors to come in the future videos
but regardless we got 95% we only did 10
epics and it really didn't take that
long so other things you can immediately
do is you can immediately begin you can
just add more layers you can make the
layers more complex that may or may not
help it might actually decrease your
accuracies and many times basically the
simplest form possible is usually the
best it's a good rule to go by but
obviously it's like a super simple like
you know some things will work but the
soup the simplest you can get away with
is pretty much always going to be the
best but anyway that's all I have for
now if you're getting any errors feel
free to post them below and I'll do my
best to kind of help you out if possible
in the coming tutorials the eminence
dataset is really cool and all but I
think the next step that needs to be
taken is working with like your own data
acquiring your own data so we're gonna
get some data from somewhere I haven't
totally decided where I want to do it or
what kind of project I want to work on
so if you're watching this tutorial and
it's fairly recent feel free to comment
below give me some suggestions or
requests on what you want to do I do
have some ideas I kind of want to work
with like I'm not even going to say I
don't want to ruin it in case that's
exactly what I do but anyways if you
have suggestions or something leave them
below we'll probably have other projects
too I think the whole neural network
thing is is fascinating enough to
warrant maybe multiple projects in it so
anyways
questions comments leaving below the
future will work on kind of our own
thing basically with this simple neural
network and the next time goes on I'll
probably add some more with
convolutional neural networks and some
of the other models as well so if you
have questions comments concerns
whatever even below otherwise thanks for
watching until next time